bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

akulakov · 2022-01-04T02:06:46Z

I'm not sure if these should be added to the docs -- there's no pyvenv.cfg section and two of the vars are currently documented in separate places and another two are not documented. It seems like someone needing them would look in pyvenv.cfg and find them; and their values are self-explanatory.

https://bugs.python.org/issue41011

vsajip · 2022-01-04T09:51:38Z

Lib/venv/__init__.py

@@ -178,6 +178,8 @@ def create_configuration(self, context):
            f.write('version = %d.%d.%d\n' % sys.version_info[:3])
            if self.prompt is not None:
                f.write(f'prompt = {self.prompt!r}\n')
+            f.write('executable = %s\n' % context.env_exec_cmd)
+            f.write('command = %s\n' % sys.executable)


That (sys.executable) is not the command (-line) which was used to create the environment - it would be the python -m venv ... or equivalent command line which need to be recorded.

Would something like this work?

args = ' '.join(sys.argv[1:]) f.write(f'command = {sys.executable} -m {context.env_name} {args}\n')

Would something like this work?

Not necessarily. The code to create a venv might be called programmatically through the EnvBuilder API. The intention is to provide a command which, if run, produces a venv with the current settings. So the thing to do is:

Run python3 -m venv -h to see what the command-line options are.

Look at the existing code to see how those options map to attributes of the EnvBuilder which are used in making the venv.

Looking at the EnvBuilder attributes of the current instance, work out what command-line would give those attributes, and synthesize the command line from there.

vsajip · 2022-01-04T09:52:34Z

Misc/NEWS.d/next/Library/2022-01-03-21-03-50.bpo-41011.uULmGi.rst

@@ -0,0 +1,3 @@
+Added two new variables to *pyvenv.cfg* which is generated by :mod:`venv`
+module: *executable* for the executable within the environment and *command*
+for the command used to create the environment.


command -> command line

bedevere-bot · 2022-01-04T09:53:55Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vsajip · 2022-01-04T16:14:31Z

Lib/venv/__init__.py

@@ -178,6 +178,8 @@ def create_configuration(self, context):
            f.write('version = %d.%d.%d\n' % sys.version_info[:3])
            if self.prompt is not None:
                f.write(f'prompt = {self.prompt!r}\n')
+            f.write('executable = %s\n' % context.env_exec_cmd)
+            f.write('command = %s\n' % sys.executable)


Would something like this work?

Not necessarily. The code to create a venv might be called programmatically through the EnvBuilder API. The intention is to provide a command which, if run, produces a venv with the current settings. So the thing to do is:

Run python3 -m venv -h to see what the command-line options are.

Look at the existing code to see how those options map to attributes of the EnvBuilder which are used in making the venv.

Looking at the EnvBuilder attributes of the current instance, work out what command-line would give those attributes, and synthesize the command line from there.

Lib/test/test_venv.py

Misc/NEWS.d/next/Library/2022-01-03-21-03-50.bpo-41011.uULmGi.rst

…sed on EnvBuilder attrs

Lib/venv/__init__.py

…t flag; expand test to check that no invalid flags are present when no args are passed to EnvBuilder

Lib/venv/__init__.py

vsajip · 2022-01-06T08:12:55Z

Lib/venv/__init__.py

+            if self.upgrade_deps:
+                args.append('--upgrade-deps')
+            if self.orig_prompt is not None:
+                args.append(f'--prompt={self.orig_prompt}')


prompt = f'"{self.orig_prompt}"' if ' ' in self.orig_prompt else self.orig_prompt args.append(f'--prompt={prompt}')

to handle the case where self.orig_prompt has spaces?

I think it's not enough to quote it on spaces, I quoted it unconditionally in case of chars like '*'.

Ah, I just realized I did not commit that change.. pushing it now.

akulakov · 2022-01-06T14:25:01Z

@vsajip I know the test got to be ugly and confusing, lmk if I can make it clearer...

vsajip · 2022-01-06T15:41:13Z

I know the test got to be ugly and confusing, lmk if I can make it clearer...

Do you mean test_config_file_command_key?

akulakov · 2022-01-06T16:11:04Z

Do you mean test_config_file_command_key?

Yes. It's also quite slow (creating new env for each flag), but checking all flags in one run would be less thorough, but it's an option we can consider.

vsajip · 2022-01-06T18:29:30Z

Yes. It's also quite slow

How about mocking the pip installation part? Does that speed things up?

akulakov · 2022-01-07T00:15:03Z

How about mocking the pip installation part? Does that speed things up?

Yep it does! Much faster now..

akulakov · 2022-01-07T13:20:51Z

@vsajip thanks for many rounds of quick reviews!! :)

pelson · 2025-01-06T16:28:19Z

Lib/venv/__init__.py

@@ -178,6 +179,29 @@ def create_configuration(self, context):
            f.write('version = %d.%d.%d\n' % sys.version_info[:3])
            if self.prompt is not None:
                f.write(f'prompt = {self.prompt!r}\n')
+            f.write('executable = %s\n' % os.path.realpath(sys.executable))


(I appreciate that this PR is long since merged, but just following-up on a surprising pyvenv.cfg file that I saw recently, and git-blamed it back to here)

I'm not at all sure why realpath is important here - the sys.executable should be good enough when being called directly. When called through the EnvBuilder, I don't see a reason for this either. 🤔

You're right, it is a difference when you're not on Windows. But thinking about it, you could view the config as recording the actual interpreter used instead of the path that could change on you in the future (e.g., some symlink later gets changed to point to another interpreter).

Thanks for this perspective (I hadn't considered it)! I'm not sure that using a config file to log information about how a venv was created is a good idea, but can appreciate the idea. Since this is a machine readable file, I would be concerned that it eventually gets used for non-log/audit purposes.

I've been digging into symlink issues with venv a lot recently, and think calling realpath is quite problematic in other context (i.e. when determining home). More details of that in #106045.

No follow-up needed here - just flagging it, and depending on the outcome of the linked issue, perhaps in the future I may request that this information become non realpath, or under a "history of how this venv was created" section of the config (perhaps in the form of a comment).

See also https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588

akulakov added 4 commits January 3, 2022 20:58

added env_exec_cmd and env_create_command; added them to the test

9857b28

add news entry

34f7943

update var names

0c7643c

unskip a few tests

e4bb85e

akulakov requested a review from vsajip as a code owner January 4, 2022 02:06

the-knights-who-say-ni added the CLA signed label Jan 4, 2022

bedevere-bot added the awaiting review label Jan 4, 2022

akulakov marked this pull request as draft January 4, 2022 04:01

use sys.executable not sys.argv

f93cae0

akulakov marked this pull request as ready for review January 4, 2022 05:32

vsajip requested changes Jan 4, 2022

View reviewed changes

bedevere-bot removed the awaiting review label Jan 4, 2022

bedevere-bot added the awaiting changes label Jan 4, 2022

update news

fabe6eb

vsajip requested changes Jan 4, 2022

View reviewed changes

update executable to be sys.executable and command to include args ba…

74bbfeb

…sed on EnvBuilder attrs

akulakov marked this pull request as draft January 5, 2022 03:44

akulakov added 4 commits January 4, 2022 23:00

unskip tests

abef4f4

fix test_defaults

1184ffb

fix test_defaults again

19d825e

update news

ac22946

vsajip reviewed Jan 5, 2022

View reviewed changes

Lib/venv/__init__.py Show resolved Hide resolved

vsajip reviewed Jan 5, 2022

View reviewed changes

Lib/venv/__init__.py Show resolved Hide resolved

brettcannon self-requested a review January 5, 2022 23:11

akulakov added 2 commits January 5, 2022 20:55

update to generate --symlinks and --copies based on the OS; add promp…

c891cac

…t flag; expand test to check that no invalid flags are present when no args are passed to EnvBuilder

add prompt to test

5afeb8a

vsajip requested changes Jan 6, 2022

View reviewed changes

akulakov added 2 commits January 6, 2022 08:51

quotes for orig_prompt

727c515

fix basic test for NT

0d44a7d

fix test config file for NT + copies flag

9f06b55

akulakov marked this pull request as ready for review January 6, 2022 15:29

mock setup pip for speed

0e519ec

vsajip changed the title ~~bpo-41011: venv -- add more variables to pyvenv.cfg~~ bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) Jan 7, 2022

vsajip merged commit f4e325c into python:main Jan 7, 2022

bedevere-bot removed the awaiting changes label Jan 7, 2022

akulakov deleted the 41011-Add-more-vars-to-pyvenv.cfg branch January 7, 2022 13:20

brettcannon mentioned this pull request May 10, 2023

[venv] record which executable and command were used to create a virtual environment #85183

Closed

pelson reviewed Jan 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

akulakov commented Jan 4, 2022 •

edited

Loading

vsajip Jan 4, 2022

akulakov Jan 4, 2022

vsajip Jan 4, 2022

vsajip Jan 4, 2022

bedevere-bot commented Jan 4, 2022

vsajip Jan 4, 2022

vsajip Jan 6, 2022

akulakov Jan 6, 2022

akulakov Jan 6, 2022

akulakov commented Jan 6, 2022

vsajip commented Jan 6, 2022

akulakov commented Jan 6, 2022

vsajip commented Jan 6, 2022

akulakov commented Jan 7, 2022

akulakov commented Jan 7, 2022

pelson Jan 6, 2025

brettcannon Jan 6, 2025

pelson Jan 8, 2025

hroncok Jan 10, 2025

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

Conversation

akulakov commented Jan 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Jan 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akulakov commented Jan 6, 2022

vsajip commented Jan 6, 2022

akulakov commented Jan 6, 2022

vsajip commented Jan 6, 2022

akulakov commented Jan 7, 2022

akulakov commented Jan 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akulakov commented Jan 4, 2022 •

edited

Loading