[Bug] `sys.path` is not accessible when subprocess is called in `pynml.execute_command_in_dir_with_realtime_output`

[adampdp]

Describe the bug

in pynml.execute_command_in_dir_with_realtime_output(),

sys.path is not accessible when subprocess is called. env['PYTHONPATH'] in subprocess overrides sys.path.

sys.path should be added to the env['PYTHONPATH'] before passing to subprocess for correct module import. See the link below:

python/cpython#105889

[sanjayankur31]

Thanks for that. We should also check all the other functions that call subprocess to see if this issue persists there too. I'll take a look ASAP.

[sanjayankur31]

@adampdp : would you have a code snippet that I can use to reproduce this please? I tried this snippet, and it does find the neuroml module, so it does seem to look at the different folders in sys.path, otherwise it wouldn't find the neuroml module:

import sys
import os
from pyneuroml.runners import execute_command_in_dir_with_realtime_output

env = os.environ

print(f'PATH is: {env["PATH"]}')
print()
print(f"sys.path is: {sys.path}")
print()
command = "python -c 'import neuroml; print(neuroml.__version__); print(neuroml.__file__); '"
execute_command_in_dir_with_realtime_output(command, directory=".")

[adampdp]

@sanjayankur31

Hi, It only happens when env is set, if env is None as it would be in this code snippet subprocess uses the sys.path for env I believe. It means if env is set to some path and passed you also need to manually add sys.path to it if you want to import other modules not on the path passed in env.

[adampdp]

@sanjayankur31

import sys
import os
from pyneuroml.runners import execute_command_in_dir_with_realtime_output

env = os.environ

print(f'PATH is: {env["PATH"]}')
print()
print(f"sys.path is: {sys.path}")
print()
sim_dir = './'
env = { "DISPLAY": os.environ.get('DISPLAY') if os.environ.get('DISPLAY') is not None else '',
            "XAUTHORITY": os.environ.get('XAUTHORITY') if os.environ.get('XAUTHORITY') is not None else '',
            "PYTHONPATH": ".:%s:%s" % (os.environ.get('PYTHONPATH', '.'), os.path.abspath(sim_dir)),
            "NEURON_MODULE_OPTIONS": "-nogui"}
command = "python -c 'import neuroml; print(neuroml.__version__); print(neuroml.__file__); '"
execute_command_in_dir_with_realtime_output(command, directory=".", env=env)

[sanjayankur31]

I looked into this. This works:

import sys
import os
from pyneuroml.runners import execute_command_in_dir_with_realtime_output

env = os.environ

print(f'PATH is: {env["PATH"]}')
print()
print(f"sys.path is: {sys.path}")
print()

sim_dir = './'
new_env = os.environ.copy()
# new_env = {}
new_env["PYTHONPATH"] =  f"{env.get('PYTHONPATH', '.')}{os.pathsep}{sim_dir}"
print(new_env)

command = "python -c 'import neuroml; print(neuroml.__version__); print(neuroml.__file__); '"

execute_command_in_dir_with_realtime_output(command, directory=".", env=new_env)

This doesn't (not copying the original environment and creating a new one):

import sys
import os
from pyneuroml.runners import execute_command_in_dir_with_realtime_output

env = os.environ

print(f'PATH is: {env["PATH"]}')
print()
print(f"sys.path is: {sys.path}")
print()

sim_dir = './'
# new_env = os.environ.copy()
new_env = {}
new_env["PYTHONPATH"] =  f"{env.get('PYTHONPATH', '.')}{os.pathsep}{sim_dir}"
print(new_env)

command = "python -c 'import neuroml; print(neuroml.__version__); print(neuroml.__file__); '"

execute_command_in_dir_with_realtime_output(command, directory=".", env=new_env)

So, there's a particular way in which the env needs to be passed. We pass it on directly to subprocess, so there's nothing that we can really do here---we don't want to tweak the environment passed by the user.

I'll close this for now, but please feel free to re-open if you think I've not understood this correctly.