pyinstaller does not take packages from virtual environment

[enricodragoni]

Hi All,

I have the following set-up:

• Python Virtual Environment created with Python 3.11.9 where, after executing the activate script, I have installed all the packages needed for my project.
• pyinstaller installed inside the virtual environment, version 6.12.0

Task: generating an EXE from a script inside my project (C:\myproject\scripts\myscript.py) and having that EXE built with the packages installed in the virtual environment.
Problem: the generated EXE crashes immediately with the message:
ModuleNotFoundError: No module named 'XXX'
where XXX is a package installed ONLY in my virtual environment.
If I run the python script not in "EXE mode", the package is obviously found.

From the generated Analysis-00.toc, I can see that the directory C:\.env\Lib\site-packages is not taken for installing the packages. The "default" Python installation in C:\Program Files\Python311\Lib is taken instead.
The log following the pyinstaller execution indeeds states:
Using Python shared library: C:\Program Files\Python311\python311.dll
even though the virtual environment is referenced multiple times in the log itself (along with the default installation):

(.env)  PS C:\myproject> pyinstaller .\scripts\myscript.spec
265 INFO: PyInstaller: 6.12.0, contrib hooks: 2025.1
265 INFO: Python: 3.11.9
333 INFO: Python environment: C:\.env
340 INFO: Module search paths (PYTHONPATH):  
['C:\\.env\\Scripts\\pyinstaller.exe',  
 'C:\\.env\\Scripts',  
 'C:\\Program Files\\Python311\\python311.zip',  
 'C:\\Program Files\\Python311\\DLLs',  
 'C:\\Program Files\\Python311\\Lib',  
 'C:\\Program Files\\Python311',  
 'C:\\.env\\Lib\\site-packages']

I have tried a bit of everything, such as:

• made sure that I am running inside the virtual environment, executing the pyinstaller inside the virtual environment etc.
• removed any Python path from the Environment Variables
• tried options such as `--paths 'C:\.env\Lib\site-packages' in the attempt to "force" the right directory

My conclusion: for some reason, pyinstaller is not able to use the site-packages of the virtual environment.

[bwoodsend]

C:\Program Files\Python311\Lib is the standard library which is shared with virtual environments. Likewise with python311.dll. If there are paths matching C:\Program Files\Python311\Lib\site-packages then that would be a concern?

What is this module XXX? Does it share a name with something else? Is it a top level import? Are you doing anything that might obfuscate the dependency (using __import__/importlib, exec(), Python subprocesses)? Does the problem go away if you use --hiddenimport=xxx? Is there an entry for xxx in build/myscript/xref-myscript.html?

[enricodragoni]

Hi, apologies for XXX. I try to clarify.
XXX, let's call it "Tools" from now on, is a local package which I have installed in my virtual environment. It is a package that is set-up through a pyproject.toml and gets installed with:
python -m pip install .
The point is that this package is only installed in the virtual environment and I would like to keep it that way. One reason is that in the virtual environment, I rely on specific installations of python packages which are different from the ones in the default python installation.
Note that the Tools package is located at C:.env\Lib\site-packages, i.e. the virtual environment. Pyinstaller is not able to get Tools as it is not looking the the venv's site-packages directory.

I have tried --hiddenimport without success.

[rokm]

The point is that this package is only installed in the virtual environment and I would like to keep it that way.

This makes absolutely no difference, as long as virtual environment's site-packages is in the python search path during analysis, which according to

340 INFO: Module search paths (PYTHONPATH):  
['C:\\.env\\Scripts\\pyinstaller.exe',  
 'C:\\.env\\Scripts',  
 'C:\\Program Files\\Python311\\python311.zip',  
 'C:\\Program Files\\Python311\\DLLs',  
 'C:\\Program Files\\Python311\\Lib',  
 'C:\\Program Files\\Python311',  
 'C:\\.env\\Lib\\site-packages']

seems to be the case (see the very last entry in the list).

Can you open build/<name>/xref-<name>.html and check if "Tools" / "XXX" appears in there? If it does and it is listed as "MissingModule", then PyInstaller indeed cannot find/resolve that module during analysis. If it is not listed at all, then the analysis is not aware that the module is supposed to be imported in the first place, for example, because of indirect/programmatic imports.

How is "Tools" imported? Directly in the main program, using import Tools, or in some other way?

[enricodragoni]

"Tools" is indeed listed in xref-myscript.html as "MissingModule". Actually, all the modules that belong to Tools are listed as missing modules (Tools is a package not a module).
Tools is imported in myscript.py, i.e. the script from which the EXE is build (but I guess this does not really make a difference, I am pretty sure that it would still be missing if imported by another script).

In myscript.py, Tools is imported as:
from Tools.Utils import process_input_data
where Utils is a module inside the Tools package (a directory containing init.py) and process_input_data is a function defined in Utils.

When I am running myscript.py normally (from cmd/power shell/ MS VSCode) using the virtual environment's python interpreter, everything is imported fine. I noticed this import issue only with pyinstaller.

[rokm]

Is your Tools dist perhaps installed in editable mode? PyInstaller is known not to be able to resolve that...

What is the output of the following snippet

import Tools
print(Tools.__spec__)
print(Tools.__file__)

I.e., what is the "true" location of the Tools?

[enricodragoni]

Issue is solved now.

That was exactly the problem: the package Tools was installed in editable mode.
Once I uninstalled the packaged and re-installed it not in editable mode, pyinstaller was able to generated the EXE correctly, i.e. with all the packages from the virtual environment (including Tools).

Thank you very much.