Is it possible to include the entire standard lib?

[jessielw]

I know this is a niche use case but I've came to a problem where I need to allow users to dynamically load plugins and if the main program hasn't imported a module from the standard lib then they won't have access to it.

I've played around with adding hidden imports in a module and in the spec file using both https://pypi.org/project/stdlib-list/#:~:text=This%20package%20includes%20lists%20of,don't%20need%20this%20library and from 3.10 on built in python methods.

However, this will include some but not all libs causing failures.

Is this possible?

Note: im bundling in one dir mode

[rokm]

Is this possible?

There's no built-in option for that, but it should be doable via hiddenimports in a way that you already attempted.

However, this will include some but not all libs causing failures.

What modules do you end up missing if you use the lists from stdlib-list? Those seem to include both top-level stdlib modules/packages and their submodules.

Whereas sys.stdlib_module_names seems to include only top-level module/package names, so you might need to combine it with PyInstaller.utils.hooks.collect_submodules (while explicitly exempting this and antigravity).

[jessielw]

Thanks for the response. I was able to sit down here and play around with things. Looks like adding them specifically in the spec file is the ideal solution here vs actually using it as a hiddenimports = in a python file. Is this the recommended solution?

Alternatively, I did some digging, stdlib_list outputs 1027 modules currently on my version of python (3.11/Windows) where as iterating/using collect_submodules returns 998. Granted, stdlib_list contains this/antigravity.

Here is the missing modules:

MISSING: __main__
MISSING: _crypt
MISSING: _curses
MISSING: _curses_panel
MISSING: _dbm
MISSING: _gdbm
MISSING: _posixshmem
MISSING: _posixsubprocess
MISSING: _scproxy
MISSING: antigravity
MISSING: crypt
MISSING: curses
MISSING: curses.ascii
MISSING: curses.has_key
MISSING: curses.panel
MISSING: curses.textpad
MISSING: fcntl
MISSING: grp
MISSING: nis
MISSING: os.path
MISSING: ossaudiodev
MISSING: posix
MISSING: pty
MISSING: pwd
MISSING: pyexpat.errors
MISSING: pyexpat.model
MISSING: readline
MISSING: resource
MISSING: spwd
MISSING: syslog
MISSING: termios
MISSING: this
MISSING: tty
MISSING: xxsubtype

I know this has nothing to do directly with Pyinstaller, but it looks like I won't be able to rely on sys.stdlib_module_names + collect_submodules.

Thanks for the response and helping me get on the right path!

Edit: It appears that most of these are unix modules. So it makes sense they'd be missing since I am on Windows. So I might just be able to reliably create the build then without using the external package.

[rokm]

Looks like adding them specifically in the spec file is the ideal solution here vs actually using it as a hiddenimports = in a python file. Is this the recommended solution?

The recommended solution is to add them to hiddenimports list that is passed to the Analysis instance in the .spec file.

I'm not sure what you mean by "hiddenimports = in a python file"?

[jessielw]

Looks like adding them specifically in the spec file is the ideal solution here vs actually using it as a hiddenimports = in a python file. Is this the recommended solution?

The recommended solution is to add them to hiddenimports list that is passed to the Analysis instance in the .spec file.

I'm not sure what you mean by "hiddenimports = in a python file"?

Thanks I just got the terminology mixed up. I was thinking a hook file that has hiddenimports = ["whatever"] in a python file. Appreciate the response and help!

Everything appears to be working well!