Pkg.gc changes Base.loaded_modules

[fonsp]

Issue

On Julia 1.11.2, I found an example where calling Pkg.gc before loading a package causes some packages to not show up in Base.loaded_modules.

Example

Here, I call import Pluto and I inspect values(Base.loaded_modules). If I did gc before import Pluto, then 20 modules do not show up in the list...

Normal – 68 modules

Details

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.2 (2024-12-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import Pluto

julia> length(Base.loaded_modules)
68

julia> foreach(println, sort(string.(values(Base.loaded_modules))))
ArgTools
Artifacts
Base
BitFlags
CodecZlib
ConcurrentUtilities
Configurations
Core
Dates
Distributed
Downloads
ExceptionUnwrapping
ExpressionExplorer
ExproniconLite
FileWatching
FuzzyCompletions
HTTP
HypertextLiteral
JLLWrappers
LazilyInitializedFields
LibCURL
LibCURL_jll
LibGit2
LibGit2_jll
LibSSH2_jll
Libdl
LinearAlgebra
Logging
LoggingExtras
MIMEs
Main
Malt
MbedTLS
MbedTLS_jll
MozillaCACerts_jll
MsgPack
NetworkOptions
OpenBLAS_jll
OpenSSL
OpenSSL_jll
OrderedCollections
Pkg
Pluto
PlutoDependencyExplorer
PrecompileSignatures
PrecompileTools
Preferences
Printf
REPLExt
Random
RegistryInstances
RelocatableFolders
SHA
Scratch
Serialization
SimpleBufferStream
Sockets
TOML
Tar
Test
TranscodingStreams
Tricks
URIs
UUIDs
Zlib_jll
libblastrampoline_jll
nghttp2_jll
p7zip_jll

With Pkg.gc before the import – 48 modules

Details

➜  ~ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.2 (2024-12-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> gc
      Active manifest files: 73 found
      Active artifact files: 198 found
      Active scratchspaces: 10 found
     Deleted no artifacts, repos, packages or scratchspaces

julia> import Pluto

julia> length(Base.loaded_modules)
48

julia> foreach(println, sort(string.(values(Base.loaded_modules))))
Artifacts
Base
BitFlags
CodecZlib
ConcurrentUtilities
Configurations
Core
Distributed
ExceptionUnwrapping
ExpressionExplorer
ExproniconLite
FileWatching
FuzzyCompletions
HTTP
HypertextLiteral
JLLWrappers
LazilyInitializedFields
Libdl
LinearAlgebra
LoggingExtras
MIMEs
Main
Malt
MbedTLS
MsgPack
OpenBLAS_jll
OpenSSL
OpenSSL_jll
OrderedCollections
Pluto
PlutoDependencyExplorer
PrecompileSignatures
PrecompileTools
Preferences
Random
RegistryInstances
RelocatableFolders
SHA
Scratch
Serialization
SimpleBufferStream
Sockets
Test
TranscodingStreams
Tricks
URIs
Zlib_jll
libblastrampoline_jll

The missing modules (e.g. Dates) are loaded and working:

julia> Pluto.Dates.isleapyear(2025)
false

Explicitly calling import Dates adds it to the list:

julia> length(Base.loaded_modules)
48

julia> import Dates

julia> length(Base.loaded_modules)
49

Environment

I created an empty env with ]add Pluto.

On Julia 1.10.7, I could not reproduce the issue.

Context

This is a reduced example from a weird bug I found today: Pluto mostly worked, but I was seeing a strange error where deserialization failed, because the Dates stdlib was not found in Base.loaded_modules:

┌ Error: Unkown error during eval_format_fetch_in_workspace
│   ex =
│    Error deserializing data from Malt.Worker on port 9314 with PID 7314:
│    
│    KeyError: key Base.PkgId(Base.UUID("ade2ca70-3891-5945-98fb-dc099432e06a"), "Dates") not found
└ @ Pluto.WorkspaceManager ~/Documents/Pluto.jl/src/evaluation/WorkspaceManager.jl:390

I eventually traced this bug back to the Pkg.gc that I did earlier today...

[IanButterworth]

It's loading Pkg via the repl switch that's causing the effect, so it's a require_stdlib issue (cc. @vtjnash)

% julia +1.11
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.2 (2024-12-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> length(Base.loaded_modules)
12

julia> import Pluto

julia> length(Base.loaded_modules)
68

ian@Ians-MacBook-Pro julia % julia +1.11
julia> length(Base.loaded_modules)
12

(@v1.11) pkg>

julia> import Pluto

julia> length(Base.loaded_modules)
48

% julia +1.11
julia> length(Base.loaded_modules)
12

julia> import Pkg

julia> import Pluto

julia> length(Base.loaded_modules)
68

[vtjnash]

The issue, as I understand it, is that _include_from_serialized only registers the newly loaded module with register_root_module when the register argument is set, and forgets to also ensure anything from maybe_loaded_precompile also got registered (and the same in _require_search_from_serialized when stalecheck is set)

[vtjnash]

Note that directly accessing values(Base.loaded_modules) will trigger UB on all Julia versions (since it is private and not thread safe), so please use one of the accessor functions instead. Regardless though, this is a bug that needs to be fixed to avoid torn environments after loading.

[bvdmitri]

Interestingly enough, Base.loaded_modules_order doesn't seem to suffer from issue and reports the correct length

(@v1.11) pkg>

julia> import Pluto

julia> length(Base.loaded_modules_order)
79

julia> length(Base.loaded_modules)
66