Concurrent compilation changes marshal output in free-threaded builds

[khmyznikov]

Bug description:

Concurrent compilation on free-threaded CPython builds can produce marshal output that differs byte-for-byte from serial compilation of the same source.

This occurs both with the GIL disabled and with the GIL explicitly re-enabled. It reproduces on Windows and Linux/WSL. Standard CPython builds used as controls did not reproduce the difference.

This was discovered while implementing threaded bytecode compilation in pypa/pip#14286.

Reproducer

A standalone standard-library-only reproducer is available here:

• Reproducer script
• Methodology and results

It does not import or execute pip. It:

• creates 64 copies of a minimized source fixture;
• compiles corresponding files serially and with a ThreadPoolExecutor, in fresh child interpreters;
• fixes logical filenames, PYTHONHASHSEED, and SOURCE_DATE_EPOCH;
• compares checked-hash .pyc files or raw marshal.dumps(compile(...)) payloads;
• unmarshals differing outputs and recursively compares documented code-object fields.

Example using the raw marshal backend:

python3.14t repro_threaded_pyc_determinism.py --gil off --backend marshal --workers 4 --output repro-ft-off-marshal --require-difference
python3.14t repro_threaded_pyc_determinism.py --gil on --backend marshal --workers 4 --output repro-ft-on-marshal --require-difference

The output directory must be new or empty.

Expected behavior

Serial and concurrent compilation of the same source, using the same logical filename and deterministic environment settings, should produce byte-identical marshal output.

Actual behavior

On free-threaded builds, some concurrently compiled files have different serialized bytes from their serial counterparts.

The differing payloads have:

• equal lengths;
• equal .pyc headers when using py_compile;
• equal values for all recursively inspected code-object fields after unmarshalling, including bytecode, constants, names, filenames, line tables, exception tables, and nested code objects.

The raw marshal backend reproduces the behavior without py_compile or .pyc headers. Its differing offsets are exactly 16 bytes earlier than the corresponding .pyc offsets, confirming that the .pyc header is not responsible.

An empty semanticDifferences result does not establish that the object identity or alias graph is identical. The precise cause has not yet been determined.

Observed results

Windows

Windows 11 prerelease VM, eight virtual CPUs, CPython 3.15.0rc1 free-threaded build and corresponding standard CPython 3.15 build:

Build	GIL	Backend	Result
Free-threaded	Off	py_compile	Reproduced; 49 differing bytes in a 3,447-byte .pyc
Free-threaded	On	py_compile	Reproduced; 49 differing bytes in a 3,447-byte .pyc
Free-threaded	Off	marshal	Reproduced; 49 differing bytes in a 3,431-byte payload
Free-threaded	On	marshal	Reproduced; 49 differing bytes in a 3,431-byte payload
Standard	On/default	Both	No difference in 10 attempts

Linux under WSL2

Ubuntu 26.04 LTS, kernel 6.18.33.2-microsoft-standard-WSL2, native ext4 storage, four available CPUs:

Build	GIL	Backend	Result
CPython 3.14.7 free-threaded	Off	py_compile	Reproduced on attempt 1
CPython 3.14.7 free-threaded	On	py_compile	Reproduced on attempt 1
CPython 3.14.7 free-threaded	Off	marshal	Reproduced on attempt 1
CPython 3.14.7 free-threaded	On	marshal	Reproduced on attempt 1
CPython 3.14.4 standard	On/default	Both	No difference in 10 attempts

The WSL differences contained 50 differing bytes per output. All inspected decoded code-object field values were equal.

Impact

Although the decoded code objects appear semantically equivalent, byte-different outputs affect:

• reproducible builds;
• artifact hashing;
• build caches;
• deterministic CI output;
• tools that record .pyc hashes.

Re-enabling the GIL on a free-threaded build does not prevent the behavior. As a temporary precaution, pip’s proposed threaded compilation path detects Py_GIL_DISABLED at build time and remains serial for free-threaded builds.

CPython versions tested on:

3.15

Operating systems tested on:

Linux