gh-151518: Avoid STW starvation of attaching threads

[tpn]

Fixes #151518.

Repeated stop-the-world requests can starve a thread returning from a detached operation. After start_the_world() restores its state to DETACHED, the next requester can suspend it again before its OS thread completes _PyThreadState_Attach().

This change distinguishes thread states suspended while detached from those suspended while attached. An active attach waiter records that it encountered the detached-origin suspension; subsequent stop-the-world scans let that waiter attach before suspending it again. The waiter remains in the stop-the-world countdown until it actually stops. Threads detached for sleep or I/O without an active attach attempt remain immediately parkable, and the uncontended attach path remains a single CAS.

The waiting flag reuses trailing _PyThreadStateImpl padding. Existing state numbers are preserved, and both suspended states resume directly to DETACHED.

The subprocess regression explicitly uses -X gil=0, waits for the collector to start, and repeatedly returns from sleep while another thread calls gc.collect() in a tight loop. It performs 50 reattachments with one second of total intentional sleep and relies on the test framework's outer timeout for stalled progress or shutdown.

Validation on Linux x86-64 after rebasing onto 89c67a98aee:

• Debug free-threaded build (--with-pydebug --enable-safety --enable-slower-safety --disable-gil): test_free_threading test_gc test_threading test_capi test_embed passed, 2,211 tests run. test_interpreters is unsupported in this build and skipped.
• Release free-threaded build: test_free_threading test_gc test_threading test_embed passed, 647 tests run.
• Debug free-threaded binary with -X gil=1: GC, threading, embedding, and free-threaded GC tests passed, 403 tests run. The final regression also passed separately with this parent setting.
• Normal GIL build, using a separate bytecode cache: test_gc test_threading test_capi test_embed test_interpreters passed, 2,132 tests run. test_free_threading skipped as expected.
• Final regression passed -R 3:3 reference-leak checking. Seven additional stress trials passed with mixed global/local pauses, repeated detach/reattach, and workers blocked waiting for collector completion.
• Compiled layout comparison matched all 64 common measurements: _PyThreadStateImpl remains 18,200 bytes, and the new flag overlays padding at offset 18,136. make patchcheck and git diff --check passed.

The original B200 experiment reported a timeout in 2/3 unpatched runs and 5/5 patched passes. On this host, current unpatched main passed all three trials of the updated regression (2.46–4.63 seconds), so these local results do not re-establish a failing baseline. Reproduction remains sensitive to scheduling and hardware.

These local builds lacked the optional _decimal, _hashlib, _ssl, and _tkinter modules; _sqlite3 was disabled.

• Issue: Free-threaded CPython can starve a thread reattaching during repeated stop-the-world GC #151518

[Copilot]

Pull request overview

Fixes a free-threaded stop-the-world (STW) fairness issue where repeated STW requests could repeatedly re-park a thread that is trying to reattach, leading to starvation (notably after returning from detached operations like time.sleep()).

Changes:

• Add a distinct thread-state for threads suspended while detached (_Py_THREAD_SUSPENDED_DETACHED) and update STW parking/unparking logic accordingly.
• Track “attach-waiters” across STW passes via a new cold _PyThreadStateImpl flag (stw_attach_waiting) reused from existing padding, so subsequent STW requests avoid re-parking those threads until they attach.
• Add a regression test exercising a tight gc.collect() loop vs. a sleeping thread, plus a NEWS entry and updated state documentation.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Python/pystate.c	Implements the new suspended-detached state and attach-waiter logic to prevent STW starvation.
Include/internal/pycore_pystate.h	Documents the new suspended-detached state and updates state constants/diagram.
Include/internal/pycore_tstate.h	Adds stw_attach_waiting in reused tail padding for free-threaded builds.
Lib/test/test_free_threading/test_gc.py	Adds a subprocess regression test for STW starvation during tight GC loops.
Misc/NEWS.d/next/Core_and_Builtins/2026-06-16-19-20-00.gh-issue-151518.e6v0Js.rst	Announces the fix in NEWS.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.