Repository navigation
Make TSAN tests pass with the GIL disabled in free-threaded builds #117657
Description
Feature or enhancement
We need to make the TSAN tests pass with the GIL disabled before we can disable the GIL by default in free-threaded builds.
This should proceed in two phases:
- Add suppressions for existing TSAN warnings.
- Triage, fix, and remove the suppressions for the warnings enumerated in (1).
How to run the TSAN tests
- Build with TSAN:
env CC=clang CXX=clang++ ./configure --disable-gil --with-thread-sanitizer --with-pydebug && make -j - Run tests:
env TSAN_OPTIONS=suppressions=<repo_root>/Tools/tsan/suppressions_free_threading.txt ./python -m test --tsan -j4
Working on a race
- Verify that the TSAN tests are passing using the steps from above.
- Pick a suppression from the section below and assign it to yourself (edit it and add your username next to it). Some of them may be related. If you find that other suppressions are related to the race you're working on please assign them to yourself or contact their owner if they're already assigned.
- Delete the suppression from
<repo_root>/Tools/tsan/suppressions_free_threading.txt, run the TSAN tests, and verify that the race is reported by TSAN. You may need to comment out unrelated functions (notably,_PyEval_EvalFrameDefault) in order to reproduce the race. - Fix the race.
Suppressions
### Tasks
- [ ] PyInterpreterState_ThreadHead
- [ ] PyThreadState_Clear
- [ ] PyThreadState_Next
- [ ] PyUnstable_InterpreterFrame_GetLine
- [ ] _PyFrame_GetCode
- [ ] _PyFrame_Initialize
- [ ] _PyObject_TryGetInstanceAttribute
- [ ] _Py_slot_tp_getattr_hook
- [x] _multiprocessing_SemLock_acquire_impl
- [x] _multiprocessing_SemLock_release_impl
- [ ] add_threadstate
- [ ] assign_version_tag
- [ ] dump_traceback
- [ ] fatal_error
- [ ] free_threadstate
- [ ] list_get_item_ref
- [x] new_reference
- [ ] pthread_create
- [ ] set_default_allocator_unlocked
- [ ] tstate_delete_common
- [ ] tstate_is_freed
- [ ] type_modified_unlocked
- [ ] write_thread_id
Linked PRs
- gh-117657: Add TSAN suppressions for the free-threaded build #117736
- gh-117657: Quiet TSAN warning about a data race between
start_the_world()andtstate_try_attach()#117828 - gh-117657: Quiet more TSAN warnings due to incorrect modeling of compare/exchange #117830
- gh-117657: Fix data races in the method cache in free-threaded builds #117954
- gh-117657: Quiet erroneous TSAN reports of data races in
_PySeqLock#117955 - gh-117657: Fix small issues with instrumentation and TSAN #118064
- gh-117657: use relaxed loads for checking dict keys immortality #118067
- [gh-117657] _Py_MergeZeroLocalRefcount isn't loading ob_ref_shared with strong enough semantics #118111
- gh-117657: Quiet TSAN warnings about remaining non-atomic accesses of
tstate->state#118165 - gh-117657: Fixes a few small TSAN issues in dictobject #118200
- [gh-117657] Fix some issues with TSAN in typeobject #118249
- gh-117657: Add a couple more TSAN suppressions #118256
- gh-117657: Fix race data race in
_Py_IsOwnedByCurrentThread()#118258 - gh-117657: Fix TSAN list set failure #118260
- gh-117657: Fix data race in
_Py_IsImmortal#118261 - gh-117657: Fix data races when writing / reading
ob_gc_bits#118292 - gh-117657: Disable the function/code cache in free-threaded builds #118301
- gh-117657: TSAN fix race on gstate->young.count #118313
- gh-117657: Don't specialize RESUME_CHECK when specialization is disabled #118349
- gh-117657: Fix TSAN race involving import lock #118523
- gh-117657: Use clang 18 in TSAN builds #118533
- gh-117657: Replace TSAN suppresions with more specific rules #118722
- gh-117657: Log TSAN warnings to separate files and archive them #118747
- gh-117657: initialize_new_array #118769
- [3.13] gh-117657: Fix data races when writing / reading
ob_gc_bits(GH-118292) #118796 - gh-117657: Acquire a critical section around
SemLock.__{enter,exit}__#118812 - gh-117657: initialize_new_array remove race #118842
- gh-117657: Fix QSBR race condition #118843
- [3.13] gh-117657: Acquire a critical section around
SemLock.__{enter,exit}__(GH-118812) #118856 - gh-117657: Fix data races reported by TSAN on
interp->threads.main#118865 - [3.13] gh-117657: Replace TSAN suppresions with more specific rules (GH-118722) #118870
- [3.13] gh-117657: Fix data races reported by TSAN on
interp->threads.main(GH-118865) #118904 - [3.13] gh-117657: Fix QSBR race condition (GH-118843) #118905
- [3.13] gh-117657: Log TSAN warnings to separate files and archive them (GH-118747) #118931
- gh-117657: Fix itertools.count thread safety #119267
- gh-117657: Fix itertools.count thread safety #119268
- gh-117657: Fix missing atomic in dict_resize #119312
- gh-117657: TSAN Fix race in
PyMember_GetandPyMember_Set, forPy_T_OBJECT_EXtype #119368 - [3.13] gh-117657: Fix missing atomic in dict_resize #119416
- [3.13] gh-117657: Fix missing atomic in dict_resize (GH-119312) #119417
- gh-117657: Fix TSAN race in free-threaded GC #119883
- gh-117657: Fix TSAN race in QSBR assertion #119887
- [3.13] gh-117657: Fix TSAN race in free-threaded GC (GH-119883) #119903
- [3.13] gh-117657: Fix TSAN race in QSBR assertion (GH-119887) #119904
- gh-117657: Fix data races report by TSAN unicode-hash #119907
- gh-117657: Add TSAN suppression for
set_discard_entry#119908 - [3.13] gh-117657: Add TSAN suppression for
set_discard_entry(GH-119908) #119914 - gh-117657: Avoid
sem_clockwaitin TSAN #119915 - gh-117657: Fix TSAN reported race in
_PyEval_IsGILEnabled. #119921 - gh-117657: Fix race involving GC and heap initialization #119923
- gh-117657: Fix race involving immortalizing objects #119927
- [3.13] gh-117657: Fix TSAN reported race in
_PyEval_IsGILEnabled. (GH-119921) #119939 - [3.13] gh-117657: Avoid
sem_clockwaitin TSAN (GH-119915) #119992 - [3.13] gh-117657: Fix race involving immortalizing objects (GH-119927) #120005
- [3.13] gh-117657: Fix itertools.count thread safety (GH-119268) #120007
- [3.13] gh-117657: Fix race involving GC and heap initialization (GH-119923) #120038
- gh-117657: Fix some simple races in instrumentation.c #120118
- gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe #120165
- [3.13] gh-117657: Fix TSAN race involving import lock (GH-118523) #120169
- gh-117657: Fix TSan reported data race on ioctl_works #120175
- gh-117657: Make PyType_HasFeature atomic #120210
- [3.13] gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (GH-120165) #120403
- [3.13] gh-117657: Make PyType_HasFeature atomic (GH-120210) #120443
- [3.13] gh-117657: Fix some simple races in instrumentation.c (GH-120118) #120444
- gh-117657: Fix data races for set_inheritable in fileutils.c with free-threaded builds #120458
- gh-117657: Make PyType_HasFeature (exported version) atomic #120484
- gh-117657: Using critical section to make _socket.socket.close thread safe #120490
- gh-117657: Add TSAN suppression for set_default_allocator_unlocked #120500
- [3.13] gh-117657: Add TSAN suppression for set_default_allocator_unlocked (GH-120500) #120510
- gh-117657: Make frame clearing thread-safe #120542
- [3.13] gh-117657: Make PyType_HasFeature (exported version) atomic (GH-120484) #120554
- gh-117657: Move static variables in fileutils.c to thread state #120559
- [3.13] gh-117657: Fix TSan reported data race on ioctl_works (GH-120175) #120654
- [3.13] gh-117657: Fix
__slots__thread safety in free-threaded build (GH-119368) #120655 - gh-117657: Enable TSAN check on macOS with free-threads build #120502
- gh-117657: Fix data races reported by TSAN in some set methods #120914
- [3.13] gh-117657: Fix data races reported by TSAN in some set methods (GH-120914) #121240
- gh-117657: Remove TSAN suppressions for _abc.c #121508
- gh-117657: Fix TSAN races in setobject.c #121511
- [3.13] gh-117657: Fix TSAN races in setobject.c (GH-121511) #121541
- gh-117657: Skip test when running under TSan #121549
- gh-117657: Fix TSan race in _PyDict_CheckConsistency #121551
- [3.13] gh-117657: Skip test when running under TSan (GH-121549) #121553
- [3.13] gh-117657: Fix TSan race in _PyDict_CheckConsistency (GH-121551) #121590
- [3.13] gh-117657: Remove TSAN suppressions for _abc.c (GH-121508) #121598
- gh-117657: Skip tests that fork with threads under TSan #121599
- gh-117657: Fix some races in tracebacks #121748
- [3.13] gh-117657: Skip tests that fork with threads under TSan (GH-121599) #121819
- gh-117657: Avoid race in
PAUSE_ADAPTIVE_COUNTERin free-threaded build #122190 - gh-117657: Skip tests that use threads after fork #122194
- [3.13] gh-117657: Skip tests that use threads after fork (GH-122194) #122198
- [3.13] gh-117657: Avoid race in
PAUSE_ADAPTIVE_COUNTERin free-threaded build (GH-122190) #122475 - gh-117657: TSAN Fix races in
PyMember_GetandPyMember_Set, for C extensions #123211 - gh-117657: fix race:sock_recv_impl suppressions for free-thread building #123697
- gh-117657: remove dump_traceback suppression from list #123871
- gh-117657: Use an atomic store to set type flags. #127588
- gh-117657: Fix data race in new_reference for free-threaded build #129665
- gh-117657: Include all of test_free_threading in TSAN tests #129749
- gh-117657: Skip tests that may cause stack overflows under TSan #129751
- gh-117657: Add test_thread_local_bytecode to TSAN tests #129753
- gh-117657: Fix data race in
dict_dict_merge#129755 - [3.13] gh-117657: Fix data race in
dict_dict_merge(gh-129755) #129808 - gh-117657: Enable test_opcache under TSAN #129831
- [3.13] gh-117657: Enable test_opcache under TSAN (GH-129831) #130597
- gh-117657: Skip some tests when running with TSAN #131555
- gh-117657: Fix TSAN data race in _PyEval_SetTrace assertion #131561
- [3.13] gh-117657: Fix TSAN data race in _PyEval_SetTrace assertion (gh-131561) #131564
- gh-117657: Fix data race in
compare_unicode_unicode_threadsafe#131746 - gh-117657: enable
test_capiunder TSAN #136269 - gh-117657: Remove remaining TSan suppressions #143499
Activity
gh-117657: Quiet TSAN warnings about remaining non-atomic accesses of…
gh-117657: Fix data race in `_Py_IsImmortal` (#118261)
gh-117657: Add a couple more TSAN suppressions (#118256)
gh-117657: Fixes a few small TSAN issues in dictobject (#118200)
Merge branch 'main' into pythongh-117657-tsan-isownedby
gh-117657: Fix race data race in `_Py_IsOwnedByCurrentThread()` (#118258
gh-117657: TSAN fix race on `gstate->young.count` (#118313)
SonicField commentedon Apr 30, 2024
I will look at initialize_new_array
gh-117657: Fix small issues with instrumentation and TSAN (#118064)
[gh-117657] Fix some issues with TSAN in typeobject (#118249)
gh-117657: Don't specialize RESUME_CHECK when specialization is disab…
Merge branch 'main' into pythongh-117657-tsan-func-setversion
pythongh-117657: Use PyMutex based recursive lock for import lock
gh-117657: Fix TSAN list set failure (#118260)
gh-117657: Disable the function/code cache in free-threaded builds (#…
Merge branch 'main' into pythongh-117657-import-lock
Merge branch 'main' into pythongh-117657-tsan-gc-bits
Merge branch 'pythongh-117657-tsan-gc-bits' into tsan-verify-gc-bits
gh-117657: Use clang 18 in TSAN builds (GH-118533)
pythongh-117657: Fix small issues with instrumentation and TSAN (pyth…
[pythongh-117657] Fix some issues with TSAN in typeobject (python#118249
pythongh-117657: Don't specialize RESUME_CHECK when specialization is…
pythongh-117657: Fix TSAN list set failure (python#118260)
pythongh-117657: Disable the function/code cache in free-threaded bui…
pythongh-117657: Use clang 18 in TSAN builds (pythonGH-118533)
pythongh-117657 initialize_new_array
pythongh-117657 initialize_new_array
pythongh-117657: tsan initialize_new_array
gh-117657: Fix data races when writing / reading `ob_gc_bits` (#118292)
pythongh-117657: Fix data races when writing / reading `ob_gc_bits` (p…
[3.13] gh-117657: Fix data races when writing / reading `ob_gc_bits` (G…
pythongh-117657: initialize_new_array
pythongh-117657: initialize_new_array
pythongh-117657: Remove unwanted file
pythongh-117657: initialize_new_array
pythongh-117657: initialize_new_array
pythongh-117657: initialize_new_array
pythongh-117657: initialize_new_array
pythongh-117657: initialize_new_array
gh-117657: Acquire a critical section around `SemLock.__{enter,exit}_…
pythongh-117657: Acquire a critical section around `SemLock.__{enter,…
[3.13] gh-117657: Acquire a critical section around `SemLock.__{enter…
SonicField commentedon May 9, 2024
I'll pick up
insertdict and lookup_tp_dict now I have initialize_new_array done. I'll bundle all three into a pull request. Sounds reasonable?
colesbury commentedon May 9, 2024
@SonicField - great! Would you please mark the
initialize_new_arrayas "ready"?gh-117657: Replace TSAN suppresions with more specific rules (#118722)
pythongh-117657: Replace TSAN suppresions with more specific rules (p…
[3.13] gh-117657: Replace TSAN suppresions with more specific rules (G…
Merge branch 'main' into pythongh-117657-tsan-is-running-main
Merge branch 'main' into pythongh-117657-tsan-log-to-file
Merge branch 'pythongh-117657-tsan-log-to-file' into tsan-verify-log-…
pythongh-117657: Acquire a critical section around `SemLock.__{enter,…
pythongh-117657: Replace TSAN suppresions with more specific rules (p…
pythongh-117657: initialize_new_array remove suppresion
gh-117657: Fix data races reported by TSAN on `interp->threads.main` (#…
pythongh-117657: Fix data races reported by TSAN on `interp->threads.…
gh-117657: Fix QSBR race condition (#118843)
pythongh-117657: Fix QSBR race condition (pythonGH-118843)
[3.13] gh-117657: Fix data races reported by TSAN on `interp->threads…
[3.13] gh-117657: Fix QSBR race condition (GH-118843) (#118905)
gh-117657: Log TSAN warnings to separate files and archive them (#118747
pythongh-117657: Log TSAN warnings to separate files and archive them (…
[3.13] gh-117657: Log TSAN warnings to separate files and archive them (
gh-117657: Fix itertools.count thread safety (#119268)
gh-117657: Fix missing atomic in dict_resize (#119312)
pythongh-117657: Fix missing atomic in dict_resize (pythonGH-119312)
[3.13] gh-117657: Fix missing atomic in dict_resize (GH-119312) (#119417
Merge branch 'main' into pythongh-117657-import-lock
pythongh-117657: Fix TSAN race in free-threaded GC
pythongh-117657: Fix TSAN race in QSBR assertion
gh-117657: Fix TSAN race in free-threaded GC (#119883)
pythongh-117657: Fix TSAN race in free-threaded GC (pythonGH-119883)
gh-117657: Fix TSAN race in QSBR assertion (#119887)
pythongh-117657: Fix TSAN race in QSBR assertion (pythonGH-119887)
[3.13] gh-117657: Fix TSAN race in free-threaded GC (GH-119883) (#119903
pythongh-117657: Fix data races report by TSAN unicode-hash
pythongh-117657: Add TSAN suppression for `set_discard_entry`
[3.13] gh-117657: Fix TSAN race in QSBR assertion (GH-119887) (#119904)
gh-117657: Add TSAN suppression for `set_discard_entry` (#119908)
pythongh-117657: Add TSAN suppression for `set_discard_entry` (python…
pythongh-117657: Remove parking lot related suppressions
[3.13] gh-117657: Add TSAN suppression for `set_discard_entry` (GH-11…
pythongh-117657: Avoid `sem_clockwait` in TSAN
pythongh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`.
pythongh-117657: Fix race involving GC and heap initialization
pythongh-117657: Fix race involving immortalizing objects
gh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`. (#119921)
pythongh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`. (p…
[3.13] gh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`. (G…
Merge branch 'main' into pythongh-117657-immortalize
Merge branch 'main' into pythongh-117657-parking-lot-suppressions
gh-117657: Fix data races report by TSAN unicode-hash (gh-119907)
pythongh-117657: Fix data races report by TSAN unicode-hash (pythongh…
[3.13] gh-117657: Fix data races report by TSAN unicode-hash (gh-119907…
Merge branch 'main' into pythongh-117657-parking-lot-suppressions
gh-117657: Avoid `sem_clockwait` in TSAN (#119915)
pythongh-117657: Avoid `sem_clockwait` in TSAN (pythonGH-119915)
[3.13] gh-117657: Avoid `sem_clockwait` in TSAN (GH-119915) (#119992)
Merge branch 'main' into pythongh-117657-immortalize
gh-117657: Fix race involving immortalizing objects (#119927)
pythongh-117657: Fix race involving immortalizing objects (python#119927
[3.13] pythongh-117657: Fix itertools.count thread safety (pythonGH-1…
[3.13] gh-117657: Fix race involving immortalizing objects (GH-119927) (
[3.13] gh-117657: Fix itertools.count thread safety (GH-119268) (#120007
pythongh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`. (p…
pythongh-117657: Fix data races report by TSAN unicode-hash (pythongh…
pythongh-117657: Avoid `sem_clockwait` in TSAN (python#119915)
pythongh-117657: Fix race involving immortalizing objects (python#119927
gh-117657: Fix race involving GC and heap initialization (#119923)
pythongh-117657: Fix race involving GC and heap initialization (pytho…
Merge branch 'main' into pythongh-117657-import-lock
[3.13] gh-117657: Fix race involving GC and heap initialization (GH-1…
pythongh-117657: Fix TSAN race in free-threaded GC (python#119883)
pythongh-117657: Fix TSAN race in QSBR assertion (python#119887)
pythongh-117657: Add TSAN suppression for `set_discard_entry` (python…
pythongh-117657: Fix TSAN reported race in `_PyEval_IsGILEnabled`. (p…
pythongh-117657: Fix data races report by TSAN unicode-hash (pythongh…
pythongh-117657: Avoid `sem_clockwait` in TSAN (python#119915)
pythongh-117657: Fix race involving immortalizing objects (python#119927
pythongh-117657: Fix race involving GC and heap initialization (pytho…
gh-117657: Fix TSAN race involving import lock (#118523)
pythongh-117657: Fix TSAN race involving import lock (pythonGH-118523)
[3.13] gh-117657: Fix TSAN race involving import lock (GH-118523) (#1…
pythongh-117657: Fix TSan reported data race on ioctl_works
gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (GH-120165)
pythongh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (pythonGH-1…
[3.13] gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (GH-120165) (
gh-117657: Make PyType_HasFeature atomic (GH-120210)
pythongh-117657: Make PyType_HasFeature atomic (pythonGH-120210)
gh-117657: Fix some simple races in instrumentation.c (GH-120118)
pythongh-117657: Fix some simple races in instrumentation.c (pythonGH…
pythongh-117657: Fix some simple races in instrumentation.c (pythonGH…
[3.13] gh-117657: Make PyType_HasFeature atomic (GH-120210) (#120443)
[3.13] gh-117657: Fix some simple races in instrumentation.c (GH-120118…
gh-117657: Add TSAN suppression for set_default_allocator_unlocked (#…
pythongh-117657: Add TSAN suppression for set_default_allocator_unloc…
[3.13] gh-117657: Add TSAN suppression for set_default_allocator_unlo…
gh-117657: Make PyType_HasFeature (exported version) atomic (#120484)
pythongh-117657: Make PyType_HasFeature (exported version) atomic (py…
[3.13] gh-117657: Make PyType_HasFeature (exported version) atomic (G…
Merge branch 'main' into pythongh-117657-set-inheritable
gh-117657: Fix TSan reported data race on ioctl_works (#120175)
pythongh-117657: Fix TSan reported data race on ioctl_works (pythonGH…
[3.13] gh-117657: Fix TSan reported data race on ioctl_works (GH-120175…
gh-117657: Fix `__slots__` thread safety in free-threaded build (#119368
pythongh-117657: Fix `__slots__` thread safety in free-threaded build (…
[3.13] gh-117657: Fix `__slots__` thread safety in free-threaded build (
pythongh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (pythonGH-1…
pythongh-117657: Make PyType_HasFeature atomic (pythonGH-120210)
pythongh-117657: Fix some simple races in instrumentation.c (pythonGH…
pythongh-117657: Add TSAN suppression for set_default_allocator_unloc…