_remote_debugging retains stale TLBC arrays in some cases

[liuzhijie-0614]

Bug description

I found stale TLBC arrays in some cases, which may result in incorrect LocationInfo.

The main issue is that tlbc_generation is incremented when thread TLBC indices are allocated or released, but when a TLBC array is modified, the profiler cannot detect the change. This may cause some problems:

1. If the TLBC array grows, a new tlbc_index may keep triggering this error:

PyErr_Format(PyExc_RuntimeError,
    "Invalid tlbc_index %d (array size %zd, corrupted remote memory)",
    tlbc_index, tlbc_entry->tlbc_array_size);

2. When a slot in the TLBC array changes from NULL to tlbc_bytecode_addr, the profiler cannot read the updated slot from its cache. A similar issue occurs when a slot is cleared.

# Linux: ./python -X gil=0 -X tlbc=1 repro_tlbc_bounds.py
import os, threading, time
from _remote_debugging import RemoteUnwinder

go = threading.Event()
stop = threading.Event()

def leaf():
    stop.wait()

threading.Thread(target=leaf, daemon=True).start()
for _ in range(16):
    threading.Thread(target=stop.wait, daemon=True).start()
threading.Thread(target=lambda: (go.wait(), leaf()), daemon=True).start()

time.sleep(0.1)
u = RemoteUnwinder(os.getpid(), all_threads=True, cache_frames=False)
u.get_stack_trace()
go.set()
time.sleep(0.1)
u.get_stack_trace()  # RuntimeError: Invalid tlbc_index 18 (array size 16, ...)

import os, threading, time
from _remote_debugging import RemoteUnwinder

go = threading.Event()
stop = threading.Event()

def leaf():
    stop.wait()

def lines(u):
    return sorted(f.location.lineno for i in u.get_stack_trace()
                  for t in i.threads for f in t.frame_info
                  if f.funcname == "leaf")

threading.Thread(target=leaf, daemon=True).start()
threading.Thread(target=lambda: (go.wait(), leaf()), daemon=True).start()
time.sleep(0.1)
u = RemoteUnwinder(os.getpid(), all_threads=True, cache_frames=False)
print("Before:", lines(u))
go.set()
time.sleep(0.1)
print("Cached:", lines(u))
print("Fresh: ", lines(RemoteUnwinder(
    os.getpid(), all_threads=True, cache_frames=False)))

Before: [8]
Cached: [-1, 8]
Fresh:  [8, 8]

CPython versions tested on

3.15

Operating systems tested on

Linux

Linked PRs

• gh-157660: Fix stale TLBC caches in _remote_debugging #157732
• [3.14] gh-157660: Fix stale TLBC caches in _remote_debugging (GH-157732) #158007

[search1ofall-maker]

Hi @liuzhijie-0614!

Thanks for the detailed report and the minimal reproduction script.

The issue makes complete sense: while global thread index allocation increments tlbc_generation, per-code co_tlbc array expansions do not. As a result, the unwinder's cache invalidation mechanism based on tlbc_generation misses per-code capacity growth.

I plan to work on a fix for this. The proposed approach is:
When the unwinder encounters a valid tlbc_index that exceeds the currently cached array capacity (i.e. tlbc_index >= cached_capacity), instead of immediately raising Invalid tlbc_index, it will force a refresh of the cached PyCodeObject metadata (rereading co_tlbc pointer and its capacity from the target process). Only if tlbc_index remains out of bounds after the cache refresh will it raise an error.

I'm taking this on and will submit a PR shortly!

[liuzhijie-0614]

Thanks for the detailed report and the minimal reproduction script.

The issue makes complete sense: while global thread index allocation increments tlbc_generation, per-code co_tlbc array expansions do not. As a result, the unwinder's cache invalidation mechanism based on tlbc_generation misses per-code capacity growth.

I plan to work on a fix for this. The proposed approach is: When the unwinder encounters a valid tlbc_index that exceeds the currently cached array capacity (i.e. tlbc_index >= cached_capacity), instead of immediately raising Invalid tlbc_index, it will force a refresh of the cached PyCodeObject metadata (rereading co_tlbc pointer and its capacity from the target process). Only if tlbc_index remains out of bounds after the cache refresh will it raise an error.

I'm taking this on and will submit a PR shortly!

One detail to watch out for: cache_tlbc_array() reads through read_ptr() and _Py_RemoteDebug_PagedReadRemoteMemory(), which can return data from the unwinder’s local page cache. Simply invalidating the TLBC cache and calling cache_tlbc_array() again may therefore still retrieve the old co_tlbc pointer or capacity.
We may need to invalidate the relevant cached pages, or bypass the page cache during the refresh, to ensure we read fresh data from the target process.

[liuzhijie-0614]

@pablogsal Could you take a look at this issue when you have a chance? I’m not sure what the best approach is for invalidating the TLBC cache when the array changes without a change to tlbc_generation. There’s a minimal reproducer above. I’d appreciate your thoughts on how to address this.