Fail PyGILState_Ensure() If Finalizing?

[ericsnowcurrently]

Feature or enhancement

Proposal:

Once the runtime or current interpreter starts finalizing, we currently exit or stall the current thread as soon as it tries to acquire the GIL, via PyThread_exit_thread() (or PyThread_hang_thread() after gh-105805). That includes in PyGILState_Ensure(). However, in the case of the latter it might make more sense (i.e. friendlier) for the call to fail instead of exiting/hanging the current thread.

FWIW, this is a hypothetical situation I noticed while looking at pystate.c. Also, the same scenario would be affected by gh-124619.

CC @gpshead

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-124622: Add PyGILState_EnsureOrFail() function #129688
• gh-124622: Add PyThreadState_Ensure() function #130012

[gpshead]

A challenge in doing this is that PyGILState_Ensure returns an enum PyGILState_STATE value (defined in Include/pystate.h). That only contains LOCKED and UNLOCKED states today, so callers of the function in existing compiled Python C API code will never expect other values. We'd need to know if they were compiled against a recent enough Python version in order to return a new enum value representing an error because the interpreter is being finalized thus no GIL acquisition could possibly succeed.

[gpshead]

I think a new GIL acquisition and release C API would be needed. The way the existing ones get used in existing C code is not amenible to suddenly bolting an error state onto; none of the existing C code is written that way. After the call they always just assume they have the GIL and can proceed. The API was designed as "it'll block and only return once it has the GIL" without any other option.

[ZeroIntensity]

This is something that seems fatal-error worthy rather than a new PyGILState_STATE value -- there's nothing much a dev can do if they want to call the C API during a finalizing interpreter.

I think a new GIL acquisition and release C API would be needed.

That's been loosely discussed recently (in the context of subinterpreters, rather than the GIL), see #123728 (comment)

[rruuaanng]

If you don't mind, I will sort out the requirements under discussion and publish some issue in the near future.

[gpshead]

Tying threads together, a new PyGILState_EnsureOrSafelyFail() is suggested for this issue in https://discuss.python.org/t/safely-using-the-c-api-when-python-might-shut-down/78850 as allowing modern extension code to actually be able to detect failure would be a good thing.

[ZeroIntensity]

I don't think it's a good idea to add new things with the term "GIL" in them. I'd prefer a totally new API design (e.g., PyThreadState_Ensure) that takes an explicit interpreter rather than an implicit one.

[ZeroIntensity]

Ok, I've been playing around with implementing this. I think we need to very clearly define two things:

• Where should Python stop allowing thread states to be created? When the interpreter starts finalizing? When the runtime starts finalizing? But, there does seem to be some desire in the discussion to use the C API in finalizers, so maybe somewhere later?
• If we go with the latter, then what should happen to threads that are created during finalization? Currently, Py_Finalize assumes that there's only one thread left, so there's no mechanism in place for getting rid of new thread states. (I do have an open PR that changes this for subinterpreters, but not the main interpreter yet.)

[vstinner]

I created #129688 to add PyGILState_EnsureOrFail() function.

[vstinner]

See also comments on a previous similar attempt: #123728