Add `PyUnstable_InterpreterState_SetEvalFrameFunc()` function

[vstinner]

C extensions should not use private functions (functions prefixed by _Py), but PEP 523 only added private functions:

• _PyFrameEvalFunction type
• _PyInterpreterState_GetEvalFrameFunc()
• _PyInterpreterState_SetEvalFrameFunc()

I propose promoting these APIs as public functions/types (use PyUnstable_ prefix):

• PyUnstable_FrameEvalFunction type
• PyUnstable_InterpreterState_GetEvalFrameFunc()
• PyUnstable_InterpreterState_SetEvalFrameFunc()

API:

typedef struct _PyInterpreterFrame PyUnstable_InterpreterFrame;

PyFrameObject* PyUnstable_InterpreterFrame_GetFrameObject(PyUnstable_InterpreterFrame *frame);

PyObject* (*PyUnstable_FrameEvalFunction)(PyThreadState *tstate, PyUnstable_InterpreterFrame *frame, int throwflag)

PyUnstable_FrameEvalFunction PyUnstable_InterpreterState_GetEvalFrameFunc(PyInterpreterState *interp)

int PyUnstable_InterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, PyUnstable_FrameEvalFunction eval_frame)

UPDATE: Add PyUnstable_InterpreterFrame and PyUnstable_InterpreterFrame_GetFrameObject.

UPDATE 2: PyUnstable_InterpreterState_SetEvalFrameFunc() can now fail, its return type is int (instead of void).

These functions are already documented at: https://docs.python.org/dev/c-api/init.html#c._PyFrameEvalFunction

• issue
• pull request

---

I hesitate to change PyUnstable_InterpreterState_SetEvalFrameFunc() return type from void to int to allow reporting an error. What do yo think? The implementation is non trivial but it cannot fail currently:

void
PyUnstable_InterpreterState_SetEvalFrameFunc(PyInterpreterState *interp,
                                             PyUnstable_FrameEvalFunction eval_frame)
{
    if (eval_frame == _PyEval_EvalFrameDefault) {
        eval_frame = NULL;
    }
    if (eval_frame == interp->eval_frame) {
        return;
    }
#ifdef _Py_TIER2
    if (eval_frame != NULL) {
        _Py_Executors_InvalidateAll(interp, 1);
    }
#endif
    RARE_EVENT_INC(set_eval_frame_func);
    _PyEval_StopTheWorld(interp);
    interp->eval_frame = eval_frame;
    _PyEval_StartTheWorld(interp);
}

[efimov-mikhail]

I propose promoting these APIs as public functions/types (just remove the _ prefix):

Nit: you possibly mean "_Py" -> "PyUnstable_" transition.

[encukou]

I commented on the issue about _PyInterpreterFrame: python/cpython#141665 (comment)

I hesitate to change PyUnstable_InterpreterState_SetEvalFrameFunc() return type from void to int to allow reporting an error. What do yo think? The implementation is non trivial but it cannot fail currently

I'd support making it int. The fact that it can't fail in practice can be an internal implementation detail (please add asserts).

[vstinner]

I added PyUnstable_InterpreterFrame and PyUnstable_InterpreterFrame_GetFrameObject() to proposed API.

[vstinner]

PyUnstable_InterpreterState_SetEvalFrameFunc() can now fail, its return type is int (instead of void).

[encukou]

Can we add PyObject *PyUnstable_InterpreterFrame_AsFrame(PyUnstable_InterpreterFrame *frame)? I'm +1 with that.

Note that PyUnstable_InterpreterFrame_GetCode already exists.
But I want users to have access to all the other PyFrame API too (even if they need an extra allocation).

[vstinner]

Oops, I meant to add PyUnstable_InterpreterFrame_GetFrameObject(), not PyUnstable_InterpreterFrame_GetCode(). Sorry, bad copy/paste.

@markshannon is against adding PyUnstable_InterpreterFrame_GetFrameObject(): python/cpython#141950 (comment)

[vstinner]

@erlend-aasland @serhiy-storchaka @zooba: What do you think of adding this API without PyUnstable_InterpreterFrame_GetFrameObject()? Users would have to access the internal C API to access the interpreter frame.

[encukou]

With that, would there be any use cases where you won't need private APIs?

If this wouldn't enable any project to avoid private APIs, what's the point?

[zooba]

I don't see the point of making any change here.

If you like, add to the docs/comments that these APIs pre-date the idea of PyUnstable and that's why they don't use it, and if/when we replace them then the new API will probably use it.

Anyone using these (such as me) will understand just as clearly the limited stability of the API.

[vstinner]

I don't see the point of making any change here.

If there is no benefit for PEP 523 use case to have a public function, I would go the opposite way and move the private _PyInterpreterState_SetEvalFrameFunc() API to the internal C API. See python/cpython#141049.

[zooba]

No, it needs to be public and unstable, and if PyUnstable had existed at the time it was created then we would've used it here.

The only correct change is to rename it to PyUnstable, but there's also no need to make that change (foolish consistency etc. etc.). So status quo should win.

[vstinner]

The only correct change is to rename it to PyUnstable, but there's also no need to make that change (foolish consistency etc. etc.). So status quo should win.

I would like that 3rd party projects stop using private functions. The status quo (keeping the existing private function) doesn't help for this goal. Or do you mean that 3rd party projects should continue using existing private undocumented/untested functions?

[zooba]

Then label this one as public despite the underscore, which is how it was initially defined. You are the one calling it private, but it was created to be public (just not stable).

For the sake of compatibility, we can just have an exception for this function from the PyUnstable naming convention.

[encukou]

FWIW, if we want third parties to use the API, I do think we should rename to PyUnstable (and keep the original name, as per PEP 689). There's almost no cost to that, and there's value in consistency of declaring all _Py-prefixed names off-limits.

But, again, this API is only usable in combination with other private API. Renaming only part of what you need doesn't make much sense.