Nondeterminism observed on pandas in typeshed primer

[hauntsaninja]

mypy_primer in typeshed has started flakily showing the introduction or removal of an error on pandas.

This typically looks like:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/computation/ops.py:328: error: Need type annotation for "_binary_ops_dict" (hint: "_binary_ops_dict: dict[<type>, <type>] = ...")  [var-annotated]

You can see this in a number of typeshed PRs https://github.com/python/typeshed/pulls?q=is%3Apr+_binary_ops_dict

Here's an example error from April 7 python/typeshed#15622 (comment). Typeshed updated to 1.20 on April 1.

The interesting thing is I suspect this will not repro with stock mypy (e.g. we haven't seen it in mypy's mypy_primer)

I think what is going in is that you also need these typeshed PRs to operator.py. I reverted these PRs for us because they are bad for our users (for other reasons): #21232 (comment)

[JelleZijlstra]

I set Codex on this a while ago but didn't submit its PR yet because I didn't feel like it was addressing the root cause. Codex thought it was because operator and _operator are part of an SCC and some types are different dependent on the order in which the files in the SCC are processed.

Depending on the order, some dict in pandas gets inferred as either dict[str, Overload(...)] or dict[str, function], and one of the two leads to errors. Codex proposed to fix that by changing how we union those two types, which felt like a bandaid.

If you want the LLM's full verbose output:

Details

Title

Nondeterministic inference for operator functions can make dict(zip(...)) alternate between precise callable types and builtins.function

Summary

I observed nondeterministic mypy output in mypy-primer on pandas. The reported diff was that this pandas error sometimes appeared and sometimes disappeared with identical code:

pandas/core/computation/ops.py:328: error: Need type annotation for "_binary_ops_dict"  [var-annotated]

The code is roughly:

import operator

ARITH_OPS_SYMS = ("+", "-", "*", "/", "**", "//", "%")
_arith_ops_funcs = (
    operator.add,
    operator.sub,
    operator.mul,
    operator.truediv,
    operator.pow,
    operator.floordiv,
    operator.mod,
)
_arith_ops_dict = dict(zip(ARITH_OPS_SYMS, _arith_ops_funcs, strict=True))

_binary_ops_dict = {}

for d in (_cmp_ops_dict, _bool_ops_dict, _arith_ops_dict):
    _binary_ops_dict.update(d)

Repeated runs against the same checkout sometimes infer _arith_ops_dict as a precise overloaded-function dict, and sometimes as a bare builtins.function dict. That then changes the loop item type for d.

Observed shapes

In one run:

for d in (...): d
# dict[str, def (Any, Any) -> Any]

In another run:

for d in (...): d
# dict[str, def (Any, Any) -> Any] | dict[str, function]

The latter shape prevents partial dict inference for _binary_ops_dict = {}, so mypy emits the annotation error.

Likely source of nondeterminism

The root appears to be interaction between typeshed’s operator / _operator stubs and mypy’s SCC processing.

In current typeshed, there is an import cycle:

# operator.pyi
from _operator import add, sub, mul, ...

# _operator.pyi
from operator import attrgetter, itemgetter, methodcaller

So operator and _operator are in the same SCC. Mypy’s build graph / SCC code passes around sets in a few places, for example SCCs and ready groups. Depending on processing order within that SCC, operator.add and friends may be seen as precise overloaded functions or as the coarser fallback type builtins.function.

That gives _arith_ops_dict two possible inferred shapes:

dict[str, Overload(...)]

or:

dict[str, function]

Then later union simplification preserves this difference, which leaks into pandas’ _binary_ops_dict inference.

Notes

• This was observed with mypy-primer using compiled mypy.
• The pandas/typeshed code was unchanged between runs.
• The typeshed PR that exposed it was unrelated (sys._jit), so it likely just perturbed processing enough to reveal an existing nondeterminism.
• A local mitigation was to make union simplification prefer a precise function-like type over bare builtins.function when the latter is only a fallback inside the same container shape. That makes the pandas loop infer one deterministic type, but it does not fully explain/fix the earlier operator/_operator shape flip.

[hauntsaninja]

Hm I kind of doubt that explanation... I will ask my Codex to investigate