Why does a dictionary union cause a type checking failure?

[dsotirho-ucsc]

https://mypy-play.net/?mypy=latest&python=3.12&gist=2c48503ebd2236fad252604292710529

Why does case c fail while a and b are okay?

type AnyJSON = JSON | str
type JSON = dict[str, AnyJSON]

a: JSON = {} | {}
b: JSON = {'': ''}
c: JSON = {'': ''} | {} # error

main.py:6: error: Incompatible types in assignment (expression has type "dict[str, str]", variable has type "JSON")  [assignment]
main.py:6: note: "dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
main.py:6: note: Consider using "Mapping" instead, which is covariant in the value type
Found 1 error in 1 file (checked 1 source file)

[Daraan]

Nice question :)

One needs to think here about assignability and when types are assigned / inferred, this involves which kind of python statements you are using.

Here are some more interesting points to look at:

b: JSON = reveal_type({'': ''})

You might expect that the revealed type is dict[str, str], but it is (pyright/mypy): dict[str, dict[str, AnyJSON] | str], actually type-checkers see it already as JSON.

Now lets look at:

from typing import cast
b: JSON = cast("dict[str, str]", {'': ''}) # error
d: dict[str, str] = {}
b = d # error

I hope you can see why this fails. Because of invariance, you cannot assign a dict[str, str] to a variable of type dict[str, AnyJSON] (i.e., JSON). I am allowed to put another JSON into b, but if b is d, I would but a dict into a dict[str, str].

Lastly lets check out:

c: JSON = {'': ''} | cast("JSON", {}) # OK

Why is this fine and {'': ''} | {} not and what is the difference to the other cases.
The answer is you use an expression |, which is evaluated first and then the result is tried to be assigned to JSON.
So if we look at the | or operator or rather how dict.__or__ is typed, the result of {'' | ''} | {} is dict[str, str] which is not assignable because of invariance.
As we involve a runtime expression for type-checkers this case is it like the error cases with b = d above. So in this case you need to either start with a JSON on the left or on the right for dict.__or__ to also produce a JSON in the outcome of the expression.

[erictraut]

There are some additional subtleties here. To understand this behavior, you need to first understand bidirectional type inference.

Normally, a static type checker infers the type of expression {'': ''} as dict[str, str]. This type is not assignable to JSON due to invariance. However, with bidirectional type inference, a type checker can take into account additional "inference context", sometimes referred to as an "expected type". In the statement b: JSON = {'': ''}, the "expected type" is JSON, which influences the inference of {'': ''}.

Bidirectional type inference can be applied to a variety of expression forms including literal container expressions (lists, sets, dicts, tuples), as well as comprehensions, assignments, unary operators, lambdas, and call expressions. In general, it cannot be applied to binary operators like | because these do not map to simple calls. The | operator maps to some combination of calls to __or__ or __ror__ depending on the types of the two operands. This can get very complicated if the operand types are unions. For this reason, type checkers do not generally apply bidirectional type inference for binary operators (or augmented assignments or ternary operators) and instead evaluate the LHS and RHS operands without the benefit of an "expected type". The operator is then evaluated using the resulting types.

In your example c: JSON = {'': ''} | {}, the LHS (without the benefit of bidirectional type inference) is evaluated as dict[str, str]. By default, pyright's default inference rules infer the RHS as dict[Unknown, Unknown], but in this case, pyright applies a limited form of bidirectional type inference and leverages the LHS type as the "expected type". It therefore evaluates the RHS type as dict[str, str]. It then evaluates the | operator, which in this case maps to the __or__ magic method and produces a result of dict[str, str]. This isn't assignable to JSON, so a type violation is reported.

Extending your example, d: JSON = b | {} type checks without error. In this case, the normal type evaluation rules (without the benefit of bidirectional type inference) evaluate the type of b | {} as dict[str, AnyJSON], which is assignable to JSON.