How to accept dicts with extra keys?

[john375639]

A lot of times I pass data around using dictionaries. When I receive the data, for example as a function argument, what I want is a dict with that contains some specifics keys. I don't mind if the dict has more data.

So, I tried typed dicts:

Movie = TypedDict('Movie', {'name': str, 'year': int})

This works:

movie = {'name': 'Blade Runner', 'year': 1982}  # type: Movie

But this throws the error Extra key 'has_sequel' for TypedDict "Movie":

movie = {'name': 'Blade Runner', 'year': 1982, 'has_sequel': True}  # type: Movie

I can understand that you can't replace for the first value, because the result of keys or items is different.

But if I am only interested in having those keys, not iterating or other stuff, what are my options (if any)?

[ilevkivskyi]

There is a cool new feature called total=False, does it work for you?
Here are some docs http://mypy.readthedocs.io/en/latest/kinds_of_types.html#totality

[gvanrossum]

This looks like a feature request for TypedDict.

[ilevkivskyi]

@gvanrossum Actually I am quite sure total=False covers this, or am I missing something?

[ilevkivskyi]

Hm, I see one problem: potentially, the list of optional keys can be very long (since one still needs to list them).

[gvanrossum]

Sadly no -- total=False allows omitting keys, it doesn't allow extra keys.

# ...imports...
A = TypedDict('A', {'x': int}, total=False)
def f(a: A) -> None:
  print(a['x'])
b: A = {'x': 0, 'y': 0}  # E: Extra key 'y' for TypedDict "A"

[emmatyping]

Yes, I think this is a valid feature request, as enumerating keys in the TypedDict could be painful.

[JukkaL]

This is already supported for function arguments:

from mypy_extensions import TypedDict

A = TypedDict('A', {'x': int})
B = TypedDict('B', {'x': int, 'y': str})

def f(x: A) -> None: ...

b: B = {'x': 1, 'y': 'foo'}
f(b)  # Ok

The issues seems specific to creating a typed dict -- mypy makes sure that no extra keys are provided (as these could be accidental typos or obsolete key names, for example).

I don't think that changing the semantics of total=False is a good idea, since we'd lose a lot of type safety. One option would be to introduce a new flag. Let's call it allow_extra=True for now. Here is an attempt to define it:

• Don't complain about extra keys when creating a typed dict object if allow_extra is true for the type. Allow arbitrary extra keys and values.
• Allow accessing arbitrary keys, such as x['whatever']. The value type would have to be Any unless the key is explicitly defined for the typed dict.
• Allow setting arbitrary keys with arbitrary values, such as x['whatever'] = <anything>.

These semantics seem kind of arbitrary to me. To seriously consider such as feature I'd like to see more concrete examples where the current behavior is not sufficient. If the structure of a typed dict object is not well specified, the current recommendation is to use Dict[str, Any] instead. TypedDict is pretty restricted by design.

If we had intersection types, a similar effect to my proposed allow_extra flag could perhaps be achieved through Intersection[MyTypedDict, Dict[str, Any]]. We don't have any concrete plans to introduce intersection types, however.

[john375639]

Thanks.

I think that I can work with the current behavior.

JukkaL's example is what I am trying to do. Something like:

from typing import Iterable
from mypy_extensions import TypedDict

NamedThing = TypedDict('NamedThing', {'name': str})
Movie = TypedDict('Movie', {'name': str, 'year': int})
Replicant = TypedDict('Replicant', {'name': str, 'model': str})

def slug(x: NamedThing) -> str:
    return x['name'].lower().replace(' ', '-')

blade_runner: Movie = {'name': 'Blade Runner', 'year': 1982}
roy: Replicant = {'name': 'Roy', 'model': 'Nexus-6'}
things: Iterable[NamedThing] = [blade_runner, roy]

for thing in things:
    print(slug(thing))

When trying Mypy I was directly assigning the values or using variables without annotations, like:

blade_runner: NamedThing = {'name': 'Blade Runner', 'year': 1982}

slug({'name': 'Blade Runner', 'year': 1982})

blade_runner = {'name': 'Blade Runner', 'year': 1982}
slug(blade_runner)

[JukkaL]

I'm closing this since it seems that the current behavior is mostly good enough, and the solution to the larger issue would be messy.

[alexjurkiewicz]

I would like this issue to be re-opened. I think extra_keys is important to have.

The current solution of Dict[str, Union[str, int, float, bool]] is significantly less expressive than what other languages offer.

Typescript example:

// Simple example
interface ElasticsearchDocument {
    _id: string
    _type: string
    _index: string
    @timestamp: number
    [key: string]: string|number|boolean;
}

// Complex example: Nested dict
interface SubObject {
  [key: string]: string|number|boolean|SubObject
}
interface DynamoDBDocument {
  index_id: number
  secondary_index_id: number
  [key: string]: string|number|boolean|SubObject
}

The above suggestion of allow_extra still has a capability gap where I cannot define the type of extra items (either the key or value).

I don't have a good sense of the restrictions on syntax that MyPy has to deal with. So this proposal might be totally unreasonable. But the syntax could be:

TypedDict("ElasticsearchDocument",
          {"_id": str},
          extra_keys={str: Union[str,int,float,bool]})

This says: "this is a dictionary with _id as a guaranteed key. There might be other keys, they have string type and value matches Union[str,int,float,bool].

Theoretically you could spec something like this:

TypedDict("ComplexDict",
          {1: str, "a": int},
          extra_keys={int: bool, str: str})

[tuukkamustonen]

I would also like to see extra keys allowed. My use case is in a HTTP/REST client:

class SomeApiClient:

    class RegisterRequest(TypedDict):
        ...

    class RegisterResponse(TypedDict):
        ...

    def register(self, payload: RegisterRequest) -> RegisterResponse
        ...
        return resp_json

1. As payload - this is API that already exists, and accepts both fixed/defined and arbitrary key-value pairs on main level. This cannot be mapped currently. Nesting dynamic data under extra key or similar:

   class RegisterRequest(TypedDict):
       static_field: str
       extra: dict

   ...cannot be done, because the API already exists and cannot be changed just like that.

2. As response - the server returns some irrelevant fields, which are really not worth mapping (but calling code might be interested about them). Or maybe the server adds a new field that the caller is interested about, before RegisterResponse is updated in the next version.

I would happily risk typoing (as explained in #6223 (comment)) just to have extra keys (it would be optional feature, anyway).

[septatrix]

This might be solveable using Protocols, Literals and __getitem__. One could have a Protocol class with a generic __getitem__ and many @overloads of said method using a Literal key and the corresponding return type. Whilst this would require many overloads it could work.

[ML-Chen]

Can this issue be reopened?

[jlumbroso]

@JukkaL As @alexjurkiewicz mentions, the advantage of allowing extra keys is that then TypedDict function as a sort of poor person's interface: "You need to provide at least these fields" with implied that extra keys would be ignored.

From a practical perspective: Python is a language that integrates many different systems, protocols, etc., it's not infrequent for various protocols to have slightly different extensions—the point of standardizations is to try to avoid that, but it's seemed inevitable. Since a lot of protocols communicate in JSON, having TypedDict's that expect several fields for sure but allow any other fields is useful. This is also the case when you have several versions of a protocol—you make the never one a superset of the older one to be backwards compatible.

This is not a duplication of the total flag: The total flag is a global way to address the possibility of optional fields.

So you could have total=False and extra=True for when you allow for certain fields to be optional—and are providing the type just as guide of which fields can be affected/tuned—but also to ignore any superset.

From a programming language theoretical perspective: The topic of typed extensible records is not new, but dates to Mitchell & Wand in the 70s. A recent paper about this can be found here:

• Gaster, B. R., & Jones, M. P. (1996). A polymorphic type system for extensible records and variants. Technical Report NOTTCS-TR-96-3, Department of Computer Science, University of Nottingham.

You may be amused to know the paper begins this way 😊🤗:

Records and variants provide flexible ways to construct datatypes, but the restrictions imposed by practical type systems can prevent them from being used in flexible ways. These limitations are often the result of concerns about efficiency or type inference, or of the difficulty in providing accurate types for key operations.

It then goes on:

Unfortunately, practical languages are often less flexible in the operations that they provide to manipulate
records and variants. For example, many languages—from C to Standard ML (SML)—will only allow the programmer to select the l component, r.l, from a record r if the type of r is uniquely determined at compile-time.

@gvanrossum I suspect having the field extra would be enable the kind of polymorphism that this paper encourages—and which is actually inherent in Python's original "duck" typing, and precisely what made Python such an attractive, and sane language to use for so many people.

[MicahLyle]

A use case that I I think I have with this is integrating with a legacy MongoDB API where a given record could have a bunch of key/value pairs, but I'm really only concerned with say 3-4 of them in the codebase I'm working with. I want to strongly type the 3-4 that I'm concerned with, and not worry about the other ones but rather just pass them around.

[hauntsaninja]

Linking this discussion on typing-sig https://mail.python.org/archives/list/[email protected]/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/#QYOBBLTWVSEWMFRRHBA2OPR5QQ4IMWOL

[JukkaL]

As mentioned in the typing-sig thread, this probably requires a PEP. The author of the PEP should figure out how the subtyping rules would need to be extended and some other technical details. I'm not against making TypedDicts more flexible with a flag that allows arbitrary keys with Any values, and there seems to quite a lot of interest in this. However, I don't care about this quite enough to write a PEP. If anybody wants to volunteer, I can probably find time to at least give feedback on the PEP draft.

[ciaransweet]

@JukkaL I am unfamiliar with all the places requests etc. go - Are you aware of this being discussed/worked on more recently?

I'm looking for the same functionality, TypedDict where it allows arbitrary other keys.

Thanks!

[peterdeme]

Anyone has a good solution for that?

[amin-nejad]

For the time being, if you know what these extra keys are that may or may not be in the dictionary ahead of time, you can include them in the definition but mark them as NotRequired (see PEP 655)

[Rocamonde]

I would also find this very useful. TypedDicts are very limiting without this. I would have to go back to using a generic Dict type instead of giving some information on keys that will certainly be available.

[JoaquimEsteves]

I second the notion that this issue be re-opened.

Coming over from Typescript this has been my biggest pain point with mypy and Python's typing in general.

Discovering this was a helpful surprise, but it's still not good enough - I'm forced to cast dictionaries I'm quite sure fullfill the TypedDict's conditions, which feels like an admission of defeat.

To bring it back into the scope of mypy (instead of delegating it to PEP) I'd suggest adding a new configuration flag.

--allow-extra-keys

[ilevkivskyi]

I don't think we need a new flag for this, I think we can simply:

• Use a dedicated error code (e.g. typeddict-unknown-key) instead of generic typeddict-item for the cases @JukkaL mentioned above
• Still type check other (known) keys, if a known one was found

Then people will be able to simply say --disable-error-code=typeddict-unknown-key. If someone submits a PR doing this, I will approve it (or maybe I can do it myself this or next weekend, but can't say for sure).

[JoaquimEsteves]

@ilevkivskyi I submitted a small PR for this.

Feel free to yell at me when you find mistakes perouse it at your own convenience.

(Apologies if pinging is considered rude)

[ilevkivskyi]

For posterity, if you want mypy to not complain about unknown/extra keys in a TypedDict you can use --disable-error-code=typeddict-unknown-key (starting mypy 1.1.0, didn't make into 1.0.0, sorry).

[StefanBrand]

For Toblerity/Fiona#1125 we are trying to mimic the GeoJSON specification. While there are some reserved keys like type and coordinates, any other keys are allowed. It is currently not possible to type this with TypedDict because it does not allow extra keys.

---

This might be solveable using Protocols, Literals and __getitem__. One could have a Protocol class with a generic __getitem__ and many @overloads of said method using a Literal key and the corresponding return type. Whilst this would require many overloads it could work.

I think this actually does not work. Are there working approaches different from the two below?

Two examples:

1. should pass, fails, playground

from typing import Any, Protocol, Literal, overload

class SomeSpecificKeysDict(Protocol):
    @overload
    def __getitem__(self, key: Literal["foo"]) -> str:
        ...
        
    @overload
    def __getitem__(self, key: Literal["bar"]) -> int:
        ...

my_dict: SomeSpecificKeysDict = { "foo": "a string", "bar": 2 }

This should pass, but it errors with

main.py:14: error: Incompatible types in assignment (expression has type "dict[Literal['foo'], str]", variable has type "SomeSpecificKeysDict")  [assignment]
main.py:14: note: Following member(s) of "dict[Literal['foo'], str]" have conflicts:
main.py:14: note:     Expected:
main.py:14: note:         @overload
main.py:14: note:         def __getitem__(self, Literal['foo'], /) -> str
main.py:14: note:         @overload
main.py:14: note:         def __getitem__(self, Literal['bar'], /) -> int
main.py:14: note:     Got:
main.py:14: note:         def __getitem__(self, Literal['foo'], /) -> str
main.py:14: error: Dict entry 1 has incompatible type "Literal['bar']": "int"; expected "Literal['foo']": "str"  [dict-item]
Found 2 errors in 1 file (checked 1 source file)

2. Should fail, passes, playground

from typing import Any, Protocol, Literal, overload

class SomeSpecificKeysDict(Protocol):
    @overload
    def __getitem__(self, key: Literal["foo"]) -> str:
        ...
        
    @overload
    def __getitem__(self, key: Literal["bar"]) -> int:
        ...

    @overload
    def __getitem__(self, key) -> Any:
        ...
        

my_dict: SomeSpecificKeysDict = { "foo": "a string", "bar": "another_string" }

Success: no issues found in 1 source file

[alecov]

@JukkaL As @alexjurkiewicz mentions, the advantage of allowing extra keys is that then TypedDict function as a sort of poor person's interface: "You need to provide at least these fields" with implied that extra keys would be ignored.

From a practical perspective: Python is a language that integrates many different systems, protocols, etc., it's not infrequent for various protocols to have slightly different extensions—the point of standardizations is to try to avoid that, but it's seemed inevitable. Since a lot of protocols communicate in JSON, having TypedDict's that expect several fields for sure but allow any other fields is useful. This is also the case when you have several versions of a protocol—you make the never one a superset of the older one to be backwards compatible.

This is not a duplication of the total flag: The total flag is a global way to address the possibility of optional fields.

So you could have total=False and extra=True for when you allow for certain fields to be optional—and are providing the type just as guide of which fields can be affected/tuned—but also to ignore any superset.

From a programming language theoretical perspective: The topic of typed extensible records is not new, but dates to Mitchell & Wand in the 70s. A recent paper about this can be found here:

• Gaster, B. R., & Jones, M. P. (1996). A polymorphic type system for extensible records and variants. Technical Report NOTTCS-TR-96-3, Department of Computer Science, University of Nottingham.

You may be amused to know the paper begins this way blushhugs:

Records and variants provide flexible ways to construct datatypes, but the restrictions imposed by practical type systems can prevent them from being used in flexible ways. These limitations are often the result of concerns about efficiency or type inference, or of the difficulty in providing accurate types for key operations.

It then goes on:

Unfortunately, practical languages are often less flexible in the operations that they provide to manipulate
records and variants. For example, many languages—from C to Standard ML (SML)—will only allow the programmer to select the l component, r.l, from a record r if the type of r is uniquely determined at compile-time.

@gvanrossum I suspect having the field extra would be enable the kind of polymorphism that this paper encourages—and which is actually inherent in Python's original "duck" typing, and precisely what made Python such an attractive, and sane language to use for so many people.

The extra argument discussed here and all the aforementioned insights are, in my opinion, the absolutely correct solution for this issue. The proposed marked as solution "throw an error and let users ignore it with a pragma" is a kludge.

The extra argument as proposed correctly encodes the intention that a certain interface allows extensions such as what is done in TypeScript:

{ [string]: any; }

The problems with the "ignore errors" approach are obvious:

1. The type itself does not encode the information that the interface is extensible, which might further cause problems down the road if this information would be useful for a certain typing construct later on.
2. It's essentially a pragma-based "ignore this warning". Linter warnings exist to tag problematic constructs, but this is not a problematic construct — it is an ordinary and well-written piece of code which does not warrant a warning. Warning ignoring is to be used sporadically in well-justified situations, otherwise it just looks and feels wrong to use it as an ordinary code feature.

Thus I believe this issue should be reopened; the proposed solution is just an unacceptable hack.

[erictraut]

@alecov, if you'd like to propose or endorse changes to the Python static type system, the python/typing discussion forum would be a better place than the mypy issue tracker. Mypy is correctly implementing support for TypedDict as it is currently spec'ed. (BTW, I agree with many of your points above.)

[bast1aan]

I also miss this a lot in the current TypedDict functionality, which makes it unsuitable to use in old code bases where I have to deal with large dicts with only a few keys that are of interest for the code that I am refactoring at that moment. Covering all keys that are not of direct importance would add way too much boilerplate to the code base. The same for using @overloaded __getitem__()s with Literals.

For new code we avoid dicts between (public) functions anyway in favor of dataclasses.

Allowing unknown keys is largely only a problem because of mutability (invariance). When I have that problem with ordinary dicts, I solve it with the Mapping[str, object] type, which disallows the mutable behaviour.
Thus, could it be that some kind of immutable TypedDict used for return types, argument types, etc, solve this?
For now, it seems that inheriting (combining) a type from TypedDict and Mapping to indicated both are supported, is not allowed by mypy.

[PIG208]

I would like this issue to be re-opened. I think extra_keys is important to have.

The current solution of Dict[str, Union[str, int, float, bool]] is significantly less expressive than what other languages offer.

Typescript example:

// Simple example
interface ElasticsearchDocument {
    _id: string
    _type: string
    _index: string
    @timestamp: number
    [key: string]: string|number|boolean;
}

// Complex example: Nested dict
interface SubObject {
  [key: string]: string|number|boolean|SubObject
}
interface DynamoDBDocument {
  index_id: number
  secondary_index_id: number
  [key: string]: string|number|boolean|SubObject
}

This opens up a hole like this:

interface A {
    value: string;
    [key: string]: string | number;
}

interface B extends A {
    foo: number;
}

const x: B = {value: "asd", foo: 12}
function mut(v: A) {
    v.foo = "asd"
}
mut(x)
const y: B = x
console.log(y)  // {"value": "asd", "foo": "asd"}

If we are bringing this in Python, do we actually want it to be a sound system, if that's actually possible?

[ikonst]

@PIG208 TypedDicts are, arguably, structural types and allow behaviors that, if you view TypedDicts as nominal types, would appear unsound (e.g. if you think of B as a "base class" of `A in the example above). TypeScript docs discuss this.

[mharding-hpe]

For those wanting this ticket reopened, I just opened a feature request which may be of interest:
#18176