Persistence: Field called "meta" in Document

[sbrandtb]

I would like to define a field called "meta" on my Document. I read and understood the documentation that states:

Having all metadata accessible through meta means that this name is reserved and you shouldn’t have a field called meta on your document. If you, however, need it you can still access the data using the get item (as opposed to attribute) syntax: post['meta'].

Let's suppose I would like to define following Document:

class MyDocument(elastic.Document):
    meta = elastic.Keyword()  

I would like to define my mapping, including the field called "meta". How do I do that? I am ripping my hair out of this since I cannot find any way to do that, like meta_ = elastic.Keyword(actual_field_name='meta'), which would be comparable to other libraries' options (like Pydantic) in DocumentOptions or anything.

I would be also happy if someone could point out a way how I could add it to the mapping in some other way that does not blow elasticsearch-dsl's internals.

Changing the field name in my schema would be possible, but a hassle and from an ideal point of view, I do not want to.

---

However, I would also be curious about the decision making in this case. If I understood correctly, we are solely talking about providing an accessor for metadata for "ORM" model instances and what we are discussing here is not at all touching Elasticsearch's internals since it happily accepts a field called meta. Furthermore, "meta" does not sound like a terrible unlikely name for a document's field. So, if I understand correctly, no one who wants to use elasticsearch-dsl and its declarative persistence can ever declare this field.

Would it not make more sense to provide metadata in some way that has less likelihood of collisions? post.meta_ or post.elastic_meta or elasticsearch.meta(post)?

[miguelgrinberg]

I would like to define my mapping, including the field called "meta". How do I do that?

This appears to work:

import os
from elasticsearch_dsl import Document, field, connections

class MyDoc(Document):
    name: str = field.Keyword()
    meta: str = field.Keyword()

    class Index:
        name = "my-index"

connections.create_connection(hosts=os.environ['ELASTICSEARCH_URL'])
MyDoc.init()
doc = MyDoc(name="susan")
doc['meta'] = "foo"
doc.save()

for doc in MyDoc.search():
    print(f'{doc.name} {doc["meta"]}')

However, I would also be curious about the decision making in this case.
Would it not make more sense to provide metadata in some way that has less likelihood of collisions?

Hard to say. I have not participated in these decisions, which were made many years ago and predate my involvement in this project. I can be wrong on this, but I do not recall this ever being raised before as a big issue, because it is assumed that some concessions have to be made to allow this package to do its work in the most idiomatic way. You always have the option to send raw requests using the regular Python client if the DSL package does not work for you.

[sbrandtb]

@miguelgrinberg Hey, and sorry for the late response.

I do not know what got into me that day, but I totally misinterpreted the issue. Yes, you are right: The mapping of fields with name "meta" just works. No problems here.

My actual problem, and what triggered me, is mypy complaining:

import elasticsearch_dsl as elastic

class MyDocument(elastic.Document):
    meta = elastic.Keyword()

doc = MyDocument(meta='foo')

4: error: Incompatible types in assignment (expression has type "Keyword", base class "ObjectBase" defined the type as "HitMeta")  [assignment]
6: error: Argument "meta" to "MyDocument" has incompatible type "str"; expected "dict[str, Any] | None"  [arg-type]

Note that we can get rid of at least the arg-type error by using elastic.M:

import elasticsearch_dsl as elastic

class MyDocument(elastic.Document):
    meta: elastic.M[str] = elastic.Keyword()

doc = MyDocument(meta='foo')

4: error: Incompatible types in assignment (expression has type "Mapped[str]", base class "ObjectBase" defined the type as "HitMeta")  [assignment]

I am not an expert in typing yoga that frameworks and especially metaclasses require, but I hope that there is a solution (other than ignoring the issue) 🙂

[miguelgrinberg]

Try this:

class MyDocument(elastic.Document):
    meta: elastic.M[str] = elastic.mapped_field(elastic.Keyword())

[sbrandtb]

That does not seem to fix it:

import elasticsearch_dsl as elastic

class MyDocument(elastic.Document):
    meta: elastic.M[str] = elastic.mapped_field(elastic.Keyword())

doc = MyDocument(meta='foo')

Same error from mypy:

4: error: Incompatible types in assignment (expression has type "Mapped[str]", base class "ObjectBase" defined the type as "HitMeta")  [assignment]

[miguelgrinberg]

Yeah, okay, I see that here as well. Unfortunately the name meta is reserved and has a type hint in one of the parent classes of Document. That is unfortunately creating a discrepancy when you create your own attribute with the same name but a different type. So I think the only way is to tell mypy to ignore the error.

[sbrandtb]

Alright, thank you 🙂 If I happen to find a way to make mypy understand, I will contribute it. So long, we can close this