MongoDB documents with non-ASCII keys break MongoEngine document instantiation

[asiebert]

We came across a DynamicEmbeddedDocument in our mongo instance that contained some accentuated keys, and ended up breaking the whole queryset retrieval through mongoengine.

As far as MongoDB's doc goes, it doesn't seem like there is any restriction on field names beyond the . and $ characters, and mongo(-shell) has no issue dealing with such (non-ASCII of UTF) fields.
http://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names

Here is a minimalist example reproducing the issue (mongoengine 0.10.0), assuming a local DB is running:

$ virtualenv venv && venv/bin/pip install mongoengine ipython
$ venv/bin/ipython

# Document ready-to-paste in iPython
from mongoengine import connect, DynamicDocument

class MongoTest(DynamicDocument):
    '''Test main document'''
    meta = {
        'collection': 'mongo_test',
    }
    connect("test")

# '>' for commands issued in mongo-shell
# ':' for commands issued in iPython

> db.mongo_test.insert({"iamakey_/}+*#!": "value"})
WriteResult({ "nInserted" : 1 })

: MongoTest.objects().all()                                                                                      │
[<MongoTest: MongoTest object>]

> db.mongo_test.insert({"iamakey_/}+*#!helloUTFé": "value"})
WriteResult({ "nInserted" : 1 })

: all = MongoTest.objects().all()
: all
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)

# --truncated stack: iPython internals--

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.pyc in __repr__(self)
     56             return '.. queryset mid-iteration ..'
     57 
---> 58         self._populate_cache()
     59         data = self._result_cache[:REPR_OUTPUT_SIZE + 1]
     60         if len(data) > REPR_OUTPUT_SIZE:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.pyc in _populate_cache(self)
     90             try:
     91                 for i in xrange(ITER_CHUNK_SIZE):
---> 92                     self._result_cache.append(self.next())
     93             except StopIteration:
     94                 self._has_more = False

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.pyc in next(self)
   1385             return self._get_as_pymongo(raw_doc)
   1386         doc = self._document._from_son(raw_doc,
-> 1387                                        _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
   1388 
   1389         if self._scalar:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in _from_son(cls, son, _auto_dereference, only_fields, created)
    729             data = dict((k, v)
    730                         for k, v in data.iteritems() if k in cls._fields)
--> 731         obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
    732         obj._changed_fields = changed_fields
    733         if not _auto_dereference:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in __init__(self, *args, **values)
    128             self._dynamic_lock = False
    129             for key, value in dynamic_data.iteritems():
--> 130                 setattr(self, key, value)
    131 
    132         # Flag initialised

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 22: ordinal not in range(128)

: all
[<MongoTest: MongoTest object>]

: len(all)
1

: all.count()
2

> db.mongo_test.find()
{ "_id" : ObjectId("558b51e14a55a1ecdbf608aa"), "iamakey_/}+*#!" : "value" }
{ "_id" : ObjectId("558b52154a55a1ecdbf608ab"), "iamakey_/}+*#!helloUTFé" : "value" }

The last instructions show that loading the queryset yields an encoding exception, but the cache is still populated, alas without the failing document.

[asiebert]

Played some more with that example, and things are also funny when walking through the queryset:

all = MongoTest.objects().all()
print all
# Boom, aforementioned UnicodeEncodeError

print len(all)
1

print all[0]
MongoTest object

print all[1]
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
…

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.pyc in __getitem__(self, key)
    161             return queryset._document._from_son(queryset._cursor[key],
    162                                                 _auto_dereference=self._auto_dereference,
--> 163                                                 only_fields=self.only_fields)
    164 
    165         raise AttributeError

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in _from_son(cls, son, _auto_dereference, only_fields, created)
    729             data = dict((k, v)
    730                         for k, v in data.iteritems() if k in cls._fields)
--> 731         obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
    732         obj._changed_fields = changed_fields
    733         if not _auto_dereference:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in __init__(self, *args, **values)
    128             self._dynamic_lock = False
    129             for key, value in dynamic_data.iteritems():
--> 130                 setattr(self, key, value)
    131 
    132         # Flag initialised

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 22: ordinal not in range(128)

Somehow the document doesn't count toward the len of the queryset, but is present nonetheless.

[MRigal]

Thanks for reporting @asiebert !
Definitely something we should try to fix soon! If you feel like submitting a PR for this, please do so, I will review it with pleasure!

[asiebert]

You're welcome! I unfortunately don't have enough time to get to the bottom of this, but I hope somebody with a good knowledge of mongoengine internals has enough information here to locate the root cause easily. Hoping the fix to be likewise simple…

2¢ from the POV of my team: this is the first and only occurrence we've ever seen, and would tend to consider non-ascii keys as bad practice (nowadays). Point is, no urgency on the matter.

[wojcikstefan]

MongoEngine's DynamicDocument turns keys from a MongoDB document into attributes of the document object, e.g.:

In [23]: Doc._get_collection().insert_one({'aaa': 'bbb'})
Out[23]: <pymongo.results.InsertOneResult at 0x104543c80>

In [24]: Doc.objects[0].aaa
Out[24]: u'bbb'

However, in Python 2.7 (which we still support), having object attributes that contain unicode characters is a no-go:

In [1]: class A():
   ...:     pass
   ...:

In [2]: a = A()

In [3]: a.ąąą = True
  File "<ipython-input-3-cc46b74fd053>", line 1
    a.ąąą = True
      ^
SyntaxError: invalid syntax

We may look into supporting unicode MongoDB doc keys/Document object attributes after dropping support for Python 2.7.