If I use exclude I can lose my information

[warvariuc]

If I use exclude I can lose my information. Here is what happens in my auth middleware:

         17         if session_key:
         18             try:
         19                 user = User.objects.exclude('crm_data').get(id=session_key)
         20                 import ipdb; ipdb.set_trace()
         21             except User.DoesNotExist:
         22                 user = None
         23             else:
    ---> 24                 if not user.is_active:
         25                     ...

    # crm_data was not loaded, but was replaced with default value
    ipdb> user.crm_data
    CrmData(notes=[], tags=[])

    # but there are values in `crm_data` in the DB
    ipdb> User.objects.get(id=session_key).crm_data
    CrmData(notes=[], tags=[u'test'])

    # if I accidentally save the instance
    ipdb> user.save()
    ...

    # data in the db is lost
    ipdb> User.objects.get(id=session_key).crm_data
    CrmData(notes=[], tags=[])

[warvariuc]

I'd say it's a critical issue. Still no comment?
only has the same behaviour:

ipdb> User.objects.get(id=session_key).crm_data
CrmData(notes=[], tags=[u'test'])
ipdb> user = User.objects.only('email').get(id=session_key)
ipdb> user.save()
User(id='54ecaa791fdf932a6053ecf5', ...)
ipdb> User.objects.get(id=session_key).crm_data
CrmData(notes=[], tags=[])
ipdb> 

mongoengine==0.8.7

[warvariuc]

mongoengine 0.10.0 does the same. looks like this happens with embedded document fields which have default values:

class User(Document):
    ...
    crm_data = me.EmbeddedDocumentField(CrmData, default=CrmData)

[touilleMan]

Hi,

Thank you for reporting the bug, I've made a test to show it (see touilleMan@cf0b28c)

However I triggered it with exclude, but not with only, can you provide a showcase for this ?

Regarding a fix, I think the trouble is the BaseQuerySet.exclude doesn't keep track of the excluded field.
On the opposite, the only method set the BaseQuerySet.only_fields (see https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/queryset/base.py#L784) which is in turn passed to the Document when created.

The simpler fix would be to mimic the behavior of only by setting only_fields:

    def exclude(self, *fields):
        fields = dict([(f, QueryFieldList.EXCLUDE) for f in fields])
        self.only_fields = get_all_fields() - fields.keys()
        return self.fields(**fields)

This mean we should find a way to retrieve the list of fields for the given collection... @MRigal any idea to do that ?

The other way (much more intrusive iyam) would be to implement an exclude_fields anlong with the only_fields

[warvariuc]

Python 2.7.8 (default, Oct 20 2014, 15:05:19) 
>>> from brain.models.auth import User

>>> user = User.objects.get(email='victor.***')

>>> user.crm_data
CrmData(notes=[], tags=[u'test'])

>>> user = User.objects.only('email').get(email='victor.***')

>>> user.crm_data
CrmData(notes=[], tags=[])

>>> user.save()
User(id='54ecaa791fdf932a6053ecf5', crm_data=CrmData(notes=[], tags=[]), email=u'victor.***', ...)

>>> User.objects.get(email='victor.***').crm_data
CrmData(notes=[], tags=[])

>>> import mongoengine

>>> mongoengine.__version__
>>> '0.10.0'

Model:

class CrmNote(EmbeddedDocument):
    text = me.StringField(required=True)
    author_id = me.ObjectIdField()
    created_at = me.DateTimeField(default=datetime.utcnow)

class CrmData(EmbeddedDocument):
    tags = me.ListField(me.StringField(min_length=2, max_length=20))
    notes = me.ListField(me.EmbeddedDocumentField(CrmNote))

class User(Document):
    crm_data = me.EmbeddedDocumentField(CrmData, default=CrmData)
    ...