'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 #64
Comments
|
After trying to find a solution myself, I looked through the forks and found this bit of code that fixed my issue: iamahuman@8b8aade I think it should be integrated if it doesn't cause any side-effects (I couldn't see any). |
|
Is this going to be merged soon? |
|
👍 Same error here with Django 1.8 |
|
this problem is still there with the last released version of django-bitfield (1.9.1) |
|
This is still a problem with django-bitfield==1.9.3 and Django 1.11.21 - I get |
|
Can you test whether this is fixed in the 1.9.5 release? I don't have time to debug myself, but I'd happily merge a patch for this and do a new release. |
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
This was referenced Apr 6, 2020
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`FieldListFilter.__init__` populates `used_parameters` as follows.
for p in self.expected_parameters():
if p in params:
value = params.pop(p)
self.used_parameters[p] = prepare_lookup_value(p, value)
`FieldListFilter.expected_parameters` simply returns a singleton list
containing `self.lookup_kwarg`, which is simply a copy of `field_path`.
`used_parameters` therefore has at most one entry with key set to
`field_path` and value from the parameters, since `prepare_lookup_value`
returns the value as-is unless the key ends with either `'__in'` or
`'__isnull'`.
`BitHandler` "represents an array of bits, each as a `Bit` object."
It is one of two classes (the other being `Bit`) that is accepted by the
specialized `exact` lookup. Here, we would only like to filter out
records which has all of the bits in the given integer parameter set.
The lookup uses the value in the given `BitHandler` as a mask, in the
form `field & mask = mask`.
The keys are not used here, since they are simply used for named bit
enumeration and manipulation. This is justified for the following
reasons:
1. The user may supply an integer with multiple bits set which `Bit`
cannot handle.
2. The class is unlikely to change to adopt a strict behavior to filter out anything not in `_keys`
3. This commit only aims to fix the bug and it would be out of the scope.
Later version may make the `keys` parameter of `BitHandler optional
or introduce some new wrapper object or a custom lookup for achieving
this.
This is a bit more efficient implementation compared to the intended one
since instead of doing `field = field | mask`, it instead chooses the
straightforward approach of `field & mask = mask`. SQL databases are
more likely to recognize the last pattern for optimization, although they
should in all way be equivalent.
Plus, empty keyword arguments for `filter` essentially means a no-op, so
in this case we just return `queryset` directly since they are meant to
be immutable anyway.
Fixes disqus#61, disqus#64, and disqus#89
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
ortholeviator
added a commit
to ortholeviator/django-bitfield
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * disqus#89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * disqus#64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * disqus#61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. disqus#26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
timabbott
pushed a commit
that referenced
this issue
Apr 6, 2020
`FieldListFilter.__init__` populates `used_parameters` as follows.
for p in self.expected_parameters():
if p in params:
value = params.pop(p)
self.used_parameters[p] = prepare_lookup_value(p, value)
`FieldListFilter.expected_parameters` simply returns a singleton list
containing `self.lookup_kwarg`, which is simply a copy of `field_path`.
`used_parameters` therefore has at most one entry with key set to
`field_path` and value from the parameters, since `prepare_lookup_value`
returns the value as-is unless the key ends with either `'__in'` or
`'__isnull'`.
`BitHandler` "represents an array of bits, each as a `Bit` object."
It is one of two classes (the other being `Bit`) that is accepted by the
specialized `exact` lookup. Here, we would only like to filter out
records which has all of the bits in the given integer parameter set.
The lookup uses the value in the given `BitHandler` as a mask, in the
form `field & mask = mask`.
The keys are not used here, since they are simply used for named bit
enumeration and manipulation. This is justified for the following
reasons:
1. The user may supply an integer with multiple bits set which `Bit`
cannot handle.
2. The class is unlikely to change to adopt a strict behavior to filter out anything not in `_keys`
3. This commit only aims to fix the bug and it would be out of the scope.
Later version may make the `keys` parameter of `BitHandler optional
or introduce some new wrapper object or a custom lookup for achieving
this.
This is a bit more efficient implementation compared to the intended one
since instead of doing `field = field | mask`, it instead chooses the
straightforward approach of `field & mask = mask`. SQL databases are
more likely to recognize the last pattern for optimization, although they
should in all way be equivalent.
Plus, empty keyword arguments for `filter` essentially means a no-op, so
in this case we just return `queryset` directly since they are meant to
be immutable anyway.
Fixes #61, #64, and #89
|
I believe this is fixed in #103. |
timabbott
pushed a commit
that referenced
this issue
Apr 6, 2020
`Field.get_db_prep_lookup` merely "prepares" the right hand side operand, and returns a (sql, params) pair. For a plain value `params` will simply be the value itself; compound expressions with `as_sql` method, however, are never designed to work with `get_db_prep_lookup` and instead needs their `as_sql` method called directly to retrieve the pair. The error message has been changed from time to time across Django versions. Related issues include: * #89 "can't adapt type 'CombinedExpression'" in admin filter Since Django 1.10 removed `Field.get_db_prep_lookup`, `Lookup.get_db_prep_lookup` implementation is simply as follows. ```python def get_db_prep_lookup(self, value, connection): return ('%s', [value]) ``` `CombinedExpression` is an expression type returned by combining exprssions with operators, such as `__add__` or `bitor`. They are never meant to be passed to the SQL engine directly; instead, it has an `as_sql` method that returns what we want. `process_rhs` first applies any bilateral transforms on both sides, and if it finds either `as_sql` or `_as_sql` method on the RHS, calls it and wraps the SQL in the pair with parentheses; otherwise, it diverts to the usual `get_db_prep_lookup` as above. * #64 'CombinedExpression' object is not iterable when using an admin filter on Django 1.8 Same as above, except `Lookup.get_db_prep_lookup` is as follows. ```python def get_db_prep_lookup(self, value, connection): return ( '%s', self.lhs.output_field.get_db_prep_lookup( self.lookup_name, value, connection, prepared=True)) ``` Following is the relevant part of `Field.get_db_prep_lookup`. ```python def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): """ Returns field's value prepared for database lookup. """ if not prepared: value = self.get_prep_lookup(lookup_type, value) prepared = True if hasattr(value, 'get_compiler'): value = value.get_compiler(connection=connection) if hasattr(value, 'as_sql') or hasattr(value, '_as_sql'): # If the value has a relabeled_clone method it means the # value will be handled later on. if hasattr(value, 'relabeled_clone'): return value if hasattr(value, 'as_sql'): sql, params = value.as_sql() else: sql, params = value._as_sql(connection=connection) return QueryWrapper(('(%s)' % sql), params) if lookup_type in ('month', 'day', 'week_day', 'hour', 'minute', 'second', 'search', 'regex', 'iregex', 'contains', 'icontains', 'iexact', 'startswith', 'endswith', 'istartswith', 'iendswith'): return [value] ``` Normally it is supposed to return the parameters for the SQL "prepared statement" as a `list`; however, if the value has either `as_sql` or `_as_sql`, it either returns the value directly if it also has `relabeled_clone` (which is the case for expressions) or wraps it in `QueryWrapper`. This is not a desired behavior, and using them would result in `TypeError` saying the object was not iterable since they were not lists in the first place. * #61 'SQLEvaluator' object is not iterable when using admin filter on Django 1.7 Same as above, except `SQLEvaluator` emerges from the SQL compiler wrapping expressions for internal use. - Update expressions: django.db.models.sql.compiler.SQLUpdateCompiler.as_sql: ... if hasattr(val, 'evaluate'): val = SQLEvaluator(val, self.query, allow_joins=False) ... - filter() resolution: django.db.models.sql.query.Query.add_filter: ... elif isinstance(value, ExpressionNode): # If value is a query expression, evaluate it value = SQLEvaluator(value, self, reuse=can_reuse) having_clause = value.contains_aggregate ... This commit also causes "regression" for the following issue; its legitimacy, however, is ambiguous considering that the lookup essentially "hijacks" the original definition of 'exact'. #26 admin BitFieldListFilter not working as expected It should be noted, however, that the original behavior is still possible to achieve by wrapping the `int` in a `BitHandler`. Fixing this correctly would involve treating `int` as `Bit` as before while showing a deprecation warning. Meanwhile `filter(flags=Value(4))` will find a record of which `flags` is exactly `4` across all versions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I'm having a problem when using
BitFieldListFilteron Django 1.8Selecting any item in the filter raises the following error:
It looks like this should be fixed for Django 1.8
The text was updated successfully, but these errors were encountered: