Deserialization error with filters aggregation when keyed=false

[mennasaleh]

Java API client version

8.17.0

Java version

17

Elasticsearch Version

8.17.0

Problem description

I'm trying to use filters aggregation with keyed=false to return sorted buckets but the java client fails to deserialize the aggregation. The exception thrown is:

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.aggregations.FiltersBucket: Property name 'key' is not in the 'type#name' format. Make sure the request has 'typed_keys' set. (JSON path: aggregations['filter#aggs_filter']['filters#aggs_nested'].buckets[0].key) (line no=1, column no=479, offset=-1)
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:123) ~[elasticsearch-java-8.17.0.jar:na]
	at co.elastic.clients.elasticsearch._types.aggregations.MultiBucketBase.lambda$setupMultiBucketBaseDeserializer$0(MultiBucketBase.java:178) ~[elasticsearch-java-8.17.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:284) ~[elasticsearch-java-8.17.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:212) ~[elasticsearch-java-8.17.0.jar:na]
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166) ~[elasticsearch-java-8.17.0.jar:na]

However, typed_keys=true is included in the request and visible in the response. Sample response from kibana:

{
  "aggregations": {
    "doc_count": 1234,
    "filter#aggs_filter": {
      "doc_count": 1234,
      "filters#aggs_nested": {
        "buckets": [
          {
            "key": "some-key",
            "doc_count": 100,
            "sum#my_metric": {
              "value": 123
            }
          }
        ]
      }
    }
  }
}

[l-trotta]

Hello, thanks for reporting this! This is weird, the client forces typed_keys as true in all requests when present. Could you send me an example request that triggers this? Thank you!

[mennasaleh]

Hi @l-trotta, thanks for your response! Here's an example request:

Map<String, Query> productCategory = Map.of(
    "antibiotics", Query.of(q -> q.term(t -> t.field("category").value("1"))),
    "vitamins", Query.of(q -> q.term(t -> t.field("category").value("2")))
);
SearchRequest searchRequest = SearchRequest.of(r -> 
    r.index("products")
     .size(0)
     .aggregations("product-category",
         a -> a.filters(fa -> 
                 fa.filters(Buckets.of(buckets -> buckets.keyed(productCategory)))
                   .keyed(false)
             )
             .aggregations("avg_price", agg -> 
                 agg.avg(aa -> aa.field("price"))
             )
             .aggregations("sort_by_avg_price", agg -> 
                 agg.bucketSort(bs -> 
                     bs.sort(s -> 
                         s.field(fs -> 
                             fs.field("avg_price").order(SortOrder.Desc)
                         )
                     )
                 )
             )
     )
);
SearchResponse<Void> response = esClient.search(searchRequest, Void.class);

The client produces a valid query and receives a proper response (as confirmed in Kibana), but it is unable to decode the response throwing the exception:

Caused by: co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.aggregations.FiltersBucket: Property name 'key' is not in the 'type#name' format. Make sure the request has 'typed_keys' set. (JSON path: aggregations.buckets[0].key) (line no=1, column no=230, offset=-1)
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:123) ~[elasticsearch-java-8.17.0.jar:na]

It occurs during deserialization of the "key" field when buckets are returned as an array of objects

[l-trotta]

@mennasaleh thanks for the additional details! It looks like the client has a bug with .keyed("false") in the filter aggregation: in theory it can handle it by performing a custom deserialization (understanding if it's the array case or map case, since keyed changes the type of the bucket field in the response), but something isn't working, I'm looking into it and will let you know as soon as I discover what's the cause!

[l-trotta]

fixed! 8.17.9 will be out soon including this fix