Projection of fields in nested lists

[fkreis]

Dear spring-data-mongodb maintenance team,
I am having trouble with the projection of fields in nested lists with spring-data-mongodb.

Given the following minimal example mongoDB:

db.nestedList.insertOne(
    {
      "flat": 1,
      "flatDescription": "one",
      "list": [
        {
          "element": 11,
          "description": "eleven"
        },
        {
          "element": 12,
          "description": "twelve"
        }
      ]
    }
 );
 db.nestedList.insertOne(
    {
      "flat": 2,
      "flatDescription": "two",
      "list": [
        {
          "element": 21,
          "description": "twentyone"
        },
        {
          "element": 22,
          "description": "twentytwo"
        }
      ]
    }
 );

Let's assume I only want to know about the fields "flat" and all "list.element". In mongoDB I can simply execute

db.nestedList
    .find({})
   .projection({ flat: 1, "list.element" : 1})

with the result

{
	"_id" : ObjectId("649a8045da17a6faee05e306"),
	"flat" : 1,
	"list" : [
		{
			"element" : 11
		},
		{
			"element" : 12
		}
	]
},

{
	"_id" : ObjectId("649a80a9da17a6faee05e307"),
	"flat" : 2,
	"list" : [
		{
			"element" : 21
		},
		{
			"element" : 22
		}
	]
}

This is exactly my expected and desired behavior!

Now my question is: How can I achieve this exact behavior with Spring Data (Java)?

The intuitive approach

ProjectionOperation projection = Aggregation.project("flat", "list.element");

generates

{ "flat" : "$flat", "element" : "$list.element"}

which produces

{
	"_id" : ObjectId("649a8045da17a6faee05e306"),
	"flat" : 1,
	"element" : [ 11, 12 ]
},
{
	"_id" : ObjectId("649a80a9da17a6faee05e307"),
	"flat" : 2,
	"element" : [ 21, 22 ]
}

which is obviously not what I need, as element is now a property in the root object and also is an array.

Slightly better

ProjectionOperation projection = Aggregation.project("flat");
projection = projection.and("list.element").as("list.element");

generates

{ "flat" : "$flat", "list.element" : "$list.element"}

which produces

{
	"_id" : ObjectId("649a8045da17a6faee05e306"),
	"list" : [
		{
			"element" : [ 11, 12 ]
		},
		{
			"element" : [ 11, 12 ]
		}
	],
	"flat" : 1
},
{
	"_id" : ObjectId("649a80a9da17a6faee05e307"),
	"list" : [
		{
			"element" : [ 21, 22 ]
		},
		{
			"element" : [ 21, 22 ]
		}
	],
	"flat" : 2
}

which is still wrong. At list the structure is correct now but list.element are still arrays.

I even tried

ProjectionOperation projection = Aggregation.project("flat");
projection = projection.and("list").nested(Fields.fields("list.element"));

generates

{ "_id" : "$_id", "list" : { "element" : "$list.element"}}

which produces

{
	"list" : [
		{
			"element" : [ 11, 12 ]
		},
		{
			"element" : [ 11, 12 ]
		}
	],
	"_id" : ObjectId("649a8045da17a6faee05e306")
},
{
	"list" : [
		{
			"element" : [ 21, 22 ]
		},
		{
			"element" : [ 21, 22 ]
		}
	],
	"_id" : ObjectId("649a80a9da17a6faee05e307")
}

which is still wrong as list.element are arrays.

Is this a bug or am I just using it wrong? In the latter case, can you please explain the proper usage? :)
I am looking forward to you response!

Kind regards,
fkreis

[christophstrobl]

Thanks @fkreis for reporting. Unfortunately there's no easy workaround that would result in what you're seeking.
The issue is related to #3435. Let me see what we can do for you to support this scenario.

[christophstrobl]

There's an early branch for this.
Meanwhile Aggregation#state could be an option.

Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }")

[fkreis]

Hello @christophstrobl ,
thank you for looking into this! 😊

I could imagine the following two approaches:

1. Change the behavior of ProjectionOperation projection = Aggregation.project("flat", "list.element"); :
   Instead of { "flat" : "$flat", "element" : "$list.element"} it should generate { "flat" : 1, "list.element" : 1} which for me would even be more intuitive.
2. Support something like CustomJsonAggregation, as suggested in Exclude projection doesn't working [DATAMONGO-2582] #3435:
   Then I could enforce my desired behavior with Aggregation.newAggregation(CustomJsonAggregation("{ "flat" : 1, "list.element" : 1}"))). However, this suggestion was from May 31, 2021. Is there any plan of integrating this kind of feature?

I am looking forward to further discussions and exchange.
Best,
Fabian

[christophstrobl]

I'm not inclined to change the behaviour of a method that has been in service this way for 10+ years.
What would be the benefit of writing Aggregation.newAggregation(CustomJsonAggregation("{ "flat" : 1, "list.element" : 1}")) over Aggregation.newAggregation(Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }"))?

[fkreis]

I'm not inclined to change the behavior of a method that has been in service this way for 10+ years.

Absolutely reasonable! If other options are available we want to avoid breaking changes of course! 👍

What would be the benefit of writing [...]

There is no benefit, I just was not aware of the Aggregation.stage feature. This is exactly what I meant with approach 2 - Thank you! 😊 Please note that #4428 (comment) was a reply to #4428 (comment). We both posted an update almost simultaneously.

[fkreis]

I just want to confirm that Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }") as suggested in #4428 (comment) works as expected in my real world project.

Thanks again for that workaround!