Open
Description
Spark has overhauled their integration points in the last few releases to make integrating with the framework easier to accomplish, especially in a pure SQL environment. We should get the ball rolling on supporting DSV2, starting with the latest Spark release. There are a number of features that we've been waiting for in Spark that are landing with these new API's, one of which looks promising for pushing down aggregations to ES.
Right now, it seems that official developer guides are a bit scarce in this regard but the API footprint is available to inspect at https://github.com/apache/spark/tree/v3.2.0/sql/catalyst/src/main/java/org/apache/spark/sql/connector