Improved N-1 join query performance for DW SQL

[lingxiao-microsoft]

Why make this change?

• Back 1 year ago, as DW SQL does not support JSON PATH for converting the execution to json format. Hence we had to use STRING_AGG as the workaround.

• Recently, we've noticed JSON PATH is now supported for outer query in DW, and we can use JSON_OBJECT + JSON_PATH to address the json conversion for N-to-1 relations, which can optimize the performance.

• For N-N relations, we're still looking into any resolutions as JSON_ARRAYAGG does not provide much performance improvements.

• For other scenarios when joins are not needed for a simple SELECT, we will JSON PATH instead of STRING_AGG for better performance.

What is this change?

This PR covers

1. Introduced a feature flag to safeguard the changes, the feature flag is default as False when not provided to avoid any regressions. It will be removed once the changes are validated in production with scoped audiences.
2. For DW query builder, use JSON_OBJECT to generate the columns for sub-queries and applied JSON PATH to handle outer query, which fully replace the need of STRING_AGG.
3. Also, for non-join queries (in which we don't need to handle the relations), used JSON PATH to replace the need of STRING_AGG for better performance as well. This will have impact on aggregations, non-join queries and pagination.
4. Added some helper functions into the unit tests module, which aims to compare the results from GraphQL & DB engine easily for deeply nested queries.

How was this tested?

• Unit Tests

  • As this change does not introduce new scenarios, so mostly added some new test cases to get more coverage when M-M / M-1 join queries are needed.

• Integration Tests

Manual Testing - Join Scenarios

Query 1-1 relation - As expected, optimization applied

Query N-1 relation - As expected, optimization applied

Query 1-N Relation - As expected, optimization not applied

Query N-N Relation - As expected, optimization not applied

Other Scenarios

We've applied the JSON PATH when there is no join in the query to replace the STRING_AGG for better performance.

Aggregation

Non-Join Query

Pagination

• N to 1, total items: 3
  
  

• N to N
  
  
  

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[Aniruddh25]

Overall looks good, but looking for better testing. And making sure we still continue to test when the feature flag is off because thats the default behavior today.
Turning it on always by default in testing keeps us at risk of regression in the false scenario.

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[Aniruddh25]

LGTM, thanks for addressing all comments and answering the questions! We should look at better DwSql testing (not using SQL2019 instance for e.g.) as a follow up to this PR.

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[lingxiao-microsoft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).