Add bucket_sort pipeline aggregation. #9605

dwelsch-esi · 2025-04-11T23:04:48Z

Description

Add bucket_sort pipeline aggregation.

Issues Resolved

Version

Frontend features

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ve aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]>

Signed-off-by: Dave Welsch <[email protected]>

github-actions · 2025-04-11T23:05:01Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

kolchfa-aws · 2025-04-14T17:33:10Z

@sandeshkr419 Could you please review this PR? Thanks!

sandeshkr419

Some minor comments. Please rebase & get rid of unwanted pipeline-agg.md changes.

sandeshkr419 · 2025-05-21T18:38:51Z

_aggregations/pipeline-agg.md

@@ -1,1260 +0,0 @@
---


Since quite a few weeks have passed since you have opened the PR (apologies for the delay in reviewing it), I think we had already migrated this page to: https://github.com/opensearch-project/documentation-website/blob/main/_aggregations/pipeline/index.md - Is that correct?

If yes, let's get rid of this change from the PR?

sandeshkr419 · 2025-05-21T18:43:39Z

_aggregations/pipeline/bucket-sort.md

+
+The `bucket_sort` aggregation is a parent aggregation that sorts buckets of a previous aggregation.
+
+You can specify several sort fields together with a sort order for each. You can sort each bucket based on its key, count, or its sub-aggregations. You can also truncate the buckets by setting `from` and `size` parameters, or use the `from` or `size` parameters without `sort` to truncate the buckets without sorting.


I find this explanation slightly redundant at the last line and had to re-read to understand it. Suggestive change:

For bucket sort aggregations, you can define multiple sort fields with individual sort orders. Each bucket can be sorted by their key, count, or its sub-aggregations. Additionally, you can use the from and size parameters to truncate results, with or without applying a sort.

Thanks for the suggestion!

sandeshkr419 · 2025-05-21T18:51:04Z

_aggregations/pipeline/bucket-sort.md

+| :--              | :--               |  :--            | :--         |
+| `gap_policy`     | Optional          | String          | The policy to apply to missing data. Valid values are `skip`, `insert_zeros`, and `keep_values`. Default is `skip`. |
+| `sort`           | Optional          | String          | A list of fields to sort. See [Sort results](https://opensearch.org/docs/latest/search-plugins/searching-data/sort/). |
+| `from`           | Optional          | String          | The index of the first result to return. Indexing starts at `0`. Must be a non-negative integer. |


Can we also specify the default is 0. I can't phrase it appropriately I guess. Something like this maybe:

The index of the first result to return. Indexing starts at default value 0. Must be a non-negative integer.

sandeshkr419 · 2025-05-21T18:52:28Z

_aggregations/pipeline/bucket-sort.md

+            "sort": [
+              { "total_bytes": { "order": "desc" } }
+            ],
+            "size": 1


Can we include from parameter as well in the request? And ofcourse, update the response accordingly.

Included in the truncation example.

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2025-05-21T20:01:48Z

@sandeshkr419 Thanks for reviewing! I addressed your comments.

sandeshkr419

Thanks for the quick revert on comments! LGTM

* Refactor pipeline aggregations to match other aggregation types. Remove aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]> * Add bucket_sort pipeline aggregation. Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add more details and a link Signed-off-by: Fanit Kolchina <[email protected]> * Remove unnecessary redirect Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> (cherry picked from commit daf7a90) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

natebower

@kolchfa-aws Please see my comments and let me know if you have any questions. Thanks!

natebower · 2025-05-22T11:18:22Z

_aggregations/pipeline/bucket-sort.md

+
+## Example
+
+The following example creates a date histogram with a one-month interval from the OpenSearch Dashboards e-commerce sample data. The `sum` subaggregation calculates the sum of all bytes for each month. Finally, the aggregation sorts the buckets in descending order of number of bytes:


"of number of bytes" => "by number of bytes"?

natebower · 2025-05-22T11:19:53Z

_aggregations/pipeline/bucket-sort.md

+
+## Example response
+
+The aggregation reorders the buckets descending order of total bytes:


"The aggregation reorders the buckets in descending order by total number of bytes"?

natebower · 2025-05-22T11:20:52Z

_aggregations/pipeline/bucket-sort.md

+
+## Example: Truncating the results 
+
+To truncate the results, provide the `from` and/or `size` parameters. The following example performs the same sort, but returns two buckets, starting with the second bucket:


2nd sentence: Delete the comma after "sort". I would also add "operation" after "sort".

I'd like to leave as is. Perform a sort is an established phrase.

dwelsch-esi and others added 3 commits April 10, 2025 16:00

Refactor pipeline aggregations to match other aggregation types. Remo…

8bb98c1

…ve aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]>

Merge branch 'opensearch-project:main' into main

2d0fafa

Add bucket_sort pipeline aggregation.

281b4c5

Signed-off-by: Dave Welsch <[email protected]>

dwelsch-esi requested review from kolchfa-aws, Naarcha-AWS, AMoo-Miki, natebower, dlvenable and epugh as code owners April 11, 2025 23:04

github-actions bot assigned kolchfa-aws Apr 11, 2025

kolchfa-aws added 3 - Tech review PR: Tech review in progress Content gap backport 2.19 labels Apr 14, 2025

kolchfa-aws added backport 3.0 and removed backport 2.19 labels May 6, 2025

sandeshkr419 added this to Performance Roadmap May 12, 2025

github-project-automation bot moved this to Todo in Performance Roadmap May 12, 2025

sandeshkr419 self-assigned this May 12, 2025

sandeshkr419 moved this from Todo to In-Review in Performance Roadmap May 12, 2025

sandeshkr419 suggested changes May 21, 2025

View reviewed changes

github-project-automation bot moved this from In-Review to In Progress in Performance Roadmap May 21, 2025

kolchfa-aws added 5 commits May 21, 2025 15:14

Merge branch 'main' into aggs-bucket_sort-1

dfe2f23

Doc review

054d6f2

Signed-off-by: Fanit Kolchina <[email protected]>

Merge branch 'main' into aggs-bucket_sort-1

909de6d

Add more details and a link

916efa1

Signed-off-by: Fanit Kolchina <[email protected]>

Remove unnecessary redirect

df0a500

Signed-off-by: Fanit Kolchina <[email protected]>

sandeshkr419 approved these changes May 21, 2025

View reviewed changes

kolchfa-aws approved these changes May 21, 2025

View reviewed changes

kolchfa-aws merged commit daf7a90 into opensearch-project:main May 21, 2025
9 checks passed

github-project-automation bot moved this from In Progress to Done in Performance Roadmap May 21, 2025

opensearch-trigger-bot bot mentioned this pull request May 21, 2025

[Backport 3.0] Add bucket_sort pipeline aggregation. #9960

Merged

github-actions bot pushed a commit that referenced this pull request May 21, 2025

Add bucket_sort pipeline aggregation. (#9605) (#9960)

8733cc4

natebower reviewed May 22, 2025

View reviewed changes

kolchfa-aws mentioned this pull request May 22, 2025

Add editorial comments to bucket sort aggregation #9963

Merged

1 task


		The `bucket_sort` aggregation is a parent aggregation that sorts buckets of a previous aggregation.

		You can specify several sort fields together with a sort order for each. You can sort each bucket based on its key, count, or its sub-aggregations. You can also truncate the buckets by setting `from` and `size` parameters, or use the `from` or `size` parameters without `sort` to truncate the buckets without sorting.


		## Example

		The following example creates a date histogram with a one-month interval from the OpenSearch Dashboards e-commerce sample data. The `sum` subaggregation calculates the sum of all bytes for each month. Finally, the aggregation sorts the buckets in descending order of number of bytes:


		## Example response

		The aggregation reorders the buckets descending order of total bytes:


		## Example: Truncating the results

		To truncate the results, provide the `from` and/or `size` parameters. The following example performs the same sort, but returns two buckets, starting with the second bucket:

Add bucket_sort pipeline aggregation. #9605

Add bucket_sort pipeline aggregation. #9605

Uh oh!

Conversation

dwelsch-esi commented Apr 11, 2025

Description

Issues Resolved

Version

Frontend features

Checklist

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

kolchfa-aws commented Apr 14, 2025

Uh oh!

sandeshkr419 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeshkr419 May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented May 21, 2025

Uh oh!

sandeshkr419 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sandeshkr419 May 21, 2025 •

edited

Loading