[Backport release-2.27] Rr/sc 60366 sparse global order reader merge (#5417) #5443
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
ihnorton
pushed a commit
that referenced
this pull request
Feb 10, 2025
…5417) (#5443) Backport of #5417 to release-2.27 --- Sc-60366 Implements "pre-process tile order" mode for the sparse global order reader. In brief, this does a first pass over the fragment metadata tile MBRs to create a single unified list of all the (fragment ID, tile ID) pairs arranged approximately in global order. When running natively, the tile order is computed a single time the first time the query is submitted, and then kept in memory. When running on the REST server, the tile order is recomputed for each query message. Evidence indicates that the overhead of this is low; we expect that it is lower than serializing the tile order would be. - Add parallel merge algorithm - Add rapidcheck library and some common scaffolding code - Add `tiledb_submit_a_b` performance testing binary - Add preprocess tile order mode to sparse global order reader, on by default --- TYPE: FEATURE | BUG | IMPROVEMENT DESC: sparse global order reader determine global order of result tiles (cherry picked from commit 3c617e3)
ihnorton
force-pushed
the
backport/pr-5417-to-release-2.27
branch
from
a0da024 to
a4e68e5
Compare
teo-tsirpanis
changed the base branch from
release-2.27
to
backport/pr-5391-to-release-2.27
…5417) (#5443) Backport of #5417 to release-2.27 --- Sc-60366 Implements "pre-process tile order" mode for the sparse global order reader. In brief, this does a first pass over the fragment metadata tile MBRs to create a single unified list of all the (fragment ID, tile ID) pairs arranged approximately in global order. When running natively, the tile order is computed a single time the first time the query is submitted, and then kept in memory. When running on the REST server, the tile order is recomputed for each query message. Evidence indicates that the overhead of this is low; we expect that it is lower than serializing the tile order would be. - Add parallel merge algorithm - Add rapidcheck library and some common scaffolding code - Add `tiledb_submit_a_b` performance testing binary - Add preprocess tile order mode to sparse global order reader, on by default --- TYPE: FEATURE | BUG | IMPROVEMENT DESC: sparse global order reader determine global order of result tiles (cherry picked from commit 3c617e3)
teo-tsirpanis
force-pushed
the
backport/pr-5417-to-release-2.27
branch
from
a4e68e5 to
1459f76
Compare
teo-tsirpanis
approved these changes
Feb 11, 2025
Base automatically changed from
backport/pr-5391-to-release-2.27
to
release-2.27
February 11, 2025 15:55
teo-tsirpanis
changed the base branch from
release-2.27
to
backport/pr-5419-to-release-2.27
teo-tsirpanis
merged commit 4bc25ea
into
backport/pr-5419-to-release-2.27
7 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #5417 to release-2.27
TYPE: IMPROVEMENT
DESC: sparse global order reader determine global order of result tiles