[Backport release-2.27] Improve readers by parallelizing I/O and compute operations (#5401) #5451
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This removes the read from waiting on all I/O operations and instead moves the I/O task to be owned by the datablock itself. If the I/O threadpool task is valid, we block on data access. This lets I/O and compute be interleaved by only blocking on data when its ready to be processed and allows for better background data loading.
This allows for copying the task/future an enabled multiple threads to check the status of the task in a thread-safe manner.
…checking. While the ThreadPool::SharedTask is designed to be used by multiple threads, its designed for copying. The data structure itself is not thread safe. A recursive mutext is needed because some functions like load_chunk_data call back into filtered_data() and would deadlock. This could be handled by also release the locking in load_chunk_data(), but a recursive_mutex is used for better safety against deadlocks.
… forward declaration issues currently
This is needed because we need to access the data buffer from inside the unfiltering task to unfilter into. We can't block on unfiltering being done from inside the unfiltering task so we need different accessors which let us bypass the check on if the unfiltering task is completed.
This is needed because zip_coordinates is called from the unfilter task itself.
This reverts commit 9647fff.
teo-tsirpanis
approved these changes
Feb 12, 2025
ypatia
approved these changes
Feb 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #5401 to release-2.27
TYPE: IMPROVEMENT
DESC: Improve readers by parallelizing I/O and compute operations