Incremental Co-operative Rebalancing Support for HDFS Connector #9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"TopicPartitionWriters" Map is being cleared on every close call, which is correct if we consider the stop the world scenario. But in Incremental mode there can be case that only few of the partitions have been revoked and task has some partitions with it to read from, In that scenario it will be throwing NullPointerException at the following part of the code. As a consequence the add or the removal of an instance create a high pressure on all instances as they need to recover from last commited file for all the task of the cluster Applying unmerged commit from original project : confluentinc#712
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
"TopicPartitionWriters" Map is being cleared on every close call, which is correct if we consider the stop the world scenario. But in Incremental mode there can be case that only few of the partitions have been revoked and task has some partitions with it to read from, In that scenario it will be throwing NullPointerException at the following part of the code.
As a consequence the add or the removal of an instance create a high pressure on all instances as they need to recover from last commited file for all the task of the cluster
Applying unmerged commit from original project : confluentinc#712
Example of CPU usage before the patch when we add / remove one instance
Will update the PR with screenshot after the patch