[improve] modify the negativeACK structure to reduce memory overhead#1410
Merged
thetumbled merged 13 commits intoapache:masterfrom Oct 18, 2025
Merged
[improve] modify the negativeACK structure to reduce memory overhead#1410thetumbled merged 13 commits intoapache:masterfrom
thetumbled merged 13 commits intoapache:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the negative acknowledgments (nack) tracker to improve memory efficiency and timing precision by implementing changes from Pulsar Java's PIP-393. The implementation replaces a simple map-based approach with a more sophisticated time-ordered structure using bitmaps to reduce memory overhead.
- Replaced map-based storage with AVL tree ordered by time and roaring bitmaps for entry tracking
- Added configurable precision bit parameter to control timing granularity and memory usage
- Updated test infrastructure to support the new nack precision bit parameter
Reviewed Changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| pulsar/negative_acks_tracker.go | Core refactoring replacing map with AVL tree and roaring bitmaps for memory-efficient nack tracking |
| pulsar/consumer.go | Added NackPrecisionBit configuration option to ConsumerOptions |
| pulsar/consumer_impl.go | Propagated nack precision bit parameter to partition consumer options |
| pulsar/consumer_partition.go | Updated partition consumer to pass nack precision bit to tracker constructor |
| pulsar/negative_acks_tracker_test.go | Updated test calls to include new nack precision bit parameter |
| pulsar/consumer_test.go | Enhanced existing nack test and added new precision bit validation test |
| pulsar/consumer_zero_queue_test.go | Updated zero queue consumer nack test to handle non-deterministic message ordering |
| go.mod | Added dependencies for roaring bitmaps and AVL tree data structures |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
RobertIndie
reviewed
Sep 4, 2025
RobertIndie
reviewed
Sep 25, 2025
Member
RobertIndie
left a comment
There was a problem hiding this comment.
Please also take a look at the Copilot comments. Seems that there are some comments that havn't been addressed.
RobertIndie
requested changes
Oct 11, 2025
RobertIndie
requested changes
Oct 13, 2025
RobertIndie
approved these changes
Oct 17, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The main content of this PR is to implement the Pulsar Java community's PIP-393 proposal in the Go client.
Negative Acknowledgement handling in now Pulsar Go client exhibits inefficiencies related to memory usage, timing precision, and batch-level contention.
This effort aims to bring parity with the refinements implemented in the Java client (and CPP client), notably around memory usage and redelivery coordination.
Modifications
Refactor the
negativeAcksTrackerto solve the above problems.Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation