[run-tests] [fix][broker] Introduce the last sent position to fix message ordering issues in Key_Shared (PIP-282) #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
equanz
force-pushed
the
pip_282_to_master
branch
from
b8fd521 to
43a2dc3
Compare
equanz
changed the title
equanz
changed the title
equanz
changed the title
equanz
changed the title
equanz
changed the title
15 tasks
… creating OwnershipCache (apache#22482)
…d fix race conditions in metricsBufferResponse mode (apache#22494)
…c loading wouldn't timeout (apache#22479)
…he#21274) Co-authored-by: Baodi Shi <[email protected]>
…xception and testIncorrectClientClock (apache#22489)
Co-authored-by: hoguni <[email protected]>
…s and Connectors (apache#22514)
…igurationMetadataStore equals localMetadataStore (apache#22519)
…sharing/caching compressed result (apache#22521)
…e when the ExtensibleLoadManager is enabled (apache#22485)
…during NamespaceEventsSystemTopicFactory init (apache#22528)
…pache#22778) Co-authored-by: Dragos Misca <[email protected]>
apache#22707) ### Motivation [PersistentTopic#getLastDispatchablePosition](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L3776-L3788) is using by [Reader#hasMessageAvailable](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Reader.java#L116) , [ConsumerImpl#hasMessageAvailable](https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L2440-L2448), [Consumer#getLastMessageIdAsync](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java#L591-L615). The current implementation is read entries from Bookkeeper(or sth else), which leads to low throughput, high latency and heavy load, this PR is for the purpose of optimization.
…terface (apache#22891) Co-authored-by: Matteo Merli <[email protected]>
…hecking if broker is active(ExtensibleLoadManagerImpl only) (apache#22899)
…osing metadata store (apache#22585) Co-authored-by: Matteo Merli <[email protected]>
…#22905) Co-authored-by: Lari Hotari <[email protected]>
…tinuously increases even when there is no backlog. (apache#22907)
…terDataImpl class. (apache#22919) Signed-off-by: jitokim <[email protected]>
…, LeastLongTermMessageRate, ModularLoadManagerImpl. (apache#22889) Implementation PR: apache#22888 ### Motivation Initially, we introduce `loadBalancerCPUResourceWeight`, `loadBalancerBandwidthInResourceWeight`, `loadBalancerBandwidthOutResourceWeight`, `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` in `ThresholdShedder` to control the resource weight for different resources when calculating the load of the broker. Then we let it work for `LeastResourceUsageWithWeight` for better bundle placement policy. But apache#19559 and apache#21168 have point out that the actual load of the broker is not related to the memory usage and direct memory usage, thus we have changed the default value of `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` to 0.0. There are still some places where memory usage and direct memory usage are used to calculate the load of the broker, such as `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We should let the resource weight work for these places so that we can set the resource weight to 0.0 to avoid the impact of memory usage and direct memory usage on the load of the broker. ### Modifications - Let resource weight work for `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`.
### Motivation Those bundles that are filtered when try to unload them should not be included in the indicator. ### Modifications Increment the metric only when the bundle are unloaded.
…r Function runtimes (apache#22910)
…calling getPartitionedTopicMetadata (apache#22838)
equanz
force-pushed
the
pip_282_to_master
branch
from
c3b318a to
8b427b3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for running tests for upstream PR #FIXME.