Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix description of Key_Shared to address PIP-282 changes #953

Merged
merged 1 commit into from
Aug 22, 2024
Merged

Fix description of Key_Shared to address PIP-282 changes #953

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions docs/administration-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ All stats below are **reset** to 0 upon broker restart or topic unloading, **exc
| lastAckedTimestamp | The latest timestamp of all the acknowledged timestamps of the consumers. |
| msgRateRedeliver | The total rate of messages redelivered on this subscription (message per second). |
| chunkedMessageRate | The chunked message dispatch rate. |
| earliestMsgPublishTimeInBacklog* | The publish time of the earliest message in the backlog for the subscription (in milliseconds). |
| earliestMsgPublishTimeInBacklog* | The publish time of the earliest message in the backlog for the subscription (in milliseconds). |
| msgBacklogNoDelayed | The number of messages in the subscription backlog that do not contain the delay messages. |
| blockedSubscriptionOnUnackedMsgs | The flag to verify if a subscription is blocked due to reaching the threshold of unacked messages. |
| msgDelayed | The number of delayed messages that are currently tracked. |
Expand All @@ -90,14 +90,16 @@ All stats below are **reset** to 0 upon broker restart or topic unloading, **exc
| replicated | Mark that the subscription state is kept in sync across different regions. |
| allowOutOfOrderDelivery | Whether out-of-order delivery is allowed on the Key_Shared subscription. |
| keySharedMode | The options of the Key_Shared subscription mode include `AUTO_SPLIT` or `STICKY`. |
| consumersAfterMarkDeletePosition | Get `recentJoinedConsumers` for the Key_Shared subscription. |
| consumersAfterMarkDeletePosition | Get `recentlyJoinedConsumers` for the Key_Shared subscription. |
| lastSentPosition | The position that all messages up to that position have been dispatched to the consumers. This is for Key_Shared subscription. |
| individuallySentPositions | Set of individually sent ranges. This is for Key_Shared subscription. |
| filterProcessedMsgCount | The number of messages processed by `EntryFilter`. |
| filterAcceptedMsgCount | The number of messages accepted by `EntryFilter`. |
| filterRejectedMsgCount | The number of messages rejected by `EntryFilter`. |
| filterRescheduledMsgCount | The number of messages rescheduled by `EntryFilter`. |
| bytesOutCounter | The total bytes delivered to a consumer. |
| msgOutCounter | The total messages delivered to a consumer. |
| backlogSize* | The size of backlog for this subscription (in bytes). |
| backlogSize* | The size of backlog for this subscription (in bytes). |
| nonContiguousDeletedMessagesRanges | The number of non-contiguous deleted messages ranges. |
| nonContiguousDeletedMessagesRangesSerializedSize | The serialized size of non-contiguous deleted messages ranges. |
| [consumers](#consumer-stats) | The list of connected consumers for this subscription. |
Expand All @@ -124,7 +126,7 @@ All stats below are **reset** to 0 upon broker restart or topic unloading, **exc
| msgRateRedeliver | The total rate of messages redelivered by this consumer (message per second). |
| chunkedMessageRate | The total rate of chunked messages delivered to this consumer. |
| avgMessagesPerEntry | The number of average messages per entry for the consumer consumed. |
| readPositionWhenJoining | The read position of the cursor when the consumer joins. |
| lastSentPositionWhenJoining | The last sent position of the cursor when the consumer joins. |
| keyHashRanges | The hash ranges assigned to this consumer if it uses Key_Shared sub mode. |
| metadata | The metadata (key/value strings) associated with this consumer. |

Expand Down
6 changes: 3 additions & 3 deletions docs/concepts-messaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -788,19 +788,19 @@ If the newly connected consumer didn't supply their ranges, or they overlap with

##### How to use mapping algorithms?

To use a mapping algorithm mentioned above, you can specify the Key Shared Mode when building the consumer:
To use a mapping algorithm mentioned above, you can specify the Key_Shared Mode when building the consumer:
* AUTO_SPLIT - Auto-split Hash Range
* STICKY - Sticky

Consistent Hashing will be used instead of Hash Range for Auto-split if the broker configuration `subscriptionKeySharedUseConsistentHashing` is enabled.

##### Preserving order of processing

Key Shared Subscription type guarantees a key will be processed by a *single* consumer at any given time. When a new consumer is connected, some keys will change their mapping from existing consumers to the new consumer. Once the connection has been established, the broker will record the current read position and associate it with the new consumer. The read position is a marker indicating that messages have been dispatched to the consumers up to this point, and after it, no messages have been dispatched yet. The broker will start delivering messages to the new consumer *only* when all messages up to the read position have been acknowledged. This will guarantee that a certain key is processed by a single consumer at any given time. The trade-off is that if one of the existing consumers is stuck and no time-out was defined (acknowledging for you), the new consumer won't receive any messages until the stuck consumer resumes or gets disconnected.
Key_Shared Subscription type guarantees a key will be processed by a *single* consumer at any given time. When a new consumer is connected, some keys will change their mapping from existing consumers to the new consumer. Once the connection has been established, the broker will record the current `lastSentPosition` and associate it with the new consumer. The `lastSentPosition` is a marker indicating that messages have been dispatched to the consumers up to this point. The broker will start delivering messages to the new consumer *only* when all messages up to the `lastSentPosition` have been acknowledged. This will guarantee that a certain key is processed by a single consumer at any given time. The trade-off is that if one of the existing consumers is stuck and no time-out was defined (acknowledging for you), the new consumer won't receive any messages until the stuck consumer resumes or gets disconnected.

That requirement can be relaxed by enabling `allowOutOfOrderDelivery` via the Consumer API. If set on the new consumer, then when it is connected, the broker will allow it to receive messages knowing some messages of that key may be still be processing in other consumers at the time, thus order may be affected for that short period of adding a new consumer.

##### Batching for Key Shared Subscriptions
##### Batching for Key_Shared Subscriptions

:::note

Expand Down