Extend driver API and add subqueuettl driver - #259
Open
ThCompiler wants to merge 3 commits into
Open
Conversation
Support driver-specific task selection through take() options. Some drivers partition tasks by driver-specific attributes and must select tasks only from the partition requested by a consumer. For example, a driver may use opts.subqueue to take tasks from a particular subqueue. Drivers can derive a consumer group from take() options and tasks. This lets the wake-up mechanism notify only consumers eligible to take a newly ready task, rather than waking an arbitrary consumer from the same tube that cannot process it.
Support driver-specific statistics. Some drivers maintain additional state or partition tasks using driver-specific attributes. They need to expose this data through the standard tube statistics API, alongside the common queue metrics. This allows each driver to report meaningful operational metrics without extending the generic statistics implementation for every driver-specific use case.
…s from named subqueues in one tube space. Add subqueuettl driver with consumer groups Some workloads need to create queues dynamically with arbitrary names. Creating a separate tube for each queue requires converting those names to valid Tarantool space names and granting access whenever a new tube is created. This adds operational overhead and makes access management harder. The utube model is otherwise well suited for this use case: it stores multiple logical queues in one space and guarantees sequential processing inside each queue. However, it does not support taking a task from a specific utube, so it cannot be used when consumers must be assigned to particular dynamically named queues. Added subqueuettl as a TTL-aware alternative. It stores multiple dynamically named subqueues in one space, and both put() and take() require a subqueue name. This keeps task selection isolated while retaining TTL, TTR, priority, delay, and ready-buffer support.
ThCompiler
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend driver API:
Added the optional consumer_group driver API so a driver can wake a consumer waiting for the same logical group instead of waking an unrelated consumer.
Added custom statistics for drivers to support showing custom statistics by them.
Subqueuettl:
Some workloads need to create queues dynamically with arbitrary names. Creating a separate tube for each queue requires converting those names to valid Tarantool space names and granting access whenever a new tube is created. This adds operational overhead and makes access management harder.
The utube model is otherwise well suited for this use case: it stores multiple logical queues in one space and guarantees sequential processing inside each queue. However, it does not support taking a task from a specific utube, so it cannot be used when consumers must be assigned to particular dynamically named queues.
Added subqueuettl as a TTL-aware alternative. It stores multiple dynamically named subqueues in one space, and both put() and take() require a subqueue name. This keeps task selection isolated while retaining TTL, TTR, priority, delay, and ready-buffer support.
If you don’t think it’s necessary to add this driver to the general library, I can remove it from this PR, but for my purposes, I need support for changes in the “abstract.lua” in order to use this driver.