|
| 1 | +# Fair-queueing spike: findings |
| 2 | + |
| 3 | +This is a throwaway spike. It ships nothing and should be deleted before any |
| 4 | +merge to main; it exists to inform a design decision, not to become code. |
| 5 | + |
| 6 | +Read the caveats section before quoting any single number. Two of them matter up |
| 7 | +front: (1) the candidate selectors are handed tenant identity (parsed from the |
| 8 | +queue name) and the exact per-tenant weights, and the fairness target is defined |
| 9 | +by those same groups and weights, so the candidate win over the tenant-blind |
| 10 | +baseline is closer to definitional than discovered. (2) The harness anchors |
| 11 | +message scores far in the past, which flattens all queue ages, so the baseline's |
| 12 | +production age bias is not exercised here; the baseline measured is closer to a |
| 13 | +uniform-random shuffle than the real one. |
| 14 | + |
| 15 | +Bottom line: at the base-queue grain, virtual-time ordering (SFQ) and stride give |
| 16 | +tight, seed-stable proportional fairness, honour weights, and cut a starved |
| 17 | +tenant's wait hard. DRR lands within noise of them (its small shortfall is a |
| 18 | +measurement artifact of the harness, not an intrinsic property). The baseline is |
| 19 | +fair on average but seed-variant and has no weight concept. The CoDel wrapper is |
| 20 | +not worth shipping as built: it is a forced no-op under bulk arrival and it |
| 21 | +actively hurt fairness on the one trickle-arrival workload that could exercise it. |
| 22 | +The biggest single result is architectural: per-concurrency-key fairness (the |
| 23 | +actual #2617 grain) cannot be expressed through the `RunQueueSelectionStrategy` |
| 24 | +interface at all; it lives below that interface, in the CK-dequeue Lua. |
| 25 | + |
| 26 | +Every number comes from the real `RunQueue` against a testcontainers Redis, one |
| 27 | +selector per run, real enqueue/dequeue/ack and real concurrency gating. Each |
| 28 | +scenario runs over 3 seeds; tables show the mean and min..max spread. Per-tenant |
| 29 | +detail (first seed) is in `results/*.json`. |
| 30 | + |
| 31 | +## Grain, and why it is not the concurrency key |
| 32 | + |
| 33 | +A tenant is the fairness group; a tenant owns one or more base queues. The |
| 34 | +adversarial scenario gives one tenant 30 queues and the light tenants one each, |
| 35 | +which is how the #2617 starvation shows up at the base-queue grain: an ordering |
| 36 | +blind to tenant identity lets the many-queue tenant win most of the selection |
| 37 | +chances. |
| 38 | + |
| 39 | +The concurrency-key grain #2617 asks for is not reachable through the strategy |
| 40 | +interface. `FairQueueSelectionStrategy` reads the master-queue members verbatim, |
| 41 | +and CK runs enqueue a single CK-wildcard entry per base queue. The per-CK pick |
| 42 | +runs later inside `dequeueMessagesFromCkQueueTracked`, where `ckIndexKey` is a |
| 43 | +ZSET of CK-queues scored by head timestamp and the Lua serves them oldest-first. |
| 44 | +That age ordering is the unfairness. Fixing it means changing that Lua or the |
| 45 | +`ckIndex` scoring, not the selection strategy. That is the follow-on spike. |
| 46 | + |
| 47 | +## How fairness is measured (and its limits) |
| 48 | + |
| 49 | +Because the sim drains every run, final throughput share is fixed by the workload |
| 50 | +and cannot tell selectors apart. Two measures do: |
| 51 | + |
| 52 | +- contention share: a tenant's share of dequeues at instants when at least two |
| 53 | + tenants have arrived, unserved work, over its expected weighted share. |
| 54 | + `contWorstS/W` is the least-served contender; 1.0 is fair, near 0 means starved |
| 55 | + while others had work. Getting this right took two corrections a review caught: |
| 56 | + it must only count a tenant once its runs have actually arrived (else poisson |
| 57 | + arrival looks like starvation), and the virtual-time floor must be monotonic |
| 58 | + (else a returning idle tenant monopolises and skews the window). Even so, when |
| 59 | + tenants have very different volumes (trickleStale: 30 runs vs 300) the |
| 60 | + low-volume tenant can legitimately be over- or under-represented in the window, |
| 61 | + so read this metric together with wait, not alone. |
| 62 | +- wait: dequeue time minus enqueue time, per tenant, in the JSON. This is the |
| 63 | + clean anti-staleness signal. (Note: `worstWaitP99` in the JSON is NOT an |
| 64 | + anti-staleness win signal; it is dominated by the highest-volume tenant, which |
| 65 | + a fair selector deliberately delays, so a fairer selector scores worse on it. |
| 66 | + Use per-tenant wait.) |
| 67 | + |
| 68 | +## Results |
| 69 | + |
| 70 | +`contWorstS/W` mean over 3 seeds (min..max). Higher is fairer. |
| 71 | + |
| 72 | +| scenario | baseline | sfq | drr | stride | codel-sfq | codel-baseline | |
| 73 | +| --------------- | ------------------- | ----- | ------------------- | ------ | --------- | -------------- | |
| 74 | +| balanced | 0.889 (0.774..0.954)| 0.985 | 0.954 (0.923..0.970)| 0.985 | 0.985 | 0.889 | |
| 75 | +| adversarialSkew | 0.288 (0.261..0.310)| 1.000 | 0.978 (0.968..0.984)| 1.000 | 1.000 | 0.288 | |
| 76 | +| weighted | 0.703 (0.679..0.719)| 1.000 | 0.990 (0.977..1.000)| 1.000 | 1.000 | 0.703 | |
| 77 | +| burst | 0.978 (0.966..0.992)| 0.992 | 0.958 (0.941..0.975)| 0.992 | 0.992 | 0.978 | |
| 78 | +| longHold | 0.828 (0.800..0.842)| 0.981 | 0.981 | 0.981 | 0.981 | 0.828 | |
| 79 | +| trickleStale | 0.208 (0.179..0.235)| 0.804 (0.769..0.826)| 0.776 (0.769..0.783)| 0.804 | 0.366 | 0.195 | |
| 80 | + |
| 81 | +Per-tenant mean wait (seed-a, logical ms), the anti-staleness signal: |
| 82 | + |
| 83 | +| scenario / selector | low-volume tenant wait | heavy tenant wait | |
| 84 | +| ------------------------ | ---------------------- | ----------------- | |
| 85 | +| adversarialSkew baseline | 1380 | 805 | |
| 86 | +| adversarialSkew sfq | 319 | 1324 | |
| 87 | +| trickleStale baseline | 1359 | 1234 | |
| 88 | +| trickleStale sfq | 19 | 1353 | |
| 89 | +| trickleStale codel-sfq | 213 | 1315 | |
| 90 | + |
| 91 | +Reading these: the fair selectors cut the light tenant's wait (skew 1380 to 319, |
| 92 | +trickle 1359 to 19) by making the heavy tenant wait its fair turn. The heavy |
| 93 | +tenant is not punished, it stops jumping the queue. CoDel undoes part of the |
| 94 | +trickle win (19 back up to 213). |
| 95 | + |
| 96 | +## Verdict per mechanism |
| 97 | + |
| 98 | +- SFQ (start-time virtual time, the start-tag form of WFQ): the strongest result. |
| 99 | + Perfect contention fairness under skew and weighting, seed-stable (zero variance |
| 100 | + across seeds), and the largest cut to the starved tenant's wait. The floor is |
| 101 | + now monotonic (a review found the earlier version let a returning idle tenant |
| 102 | + monopolise; fixed). Recommended as the leaf ordering. |
| 103 | +- Stride: identical to SFQ to the decimal on every scenario. The spike does not |
| 104 | + separate them. Stride carries slightly less state. |
| 105 | +- DRR: within noise of SFQ. It trails by a couple of points on balanced (0.954) |
| 106 | + and burst (0.958) and matches SFQ elsewhere. That small shortfall is a |
| 107 | + measurement artifact, not an intrinsic property: the driver drains a whole |
| 108 | + capacity batch from a single strategy snapshot and only advances DRR's deficit |
| 109 | + after the batch (via `onServiced`), so DRR's current-winner group, whose queues |
| 110 | + it fronts together, grabs several slots before its deficit updates and its |
| 111 | + deficit runs negative. Served one-at-a-time DRR is exactly fair (see |
| 112 | + `drr.test.ts`). Note the earlier claim that "virtual-time sorts an over-served |
| 113 | + group's queues to the back and so avoids this" was wrong: at a tie all of a |
| 114 | + group's queues share one clock, so SFQ fronts them together too; the schemes |
| 115 | + only separate after their state advances. DRR is O(1) and composes weight |
| 116 | + trivially, so it is a fine choice if per-op cost matters, subject to that |
| 117 | + caveat. |
| 118 | +- CoDel wrapper: do not ship as built. Under bulk arrival it is a forced no-op: |
| 119 | + all of a queue's runs share one enqueue timestamp, so every tenant's sojourn is |
| 120 | + identical and they all cross the target together, so hoisting everyone collapses |
| 121 | + to the base order (this is why codel-sfq equals sfq and codel-baseline equals |
| 122 | + baseline to the decimal on those scenarios; it is one workload shape confirming |
| 123 | + a null result, not five independent tests). On trickleStale, the one scenario |
| 124 | + where sojourns diverge, the sojourn-hoist overshoots: it drops SFQ from 0.804 to |
| 125 | + 0.366 and pushes the trickle tenant's wait from 19 back to 213. A staleness |
| 126 | + monitor may still help on top of an unfair base or behind a hard concurrency |
| 127 | + wall, but that needs a different construction and this spike does not support it. |
| 128 | +- Baseline (`FairQueueSelectionStrategy`): fair on average on the easy scenarios |
| 129 | + but seed-variant (balanced 0.774..0.954), no weight concept (weighted 0.703), |
| 130 | + and it starves a light tenant under queue-count skew (0.288) and under trickle |
| 131 | + arrival (0.208, trickle wait 1359). Remember its age bias is not exercised here |
| 132 | + (see caveats), so this is a floor on its unfairness, not the production picture. |
| 133 | + |
| 134 | +## Caveats |
| 135 | + |
| 136 | +- Grain is base queues, not concurrency keys. The disciplines are grain-agnostic |
| 137 | + so the ranking should carry over, but the #2617 gap itself needs the CK-Lua |
| 138 | + spike. adversarialSkew is a proxy for that gap, not a measurement of it. |
| 139 | +- Definitional advantage: candidates get tenant identity and exact weights the |
| 140 | + real interface does not carry; the baseline structurally cannot. |
| 141 | +- Baseline age bias inert: scores are anchored ~600s in the past, so all queue |
| 142 | + ages are near-equal and the baseline degenerates to near-uniform selection. The |
| 143 | + production age bias (which would give a heavy tenant's older heads more weight, |
| 144 | + i.e. make skew worse) is not measured. |
| 145 | +- Selection-only seam: the driver feeds serviced descriptors back via an |
| 146 | + `onServiced` hook; production would advance selector state inside the ack/dequeue |
| 147 | + Lua. The spike proves ordering logic, not that wiring. |
| 148 | +- Cost was not rigorously measured. `selectionRounds` is roughly equal across |
| 149 | + selectors (646..729) but is not comparable between them (a candidate reads all |
| 150 | + queues per call; the baseline short-circuits at capacity), and there is no load |
| 151 | + benchmark. DRR's "O(1)" advantage is a theory claim, not a spike measurement. |
| 152 | +- Scenario quality varies. balanced best shows the baseline's variance; |
| 153 | + adversarialSkew and weighted carry the clear separation; longHold and burst |
| 154 | + barely separate the candidates; trickleStale's contention number only became |
| 155 | + meaningful after two metric fixes and should be read with wait. Per-tenant p99 |
| 156 | + equals max for the small (20 to 30 run) tenants, so "p99" there is just the max. |
| 157 | +- Single Redis shard; single sequential consumer (not the multi-consumer, |
| 158 | + Redis-hash-state design the spec sketched); simulated holds on a logical clock. |
| 159 | + Three seeds shows the baseline's variance and the virtual-time schemes' |
| 160 | + stability but is not a statistical study. |
| 161 | + |
| 162 | +## Recommended direction |
| 163 | + |
| 164 | +Use virtual-time (SFQ, or stride) for leaf ordering and compose weight with it. |
| 165 | +DRR is an acceptable O(1) fallback given the batch caveat. Do not adopt the CoDel |
| 166 | +wrapper as built. Then run the follow-on spike against the CK-dequeue Lua / |
| 167 | +`ckIndex` scoring, because that is where per-tenant fairness actually has to land |
| 168 | +in the current design. |
0 commit comments