Replication Flow Control – Prioritizing replication traffic in the replica

[xbasel]

Overview

This PR introduces Replication Flow Control (repl-flow-control), a dynamic mechanism that prioritizes replication traffic on the replica side. By detecting replication pressure and adjusting read frequency adaptively, it reduces the risk of primary buffer overflows and full syncs.

Problem

In high-load scenarios, a replica might not consume replication data fast enough, leading to backpressure on the primary. When the primary’s buffer overflows, it drops the replica connection, triggering a full sync, a costly operation that impacts system performance.

Without this feature:

• Replication reads occur at a fixed rate, irrespective of data pressure.
• If the replica falls behind, the primary accumulates replication data leading to higher memory utilization.
• Once the primary buffer overflows, the connection drops, forcing a full sync.
• Full syncs cause high memory, CPU, and I/O spikes.

Solution: Replication Flow Control

repl-flow-control enables the replica to dynamically increase its replication read rate if it detects that replication data is accumulating. The mechanism operates as follows:

Detecting replication pressure
Each read from the primary is checked against the max real byte limit. If the read hit the limit, suggesting more data is likely available.

Prioritizing replication reads
If replication pressure is detected, the replica invokes multiple reads per I/O event instead of a single one. This allows the replica to catch up faster, reducing memory consumption and buffer overflows on the primary.

Adaptive flow control
The read limit ramps up gradually if full reads continue, avoiding sudden spikes. If a read does not fill the buffer, the limit is reduced. Increases are rate-limited to once every 100ms, avoiding over-aggressive read bursts. There's a configurable maximum number of reads.

Performance Impact

Test setup:

1. Bombard the replica with expensive commands, leading to high CPU utilization
2. Write to the main database to trigger replication traffic

Latency and Throughput Changes

Metric	Before (repl-flow-control Disabled)	After (repl-flow-control Enabled)
Throughput (requests/sec)	941.71	760.98
Avg Latency (ms)	52.865	65.534
p50 Latency (ms)	59.743	68.543
p95 Latency (ms)	79.231	106.687
p99 Latency (ms)	90.303	126.527
Max Latency (ms)	188.031	385.535

📌 Observations:

• Replication stability improves,no full syncs were observed after enabling flow control.
• Higher latency for normal clients due to increased resource allocation for replication.
• CPU and memory usage remain stable, with no major overhead.
• Replica throughput slightly decreases as replication takes priority.

TODO

• Consider limiting the maximum number of reads per event to a ratio of the total number of events returned by the epoll cycle. For example, if the ratio is 20% and EPOLL returns 100 events, the replica can read from the primary up to 20 times per primary I/O event.

Implements #1596

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 70.98%. Comparing base (bcd2f95) to head (e846d9e).

Files with missing lines	Patch %	Lines
src/config.c	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           unstable    #1838   +/-   ##
=========================================
  Coverage     70.97%   70.98%           
=========================================
  Files           123      123           
  Lines         65665    65686   +21     
=========================================
+ Hits          46608    46628   +20     
- Misses        19057    19058    +1     

Files with missing lines	Coverage Δ
src/networking.c	88.95% <100.00%> (+0.06%)	⬆️
src/server.c	87.54% <100.00%> (-0.03%)	⬇️
src/server.h	100.00% <ø> (ø)
src/config.c	78.14% <0.00%> (-0.25%)	⬇️

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hwware]

I have reservations about the default value is yes due to the lower performance.

[hwware]

It looks like a boolean variable (I check the codes, the variable is 1 or 0), how about change to is_qb_full_read or similar? It is easy to read for others.

[hwware]

I do not find where this variable repl_cur_reads_per_io_event initialization, did you do in somewhere else? If not, it is dangerous, please initialize it.

[madolson]

I suppose I don't really understand why this needs to be adaptive. As long as we got a full read, why can't we repeat up until the max io event size? There is a comment about "The read limit ramps up gradually if full reads continue, avoiding sudden spikes.", but we are just deferring the spikes until later. Most users don't like replication lag, they would rather have the most up to date data if possible.

[xbasel]

I suppose I don't really understand why this needs to be adaptive. As long as we got a full read, why can't we repeat up until the max io event size? There is a comment about "The read limit ramps up gradually if full reads continue, avoiding sudden spikes.", but we are just deferring the spikes until later. Most users don't like replication lag, they would rather have the most up to date data if possible.

I have no strong opinion here, probably reading up to the max reads is enough (and simpler).

[zuiderkwast]

This looks to me like we're fixing a stability issue. Why would anyone want to disable it? Let's discuss if we actually need a config for this. I think we maybe don't need it and we can just keep this always enabled.

I think the benchmark numbers don't give a fair picture. Without this feature, there is a problem of replication lag and for the full sync, it means extra resources used by replica and primary and maybe even extra latency for commands sent to the primary.

Also, this affects only the latency of read-from-replicas. I think this is less important than read-from-primary. If the replica is not fast enough, the client can read from the primary or from another replica. If they do that, it's good because it will reduce the load on the overloaded replica.