Skip to content

feat: Remove permanent failure modes from FDv1 following RETRY spec - #519

Open
jsonbailey wants to merge 16 commits into
mainfrom
jb/sdk-2792/retry-conformance
Open

jsonbailey wants to merge 16 commits into
mainfrom
jb/sdk-2792/retry-conformance

Conversation

@jsonbailey

@jsonbailey jsonbailey commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

BEGIN_COMMIT_OVERRIDE
feat: Retry indefinitely after a data source failure instead of stopping permanently
fix: Warn and use the documented default for an invalid poll interval or initial reconnect delay
END_COMMIT_OVERRIDE

Summary

Brings the FDv1 streaming and polling data sources into conformance with the RETRY specification. No HTTP response and no transport-level failure stops a data source permanently any more.

Every failure is classified normal or unexpected. A normal failure retries on the existing curve — 1s doubling to a 30s ceiling for streaming, the poll interval for polling. An unexpected failure (401, 403, any other 4xx outside 400/408/429) moves to a longer regime starting at 5 minutes and doubling to a 1-hour ceiling, and keeps retrying there until the condition clears.

The retry state machine itself landed in #522; the delay-source plumbing landed in #521. This PR connects them to the four data sources and removes the permanent-stop paths.

Behaviour changes for release notes

  1. A bad or revoked SDK key no longer fails fast. LDClient(config, start_wait=N), postfork(start_wait=N) and await client.start(start_wait=N) now block for the full start_wait and return with is_initialized() false, rather than returning at once. The SDK keeps retrying in the background.
  2. DataSourceState.OFF is now reserved for explicit shutdown and unparseable configuration. HTTP errors produce INTERRUPTED.
  3. A rejected SDK key now logs at error roughly hourly, indefinitely, rather than once. An SDK retrying with a rejected credential consumes resources, and the condition needs a person to fix it.
  4. A server-initiated stream close now logs a warning where it previously logged nothing, and backs off rather than reconnecting immediately.
  5. Failure logs now state the actual delayReceived HTTP error 401 (invalid SDK key) for stream connection - will retry in 300.0s. Previously the SDK said only "will retry", and the delay was logged separately at info by the SSE client, so it was invisible at default log levels.
  6. An invalid poll_interval or initial_reconnect_delay now logs a warning and uses the documented default. poll_interval was silently clamped and initial_reconnect_delay was not checked at all.

What changed

  • impl/datasource/{streaming,async_streaming,polling,async_polling}.py — the permanent-stop paths are gone. Each failure is classified, the retry state advances, status becomes INTERRUPTED, and the wait is interruptible by stop(), which matters now that a wait can be an hour long.
  • config.py / async_config.py — both intervals are validated. Config previously clamped poll_interval with max(), which let NaN and inf through, because every comparison against NaN is false. A NaN interval reached Event.wait() and the delay arithmetic. The 30-second minimum still applies on top of validation.
  • impl/util.pyvalidate_positive_finite, beside the validators Config already imports, so config and retry share it without either importing the other.
  • impl/retry.py — uses the shared validator; the two configurable defaults now live in config.py next to DEFAULT_STREAM_URI.
  • impl/datasource/datasource_common.py, interfaces.py, client.py / async_client.py — docstrings and status handling updated for the above.
  • impl/aio/transport.py — the async transport's own retry is driven by the shared state.
  • contract-tests/ — both services declare the conformance capability.

Known gaps, deliberately out of scope

  • FDv2 (impl/datasourcev2/**, impl/datasystem/**) still stops permanently on an unexpected response. Tracked as SDK-2776.
  • The event processor's _disabled permanent stop stays. No spec binding exists for it yet. Also SDK-2776.
  • SSE action loops have no except. "A data source never stops" holds via ld_eventsource internals rather than by construction. No reachable escape path was found at the pinned version.

Testing

make test: 1681 passed. make lint: clean across 228 files.

Contract tests were run out of band against harness v2.41.0 — streaming's eight conformance subtests and polling's four all pass. Note the conformance scenarios need -enable-long-running-tests, which the Makefile does not pass, so they do not run in CI.


Note

Overview
Aligns FDv1 streaming and polling (sync and async) with the RETRY spec: data source failures no longer shut down permanently. Failures are classified as normal vs unexpected (e.g. most 4xx including invalid SDK keys), drive shared RetryState backoff, report INTERRUPTED, and keep retrying—including multi-minute/hour waits that stop() can interrupt.

Retry ownership moves to the SDK: SSE clients are created with sdk_managed_retry so reconnect delays come from for_streaming / for_polling, not the eventsource library. Server-initiated stream closes are treated as StreamClosedError with normal backoff (not ignored tight reconnect loops).

User-visible behavior shifts: bad SDK keys no longer fail fast during start_wait—the client returns uninitialized but keeps retrying in the background. DataSourceState.OFF is reserved for explicit shutdown (not HTTP errors). Logs now include the actual retry delay; invalid poll_interval / initial_reconnect_delay warn and fall back to shared defaults in config.py.

Contract test services advertise retry-conformance-fdv1-* capabilities; docs and tests are updated across processors, transport, and e2e cases.

Reviewed by Cursor Bugbot for commit 30bb8ec. Bugbot is set up for automated code reviews on this repo. Configure here.

@jsonbailey
jsonbailey force-pushed the jb/sdk-2792/retry-conformance branch from 37cdb27 to 9413155 Compare September 14, 2026 15:02
…loor

Streaming's operating cadence is zero rather than absent, so a healthy
stream schedules no wait and a retry delay has no floor. Polling is the
only data source that reads next_delay as a DelaySource, and its cadence
floor keeps that from reaching zero.

Make the seven observational properties private. No data source reads
them; they are test instrumentation, and as public API they invite
callers to attach logic to unsynchronized state.
…formance

# Conflicts:
#	ldclient/impl/datasource/async_polling.py
#	ldclient/impl/datasource/polling.py
#	ldclient/testing/impl/datasource/test_async_polling.py
Config accepted any initial_reconnect_delay and clamped poll_interval
with max(), which let NaN and inf through because every comparison
against NaN is false. A NaN interval reaches Event.wait() and the delay
arithmetic downstream. Both are now validated, warn, and fall back to
the documented default; the poll interval keeps its 30s minimum on top.

Move the check to impl/util.py as validate_positive_finite, next to the
validators Config already imports, so config and retry share it without
either importing the other. The retry factories keep their own call: a
RetryState can be built without going through Config.

Also name the delay bounds after the spec's ceiling vocabulary.
…e delay

The exponent driver was _n and a second counter held the name attempts,
which is what Requirement 1.4.1 calls the exponent driver. A reviewer
reading self.attempts against the spec was reading the wrong field.

The second counter is gone. It had no reader outside tests, not even a
logger, and every test that used it recorded only normal failures --
where the two counters are equal by construction.
…formance

# Conflicts:
#	ldclient/impl/retry.py
#	ldclient/testing/impl/test_retry.py
Config now validates both intervals, so the factories' guards are no
longer the only check. They still matter -- a RetryState can be built
without going through Config -- but the docstrings described the old
state, where Config ignored initial_reconnect_delay and only clamped
poll_interval.
@jsonbailey
jsonbailey force-pushed the jb/sdk-2792/retry-conformance branch from 6c277e3 to 8e11162 Compare September 18, 2026 15:39
@jsonbailey
jsonbailey marked this pull request as ready for review September 18, 2026 16:56
@jsonbailey
jsonbailey requested a review from a team as a code owner September 18, 2026 16:56
@jsonbailey
jsonbailey force-pushed the jb/sdk-2792/retry-conformance branch from 8e11162 to e1f29df Compare September 18, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant