feat: Remove permanent failure modes from FDv1 following RETRY spec - #519
Open
jsonbailey wants to merge 16 commits into
Open
jsonbailey wants to merge 16 commits into
jsonbailey wants to merge 16 commits into
Conversation
jsonbailey
force-pushed
the
jb/sdk-2792/retry-conformance
branch
from
September 14, 2026 15:02
37cdb27 to
9413155
Compare
…loor Streaming's operating cadence is zero rather than absent, so a healthy stream schedules no wait and a retry delay has no floor. Polling is the only data source that reads next_delay as a DelaySource, and its cadence floor keeps that from reaching zero. Make the seven observational properties private. No data source reads them; they are test instrumentation, and as public API they invite callers to attach logic to unsynchronized state.
…formance # Conflicts: # ldclient/impl/datasource/async_polling.py # ldclient/impl/datasource/polling.py # ldclient/testing/impl/datasource/test_async_polling.py
Config accepted any initial_reconnect_delay and clamped poll_interval with max(), which let NaN and inf through because every comparison against NaN is false. A NaN interval reaches Event.wait() and the delay arithmetic downstream. Both are now validated, warn, and fall back to the documented default; the poll interval keeps its 30s minimum on top. Move the check to impl/util.py as validate_positive_finite, next to the validators Config already imports, so config and retry share it without either importing the other. The retry factories keep their own call: a RetryState can be built without going through Config. Also name the delay bounds after the spec's ceiling vocabulary.
…e delay The exponent driver was _n and a second counter held the name attempts, which is what Requirement 1.4.1 calls the exponent driver. A reviewer reading self.attempts against the spec was reading the wrong field. The second counter is gone. It had no reader outside tests, not even a logger, and every test that used it recorded only normal failures -- where the two counters are equal by construction.
…formance # Conflicts: # ldclient/impl/retry.py # ldclient/testing/impl/test_retry.py
Config now validates both intervals, so the factories' guards are no longer the only check. They still matter -- a RetryState can be built without going through Config -- but the docstrings described the old state, where Config ignored initial_reconnect_delay and only clamped poll_interval.
jsonbailey
force-pushed
the
jb/sdk-2792/retry-conformance
branch
from
September 18, 2026 15:39
6c277e3 to
8e11162
Compare
jsonbailey
marked this pull request as ready for review
September 18, 2026 16:56
jsonbailey
force-pushed
the
jb/sdk-2792/retry-conformance
branch
from
September 18, 2026 21:36
8e11162 to
e1f29df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BEGIN_COMMIT_OVERRIDE
feat: Retry indefinitely after a data source failure instead of stopping permanently
fix: Warn and use the documented default for an invalid poll interval or initial reconnect delay
END_COMMIT_OVERRIDE
Summary
Brings the FDv1 streaming and polling data sources into conformance with the RETRY specification. No HTTP response and no transport-level failure stops a data source permanently any more.
Every failure is classified
normalorunexpected. A normal failure retries on the existing curve — 1s doubling to a 30s ceiling for streaming, the poll interval for polling. An unexpected failure (401,403, any other4xxoutside400/408/429) moves to a longer regime starting at 5 minutes and doubling to a 1-hour ceiling, and keeps retrying there until the condition clears.The retry state machine itself landed in #522; the delay-source plumbing landed in #521. This PR connects them to the four data sources and removes the permanent-stop paths.
Behaviour changes for release notes
LDClient(config, start_wait=N),postfork(start_wait=N)andawait client.start(start_wait=N)now block for the fullstart_waitand return withis_initialized()false, rather than returning at once. The SDK keeps retrying in the background.DataSourceState.OFFis now reserved for explicit shutdown and unparseable configuration. HTTP errors produceINTERRUPTED.errorroughly hourly, indefinitely, rather than once. An SDK retrying with a rejected credential consumes resources, and the condition needs a person to fix it.Received HTTP error 401 (invalid SDK key) for stream connection - will retry in 300.0s. Previously the SDK said only "will retry", and the delay was logged separately atinfoby the SSE client, so it was invisible at default log levels.poll_intervalorinitial_reconnect_delaynow logs a warning and uses the documented default.poll_intervalwas silently clamped andinitial_reconnect_delaywas not checked at all.What changed
impl/datasource/{streaming,async_streaming,polling,async_polling}.py— the permanent-stop paths are gone. Each failure is classified, the retry state advances, status becomesINTERRUPTED, and the wait is interruptible bystop(), which matters now that a wait can be an hour long.config.py/async_config.py— both intervals are validated.Configpreviously clampedpoll_intervalwithmax(), which letNaNandinfthrough, because every comparison againstNaNis false. ANaNinterval reachedEvent.wait()and the delay arithmetic. The 30-second minimum still applies on top of validation.impl/util.py—validate_positive_finite, beside the validatorsConfigalready imports, soconfigandretryshare it without either importing the other.impl/retry.py— uses the shared validator; the two configurable defaults now live inconfig.pynext toDEFAULT_STREAM_URI.impl/datasource/datasource_common.py,interfaces.py,client.py/async_client.py— docstrings and status handling updated for the above.impl/aio/transport.py— the async transport's own retry is driven by the shared state.contract-tests/— both services declare the conformance capability.Known gaps, deliberately out of scope
impl/datasourcev2/**,impl/datasystem/**) still stops permanently on an unexpected response. Tracked as SDK-2776._disabledpermanent stop stays. No spec binding exists for it yet. Also SDK-2776.except. "A data source never stops" holds viald_eventsourceinternals rather than by construction. No reachable escape path was found at the pinned version.Testing
make test: 1681 passed.make lint: clean across 228 files.Contract tests were run out of band against harness v2.41.0 — streaming's eight conformance subtests and polling's four all pass. Note the conformance scenarios need
-enable-long-running-tests, which theMakefiledoes not pass, so they do not run in CI.Note
Overview
Aligns FDv1 streaming and polling (sync and async) with the RETRY spec: data source failures no longer shut down permanently. Failures are classified as normal vs unexpected (e.g. most 4xx including invalid SDK keys), drive shared
RetryStatebackoff, reportINTERRUPTED, and keep retrying—including multi-minute/hour waits thatstop()can interrupt.Retry ownership moves to the SDK: SSE clients are created with
sdk_managed_retryso reconnect delays come fromfor_streaming/for_polling, not the eventsource library. Server-initiated stream closes are treated asStreamClosedErrorwith normal backoff (not ignored tight reconnect loops).User-visible behavior shifts: bad SDK keys no longer fail fast during
start_wait—the client returns uninitialized but keeps retrying in the background.DataSourceState.OFFis reserved for explicit shutdown (not HTTP errors). Logs now include the actual retry delay; invalidpoll_interval/initial_reconnect_delaywarn and fall back to shared defaults inconfig.py.Contract test services advertise
retry-conformance-fdv1-*capabilities; docs and tests are updated across processors, transport, and e2e cases.Reviewed by Cursor Bugbot for commit 30bb8ec. Bugbot is set up for automated code reviews on this repo. Configure here.