Add disabled offline Feature Flagging contract#7305
Conversation
|
|
|
|
|
||
| Args: | ||
| configuration: Source-independent configuration retained for existing test clients. | ||
| offline_configuration: UFC JSON bytes supplied directly at provider startup. |
There was a problem hiding this comment.
Is this supposed to be the wire format? https://datadoghq.atlassian.net/wiki/spaces/PANA/pages/5141725646/ConfigurationWire
https://docs.google.com/document/d/1OWNBtXtSk535VXqf-9fqsAmU9W8kpFLAwxYi2y1qyQQ/edit?tab=t.0#heading=h.oa3go2ysd90q
Or is it expected that a conversion happens somewhere downstream?
| class Test_Feature_Flag_Offline_Evaluation: | ||
| """Validate that offline delivery preserves the canonical UFC evaluation corpus.""" | ||
|
|
||
| @parametrize("test_case_file", ALL_TEST_CASE_FILES) |
There was a problem hiding this comment.
The offline coverage in rules/UFC-only, which is fine, but it should probably be clarified somewhere that offline init for precomputed assignments are not covered yet, e.g. testing for not leaking mismatched precomputed assignments before/after a eval context switch (precomputed for Alice, app starts offline, context becomes Bob, provider returns Alice’s assignment instead of default/error/not-ready)
cbeauchesne
left a comment
cbeauchesne
left a comment
There was a problem hiding this comment.
A small RC for framwork usage
| from tests.parametric.conftest import APMLibrary | ||
| from tests.parametric.test_ffe.test_dynamic_evaluation import _set_and_wait_ffe_rc, _ffe_evaluate_with_rc_retry | ||
| from tests.parametric.test_ffe.utils import UFC_FIXTURE_DATA, evaluate_with_configuration_retry | ||
| from tests.parametric.test_ffe.test_dynamic_evaluation import _set_and_wait_ffe_rc |
There was a problem hiding this comment.
could you move _set_and_wait_ffe_rc into tests.parametric.test_ffe.utils? a test file must not import anything from another test file (CI orchestrator constraint).
3 participants
Motivation
Define the cross-SDK contract for startup-only offline Feature Flagging on
mainbefore each SDK implementation exists. Offline is not a cached agentless startup: the application supplies UFC JSON bytes when it creates the provider, evaluations must be ready immediately, and Feature Flagging must not activate either network configuration source.Today, system-tests can deliver the canonical UFC fixtures through Remote Configuration or the mocked agentless HTTP endpoint, but it has no cross-SDK path for passing raw UFC bytes into provider construction. That leaves offline initialization, immediate readiness, immutability, and network isolation untestable.
flowchart LR fixtures["Canonical UFC fixtures"] subgraph available["Capabilities available before this PR"] rc["Inject UFC through Remote Configuration"] agentless["Serve UFC through the agentless mock endpoint"] async["Run evaluation assertions after asynchronous readiness"] rc --> async agentless --> async end subgraph missing["Missing offline capability"] noinput["No parametric input for UFC startup bytes"] noinit["Cannot construct a provider in offline mode"] noproof["Cannot prove immediate readiness, immutability, or FFE network isolation"] noinput --> noinit --> noproof end fixtures --> rc fixtures --> agentless fixtures -.-> noinputThe tests remain disabled in every SDK manifest. The stacked Java reference in DataDog/dd-trace-java#11933 exists only to prove that this contract and the harness work end to end before SDK owners implement it.
Changes
/ffe/startFFE_FLAGSRemote Configuration payload and prove that it cannot replace the startup bytesFFE_FLAGSproduct and capability, while allowing unrelated tracer RC traffic or no RC traffic at allutilsso framework code does not import the parametric test packagedd-openfeatureartifact required for local Java provider testingflowchart TB before["Delivery-specific fixture loading and evaluation logic"] shared["Shared UFC fixture bytes and evaluation runner"] before --> shared subgraph delivery["Configuration source adapters"] rc["Remote Configuration injection"] agentless["Agentless mock HTTP endpoint"] offline["Offline bytes sent through ffe start"] end shared --> rc shared --> agentless shared --> offline rc --> retry["Asynchronous readiness retry"] agentless --> retry offline --> immediate["First evaluation must be ready"] retry --> semantics["Canonical UFC and OpenFeature assertions"] immediate --> semantics offline --> failed["Empty or malformed bytes fail closed"] offline --> immutable["Conflicting FFE RC cannot replace startup configuration"] offline --> isolated["No agentless request and no FFE RC subscription"] semantics --> feature["Parity feature 562 Offline init"] failed --> feature immutable --> feature isolated --> feature feature --> manifests["All SDK manifests stay disabled until native implementations land"]Tests added
The file contributes 26 pytest items. Each canonical fixture group can contain multiple flag-evaluation assertions.
test_offline_configuration_evaluates_fixturetest_invalid_startup_configuration_fails_closed[empty]test_invalid_startup_configuration_fails_closed[malformed]test_offline_configuration_is_immutable_and_network_isolatedgreen, injects a conflicting RC fixture whose same variation evaluates toreplacement, observes both network surfaces, then evaluates again.green; the agentless endpoint receives zero requests; all observed RC requests omitFFE_FLAGSand its capability. Unrelated tracer RC polling is permitted.Decisions
Validation
./format.sh— passed mypy, Ruff, YAML formatting/lint, parser checks, ShellCheck, and Node.js linters./run.sh TEST_THE_TEST tests/test_the_test/test_conventions.py tests/test_the_test/test_mock_ffe_agentless_backend.py tests/test_the_test/test_features.py tests/test_the_test/test_manifest.py -q—29 passed, 1 deselectedwith locally built artifacts matching DataDog/dd-trace-java#11933 head
8d654e23b7:Result:
26 passed in 32.29s.