diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4cdde19..2a94829 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,22 @@ Thank you for your interest in contributing to the FFE system test data repository! +## Changing the contract + +Read [contract ownership](README.md#contract-ownership) before changing an +expectation. These fixtures define behavior for downstream implementations; +they are not snapshots of whichever SDK or backend happens to be deployed. + +Describe the rule being added or corrected, include a concrete input and expected +result, and explain the effect on consumers. Validate against actual native +implementations where available and record the tested revisions and capability +coverage. A consumer mismatch calls for investigation; it does not by itself +justify changing the expected result. Follow [handling a mismatch](README.md#handling-a-mismatch). + +Keep deployment prerequisites, release versions, and temporary consumer workarounds +in downstream changes. Contract documentation should remain valid as those +projects ship support. + ## Getting Started 1. Fork the repository @@ -76,12 +92,24 @@ merge by itself. ## Updating Downstream Repositories -After your changes are merged to this repository, update the submodule references in downstream repos. Each downstream repository should: - -1. Keep `ffe-system-test-data` as a git submodule adjacent to the OpenFeature/FFE tests -2. Configure Dependabot with `package-ecosystem: "gitsubmodule"` so fixture updates are proposed automatically -3. Load `ufc-config.json` and loop over every `evaluation-cases/*.json` file in unit tests -4. Avoid copied fixture directories and programmatic-only shared evaluator cases +After your changes are merged to this repository, update the pinned fixture +revision in downstream repos. Each rules-based consumer should: + +1. Pin an exact fixture revision, normally with a git submodule adjacent to the + OpenFeature/FFE tests. Generated snapshots must retain the source repository + and commit SHA and be refreshed from that source. +2. Automate fixture updates. For submodules, configure Dependabot with + `package-ecosystem: "gitsubmodule"`; for snapshots, automate regeneration from + the source revision. +3. Load `ufc-config.json` and loop over every `evaluation-cases/*.json` file in + unit tests. +4. Run the cases for every implemented capability as well as the baseline; + follow the [runner requirements](README.md#implement-the-rules-based-runner). +5. Preserve shared expectations unchanged. Avoid hand-maintained copies and + programmatic-only replacements for shared evaluator cases. + +Precomputed-assignment consumers use the same pinning and provenance rules with +the [precomputed fixture runner](precomputed-assignments/README.md). Current downstream consumers are `system-tests`, `dd-trace-go`, `dd-trace-java`, `dd-trace-js`, `dd-trace-py`, `dd-trace-rb`, `dd-trace-dotnet`, `dd-trace-php`, `libdatadog`, and `openfeature-js-client`. diff --git a/README.md b/README.md index 1969b1a..191bb98 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,66 @@ # ffe-system-test-data -Canonical test data for Datadog's Feature Flags & Experimentation (FFE) SDK implementations. +The canonical behavioral contract for Datadog's Feature Flags & Experimentation +(FFE) implementations, expressed as shared test inputs and expected results. ## Overview -This repository contains the canonical set of flag configurations and evaluation test cases used to validate FFE SDK implementations across multiple languages. It serves as a single source of truth consumed via git submodules by: +A fixture describes what an implementation must do with a particular +configuration, evaluation context, and set of options. The expected result can +include a value, reason, error code, selected allocation metadata, or emitted +events. The fixtures and their documented rules define the shared behavior +covered here; downstream projects implement that behavior in their own language. + +## Contract ownership + +**Downstream implementations must conform to this contract.** A language's +standard library, an existing SDK behavior, or the current backend deployment +does not redefine an expected result. For example, if a timestamp boundary test +distinguishes microseconds, the consumer must preserve that precision even when +its usual date type only stores milliseconds. + +```mermaid +flowchart TD + C["Canonical contract
Fixtures, schemas, documented rules"] + P["Configuration producers"] + E["SDK and backend evaluators"] + T["Downstream conformance tests"] + C -->|implement required configuration semantics| P + C -->|implement required behavior| E + C -->|load inputs and expected results unchanged| T + P -->|produce configuration| E + E -->|return actual results| T +``` + +Configuration producers must emit the required fields with their defined +meaning. Evaluators implement parsing and evaluation. A downstream test adapter +maps the shared inputs and results to native APIs and checks the implementation +against the fixture. Rules-based tests can load `ufc-config.json` directly; +they do not need a running configuration compiler or deployed backend. + +The contract can be extended or corrected through review in this repository. +An implementation mismatch is evidence to investigate, not an automatic reason +to change an expectation. See [handling a mismatch](#handling-a-mismatch). + +Contract changes can land before every consumer implements them. SDK release +readiness, backend rollout order, and supported product versions belong in the +downstream projects' changes and release plans. This README describes stable +behavior and does not need an update whenever a consumer ships support. + +## Start here + +| What you are implementing | Contract to read | Test integration | +| --- | --- | --- | +| An SDK or backend that evaluates UFC rules | [`ufc-config.json`](ufc-config.json), [`evaluation-cases/`](evaluation-cases/), and [optional capabilities](#optional-capabilities) | Load the configuration through the real ingestion path and evaluate each case with the native evaluator. | +| An SDK that consumes precomputed assignments | [Precomputed fixture guide](precomputed-assignments/README.md) and [schema](schemas/precomputed-assignment.schema.json) | Supply the mocked response, run the ordered evaluations, and check results and emitted events. | + +These families have different formats and runners. Passing one does not +establish coverage of the other. When changing the contract, also read +[CONTRIBUTING.md](CONTRIBUTING.md). + +## Consumers + +Downstream projects include: - [system-tests](https://github.com/DataDog/system-tests) - Parametric tests - [dd-trace-py](https://github.com/DataDog/dd-trace-py) - Python tracer @@ -21,22 +77,19 @@ This repository contains the canonical set of flag configurations and evaluation ``` ffe-system-test-data/ -├── ufc-config.json # Master flag configuration (UFC format) +├── ufc-config.json # Shared rules-based configuration, including malformed test inputs ├── evaluation-cases/ │ └── test-*.json # Evaluation test case files ├── precomputed-assignments/ │ ├── README.md # Precomputed assignment fixture format -│ └── cases/*.json # Precomputed assignment test fixtures -└── regex-conformance/ - ├── targeting-regex-conformance.json # FFE authoring and matching contract - ├── targeting-regex-conformance.sha256 # SHA-256 of the JSON bytes - ├── validate-targeting-regex-conformance.jq # Canonical schema validator - └── test-validate-targeting-regex-conformance.sh # Validator regression tests +│ └── cases/ # Precomputed assignment fixtures, discovered recursively +├── schemas/ # Precomputed assignment fixture schema +└── ci/ # Static validators and downstream comparison runner ``` ## Usage -### As a Git Submodule +### Pin the fixture revision Add this repository as a submodule to your project: @@ -52,31 +105,75 @@ git clone --recurse-submodules git submodule update --init --recursive ``` -### In Tests - -1. Load `ufc-config.json` to initialize your UFC evaluator -2. For each file in `evaluation-cases/`, parse the JSON array -3. For each test case, call your evaluator with `flag`, `defaultValue`, `targetingKey`, and `attributes` -4. Assert the result matches `result.value` and `result.reason` +Commit the submodule pointer in the downstream repository so local runs and CI +use the same fixture commit. During development, a consumer can pin a fixture PR +commit; update to the merged fixture revision before landing the consumer change. +If a repository uses a generated snapshot instead, retain the source repository +and exact commit SHA and refresh it from that source. Do not hand-edit expected +results in a downstream copy. + +### Implement the rules-based runner + +1. Load `ufc-config.json` through the consumer's real configuration parser and + initialize its evaluator. Preserve the malformed inputs used by isolation tests. +2. Discover every `evaluation-cases/*.json` file and parse its array. Keep case + order within each file, including repeated evaluations. Fail if the fixture + directory is missing or empty. +3. Run every baseline case with default provider options, using `flag`, + `variationType`, `defaultValue`, `targetingKey`, and `attributes` as supplied. + Assert the expected value and reason, and any specified error code. Consumers + exposing selected metadata must also check the supplied `serialId` and `doLog` + expectations against the actual selection. +4. For each capability the consumer implements, map its name to the native + option and rerun every case carrying that capability. Assert its expected + result instead of the baseline result. Run capabilities separately. +5. Report the fixture SHA and executed case counts, including counts per + supported capability. Fail if a supported capability discovers no cases. + Skipping every capability case must not look like successful capability support. + +The adapter may translate API names and native result containers. It must +preserve the meaning and precision of the inputs and expected results, including +`false`, `0`, empty values, and the distinction between absent and null fields. +Use SDK-local tests for behavior outside these fixtures, such as concurrency, +transport failures, and provider lifecycle. ## File Formats ### UFC Config (`ufc-config.json`) -The UFC (Unified Flag Configuration) file contains flag definitions in the format used by Datadog's Remote Configuration: +The UFC (Universal Flag Configuration) file contains a document envelope and a +`flags` map. This is a complete minimal example of the rules-based format: ```json { - "flag-key": { - "key": "flag-key", - "enabled": true, - "variationType": "STRING|BOOLEAN|INTEGER|NUMERIC|JSON", - "variations": { ... }, - "allocations": [ ... ] + "createdAt": "2026-01-01T00:00:00Z", + "format": "SERVER", + "environment": { "name": "test" }, + "flags": { + "flag-key": { + "key": "flag-key", + "enabled": true, + "variationType": "BOOLEAN", + "variations": { "on": { "key": "on", "value": true } }, + "allocations": [ + { + "key": "ordinary", + "rules": [], + "splits": [{ "variationKey": "on", "shards": [] }], + "doLog": false + } + ] + } } } ``` +Valid allocations require an explicit boolean `doLog`. `false` is a valid +logging decision; omission and `null` are not valid substitutes. This input +requirement is separate from the optional `result.doLog` test expectation below. +Some fixtures deliberately contain malformed configuration to test error +handling; their presence does not make that input valid producer output. + ### Evaluation Test Cases (`evaluation-cases/test-*.json`) Each evaluation case uses a universal schema with the following fields: @@ -101,11 +198,11 @@ Example: [ { "flag": "flag-key", - "variationType": "STRING", - "defaultValue": "default", + "variationType": "BOOLEAN", + "defaultValue": false, "targetingKey": "user-123", - "attributes": { "country": "US" }, - "result": { "value": "expected-value", "reason": "TARGETING_MATCH" } + "attributes": {}, + "result": { "value": true, "reason": "STATIC" } } ] ``` @@ -117,6 +214,11 @@ For each supported entry in `capabilities`, enable that capability and repeat th same evaluation. Check its expected result instead. Ignore unsupported entries, not the baseline case. Capabilities run separately, not in combination. +A capability is a named optional behavior, not a release-status flag or a way +to exempt an implemented feature from conformance. A consumer that implements +the feature must wire its capability into the runner; updating the fixture pin +alone only adds baseline coverage. + `experimentationOptOutImmediate` enables immediate experimentation opt-out. The input stays in `attributes.dd_experimentation_opt_out_at`. For example: @@ -134,8 +236,8 @@ input stays in `attributes.dd_experimentation_opt_out_at`. For example: } ``` -The new cases reuse `ufc-config.json`. Immediate mode skips allocations whose -`doLog` is true (or omitted), then continues ordinary allocations. A valid +The cases reuse `ufc-config.json`. Immediate mode skips allocations whose +`doLog` is true, then continues ordinary allocations. A valid RFC3339 timestamp excludes immediately, including a future timestamp. A missing attribute leaves evaluation unchanged. An invalid value returns the caller default with `ERROR` / `INVALID_CONTEXT`. Default mode ignores the attribute. @@ -154,6 +256,42 @@ behavior. A supporting SDK adds the capability mapping to its existing reader; updating the submodule alone does not test the enabled behavior. SDK tests retain responsibility for telemetry, cache state, and provider lifecycle. +#### Future experiments only + +`experimentationOptOutFutureExperimentsOnly` uses the same subject timestamp. +Map it to the provider's future-experiments-only mode, or +`DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE=future_experiments_only`. + +| Allocation | With a valid subject cutoff | +| --- | --- | +| `doLog=false` | Evaluate normally. | +| `doLog=true`, activation before cutoff | Evaluate normally, including normal exposures. | +| `doLog=true`, activation at or after cutoff | Skip before targeting and sharding. Continue later allocations. | +| `doLog=true`, missing or null activation | Skip. Unknown activation does not establish an older experiment. | + +The allocation field is `randomizationActivatedAt`, an optional RFC3339 timestamp. +Compare instants and preserve the fractional precision exercised by the cases; +rounding or truncating to milliseconds changes the required result. Choose a +native representation that retains the required precision. Do not use `startAt`, +`endAt`, the configuration's `createdAt`, or the SDK's clock as a substitute. +Existing schedule checks still apply to eligible allocations. No subject history +or SDK persistence is required. + +The cutoff concerns allocation activation, not subject enrollment. An older +experiment can still assign a subject for the first time. Holdout allocations +use the same rule; the compiler supplies their activation metadata. + +An absent subject timestamp preserves normal evaluation. Invalid subject input +returns `ERROR` / `INVALID_CONTEXT` before allocation selection, as in immediate mode. +Supply the time the subject opted out. A later cutoff keeps experiments activated +before that later time eligible, including experiments started after the subject +opted out. Targeting and schedules still apply. +Removing the attribute permits experimentation again. + +The future-mode cases reuse existing flags. Added activation fields do not change +baseline or immediate results. Malformed activation parsing, configuration refresh, +exposure deduplication, and telemetry independence also need SDK-local tests. + ### Selected split and allocation metadata `test-case-serial-id-metadata.json` and `test-case-span-enrichment-inputs.json` @@ -180,9 +318,12 @@ capacity-limit assertions. ### SDK-Specific Fields -The shared fixtures intentionally exclude SDK-specific fields such as `variant` and `flagMetadata`. SDKs that need these fields should compute them at test load time from the universal fields. For example: +The shared fixtures use universal result fields rather than SDK-specific +containers such as `variant` and `flagMetadata`. Adapters should read the native +evaluation result and map it to these fields. For additional SDK-local assertions: -- **variant**: Derive from the flag configuration in `ufc-config.json` by matching the result value +- **variant**: Use the selected variation key. Equal result values do not + necessarily identify the same variation. - **flagMetadata**: Map the selected split's `serialId` and allocation's `doLog` to the SDK's metadata keys; assert the optional expected fields above directly @@ -197,7 +338,7 @@ expected exposure/flagevaluation emission counts. |-------|------|-------------| | `name` | string | Fixture identifier (matches filename stem) | | `description` | string | Human-readable description of what the fixture covers | -| `context` | object | Evaluation context: `targetingKey` (string) and `attributes` (object) | +| `context` | object | Evaluation context: `targetingKey` (string or null) and `attributes` (object) | | `response` | object | Full precomputed-assignments JSON:API envelope served by the mock CDN | | `_skip` | array? | Skip fixture for listed platforms: `[{"platform": "web", "reason": "..."}]` | | `_include` | array? | Run fixture only for listed platforms | @@ -213,29 +354,6 @@ expected exposure/flagevaluation emission counts. See [precomputed-assignments/README.md](precomputed-assignments/README.md) for further detail. -### Targeting Regex Conformance - -`regex-conformance/targeting-regex-conformance.json` is a standalone, versioned -contract for authoring targeting regular expressions in FFE. It is intentionally -outside `evaluation-cases/`; consumers of that directory parse every JSON file -as a complete UFC evaluation case. - -The fixture records native observations for four implementations: Go `regexp`, -RE2JS, the Rust rules-based evaluator, and the Rust rkyv evaluator. The accepted -authoring subset is narrower: accepted cases must also evaluate consistently in -the shipped Java, JavaScript, and .NET SDK evaluators. Several SDKs share the -Rust evaluator, so agreement across those SDKs is not evidence from independent -regex engines. - -Each regex case has a stable ID, an FFE authoring `contract`, raw and normalized -patterns, native compile observations, an input, and an unanchored match -observation when the modeled engines agree. Go and RE2JS consumers compile -`normalizedPattern`; Rust consumers compile `rawPattern`. Cases with differences -between modeled engines include per-engine expectations. Downstream SDK checks -must require consistent behavior for accepted cases. A native engine accepting -rejected syntax does not change the authoring contract. The adjacent SHA-256 -file lets downstream tests detect fixture drift. - ## Automated Validation Pull requests run a blocking static validation check over `ufc-config.json` @@ -247,7 +365,7 @@ context validation precedes flag lookup. The baseline still requires `FLAG_NOT_F The validator deliberately does not fully schema-check allocation internals. Some fixtures contain malformed flag fields on purpose to verify that consumers -reject only the affected flag. Run the same check locally with: +reject only the affected flag. Consumers exclude malformed flags from the active evaluation map while retaining their rejected keys for the current configuration. Evaluating a rejected key @@ -257,8 +375,41 @@ maps atomically so fixed or deleted flags do not leave stale rejection entries. ```bash python3 ci/validate-fixtures.py +python3 -m unittest discover -s ci -p 'test_*.py' ``` +For precomputed fixtures, install `jsonschema` and run +`python3 ci/validate-precomputed-fixtures.py`. See the +[precomputed fixture guide](precomputed-assignments/README.md#schema-and-validation). +Static validation checks fixture structure; it does not execute an SDK or prove +the expected behavior is correct. + +The [downstream compatibility workflow](.github/workflows/downstream-compatibility.yml) +runs native consumer tests against the base and proposed fixture revisions. +Its results are advisory evidence for review, even when a new mismatch makes +the comparison job fail. Inspect the tested consumer revision, runner, and +capability coverage; a passing baseline run does not prove a new capability was +executed. Local or CI conformance also does not prove deployment readiness. + +## Handling a mismatch + +1. Record the fixture commit, consumer commit, case file and index, enabled + capability, expected result, and actual result. Run through the real parser + and evaluator to distinguish an adapter bug from an implementation bug. +2. Check the documented rule and reproduce the relevant boundary. Once the + expectation is confirmed, fix the downstream implementation to meet it. + Keep any temporary downstream exclusion explicit and tracked; it does not + change the shared expectation or establish conformance. +3. If the fixture is wrong or the intended behavior needs to change, propose + that correction here with a concrete example, rationale, and consumer impact. + Obtain contract review before propagating changed expectations downstream. + +For example, a consumer treating `.123001` seconds as equal to `.123` has lost +precision required by the opt-out boundary cases. Fix its timestamp handling; +do not round the fixture to make that implementation pass. Likewise, backend +support that has not shipped yet is a rollout dependency, not a defect in a +fixture defining the intended behavior. + ## Evaluation Cases | File | Description | @@ -269,6 +420,9 @@ python3 ci/validate-fixtures.py | `test-case-disabled-flag.json` | Disabled flag returning DISABLED reason | | `test-case-empty-flag.json` | Flag with empty configuration | | `test-case-empty-string-variation.json` | Flag returning empty string as value | +| `test-case-experimentation-opt-out.json` | Baseline, immediate, and future opt-out behavior, including invalid subject input and ordinary fallbacks | +| `test-case-experimentation-opt-out-future.json` | Activation cutoffs, fractional precision, timezone offsets, and unknown activation | +| `test-case-experimentation-opt-out-do-log.json` | Ordinary allocations remain eligible when activation equals or follows the cutoff | | `test-case-falsy-value-assignments.json` | Flags returning falsy values (0, false, empty) | | `test-case-flag-with-empty-string.json` | Flag with empty string in configuration | | `test-case-integer-flag.json` | Integer-typed flag evaluation | @@ -296,6 +450,8 @@ python3 ci/validate-fixtures.py | `test-case-semver-validation-flag.json` | SemVer 2 syntax boundaries retained by the extended parser, invalid syntax, non-string attributes, and invalid configured comparands | | `test-case-semver-invalid-comparand-waterfall.json` | An invalid configured SemVer comparand in a non-last allocation of a waterfall must reject the whole flag at parse time rather than skipping it and falling through to a later valid default | | `test-case-semver-invalid-comparand-categories.json` | Configured SemVer comparand boundaries: a valid hyphen-heavy prerelease matches, while invalid syntax (including consecutive dots and a Ruby-style `.DEV` suffix), v-prefixed, leading-zero, and non-string comparands reject their flag at parse time with PARSE_ERROR without poisoning the rest of the configuration | +| `test-case-serial-id-metadata.json` | Selected split IDs, including zero, absence, targeting, and repeated evaluations | +| `test-case-span-enrichment-inputs.json` | Selected IDs and logging decisions used by downstream span-enrichment tests | | `test-case-start-and-end-date-flag.json` | Flag with start/end date time bounds | | `test-case-unknown-fields-tolerance.json` | Unknown UFC object fields are ignored | | `test-case-unknown-operator-isolation.json` | A flag with an unknown operator is removed without poisoning valid flags | @@ -307,11 +463,16 @@ python3 ci/validate-fixtures.py ## Origin -These fixtures are derived from the Go SDK (`dd-trace-go`) reference implementation, which was the first to implement and validate the OpenFeature `reason` field. The Go fixtures serve as the canonical source of truth for expected evaluation behavior across all Datadog SDKs. +These fixtures originated in the Go SDK (`dd-trace-go`). This repository now +owns the shared contract; Go and every other downstream implementation consume +it and are subject to the same conformance requirements. ## Downstream Updates -Downstream repositories should consume this repository as a git submodule and run their fixture coverage by loading every JSON file in `evaluation-cases/`. Shared evaluator behavior should be added here first, then downstream repositories should update their submodule SHA. Do not add copied JSON fixture directories or language-only programmatic cases for behavior that belongs in this shared fixture set. +Add shared evaluator behavior here, then update each consumer's +[pinned fixture revision](#pin-the-fixture-revision) and native test runner. +See [Updating Downstream Repositories](CONTRIBUTING.md#updating-downstream-repositories) +for the integration checklist. ### Informational Compatibility Preview diff --git a/evaluation-cases/test-case-experimentation-opt-out-do-log.json b/evaluation-cases/test-case-experimentation-opt-out-do-log.json new file mode 100644 index 0000000..0d2e8d5 --- /dev/null +++ b/evaluation-cases/test-case-experimentation-opt-out-do-log.json @@ -0,0 +1,22 @@ +[ + { + "description": "ordinary allocation remains eligible when activation equals cutoff", + "flag": "no-log-flag", "variationType": "BOOLEAN", "defaultValue": false, "targetingKey": "subject", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, + "result": { "value": true, "reason": "STATIC" }, + "capabilities": { + "experimentationOptOutImmediate": { "value": true, "reason": "STATIC" }, + "experimentationOptOutFutureExperimentsOnly": { "value": true, "reason": "STATIC" } + } + }, + { + "description": "ordinary allocation remains eligible when activation is after cutoff", + "flag": "no-log-flag", "variationType": "BOOLEAN", "defaultValue": false, "targetingKey": "subject", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T11:59:59Z" }, + "result": { "value": true, "reason": "STATIC" }, + "capabilities": { + "experimentationOptOutImmediate": { "value": true, "reason": "STATIC" }, + "experimentationOptOutFutureExperimentsOnly": { "value": true, "reason": "STATIC" } + } + } +] diff --git a/evaluation-cases/test-case-experimentation-opt-out-future.json b/evaluation-cases/test-case-experimentation-opt-out-future.json new file mode 100644 index 0000000..ea33f20 --- /dev/null +++ b/evaluation-cases/test-case-experimentation-opt-out-future.json @@ -0,0 +1,146 @@ +[ + { + "description": "older allocation continues", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-01T12:00:00.000001Z" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } + } + }, + { + "description": "equal activation is excluded", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-01T12:00:00Z" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "newer allocation is excluded", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-01T11:59:59.999999Z" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "equal instant with negative offset", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-01T08:00:00-04:00" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "equal instant with positive offset", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-01T14:00:00+02:00" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "future cutoff is not an evaluation schedule", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "dd_experimentation_opt_out_at": "2099-01-01T00:00:00Z" }, + "result": { "value": 1, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } + } + }, + { + "description": "skip newer targeted allocation and continue older randomization", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "country": "US", "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, + "result": { "value": 3, "reason": "TARGETING_MATCH" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } + } + }, + { + "description": "targeted allocation remains eligible before cutoff", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "country": "US", "dd_experimentation_opt_out_at": "2026-10-21T12:00:00Z" }, + "result": { "value": 3, "reason": "TARGETING_MATCH" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 3, "reason": "TARGETING_MATCH" } + } + }, + { + "description": "skip both allocations", + "flag": "integer-flag", "variationType": "INTEGER", "defaultValue": 0, "targetingKey": "5", + "attributes": { "country": "US", "dd_experimentation_opt_out_at": "2026-10-01T12:00:00Z" }, + "result": { "value": 3, "reason": "TARGETING_MATCH" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "fractional activation equals cutoff", + "flag": "special-characters", "variationType": "JSON", "defaultValue": {}, "targetingKey": "ben", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00.123Z" }, + "result": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } + } + }, + { + "description": "allocation is one microsecond older", + "flag": "special-characters", "variationType": "JSON", "defaultValue": {}, "targetingKey": "ben", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00.123001Z" }, + "result": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" } + } + }, + { + "description": "allocation is one microsecond newer", + "flag": "special-characters", "variationType": "JSON", "defaultValue": {}, "targetingKey": "ben", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00.122999Z" }, + "result": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } + } + }, + { + "description": "missing activation is not treated as older", + "flag": "kill-switch", "variationType": "BOOLEAN", "defaultValue": false, "targetingKey": "alice", + "attributes": { "country": "US", "dd_experimentation_opt_out_at": "2099-01-01T00:00:00Z" }, + "result": { "value": true, "reason": "TARGETING_MATCH" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": false, "reason": "DEFAULT" } + } + }, + { + "description": "null activation is not treated as older", + "flag": "numeric_flag", "variationType": "NUMERIC", "defaultValue": 0, "targetingKey": "alice", + "attributes": { "dd_experimentation_opt_out_at": "2099-01-01T00:00:00Z" }, + "result": { "value": 3.1415926, "reason": "STATIC" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } + } + }, + { + "description": "activation is not startAt or document createdAt", + "flag": "start-and-end-date-test", "variationType": "STRING", "defaultValue": "default", "targetingKey": "subject", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, + "result": { "value": "current", "reason": "DEFAULT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": "default", "reason": "DEFAULT" } + } + }, + { + "description": "old activation does not override endAt or startAt", + "flag": "start-and-end-date-test", "variationType": "STRING", "defaultValue": "default", "targetingKey": "subject", + "attributes": { "dd_experimentation_opt_out_at": "2026-10-21T12:00:00Z" }, + "result": { "value": "current", "reason": "DEFAULT" }, + "capabilities": { + "experimentationOptOutFutureExperimentsOnly": { "value": "current", "reason": "DEFAULT" } + } + } +] diff --git a/evaluation-cases/test-case-experimentation-opt-out.json b/evaluation-cases/test-case-experimentation-opt-out.json index 8120c5f..daa4f72 100644 --- a/evaluation-cases/test-case-experimentation-opt-out.json +++ b/evaluation-cases/test-case-experimentation-opt-out.json @@ -4,7 +4,8 @@ "attributes": { "dd_experimentation_opt_out_at": "not-a-timestamp" }, "result": { "value": "default", "reason": "ERROR", "errorCode": "FLAG_NOT_FOUND" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -12,7 +13,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": {}, "reason": "ERROR", "errorCode": "TARGETING_KEY_MISSING" }, "capabilities": { - "experimentationOptOutImmediate": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } + "experimentationOptOutImmediate": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" }, + "experimentationOptOutFutureExperimentsOnly": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } } }, { @@ -20,7 +22,8 @@ "attributes": { "country": "UK" }, "result": { "value": "red", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "red", "reason": "TARGETING_MATCH" } + "experimentationOptOutImmediate": { "value": "red", "reason": "TARGETING_MATCH" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "red", "reason": "TARGETING_MATCH" } } }, { @@ -28,7 +31,8 @@ "attributes": { "country": "UK", "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": "red", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "default", "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": "default", "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "red", "reason": "TARGETING_MATCH" } } }, { @@ -36,7 +40,8 @@ "attributes": { "email": "alice@mycompany.com", "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": "green", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "green", "reason": "TARGETING_MATCH" } + "experimentationOptOutImmediate": { "value": "green", "reason": "TARGETING_MATCH" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "green", "reason": "TARGETING_MATCH" } } }, { @@ -44,7 +49,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } + "experimentationOptOutImmediate": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" }, + "experimentationOptOutFutureExperimentsOnly": { "value": { "a": "kümmert", "b": "schön" }, "reason": "STATIC" } } }, { @@ -52,7 +58,8 @@ "attributes": {}, "result": { "value": 2, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 2, "reason": "SPLIT" } + "experimentationOptOutImmediate": { "value": 2, "reason": "SPLIT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 2, "reason": "SPLIT" } } }, { @@ -60,7 +67,8 @@ "attributes": {}, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 1, "reason": "SPLIT" } + "experimentationOptOutImmediate": { "value": 1, "reason": "SPLIT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } } }, { @@ -68,7 +76,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": 2, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 2, "reason": "SPLIT" } } }, { @@ -76,7 +85,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } } }, { @@ -84,7 +94,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": 3.1415926, "reason": "STATIC" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "DEFAULT" } } }, { @@ -92,7 +103,8 @@ "attributes": { "country": "US", "dd_experimentation_opt_out_at": "2026-10-19T12:00:00Z" }, "result": { "value": true, "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": false, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": false, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": false, "reason": "DEFAULT" } } }, { @@ -100,7 +112,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T12:00:00.123456789Z" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } } }, { @@ -108,7 +121,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2026-10-19T08:00:00-04:00" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } } }, { @@ -116,7 +130,8 @@ "attributes": { "dd_experimentation_opt_out_at": "2099-01-01T00:00:00Z" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 1, "reason": "SPLIT" } } }, { @@ -124,7 +139,8 @@ "attributes": { "dd_experimentation_opt_out_at": "not-a-timestamp" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -132,7 +148,8 @@ "attributes": { "dd_experimentation_opt_out_at": "" }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -140,7 +157,8 @@ "attributes": { "dd_experimentation_opt_out_at": true }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -148,7 +166,8 @@ "attributes": { "dd_experimentation_opt_out_at": null }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -156,7 +175,8 @@ "attributes": { "dd_experimentation_opt_out_at": 1792411200 }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -164,7 +184,8 @@ "attributes": { "email": "alice@mycompany.com", "dd_experimentation_opt_out_at": "not-a-timestamp" }, "result": { "value": "green", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -172,7 +193,8 @@ "attributes": { "email": "alice@mycompany.com", "dd_experimentation_opt_out_at": false }, "result": { "value": "green", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -180,7 +202,8 @@ "attributes": { "email": "alice@mycompany.com", "dd_experimentation_opt_out_at": null }, "result": { "value": "green", "reason": "TARGETING_MATCH" }, "capabilities": { - "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": "default", "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -188,7 +211,8 @@ "attributes": { "dd_experimentation_opt_out_at": "not-a-timestamp" }, "result": { "value": { "a": "піклуватися", "b": "любов" }, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": {}, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": {}, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": { }, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -196,7 +220,8 @@ "attributes": { "dd_experimentation_opt_out_at": { "timestamp": "2026-10-19T12:00:00Z" } }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } }, { @@ -204,7 +229,8 @@ "attributes": { "dd_experimentation_opt_out_at": ["2026-10-19T12:00:00Z"] }, "result": { "value": 1, "reason": "SPLIT" }, "capabilities": { - "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } + "experimentationOptOutImmediate": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" }, + "experimentationOptOutFutureExperimentsOnly": { "value": 0, "reason": "ERROR", "errorCode": "INVALID_CONTEXT" } } } ] diff --git a/ufc-config.json b/ufc-config.json index a436b88..e435a37 100644 --- a/ufc-config.json +++ b/ufc-config.json @@ -65,7 +65,8 @@ "shards": [] } ], - "doLog": true + "doLog": true, + "randomizationActivatedAt": null } ] }, @@ -1016,7 +1017,8 @@ } ], "endAt": "2002-10-31T09:00:00.594Z", - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2000-01-01T00:00:00Z" }, { "key": "future-versions", @@ -1027,7 +1029,8 @@ } ], "startAt": "2052-10-31T09:00:00.594Z", - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-01T12:00:00Z" }, { "key": "current-versions", @@ -1039,7 +1042,8 @@ ], "startAt": "2022-10-31T09:00:00.594Z", "endAt": "2050-10-31T09:00:00.594Z", - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-20T12:00:00Z" } ] }, @@ -1278,7 +1282,8 @@ ] } ], - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-01T12:00:00Z" }, { "key": "rollout", @@ -1384,7 +1389,8 @@ ] } ], - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-20T12:00:00Z" }, { "key": "50/50 split", @@ -1421,7 +1427,8 @@ ] } ], - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-01T12:00:00Z" } ] }, @@ -1644,7 +1651,8 @@ ] } ], - "doLog": true + "doLog": true, + "randomizationActivatedAt": "2026-10-19T12:00:00.123Z" }, { "key": "allocation-default", @@ -5341,6 +5349,7 @@ "allocations": [ { "key": "no-log-allocation", + "randomizationActivatedAt": "2026-10-19T12:00:00Z", "rules": [], "splits": [ {