Repository navigation
test(ffe): verify experimentation opt-out against Lean contracts - #8135
Closed
leoromanovsky wants to merge 17 commits into
Closed
leoromanovsky wants to merge 17 commits into
leoromanovsky wants to merge 17 commits into
Conversation
Contributor
|
|
9 tasks
🎉 All green!🧪 All tests passed 🔗 Commit SHA: d4c4308 | Docs | View more details | Give us feedback! |
leoromanovsky
marked this pull request as ready for review
October 21, 2026 16:02
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
leoromanovsky
added a commit
to DataDog/dd-trace-go
that referenced
this pull request
Oct 22, 2026
### What does this PR do?
Adds `ExperimentationOptOutFutureExperimentsOnly` and the environment
value `future_experiments_only`.
For a subject with `dd_experimentation_opt_out_at`, compare the cutoff
with each exposure-logging allocation's `randomizationActivatedAt`:
| Allocation activation | Result |
| --- | --- |
| Before the cutoff | Continue normal evaluation and exposure logging. |
| At or after the cutoff, or unknown | Skip the allocation and continue
ordinary delivery. |
Ordinary allocations remain eligible. Missing subject input preserves
normal evaluation. Malformed subject input returns `INVALID_CONTEXT`, as
in immediate mode. No new telemetry or subject history is added.
```sh
DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE=future_experiments_only
```
```go
// ProviderConfig{} uses the environment setting.
provider, err := ddflags.NewDatadogProvider(ddflags.ProviderConfig{})
if err != nil {
return err
}
if err := of.SetNamedProviderAndWait("flags", provider); err != nil {
return err
}
client := of.NewClient("flags")
attributes := map[string]any{}
if subject.ExperimentationOptOutAt != nil {
attributes["dd_experimentation_opt_out_at"] =
subject.ExperimentationOptOutAt.UTC().Format(time.RFC3339Nano)
}
enabled, err := client.BooleanValue(ctx, "new-route-planner", false,
of.NewEvaluationContext(subject.ID, attributes))
```
An explicit `ProviderConfig{ExperimentationOptOutMode:
ddflags.ExperimentationOptOutFutureExperimentsOnly}` overrides the
environment setting.
### Motivation
**Review scope:** [Downscoped in
e282fff](e282fff)
to OpenFeature implementation, tests, and documentation. The benchmark
CI configuration is identical to #5666.
`BenchmarkExperimentationExclusion` and
`BenchmarkBooleanEvaluationExclusion` remain available locally;
registering them in CI and assessing performance belong to #5666 after
this child is folded in. That follow-up remains open; existing approvals
are not approval of the future CI changes.
Customers can stop new experiment participation while continuing older
experiments. The cutoff describes allocation activation, not previous
subject enrollment. An older experiment can still assign a subject for
the first time.
**Stack:** #5666 supplies immediate mode and the configuration API. This
PR adds the activation comparison. [Shared fixtures
#52](DataDog/ffe-system-test-data#52) defines 42
capability checks; update the pin after that PR merges.
Enablement depends on the backend emitting stable activation metadata
and retaining ordinary fallbacks. This PR does not deploy those changes.
Allocations without metadata are skipped for opted-out subjects.
### Test coverage
Shared fixtures cover cutoff boundaries, timezone offsets, fractional
seconds, invalid input, missing metadata, and fallback delivery. Local
tests cover malformed activation metadata and valid-neighbor isolation
in all modes, refresh, exposure deduplication, telemetry independence,
and concurrent randomized subjects. Baseline and immediate-mode behavior
remain covered. A benchmark covers all three modes.
[System tests #8135](DataDog/system-tests#8135),
stacked on system-tests #8080, add Remote Config, holdout assignment,
exposure, and telemetry checks across all three modes. Release gating
remains provisional until supported SDK releases are known.
### Validation
| Tested commit | Validation | Result |
| --- | --- | --- |
|
[e282fff](e282fff)
| Local OpenFeature package and CI-selection tests after syncing the
base and removing the child CI delta. | Both passed. Remote CI is
tracked separately. |
|
[febcc18](febcc18)
| System tests: disabled, immediate, and future-only modes; cutoff
boundaries, Remote Config refresh, holdout assignments, exposure
deduplication, and independent evaluation telemetry. Linux/amd64;
[system-tests
6418472](DataDog/system-tests@6418472).
| 12 passed; 0 failed; 0 skipped. |
### Reviewer's Checklist
- [ ] Changed code has unit tests for its functionality at or near 100%
coverage.
- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering
this feature have been added and enabled with the va.b.c-dev version
tag.
- [ ] There is a benchmark for any new code, or changes to existing
code.
- [ ] If this interacts with the agent in a new way, a system test has
been added.
- [ ] New code is free of linting errors. You can check this by running
`make lint` locally.
- [ ] New code doesn't break existing tests. You can check this by
running `make test` locally.
- [ ] Add an appropriate team label so this PR gets put in the right
place for the release notes.
- [ ] All generated files are up to date. You can check this by running
`make generate` locally.
- [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed
by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to
date by running `make fix-modules` locally.
Unsure? Have a question? Request a review!
leoromanovsky
changed the title
Base automatically changed from
leoromanovsky/evaluation-telemetry-opt-out
to
main
October 23, 2026 18:02
greghuels
removed their request for review
October 27, 2026 10:33
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Verify experimentation opt-out through running SDKs, Remote Config, and captured telemetry.
Changes
Ruby uses the temporary libdatadog branch at
ead1d3db2aa16e494f23e47b946ff12a7a1be53a. Stacked on #8080. Release manifests remain provisional. Java coverage uses the legacy provider; standalone transport remains untested.Validation
dd-trace-go@54b5ad7fe9818966d32eeceabb6c048005f077f6system-tests@23b69bd59edaf9a6a3fdc71873d18d14616f550c.dd-trace-java@578ffa77cbc52e6d5d8950657a05863a449f59absystem-tests@23b69bd59edaf9a6a3fdc71873d18d14616f550c.|
DataDog/dd-trace-rb@6c4e268d18e68523f0e5e6546a86d6754c350e3a| System tests: three opt-out modes, policy and lookup cases, recovery, Remote Config, holdouts, exposures, and independent telemetry; Ruby 4.0, Linux/amd64;system-tests@d4c4308b3600a48885564f687c37800b4eeb5c19; isolated net-imap 0.6.2 harness. | 21 passed; 0 failed; 0 skipped. |Workflow
system-tests-reviewers;system-tests-corefor scenario integration.FFE reviewer selection
Reviewers: @greghuels, @sameerank
Why: Global fallback — The PR adds system tests for per-subject experimentation opt-out: an activation cutoff that suppresses exposures, plus a separate per-subject suppression of evaluation telemetry. Global reviewers were selected because no project or SME pod matched.
🤖 Automated selection.
Selection details
Initiative: global / global
Why this match: The PR adds system tests for per-subject experimentation opt-out: an activation cutoff that suppresses exposures, plus a separate per-subject suppression of evaluation telemetry. The expected results come from a pinned Lean contract and cover Remote Config updates, holdouts and EVP delivery. None of the supplied projects fits. flag-evaluation-evp explicitly excludes per-context suppression of evaluation and exposure egress and assigns it to ufc-compliance, which is not in the catalog. The work has no bandit purpose. The weblog changes that return OpenFeature evaluation details only support the opt-out assertions and are not onboarding diagnostics. No supported primary initiative fits, so this uses global ownership.
The PR author was excluded. Existing reviewers and teams were preserved.
Debug or fix a problem
Workflow run · Bits Code session
dry_run=false. Unchanged completed runs are skipped. If the receipt saysneeds_reconciliationor a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.Evidence and run details
✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.
Generated asynchronously by the FFE reviewer-routing workflow.
Changed paths:
tests/ffe/test_experimentation_opt_out.py,tests/ffe/test_evaluation_telemetry_opt_out.py,tests/ffe/utils/experimentation_opt_out.py,tests/ffe/utils/lean_experimentation_opt_out.py,tests/ffe/README.md,utils/scripts/generate_ffe_opt_out_contract.pyEvaluated head:
d4c4308b3600a48885564f687c37800b4eeb5c19Registry revision:
revision:1946aa2a-530e-487a-84a7-83aff44a1089Pinned skill source:
d214cfc65ae154b97d39af5cf32caf48aaec9bf0Instructions: workflow-injected snapshot (no plugin loaded)
Instruction snapshot SHA-256:
f23b2a88c13c0da90e2dd58246a14f257af91e2195c555039bd555b52e6f3516