Skip to content

feat(openfeature): support immediate per-subject experimentation opt-out - #21112

Open
leoromanovsky wants to merge 5 commits into
mainfrom
leoromanovsky/experimentation-opt-out-immediate
Open

leoromanovsky wants to merge 5 commits into
mainfrom
leoromanovsky/experimentation-opt-out-immediate

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Customers need to exclude selected subjects from experimentation without blocking ordinary feature delivery.

Changes and Decisions

  • Add DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE, defaulting to disabled. Explicit provider configuration takes precedence.
  • In immediate mode, validate the subject's timestamp and skip experiment allocations before targeting. Continue through ordinary allocations.
  • Return the application default with INVALID_CONTEXT for invalid timestamps. Keep evaluation telemetry and exposure deduplication unchanged.
  • Use the shared Rust evaluator's existing allocation timestamp parser and the merged fixture capability contract.
  • Preserve invalid object and array opt-out attributes at the native boundary without changing the customer's context.
export DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE=immediate
from ddtrace.openfeature import DataDogProvider
from openfeature import api
from openfeature.evaluation_context import EvaluationContext

api.set_provider(DataDogProvider())
attributes = {}
if athlete.experimentation_opt_out_at is not None:
    attributes["dd_experimentation_opt_out_at"] = (
        athlete.experimentation_opt_out_at.isoformat()
    )
enabled = api.get_client().get_boolean_value(
    "new-route-planner",
    False,
    EvaluationContext(targeting_key=athlete.id, attributes=attributes),
)

The timestamp must include its time zone. A future timestamp also takes effect immediately.

Dependency: The shared libdatadog evaluator change must land and receive an approved dependency version before this draft can merge. This draft pins DataDog/libdatadog@020004781291bd9178cd91868f53291bfed550f7.

Validation

Tested commit Validation Result
DataDog/dd-trace-py@ac6a36954750d31fb423d42d562200797b8db865 System tests: environment modes, ordinary fallback, invalid-context defaults, and exposure deduplication; Linux/arm64 Not rerun. Local Docker storage reports filesystem I/O errors. The new object/array provider tests have not run locally.
DataDog/dd-trace-py@f460cd9276f220d1c4f33cf36b6472c87bfc84dc Previous-head system tests: environment modes, ordinary fallback, invalid-context defaults, and exposure deduplication; Linux/arm64 3/3 passed. This does not validate the current head. Evaluation-event delivery was not validated.

FFE reviewer selection

Reviewers: @vjfridge, @btthomas

Why: server py — Adds a per-subject experimentation opt-out mode to the Python OpenFeature provider. Reviewers come from the matched pod.

🤖 Automated selection.

Selection details

Initiative: fallback / server-py

Why this match: Adds a per-subject experimentation opt-out mode to the Python OpenFeature provider. In immediate mode it validates dd_experimentation_opt_out_at, skips experiment allocations and returns INVALID_CONTEXT for invalid timestamps. It does not suppress evaluation or exposure events: the PR keeps telemetry and exposure deduplication unchanged, so ufc-compliance does not fit. It adds no layers, mutual exclusion, holdouts or bandits. No catalog project matches this behavior, so it falls to the Python Feature Flags/OpenFeature subject pod.

  • @vjfridge — primary (server-python): Matched pod member. Review requested.
  • @btthomas — floating (server-python): Coverage and knowledge sharing. Review requested.

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

  • Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
  • Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
  • Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
  • Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.
Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: ddtrace/internal/openfeature/_experimentation_opt_out.py, ddtrace/internal/openfeature/_provider.py, ddtrace/internal/openfeature/_native.py, ddtrace/internal/settings/openfeature.py, ddtrace/openfeature/__init__.py, tests/openfeature/test_experimentation_opt_out.py, tests/openfeature/test_provider_fixtures.py

Evaluated head: ac6a36954750d31fb423d42d562200797b8db865

Registry revision: revision:1946aa2a-530e-487a-84a7-83aff44a1089

Pinned skill source: d214cfc65ae154b97d39af5cf32caf48aaec9bf0

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: f23b2a88c13c0da90e2dd58246a14f257af91e2195c555039bd555b52e6f3516

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 135 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 135 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.litellm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
ddtrace/internal/openfeature/_experimentation_opt_out.py                @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/openfeature/_native.py                                 @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/openfeature/_provider.py                               @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/settings/_supported_configurations.py                  @DataDog/apm-python
ddtrace/internal/settings/openfeature.py                                @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/openfeature/__init__.py                                         @DataDog/feature-flagging-and-experimentation-sdk
docs/configuration.rst                                                  @DataDog/python-guild
releasenotes/notes/feature-flags-experimentation-opt-out-549474088d1582eb.yaml  @DataDog/apm-python
src/native/Cargo.lock                                                   @DataDog/apm-core-python
src/native/Cargo.toml                                                   @DataDog/apm-core-python
src/native/ffe.rs                                                       @DataDog/feature-flagging-and-experimentation-sdk
supported-configurations.json                                           @DataDog/apm-python
tests/openfeature/ffe-system-test-data/SOURCE.md                        @DataDog/feature-flagging-and-experimentation-sdk
tests/openfeature/ffe-system-test-data/evaluation-cases/test-case-experimentation-opt-out.json  @DataDog/feature-flagging-and-experimentation-sdk
tests/openfeature/test_experimentation_opt_out.py                       @DataDog/feature-flagging-and-experimentation-sdk
tests/openfeature/test_provider_fixtures.py                             @DataDog/feature-flagging-and-experimentation-sdk

@pr-commenter

pr-commenter Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-10-20 22:49:26

Comparing candidate commit ac6a369 in PR branch leoromanovsky/experimentation-opt-out-immediate with baseline commit ef12f0a in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 543 metrics, 10 unstable metrics, 6 known flaky benchmarks, 18 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-tracecontext_headers

  • 🟥 execution_time [+3.158µs; +3.396µs] or [+10.094%; +10.855%]

scenario:iastaspects-add_aspect

  • 🟥 execution_time [+10.169µs; +11.952µs] or [+11.373%; +13.366%]

scenario:iastaspects-capitalize_noaspect

  • 🟥 execution_time [+36.884µs; +41.521µs] or [+19.276%; +21.699%]

scenario:iastaspects-ljust_aspect

  • 🟥 execution_time [+147.169µs; +152.879µs] or [+30.311%; +31.488%]

scenario:iastaspectsremodule-re_expand_aspect

  • 🟥 execution_time [+96.877µs; +103.759µs] or [+15.592%; +16.699%]

scenario:msgpackencoderscenario-simple_one_span

  • 🟥 execution_time [+345.504ns; +431.268ns] or [+8.632%; +10.774%]

scenario:otelspan-start

  • 🟥 execution_time [+2.447ms; +3.309ms] or [+9.280%; +12.546%]

scenario:recursivecomputation-shallow

  • 🟥 execution_time [+51.217µs; +54.833µs] or [+7.380%; +7.901%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-689.545ns; +785.486ns] or [-6.464%; +7.363%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-38.022ns; +39.792ns] or [-5.582%; +5.842%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1847.358ns; +1985.131ns] or [-9.285%; +9.977%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1850.048ns; +2125.408ns] or [-9.578%; +11.004%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-350.737ns; +408.033ns] or [-8.227%; +9.571%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-232.954ns; +226.117ns] or [-8.640%; +8.386%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-42.287ns; +153.629ns] or [-3.025%; +10.992%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-4347.049ns; +5335.444ns] or [-8.748%; +10.736%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-998.462ns; +951.101ns] or [-9.464%; +9.015%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-54.399µs; +57.681µs] or [-9.435%; +10.005%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.500µs; +2.653µs] or [+18.239%; +19.354%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+134.232µs; +139.837µs] or [+35.367%; +36.844%]

scenario:iastaspectssplit-rsplit_aspect

  • 🟥 execution_time [+30.072µs; +33.271µs] or [+19.083%; +21.113%]

scenario:span-start

  • 🟥 execution_time [+1.034ms; +1.328ms] or [+9.072%; +11.652%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+410.554ns; +455.593ns] or [+21.131%; +23.449%]

scenario:tracer-small

  • 🟥 execution_time [+43.256µs; +44.727µs] or [+18.544%; +19.174%]

Known flaky benchmarks without significant changes:

  • scenario:errortrackingflasksqli-baseline
  • scenario:flasksimple-iast-get
  • scenario:iastaspects-casefold_aspect
  • scenario:iastaspects-casefold_noaspect
  • scenario:iastaspects-index_aspect
  • scenario:iastaspects-ljust_noaspect
  • scenario:iastaspects-lower_aspect
  • scenario:iastaspects-replace_aspect
  • scenario:iastaspects-rstrip_aspect
  • scenario:iastaspects-swapcase_aspect
  • scenario:iastaspects-title_noaspect
  • scenario:iastaspects-translate_aspect
  • scenario:iastaspects-translate_noaspect
  • scenario:iastaspects-upper_noaspect
  • scenario:packagespackageforrootmodulemapping-cache_off
  • scenario:packagespackageforrootmodulemapping-cache_on
  • scenario:sethttpmeta-all-enabled
  • scenario:telemetryaddmetric-record-100-metrics

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ac6a369 | Docs | View more details | Give us feedback!

@leoromanovsky
leoromanovsky marked this pull request as ready for review October 20, 2026 22:28
@leoromanovsky
leoromanovsky requested review from a team as code owners October 20, 2026 22:28
@leoromanovsky
leoromanovsky requested review from Yun-Kim and brettlangdon and removed request for a team October 20, 2026 22:28
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T22:32:02.360199Z ac6a369 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-20T22:32:32.486443Z ac6a369 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

No concrete defect was identified in configuration precedence, invalid-context handling, or ordinary-allocation fallback. Native and system runtime behavior remains unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit ac6a369

@brettlangdon brettlangdon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit about keeping Cargo.toml crates all in sync on an actual libdatadog version, otherwise config/release note changes lgtm

Comment thread src/native/Cargo.toml
smallvec = "1"
tokio = "1"
libdd-ffe = { git = "https://github.com/DataDog/libdatadog", rev = "v46.0.0", optional = true, features = ["pyo3"] }
libdd-ffe = { git = "https://github.com/DataDog/libdatadog", rev = "020004781291bd9178cd91868f53291bfed550f7", optional = true, features = ["pyo3"] }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to keep this on a synced release across the other crates... if we can.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I'll wait for a release.

@greghuels
greghuels removed their request for review October 27, 2026 10:33
@leoromanovsky

leoromanovsky commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

FFE reviewer selection

Reviewers: @btthomas, @sameerank

Why: server py — Adds an immediate per-subject experimentation opt-out mode to the Python OpenFeature DataDogProvider. Reviewers come from the matched pod and the required specialist pods.

🤖 Automated selection.

Selection details

Initiative: fallback / server-py

Why this match: Adds an immediate per-subject experimentation opt-out mode to the Python OpenFeature DataDogProvider. A new setting and constructor option control it. When it is on, the provider skips experiment allocations for opted-out subjects and returns INVALID_CONTEXT for invalid timestamps. Evaluation telemetry and exposure handling stay the same. Experiment-allocation eligibility is explicitly excluded from ufc-compliance, and no other registered project fits, so the Python OpenFeature SME owns this.

Why these owners: Eligible reviewers have already approved the current head; no additional reviewers are needed for the responsibilities those approvals cover.

  • @btthomas — floating (server-python): Coverage and knowledge sharing. Existing review coverage preserved.
  • @sameerank — specialist (openfeature-architecture): Additional subject expertise. Covers: openfeature-architecture. Existing review coverage preserved.

Additional review: openfeature-architecture — Changes the contract at the native evaluator boundary. The PyO3 resolve_value now takes an experimentation_opt_out_immediate option and calls the shared libdatadog eval_flag_with_options with EvaluationOptions in place of get_assignment. It maps the new EvaluationError::InvalidContext to the INVALID_CONTEXT error code. The Python wrapper now replaces invalid object or array opt-out attributes with null before they reach the native layer. Coverage: @sameerank. Evidence: src/native/ffe.rs, ddtrace/internal/openfeature/_native.py, ddtrace/internal/native/_native.pyi

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

  • Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
  • Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
  • Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
  • Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.
Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: ddtrace/internal/openfeature/_provider.py, ddtrace/internal/openfeature/_experimentation_opt_out.py, ddtrace/internal/openfeature/_native.py, ddtrace/internal/settings/openfeature.py, ddtrace/openfeature/__init__.py, tests/openfeature/test_experimentation_opt_out.py, tests/openfeature/test_provider_fixtures.py

Evaluated head: ac6a36954750d31fb423d42d562200797b8db865

Registry revision: revision:5067899c-c8bf-4370-84a0-3e0cb0c89e02

Pinned skill source: 7e3283c602f6b18a7e3e623eecb52d3a9c091a2e

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: 52ed8cb68675dde1a47addaefe4a9500891c24014abf7dade6dfe38cf836c1d2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants