Skip to content

Add provisional per-subject evaluation telemetry opt-out tests #8080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leoromanovsky wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add provisional per-subject evaluation telemetry opt-out tests #8080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
89aa7bc
test(ffe): cover per-subject evaluation telemetry opt-out
leoromanovsky Oct 18, 2026
35686a7
test(ffe): associate telemetry opt-out coverage with feature 649
leoromanovsky Oct 18, 2026
20ce91e
test(ffe): address telemetry opt-out review feedback
leoromanovsky Oct 18, 2026
7cde918
test(ffe): run telemetry opt-out scenarios in end-to-end workflow
leoromanovsky Oct 21, 2026
3dc19c5
test(ffe): harden telemetry opt-out scenarios after review
leoromanovsky Oct 21, 2026
b81727f
test(ffe): preserve subject identity in opt-out assertions
leoromanovsky Oct 21, 2026
4f19883
test(ffe): use portable subject targeting in opt-out fixture
leoromanovsky Oct 21, 2026
cdbf7ac
test(ffe): cover error and per-evaluation telemetry opt-out
leoromanovsky Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/run-end-to-end.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,12 @@ jobs:
- name: Run FEATURE_FLAGGING_AND_EXPERIMENTATION scenario
if: steps.build.outcome == 'success' && !cancelled() && contains(inputs.scenarios, '"FEATURE_FLAGGING_AND_EXPERIMENTATION"')
run: ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
- name: Run FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT scenario
if: steps.build.outcome == 'success' && !cancelled() && contains(inputs.scenarios, '"FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT"')
run: ./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT
- name: Run FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED scenario
if: steps.build.outcome == 'success' && !cancelled() && contains(inputs.scenarios, '"FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED"')
run: ./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED
- name: Run FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS scenario
if: steps.build.outcome == 'success' && !cancelled() && contains(inputs.scenarios, '"FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS"')
run: ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS
Expand Down
1 change: 1 addition & 0 deletions manifests/c.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ manifest:
tests/ai_guard/: irrelevant (dd-trace-c does not implement AI Guard)
tests/appsec/: irrelevant (dd-trace-c does not implement AppSec or IAST)
tests/cws/test_thread_context_sharing.py::Test_ThreadContextSharing: missing_feature (missing /security/thread_context_sharing endpoint on weblog)
tests/ffe/test_evaluation_telemetry_opt_out.py: incomplete_test_app (The Perl Mojolicious workload does not provide the /ffe endpoint)
tests/integrations/test_dbm.py: incomplete_test_app (The Perl Mojolicious workload does not provide database endpoints)
tests/parametric/otel_env_vars/test_otel_exporter_otlp_compression.py::Test_OTEL_EXPORTER_OTLP_COMPRESSION: missing_feature
tests/parametric/otel_env_vars/test_otel_exporter_otlp_headers.py::Test_OTEL_EXPORTER_OTLP_HEADERS: missing_feature
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (only parametric tests are run for cpp)
tests/ffe/test_flag_eval_evp.py: irrelevant (only parametric tests are run for cpp)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp_httpd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ manifest:
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_agentless_configuration.py: missing_feature (C++ is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (C++ does not implement per-subject evaluation telemetry opt-out)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp_nginx.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ manifest:
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: missing_feature
tests/ffe/test_agentless_configuration.py: missing_feature (C++ is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (C++ does not implement per-subject evaluation telemetry opt-out)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
2 changes: 2 additions & 0 deletions manifests/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -822,6 +822,8 @@ manifest:
weblog: [poc, uds]
- declaration: bug (FFL-2184)
excluded_weblog: [poc, uds]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_evaluation_telemetry_opt_out.py::Test_FFE_Evaluation_Telemetry_Opt_Out_EVP: missing_feature (Requires a release containing dd-trace-dotnet#9351; FFL-2446)
tests/ffe/test_exposure_egress.py: v3.36.0
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct: missing_feature (Not yet implemented)
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct_Shutdown: missing_feature (Not yet implemented)
Expand Down
1 change: 1 addition & 0 deletions manifests/golang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1728,6 +1728,7 @@ manifest:
weblog: [echo, gin, chi, net-http-orchestrion, net-http, uds-echo]
- declaration: bug (FFL-2182)
excluded_weblog: [echo, gin, chi, net-http-orchestrion, net-http, uds-echo]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: v2.6.0-dev # Easy win for chi, echo, gin, net-http, net-http-orchestrion, uds-echo and version 2.5.0
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct: v2.12.0-dev
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct_Shutdown:
Expand Down
1 change: 1 addition & 0 deletions manifests/java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3476,6 +3476,7 @@ manifest:
weblog: [spring-boot]
- declaration: bug (FFL-2184)
excluded_weblog: [spring-boot]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/java_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: missing_feature
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/test_resource_renaming.py: missing_feature
tests/test_rum_injection.py: irrelevant (RUM injection only supported for Java)
Expand Down
1 change: 1 addition & 0 deletions manifests/java_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1871,6 +1871,7 @@ manifest:
weblog: [express5]
- declaration: bug (FFL-2182)
excluded_weblog: [express5]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
Comment thread
leoromanovsky marked this conversation as resolved.
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": incomplete_test_app
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: missing_feature
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/test_resource_renaming.py: missing_feature
tests/test_rum_injection.py: irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/php.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1240,6 +1240,7 @@ manifest:
"*": v1.21.0-dev
laravel11x: incomplete_test_app
symfony7x: incomplete_test_app
tests/ffe/test_evaluation_telemetry_opt_out.py: incomplete_test_app (Deferred until the PHP OpenFeature test-app foundation is available)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": v1.21.0-dev
Expand Down
1 change: 1 addition & 0 deletions manifests/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1499,6 +1499,7 @@ manifest:
flask-poc: v4.14.0
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Down_From_Start: v4.0.0
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: flaky (FFL-1622)
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: v4.2.0-dev
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct:
- weblog_declaration:
Expand Down
1 change: 1 addition & 0 deletions manifests/python_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/python_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/ruby.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2137,6 +2137,7 @@ manifest:
- weblog_declaration:
"*": irrelevant
rails72: v2.23.0-dev
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/ruby_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,5 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking::test_html_template_v2: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: v3.30.0
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: v3.29.0
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
1 change: 1 addition & 0 deletions manifests/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ manifest:
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: missing_feature
tests/ffe/test_agentless_configuration.py: missing_feature (Rust is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
Loading
Loading