Repository navigation
test: add future experimentation opt-out capability cases - #52
Merged
Merged
Conversation
This was referenced Oct 21, 2026
leoromanovsky
commented
Oct 21, 2026
Comment on lines
-71
to
+80
| "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" } | ||
| "experimentationOptOutImmediate": { "value": 0, "reason": "DEFAULT" }, | ||
| "experimentationOptOutFutureExperimentsOnly": { "value": 2, "reason": "SPLIT" } |
Collaborator
Author
There was a problem hiding this comment.
☝️ canonical example
This was referenced Oct 21, 2026
danyal002
reviewed
Oct 22, 2026
There was a problem hiding this comment.
The cases are precise. Activation equal to the cutoff is excluded, one microsecond either way flips the result, offsets compare as instants, and missing and null activations are tested separately. The downstream runs show that every current SDK accepts the new field.
Question for the compiler work: the activation instant must not change after an allocation starts randomizing. If it moves, subjects move into or out of experiments after the fact. Holdout allocations also need a definition: is their activation the holdout start, or the covered allocation's activation?
Review by Claude on Danyal's behalf.
danyal002
reviewed
Oct 22, 2026
danyal002
reviewed
Oct 22, 2026
hhan2024
reviewed
Oct 22, 2026
3 of 5 tasks
Keep doLog=false allocations eligible when activation equals or follows the cutoff. Environment: Datadog workspace
Document contract ownership, fixture integration, and mismatch handling for new contributors. Add an ownership diagram and executable examples. Environment: Datadog workspace
danyal002
approved these changes
Oct 22, 2026
leoromanovsky
added a commit
to DataDog/dd-trace-go
that referenced
this pull request
Oct 22, 2026
### What does this PR do?
Adds `ExperimentationOptOutFutureExperimentsOnly` and the environment
value `future_experiments_only`.
For a subject with `dd_experimentation_opt_out_at`, compare the cutoff
with each exposure-logging allocation's `randomizationActivatedAt`:
| Allocation activation | Result |
| --- | --- |
| Before the cutoff | Continue normal evaluation and exposure logging. |
| At or after the cutoff, or unknown | Skip the allocation and continue
ordinary delivery. |
Ordinary allocations remain eligible. Missing subject input preserves
normal evaluation. Malformed subject input returns `INVALID_CONTEXT`, as
in immediate mode. No new telemetry or subject history is added.
```sh
DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE=future_experiments_only
```
```go
// ProviderConfig{} uses the environment setting.
provider, err := ddflags.NewDatadogProvider(ddflags.ProviderConfig{})
if err != nil {
return err
}
if err := of.SetNamedProviderAndWait("flags", provider); err != nil {
return err
}
client := of.NewClient("flags")
attributes := map[string]any{}
if subject.ExperimentationOptOutAt != nil {
attributes["dd_experimentation_opt_out_at"] =
subject.ExperimentationOptOutAt.UTC().Format(time.RFC3339Nano)
}
enabled, err := client.BooleanValue(ctx, "new-route-planner", false,
of.NewEvaluationContext(subject.ID, attributes))
```
An explicit `ProviderConfig{ExperimentationOptOutMode:
ddflags.ExperimentationOptOutFutureExperimentsOnly}` overrides the
environment setting.
### Motivation
**Review scope:** [Downscoped in
e282fff](e282fff)
to OpenFeature implementation, tests, and documentation. The benchmark
CI configuration is identical to #5666.
`BenchmarkExperimentationExclusion` and
`BenchmarkBooleanEvaluationExclusion` remain available locally;
registering them in CI and assessing performance belong to #5666 after
this child is folded in. That follow-up remains open; existing approvals
are not approval of the future CI changes.
Customers can stop new experiment participation while continuing older
experiments. The cutoff describes allocation activation, not previous
subject enrollment. An older experiment can still assign a subject for
the first time.
**Stack:** #5666 supplies immediate mode and the configuration API. This
PR adds the activation comparison. [Shared fixtures
#52](DataDog/ffe-system-test-data#52) defines 42
capability checks; update the pin after that PR merges.
Enablement depends on the backend emitting stable activation metadata
and retaining ordinary fallbacks. This PR does not deploy those changes.
Allocations without metadata are skipped for opted-out subjects.
### Test coverage
Shared fixtures cover cutoff boundaries, timezone offsets, fractional
seconds, invalid input, missing metadata, and fallback delivery. Local
tests cover malformed activation metadata and valid-neighbor isolation
in all modes, refresh, exposure deduplication, telemetry independence,
and concurrent randomized subjects. Baseline and immediate-mode behavior
remain covered. A benchmark covers all three modes.
[System tests #8135](DataDog/system-tests#8135),
stacked on system-tests #8080, add Remote Config, holdout assignment,
exposure, and telemetry checks across all three modes. Release gating
remains provisional until supported SDK releases are known.
### Validation
| Tested commit | Validation | Result |
| --- | --- | --- |
|
[e282fff](e282fff)
| Local OpenFeature package and CI-selection tests after syncing the
base and removing the child CI delta. | Both passed. Remote CI is
tracked separately. |
|
[febcc18](febcc18)
| System tests: disabled, immediate, and future-only modes; cutoff
boundaries, Remote Config refresh, holdout assignments, exposure
deduplication, and independent evaluation telemetry. Linux/amd64;
[system-tests
6418472](DataDog/system-tests@6418472).
| 12 passed; 0 failed; 0 skipped. |
### Reviewer's Checklist
- [ ] Changed code has unit tests for its functionality at or near 100%
coverage.
- [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering
this feature have been added and enabled with the va.b.c-dev version
tag.
- [ ] There is a benchmark for any new code, or changes to existing
code.
- [ ] If this interacts with the agent in a new way, a system test has
been added.
- [ ] New code is free of linting errors. You can check this by running
`make lint` locally.
- [ ] New code doesn't break existing tests. You can check this by
running `make test` locally.
- [ ] Add an appropriate team label so this PR gets put in the right
place for the release notes.
- [ ] All generated files are up to date. You can check this by running
`make generate` locally.
- [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed
by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to
date by running `make fix-modules` locally.
Unsure? Have a question? Request a review!
9 tasks
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
SDKs need one shared contract for excluding subjects from future experiments while continuing older experiments.
Changes
experimentationOptOutFutureExperimentsOnlycapability expectations. Reuse the existing flags and subject timestamp.randomizationActivatedAtwithdd_experimentation_opt_out_at. Skip exposure-logging allocations activated at or after the cutoff. Skip allocations with unknown activation.Decisions
Keep ordinary allocations eligible. Preserve existing scheduling, targeting, hashing, baseline results, and immediate-mode results. The cutoff describes allocation activation, not prior subject enrollment.
Consumers:
b28876200152f00c7d5eb04616f721afa5955c1drevision and executes all 42 future-mode cases.Merge these fixtures first. Update each consumer to the merged fixture commit before landing it.
Checklist
ufc-config.json.