Repository navigation
feat(ffe): support subject experimentation opt-out modes - #2698
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits into
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 975c8e4 | Docs | View more details | Give us feedback! |
BenchmarksComparisonBenchmark execution time: 2026-10-26 19:49:43 Comparing candidate commit 975c8e4 in PR branch Found 4 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 975c8e4 | 1793043480 | leoromanovsky/experimentation-opt-out-immediate |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 34.101ms | 34.378ms ± 0.114ms | 34.375ms ± 0.078ms | 34.462ms | 34.558ms | 34.657ms | 34.683ms | 0.90% | -0.019 | -0.226 | 0.33% | 0.008ms | 1 | 200 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 72081.055op/s | 72721.093op/s ± 241.823op/s | 72727.889op/s ± 164.948op/s | 72876.593op/s | 73131.037op/s | 73277.228op/s | 73312.631op/s | 0.80% | 0.037 | -0.228 | 0.33% | 17.099op/s | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 181.357ms | 182.330ms ± 0.384ms | 182.321ms ± 0.237ms | 182.593ms | 182.961ms | 183.139ms | 183.183ms | 0.47% | -0.032 | -0.415 | 0.21% | 0.027ms | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 5459.030op/s | 5484.593op/s ± 11.563op/s | 5484.825op/s ± 7.128op/s | 5491.664op/s | 5504.089op/s | 5509.199op/s | 5513.992op/s | 0.53% | 0.042 | -0.413 | 0.21% | 0.818op/s | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 834.082µs | 835.551µs ± 0.696µs | 835.437µs ± 0.459µs | 836.037µs | 836.826µs | 837.328µs | 838.270µs | 0.34% | 0.572 | 0.639 | 0.08% | 0.049µs | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 119293.297op/s | 119681.560op/s ± 99.588op/s | 119697.768op/s ± 65.792op/s | 119749.355op/s | 119823.066op/s | 119875.671op/s | 119892.226op/s | 0.16% | -0.566 | 0.626 | 0.08% | 7.042op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [34.362ms; 34.394ms] or [-0.046%; +0.046%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [72687.578op/s; 72754.607op/s] or [-0.046%; +0.046%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [182.276ms; 182.383ms] or [-0.029%; +0.029%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [5482.990op/s; 5486.195op/s] or [-0.029%; +0.029%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [835.455µs; 835.648µs] or [-0.012%; +0.012%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [119667.758op/s; 119695.362op/s] or [-0.012%; +0.012%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 975c8e4 | 1793043480 | leoromanovsky/experimentation-opt-out-immediate |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 155.677ns | 159.080ns ± 2.894ns | 158.695ns ± 1.615ns | 160.177ns | 164.386ns | 168.938ns | 170.651ns | 7.53% | 1.421 | 2.645 | 1.81% | 0.205ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [158.679ns; 159.481ns] or [-0.252%; +0.252%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 975c8e4 | 1793043480 | leoromanovsky/experimentation-opt-out-immediate |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 260.065µs | 266.301µs ± 12.818µs | 263.085µs ± 1.319µs | 265.314µs | 278.067µs | 314.771µs | 397.950µs | 51.26% | 6.919 | 59.835 | 4.80% | 0.906µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [264.525µs; 268.077µs] or [-0.667%; +0.667%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 975c8e4 | 1793043480 | leoromanovsky/experimentation-opt-out-immediate |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 30.423ms | 30.629ms ± 0.076ms | 30.625ms ± 0.046ms | 30.672ms | 30.747ms | 30.831ms | 30.951ms | 1.06% | 0.493 | 1.893 | 0.25% | 0.005ms | 1 | 200 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 80771.822op/s | 81622.313op/s ± 201.953op/s | 81631.853op/s ± 121.889op/s | 81742.971op/s | 81942.720op/s | 82041.520op/s | 82175.494op/s | 0.67% | -0.466 | 1.824 | 0.25% | 14.280op/s | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 188.086ms | 189.688ms ± 1.916ms | 188.831ms ± 0.305ms | 189.469ms | 194.191ms | 195.843ms | 196.887ms | 4.27% | 1.845 | 2.202 | 1.01% | 0.135ms | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 5079.060op/s | 5272.330op/s ± 52.283op/s | 5295.750op/s ± 8.571op/s | 5302.476op/s | 5308.476op/s | 5312.431op/s | 5316.727op/s | 0.40% | -1.822 | 2.092 | 0.99% | 3.697op/s | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 682.521µs | 684.675µs ± 1.112µs | 684.659µs ± 0.489µs | 685.104µs | 685.745µs | 686.331µs | 696.605µs | 1.74% | 6.118 | 64.436 | 0.16% | 0.079µs | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 143553.342op/s | 146055.094op/s ± 234.895op/s | 146058.170op/s ± 104.344op/s | 146169.340op/s | 146324.643op/s | 146408.290op/s | 146515.543op/s | 0.31% | -5.978 | 62.511 | 0.16% | 16.610op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [30.619ms; 30.640ms] or [-0.034%; +0.034%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [81594.325op/s; 81650.302op/s] or [-0.034%; +0.034%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [189.423ms; 189.954ms] or [-0.140%; +0.140%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [5265.084op/s; 5279.576op/s] or [-0.137%; +0.137%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [684.521µs; 684.829µs] or [-0.023%; +0.023%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [146022.540op/s; 146087.648op/s] or [-0.022%; +0.022%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 1adb233 | 1793041762 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 35.075ms | 35.317ms ± 0.098ms | 35.316ms ± 0.059ms | 35.366ms | 35.471ms | 35.614ms | 35.692ms | 1.06% | 0.594 | 1.366 | 0.28% | 0.007ms | 1 | 200 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 70043.990op/s | 70787.054op/s ± 195.130op/s | 70788.617op/s ± 118.665op/s | 70913.634op/s | 71065.700op/s | 71193.967op/s | 71275.793op/s | 0.69% | -0.570 | 1.316 | 0.27% | 13.798op/s | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 193.943ms | 194.932ms ± 0.386ms | 194.871ms ± 0.261ms | 195.193ms | 195.583ms | 195.955ms | 196.225ms | 0.69% | 0.417 | 0.089 | 0.20% | 0.027ms | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 5096.184op/s | 5130.020op/s ± 10.149op/s | 5131.602op/s ± 6.862op/s | 5137.627op/s | 5143.671op/s | 5151.207op/s | 5156.164op/s | 0.48% | -0.405 | 0.074 | 0.20% | 0.718op/s | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 851.041µs | 853.426µs ± 0.953µs | 853.384µs ± 0.658µs | 854.049µs | 854.983µs | 855.865µs | 856.128µs | 0.32% | 0.160 | -0.027 | 0.11% | 0.067µs | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 116804.917op/s | 117174.891op/s ± 130.884op/s | 117180.568op/s ± 90.289op/s | 117258.478op/s | 117401.081op/s | 117460.451op/s | 117503.176op/s | 0.28% | -0.154 | -0.031 | 0.11% | 9.255op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [35.304ms; 35.331ms] or [-0.038%; +0.038%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [70760.011op/s; 70814.097op/s] or [-0.038%; +0.038%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [194.878ms; 194.985ms] or [-0.027%; +0.027%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [5128.613op/s; 5131.426op/s] or [-0.027%; +0.027%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [853.294µs; 853.558µs] or [-0.015%; +0.015%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [117156.752op/s; 117193.030op/s] or [-0.015%; +0.015%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 1adb233 | 1793041762 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 158.213ns | 161.662ns ± 2.671ns | 161.278ns ± 1.642ns | 163.042ns | 166.188ns | 171.062ns | 172.930ns | 7.22% | 1.376 | 2.788 | 1.65% | 0.189ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [161.292ns; 162.032ns] or [-0.229%; +0.229%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 1adb233 | 1793041762 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 257.305µs | 262.931µs ± 12.274µs | 259.948µs ± 1.043µs | 261.537µs | 273.363µs | 309.580µs | 391.179µs | 50.48% | 7.247 | 64.389 | 4.66% | 0.868µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [261.230µs; 264.632µs] or [-0.647%; +0.647%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz | 1adb233 | 1793041762 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 31.500ms | 31.705ms ± 0.109ms | 31.694ms ± 0.063ms | 31.754ms | 31.930ms | 31.985ms | 32.162ms | 1.47% | 0.853 | 1.175 | 0.34% | 0.008ms | 1 | 200 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 77732.195op/s | 78852.133op/s ± 270.016op/s | 78878.231op/s ± 156.406op/s | 79034.945op/s | 79241.032op/s | 79319.143op/s | 79364.079op/s | 0.62% | -0.828 | 1.101 | 0.34% | 19.093op/s | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 200.682ms | 201.434ms ± 1.220ms | 201.148ms ± 0.133ms | 201.289ms | 202.275ms | 206.686ms | 207.007ms | 2.91% | 3.907 | 13.918 | 0.60% | 0.086ms | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 4830.765op/s | 4964.581op/s ± 29.377op/s | 4971.459op/s ± 3.294op/s | 4974.385op/s | 4979.155op/s | 4981.582op/s | 4983.003op/s | 0.23% | -3.896 | 13.852 | 0.59% | 2.077op/s | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 694.737µs | 696.615µs ± 0.772µs | 696.543µs ± 0.511µs | 697.137µs | 697.901µs | 698.342µs | 698.561µs | 0.29% | 0.155 | -0.353 | 0.11% | 0.055µs | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 143151.509op/s | 143551.554op/s ± 159.073op/s | 143566.211op/s ± 105.330op/s | 143651.269op/s | 143810.857op/s | 143877.646op/s | 143939.438op/s | 0.26% | -0.150 | -0.354 | 0.11% | 11.248op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [31.690ms; 31.720ms] or [-0.048%; +0.048%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [78814.711op/s; 78889.555op/s] or [-0.047%; +0.047%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [201.265ms; 201.603ms] or [-0.084%; +0.084%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [4960.510op/s; 4968.652op/s] or [-0.082%; +0.082%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [696.508µs; 696.722µs] or [-0.015%; +0.015%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [143529.508op/s; 143573.600op/s] or [-0.015%; +0.015%] | None | None | None |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
ec56508 to
3eb48e6
Compare
yannham
left a comment
There was a problem hiding this comment.
Can't say much about the logic, but LGTM from common component's side
# What does this PR do? Changes to the FFE C example currently inherit general example ownership. Assign `examples/ffi/ffe.c` to `@DataDog/feature-flagging-and-experimentation-sdk`, matching ownership of the FFE crates. # Motivation Follow up on the ownership discussion around [#2698](#2698) so the product team owns its example. # Additional Notes The file-specific rule sits in the existing overrides section, after the general `/examples` rule. Other examples and shared build files retain their owners. # How to test the change? Confirmed that the target file exists, the override is unique and follows the general rule, and `git diff --check` passes. CI's CODEOWNERS validator checks the configuration; no source code changed. Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
danyal002
left a comment
There was a problem hiding this comment.
The exclusion logic matches the shared contract. It runs before targeting and randomization. doLog=false allocations are never excluded, immediate mode skips every logging allocation, and future mode skips activations at or after the cutoff, and missing ones. Disabled mode and the existing C entry points keep their current behavior, and an invalid cutoff returns INVALID_CONTEXT. One item to fix and one question, below.
Review by Claude on Danyal's behalf.
Use one strict parser for cutoffs and activation while preserving lowercase timestamps, nanosecond truncation, and existing allocation scheduling behavior.
0ea359a
into
main
What does this PR do?
Adds
Disabled,Immediate, andFutureExperimentsOnlyevaluation modes usingdd_experimentation_opt_out_at. SDK wrappers resolve provider configuration; the evaluator reads no process settings.PARSE_ERRORonly for future mode with a valid cutoff, including unreachable allocations.Motivation
Customers need to exclude subjects from experimentation immediately or from experiments activated at or after their opt-out time.
Additional Notes
The parser refinement follows the review feedback and the scoped D005/D006/D007 contract. The cross-SDK lowercase disagreement remains separately tracked.
How to test the change?
Native tests cover all 90 shared policy cases, malformed input, unreachable allocations, valid neighboring flags, timestamp boundaries, inactive exclusion calls, and panic containment. Shared fixtures use the merged #52 commit.
DataDog/libdatadog@ead1d3db2aa16e494f23e47b946ff12a7a1be53aDownstream SDK runtime and telemetry delivery were not tested. Shared decisions and scoped adoption.
Parser follow-up at
975c8e4fe: all 50 timestamp cases pass through both production paths across 464 evaluations, including all seven reviewer examples. The new adapter fails against the prior parser. All 110 native tests pass, including the unchanged 90 policy cases and FFI tests; check, formatting, Clippy, docs and shared-model verification pass. The C integration receipt above remains historical evidence for its stated commit, not this follow-up. Parser validation and publication receipt.