Send agentless FFE events to the mocked backend by cbeauchesne · Pull Request #8211 · DataDog/system-tests

cbeauchesne · 2026-10-28T16:32:09Z

Motivation

Second step after #8186: remove the datadog_sidecar and datadog_direct interfaces. Agentless FFE exposures and flag evaluations go to the mocked backend, like the rest of the agentless traffic, and the sender is identified by its API key.

Changes

The end-to-end mocked backend always serves HTTPS (MockedBackendContainer(tls=True)) on a single domain, mocked-backend.invalid (MOCKED_BACKEND_DOMAIN): the agent, serverless-init and the weblogs reach it on https://mocked-backend.invalid, and it is the DD_SITE of the agentless senders, the intake hostnames derived from it being its network aliases. The certificate is regenerated for these hostnames. Its certificate is signed by the mitmproxy CA, already trusted by the agent and by the agentless weblogs (CA bundle mount, NODE_EXTRA_CA_CERTS), which now get it in every agentless scenario since the configuration is fetched over HTTPS.
Egress scenarios (_DIRECT, _SERVERLESS) no longer use the proxy. serverless-init owns its API key from the scenario (sidecar_api_key), and sends to the mocked backend.
Captures are read from interfaces.backend_v2, filtered by API key: the SDK key for direct, the serverless-init key for sidecar. "Not sent by the other route" becomes "not sent with the other API key". The intake host is asserted from the Host header.
Mocked backend: records timestamp_start (used by the shutdown-flush evidence) and answers 202 on /api/v2/exposures and /api/v2/flagevaluation.
Removed: datadog_direct / datadog_sidecar interfaces, proxy ports 8201/8202 and their routing.

Known caveat, to check: Docker rejects network aliases on GitLab's default bridge network, and every scenario using the mocked backend now relies on them (including the agent ones). If they run on GitLab, the senders need extra_hosts instead.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

🤖 Generated with Claude Code

The direct and serverless agentless FFE scenarios captured exposures and flag evaluations with the proxy (datadog_direct / datadog_sidecar interfaces). They now go to the mocked backend, as the rest of the agentless traffic, and are read with interfaces.backend_v2, the sender being identified by its API key. - the end-to-end mocked backend always serves HTTPS, also under the intake hostnames derived from DD_SITE=mock-intake.invalid. The agent and the agentless weblogs trust its certificate (mitmproxy CA) - the egress scenarios no longer use the proxy; serverless-init sends to the mocked backend - the mocked backend records when a request started, and answers 202 on the exposures and flag evaluation intake paths - remove the datadog_direct and datadog_sidecar interfaces and their proxy ports Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Temporary: only run the scenarios using the mocked backend, on the libraries running the agentless FFE egress tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-28T16:32:43Z

CODEOWNERS have been resolved as:

utils/proxy/.mitmproxy/mitmproxy-ca-truststore.p12                      @DataDog/system-tests-core
.cursor/rules/system-tests-overview.mdc                                 @DataDog/system-tests-core
tests/ffe/test_exposure_egress.py                                       @DataDog/feature-flagging-and-experimentation-sdk
tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk
tests/ffe/utils/evp.py                                                  @DataDog/feature-flagging-and-experimentation-sdk
tests/test_the_test/test_docker_scenario.py                             @DataDog/system-tests-reviewers
tests/test_the_test/test_ffe_tests.py                                   @DataDog/feature-flagging-and-experimentation-sdk
tests/test_the_test/test_group_rules.py                                 @DataDog/system-tests-reviewers
tests/test_the_test/test_mock_backend_v2.py                             @DataDog/system-tests-reviewers
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/feature-flagging-and-experimentation-sdk
utils/_context/_scenarios/_core/endtoend.py                             @DataDog/system-tests-core
utils/_context/_scenarios/agentless_endtoend.py                         @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/interfaces/__init__.py                                            @DataDog/system-tests-core
utils/interfaces/_backend_v2.py                                         @DataDog/system-tests-core
utils/mocked_backend/_backend_ffe.py                                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/mocked_backend/_request_handler.py                                @DataDog/system-tests-core
utils/mocked_backend/tls/README.md                                      @DataDog/system-tests-core
utils/mocked_backend/tls/mocked-backend.pem                             @DataDog/system-tests-core
utils/proxy/_deserializer.py                                            @DataDog/system-tests-core
utils/proxy/core.py                                                     @DataDog/system-tests-core
utils/proxy/ports.py                                                    @DataDog/system-tests-core

wait_for gets the same with_api_key parameter as get_data, replacing the _evp_capture_matcher helper of the agentless FFE scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Revert the with_api_key parameter of interfaces.backend_v2.wait_for: each matcher of the agentless FFE scenario checks the API key itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

weblog_api_key and sidecar_api_key tell which sender a mocked backend request comes from, instead of dd_api_key and the serverless-init container environment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The key is deterministic with the mocked backend: no reason to skip the check in replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The scenario defines the serverless-init API key, and gives it to the container it creates, instead of a constant in containers.py and a container built by the weblog infra. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The agent, serverless-init and the weblogs reach the mocked backend on https://mocked-backend.invalid, and DD_SITE is mocked-backend.invalid: the intake hostnames derived from it are network aliases of the mocked backend. One constant, MOCKED_BACKEND_DOMAIN, replaces MOCK_INTAKE_SITE and the container name in the URLs. The certificate is regenerated for these hostnames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The CI is green with the hard-coded set: back to the computed libraries and scenarios. This reverts commit b746bd5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-10-28T18:17:38Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-28T18:23:51.190898Z	`7d9e099`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7d9e099253

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

It was stopped after the mocked backend: what it flushed while shutting down was lost. It now stops right after the weblog it relays, while the mocked backend still receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

datadog-prod-us1-4 · 2026-10-28T19:51:27Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

Testing the test | System Tests (ruby, dev) / parametric / parametric (1) — 🔄 Retry may pass, looks flaky

View more details · View in GitHub Actions

Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c9155a5 | Docs | View more details | Give us feedback!}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The JVM does not use the system CA bundle: with the mocked backend now serving HTTPS, Java weblogs failed to fetch the agentless configuration (PKIX path building failed). They get the mitmproxy CA in a PKCS12 trust store (built with keytool), through JDK_JAVA_OPTIONS, which does not replace the JAVA_TOOL_OPTIONS set by the weblogs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Golden rule, with the TLS mocked backend as example: make clients trust its certificate instead of falling back to plain HTTP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne and others added 2 commits October 28, 2026 17:31

[DO NOT MERGE] Hard-code the CI libraries and scenarios

b746bd5

Temporary: only run the scenarios using the mocked backend, on the libraries running the agentless FFE egress tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne and others added 8 commits October 28, 2026 17:54

Filter interfaces.backend_v2.wait_for by API key

02d7d2f

wait_for gets the same with_api_key parameter as get_data, replacing the _evp_capture_matcher helper of the agentless FFE scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Check the API key inside the EVP capture matchers

d3ff702

Revert the with_api_key parameter of interfaces.backend_v2.wait_for: each matcher of the agentless FFE scenario checks the API key itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use interfaces.backend_v2 directly in the agentless FFE scenario

060e02e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Name the API keys of the agentless FFE senders

ce9d322

weblog_api_key and sidecar_api_key tell which sender a mocked backend request comes from, instead of dd_api_key and the serverless-init container environment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Check the weblog API key in replay mode too

61d478d

The key is deterministic with the mocked backend: no reason to skip the check in replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use serverless_init_container in the agentless topology test

d00d2d3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne mentioned this pull request Oct 28, 2026

Check the mocked backend network aliases on GitLab #8212

Open

Revert "[DO NOT MERGE] Hard-code the CI libraries and scenarios"

7d9e099

The CI is green with the hard-coded set: back to the computed libraries and scenarios. This reverts commit b746bd5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne marked this pull request as ready for review October 28, 2026 18:17

cbeauchesne requested review from a team as code owners October 28, 2026 18:17

chatgpt-codex-connector Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread utils/_context/_scenarios/agentless_endtoend.py Outdated

Stop serverless-init right after the weblog

d4cefd0

It was stopped after the mocked backend: what it flushed while shutting down was lost. It now stops right after the weblog it relays, while the mocked backend still receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pawelchcki approved these changes Oct 28, 2026

View reviewed changes

mark-caii approved these changes Oct 28, 2026

View reviewed changes

cbeauchesne and others added 3 commits October 29, 2026 00:13

tautologic assertion by design.

fc83ad4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

AI rules: stay as close as possible to customer infrastructure

c9155a5

Golden rule, with the TLS mocked backend as example: make clients trust its certificate instead of falling back to plain HTTP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne merged commit 8338c2a into main Oct 29, 2026
2253 of 2255 checks passed

cbeauchesne deleted the ffe-agentless-mocked-backend branch October 29, 2026 09:55

Navigation Menu

Repository navigation

Send agentless FFE events to the mocked backend - #8211

Send agentless FFE events to the mocked backend#8211
cbeauchesne merged 15 commits into
mainfrom
ffe-agentless-mocked-backend

cbeauchesne commented Oct 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Oct 28, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cbeauchesne commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented Oct 28, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cbeauchesne commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Oct 28, 2026 •

edited by datadog-prod-us1-6 Bot

Loading