Check the mocked backend network aliases on GitLab by cbeauchesne · Pull Request #8212 · DataDog/system-tests

cbeauchesne · 2026-10-28T18:09:45Z

Motivation

Stacked on #8211. There, every scenario using the mocked backend reaches it through Docker network aliases (mocked-backend.invalid and the intake hostnames derived from it). A comment in MockedBackendContainer warns that Docker rejects aliases on the default bridge network, which GitLab uses (_NETWORK_NAME = "bridge" when GITLAB_CI is set). This PR checks it.

Changes

.gitlab-ci.yml gets a new system_tests_python_mocked_backend job. On every PR (non-default branches), whatever files changed, it runs these scenarios on python flask-poc (prod tracer) in a child pipeline:

APM_TRACING_E2E_SINGLE_SPAN: the agent can only reach the mocked backend through the mocked-backend.invalid alias.
FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS, _DIRECT and _SERVERLESS: the weblog (and serverless-init) reach it through the alias and the intake hostnames.

It works the same way as system_tests_php_thread_context_sharing: it triggers its own child pipeline from utils/ci/gitlab/main.yml. The main e2e include is left unchanged, so every other GitLab run is unaffected.

Note: locally, the _DIRECT scenario already failed with the python prod tracer (4.16.1), also on main: the released tracer does not deliver direct EVP events. A failure there on the EVP waits is not an alias problem; a container that does not start, or a name that does not resolve, is.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

🤖 Generated with Claude Code

The direct and serverless agentless FFE scenarios captured exposures and flag evaluations with the proxy (datadog_direct / datadog_sidecar interfaces). They now go to the mocked backend, as the rest of the agentless traffic, and are read with interfaces.backend_v2, the sender being identified by its API key. - the end-to-end mocked backend always serves HTTPS, also under the intake hostnames derived from DD_SITE=mock-intake.invalid. The agent and the agentless weblogs trust its certificate (mitmproxy CA) - the egress scenarios no longer use the proxy; serverless-init sends to the mocked backend - the mocked backend records when a request started, and answers 202 on the exposures and flag evaluation intake paths - remove the datadog_direct and datadog_sidecar interfaces and their proxy ports Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Temporary: only run the scenarios using the mocked backend, on the libraries running the agentless FFE egress tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

wait_for gets the same with_api_key parameter as get_data, replacing the _evp_capture_matcher helper of the agentless FFE scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Revert the with_api_key parameter of interfaces.backend_v2.wait_for: each matcher of the agentless FFE scenario checks the API key itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

weblog_api_key and sidecar_api_key tell which sender a mocked backend request comes from, instead of dd_api_key and the serverless-init container environment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The key is deterministic with the mocked backend: no reason to skip the check in replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The scenario defines the serverless-init API key, and gives it to the container it creates, instead of a constant in containers.py and a container built by the weblog infra. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The agent, serverless-init and the weblogs reach the mocked backend on https://mocked-backend.invalid, and DD_SITE is mocked-backend.invalid: the intake hostnames derived from it are network aliases of the mocked backend. One constant, MOCKED_BACKEND_DOMAIN, replaces MOCK_INTAKE_SITE and the container name in the URLs. The certificate is regenerated for these hostnames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Checks that the mocked backend network aliases (mocked-backend.invalid) work on GitLab's Docker network, where the agent can only reach the mocked backend through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-28T18:10:36Z

CODEOWNERS have been resolved as:

.gitlab-ci.yml                                                          @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-reviewers
tests/test_the_test/test_gitlab_library_weblog_scenarios.py             @DataDog/system-tests-reviewers
utils/ci/gitlab/build_pipeline.py                                       @DataDog/system-tests-core
utils/ci/gitlab/main.yml                                                @DataDog/system-tests-core
utils/scripts/compute_libraries_and_scenarios.py                        @DataDog/system-tests-core

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The CI is green with the hard-coded set: back to the computed libraries and scenarios. This reverts commit b746bd5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

datadog-prod-us1-5 · 2026-10-28T18:23:30Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 707fb61 | Docs | View more details | Give us feedback!}

It was stopped after the mocked backend: what it flushed while shutting down was lost. It now stops right after the weblog it relays, while the mocked backend still receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…-backend-gitlab

…backend-gitlab # Conflicts: # tests/test_the_test/test_mock_ffe_agentless_backend.py # utils/_context/_scenarios/agentless_endtoend.py

chatgpt-codex-connector · 2026-10-29T09:58:42Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T13:04:21.350578Z	`707fb61`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

GitLab runs containers on the default bridge network, so check there that the mocked backend network aliases work. On GitLab PR pipelines, python is now activated with a weblog map: flask-poc with APM_TRACING_E2E_SINGLE_SPAN and the agentless FFE scenarios. The map is filtered by the scenarios selected from the diff, using libraries_and_scenarios_rules.yml, and passed to the e2e pipeline as library_weblog_scenarios. library_weblog_scenarios can now be combined with libraries, scenarios, scenarios_groups and weblogs, which apply to the other libraries, so the selection of the other libraries is unchanged. The GitHub selection and the GitLab main selection are unchanged too.

cbeauchesne and others added 11 commits October 28, 2026 17:31

[DO NOT MERGE] Hard-code the CI libraries and scenarios

b746bd5

Temporary: only run the scenarios using the mocked backend, on the libraries running the agentless FFE egress tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Filter interfaces.backend_v2.wait_for by API key

02d7d2f

wait_for gets the same with_api_key parameter as get_data, replacing the _evp_capture_matcher helper of the agentless FFE scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Check the API key inside the EVP capture matchers

d3ff702

Revert the with_api_key parameter of interfaces.backend_v2.wait_for: each matcher of the agentless FFE scenario checks the API key itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use interfaces.backend_v2 directly in the agentless FFE scenario

060e02e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Name the API keys of the agentless FFE senders

ce9d322

weblog_api_key and sidecar_api_key tell which sender a mocked backend request comes from, instead of dd_api_key and the serverless-init container environment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Check the weblog API key in replay mode too

61d478d

The key is deterministic with the mocked backend: no reason to skip the check in replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use serverless_init_container in the agentless topology test

d00d2d3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

[DO NOT MERGE] Run APM_TRACING_E2E_SINGLE_SPAN on GitLab

362e2f0

Checks that the mocked backend network aliases (mocked-backend.invalid) work on GitLab's Docker network, where the agent can only reach the mocked backend through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne and others added 2 commits October 28, 2026 19:12

[DO NOT MERGE] Also run the agentless FFE scenarios on GitLab

015670e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Revert "[DO NOT MERGE] Hard-code the CI libraries and scenarios"

7d9e099

The CI is green with the hard-coded set: back to the computed libraries and scenarios. This reverts commit b746bd5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cbeauchesne and others added 3 commits October 28, 2026 19:26

Stop serverless-init right after the weblog

d4cefd0

It was stopped after the mocked backend: what it flushed while shutting down was lost. It now stops right after the weblog it relays, while the mocked backend still receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tautologic assertion by design.

fc83ad4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'ffe-agentless-mocked-backend' into ffe-agentless-mocked…

3f5d145

…-backend-gitlab

cbeauchesne changed the base branch from ffe-agentless-mocked-backend to main October 29, 2026 09:53

Merge remote-tracking branch 'origin/main' into ffe-agentless-mocked-…

b998905

…backend-gitlab # Conflicts: # tests/test_the_test/test_mock_ffe_agentless_backend.py # utils/_context/_scenarios/agentless_endtoend.py

cbeauchesne changed the title ~~[DO NOT MERGE] Check the mocked backend network aliases on GitLab~~ Check the mocked backend network aliases on GitLab Oct 29, 2026

cbeauchesne marked this pull request as ready for review October 29, 2026 09:58

cbeauchesne requested a review from a team as a code owner October 29, 2026 09:58

Merge branch 'main' into ffe-agentless-mocked-backend-gitlab

ae4c6ef

cbeauchesne marked this pull request as draft October 30, 2026 10:37

nccatoni force-pushed the ffe-agentless-mocked-backend-gitlab branch 2 times, most recently from cad91a2 to 707fb61 Compare October 30, 2026 12:53

cbeauchesne marked this pull request as ready for review October 30, 2026 13:00

cbeauchesne requested a review from a team as a code owner October 30, 2026 13:00

nccatoni approved these changes Oct 30, 2026

View reviewed changes

cbeauchesne enabled auto-merge (squash) October 30, 2026 13:13

cbeauchesne merged commit 5b73d5d into main Oct 30, 2026
1271 of 1650 checks passed

cbeauchesne deleted the ffe-agentless-mocked-backend-gitlab branch October 30, 2026 14:37

Navigation Menu

Repository navigation

Check the mocked backend network aliases on GitLab - #8212

Check the mocked backend network aliases on GitLab#8212
cbeauchesne merged 19 commits into
mainfrom
ffe-agentless-mocked-backend-gitlab

cbeauchesne commented Oct 28, 2026 •

edited by nccatoni

Loading

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-5 Bot commented Oct 28, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cbeauchesne commented Oct 28, 2026 • edited by nccatoni Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-5 Bot commented Oct 28, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cbeauchesne commented Oct 28, 2026 •

edited by nccatoni

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Oct 28, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

chatgpt-codex-connector Bot commented Oct 29, 2026 •

edited

Loading