feat(standard-tests): lint airbyte_secret marking and secret leaks

[Anatolii Yatsuk (tolik0)]

What

👉 TL;DR: The standard connector tests now fail when a connector's configuration form leaves a credential field unmasked, and when a connection check prints a secret from its config. Neither check needs real credentials, and both run on the path Python connectors use in CI today. Specifically, test_docker_image_build_and_spec and SourceTestSuiteBase.test_spec lint airbyte_secret marking in the emitted spec, and test_docker_image_build_and_check plus both in-process test_check tests assert that no airbyte_secret value from the scenario config appears in the output.

Important

Release after the connector fixes. Do not release a CDK version with this change until these monorepo PRs are merged; together they fix all 12 connectors (13 properties) that the marking lint fails:

• fix(connectors): mark unmasked credential fields as airbyte_secret airbyte#88404 (8 connectors)
• fix(destination-kvdb): mark secret_key as airbyte_secret and fix the image entrypoint airbyte#88413 (destination-kvdb)
• fix(source-convertkit, source-mailjet-mail, source-n8n): mark credential fields as airbyte_secret airbyte#88414 (source-convertkit, source-mailjet-mail, source-n8n)

Part of https://github.com/airbytehq/airbyte-internal-issues/issues/17136 (epic airbytehq/airbyte-internal-issues#17129). This is slice 1 of the issue's landing order: secret marking and leakage. test_config_match_spec and the structural lints follow in separate PRs.

How

The checks hang off the Docker spec and check tests because airbyte-cdk image test, the only standard-test path Python connectors run in monorepo CI, builds with no_verify=True and so never checks the spec at build time. The same assertions run in-process for manifest-only connectors. The marking lint walks the spec's connectionSpecification and matches property names against a secret name list and suffixes. It follows the CDK's runtime secret filter (get_secret_paths/get_secrets), so a flag the lint accepts is one the runtime masks. The leak check runs spec to learn which config paths are secret, then searches every non-CONTROL message from check for those values.

The leak check found a real CDK leak, fixed here: the CDK's own Config validation error CONNECTION_STATUS quoted the rejected value without the secret filter.

Changes

• CDK fix: AirbyteTracedException.as_connection_status_message passes its message through filter_secrets. jsonschema quotes the rejected value, so a secret that failed a pattern, enum or length check was printed in clear in the check output. AirbyteEntrypoint.check now also emits a connector-raised config error as a sanitized TRACE, and Destination.run_cmd registers the config's secrets with the filter before validating, as the source entrypoint already did.
• New private module airbyte_cdk/test/standard_tests/_spec_lint.py:
  • find_secret_marking_errors walks properties, oneOf/anyOf/allOf and items, and reports four cases:
    • a secret-named property that is not marked airbyte_secret: true, on itself or on a oneOf variant;
    • a property marked secret whose type (boolean, object, array, or a const) cannot hold a secret;
    • any non-boolean airbyte_secret value, such as the string "true", which the CDK's secret filter ignores;
    • airbyte_secret: true under anyOf, allOf or a top-level combinator, where the runtime filter cannot resolve it.
  • A pydantic-v2 Optional[str] secret (flag next to anyOf: [{type: string}, {type: null}]) passes: variants are not linted as the property, and a property typed only by its variants is checked against their types.
  • An explicit airbyte_secret: false records a reviewed non-secret, such as a sort key. The failure text says so. This lives in the spec rather than on the suite class because monorepo CI generates the suite class per connector.
  • get_single_spec asserts exactly one SPEC message and returns its spec.
  • find_config_secrets returns every airbyte_secret value in a config with its config path. Unlike get_secrets, it also walks array items and anyOf/allOf variants. Of a oneOf, it walks only the variant the config selects by its const (or single-value enum) discriminators, and all variants when none matches. It yields the values of a secret object, not its field names.
  • find_leaked_secrets / assert_no_secrets_in_output search every non-CONTROL message, numbers included, and stderr on the Docker path. The failure names the config path of each leaked secret and where it was found (message number, type, stream, or stderr line). It contains no text from the output. A hit in a CONNECTION_STATUS that starts with Config validation error: is labeled as built by the CDK, with a hint to upgrade airbyte-cdk or fix the scenario config, since no connector code ran.
• DockerConnectorTestSuite.test_docker_image_build_and_spec now parses the image's spec output, requires exactly one SPEC message, and lints it. Before this PR it only checked that the command did not error.
• DockerConnectorTestSuite.test_docker_image_build_and_check runs the leak check before it raises on errors or asserts the outcome, because those assertions print raw messages. The spec run it needs is cached per image for the session, failures included, so there is no extra docker run per scenario. If spec fails, each check scenario is skipped with a reason; the spec test reports the failure once.
• SourceTestSuiteBase.test_spec uses the same single-spec assertion and lint in-process. ConnectorTestSuiteBase.test_check and SourceTestSuiteBase.test_check run the leak check through a new inspect_output callback on run_test_job, which runs before any of the job's own assertions. The helper is private (_assert_no_secrets_in_output). It runs an extra in-process spec per check scenario, uncached because create_connector takes the scenario; if that spec fails or emits other than one SPEC message, the check test is skipped with a reason.
• EntrypointOutput gains an optional stderr, filled by run_docker_airbyte_command.
• 127 unit tests in unit_tests/test/test_spec_lint.py, including Docker-path tests against a faked docker run and in-process tests against a fake source, plus masking tests in test_entrypoint.py, test_destination.py and test_traced_exception.py.
• cdk-migrations.md: a section on the two new checks and how to fix each failure.

Recommended Review Order

1. airbyte_cdk/test/standard_tests/_spec_lint.py
2. airbyte_cdk/test/standard_tests/docker_base.py
3. airbyte_cdk/test/standard_tests/connector_base.py
4. airbyte_cdk/test/standard_tests/source_base.py
5. airbyte_cdk/test/standard_tests/_job_runner.py
6. airbyte_cdk/test/entrypoint_wrapper.py and airbyte_cdk/utils/docker.py
7. airbyte_cdk/utils/traced_exception.py, airbyte_cdk/entrypoint.py, airbyte_cdk/destinations/destination.py (the masking fix)

The decisions most likely to need discussion are the secret name list and suffixes (SECRET_PROPERTY_NAMES, SECRET_PROPERTY_NAME_SUFFIXES), which decide which connectors go red, and the leak heuristic (_secret_variants, find_leaked_secrets), all in _spec_lint.py.

Fleet impact: 12 community connectors go red on their next PR

Monorepo CI installs airbyte-cdk[dev] unpinned, so these tests reach every connector on the next CDK release without opt-in. The marking lint was dry-run against every connector on airbyte master (3ab60a0e19f, re-checked on a65948f9c42, and on 239eafd614e after the review changes, with the same result). It used the committed spec (manifest.yaml spec block, or the package spec.json/spec.yaml), and the published OSS registry spec for Python connectors whose spec is built in code. Only Python and manifest-only connectors run these tests.

Language	Support level	Connectors	Would fail
manifest-only	certified	42	0
manifest-only	community	459	8
python	certified	11	0
python	community	77	4
manifest-only and python	archived or unknown	20 (11 more archived Python connectors have no spec)	0
java and other	all	71	not run

No certified connector fails. The 12 failures cover 13 properties:

• Python: destination-convex (access_key is marked with the string "true"), destination-kvdb (secret_key), destination-surrealdb (surrealdb_password), destination-typesense (api_key).
• Manifest-only: source-convertkit (api_secret), source-keka (api_key), source-kisi (api_key), source-mailjet-mail (api_key), source-n8n (api_key), source-nexus-datasets (api_key, secret_key), source-shippo (shippo_token), source-track-pms (api_key).

destination-kvdb is disabled in both registries, so it has no published spec, and the first dry run missed it. Its image cannot run spec today either (the console script points at a missing destination_kvdb.run), so it is already red; whoever fixes that will hit this lint next. source-100ms app_access_key is no longer flagged: it sits next to the marked app_secret and is the public half of that key pair.

Each remaining finding is a credential that the platform shows in plain text in the UI and that the CDK does not filter from logs. source-convertkit's api_secret is a hidden legacy field, but it still carries an API key. No connector fails the reverse check (marked secret on a type that cannot hold one), and the camelCase matching adds no failures.

Risks

• Rollout order. Do not release this CDK version until fix(connectors): mark unmasked credential fields as airbyte_secret airbyte#88404, fix(destination-kvdb): mark secret_key as airbyte_secret and fix the image entrypoint airbyte#88413 and fix(source-convertkit, source-mailjet-mail, source-n8n): mark credential fields as airbyte_secret airbyte#88414 are merged. They mark the 13 properties above in the 12 connectors listed; without them each of those connectors fails CI on its next unrelated change. A CDK major version would not shield anyone, since monorepo CI installs the CDK unpinned. #88404 and #88413 are approved; #88414 waits on refreshed GSM test secrets for its three connectors.
• Leak check with real credentials. Run locally from this branch with GSM configs against the published :latest images, through the real test_docker_image_build_and_check and, for manifest-only connectors, the in-process test_check: source-gcs, source-s3, source-klaviyo, source-sentry, source-stripe (certified) and source-recharge. Static API keys, AWS keys and a service-account key only; no rotating refresh tokens. 16 scenarios, 10 of them with a searchable secret: 0 leaks, and every scenario passed except source-s3 config_iam_role.json, whose role assumption cannot run outside CI (its secrets sit outside the spec, so the leak check has nothing to search there). Without credentials, only 27 of the 240 committed check scenarios carry a searchable secret.
• Merge conflicts with sibling PRs.
  • feat(standard-tests): check spec backward compatibility against the published version #1220 (spec backward compatibility) conflicts in docker_base.py: both PRs add a spec-running helper after _assert_check_outcome and both edit the import block. Whichever lands second should keep one helper that is both cached per image and deployment-mode aware.
  • feat(standard-tests): assert discovered catalog properties in test_discover #1217 (discover assertions) touches source_base.py (test_discover only) and merges cleanly there, but both PRs add a ## Upgrading to the next minor release (unreleased) section to the top of cdk-migrations.md; resolve by keeping one heading with both subsections.
  • feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142 and feat(test): persist configs rotated during standard tests to secrets/updated_configurations #1199 (drafts) conflict only where both sides add code: the docker_base.py import block, a module-level helper next to feat(test): persist configs rotated during standard tests to secrets/updated_configurations #1199's _run_docker_airbyte_command_with_config_updates, and one string literal in connector_base.py (feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142). Each resolves by keeping both sides.
  • Suggested landing order: feat(standard-tests): assert discovered catalog properties in test_discover #1217, then this PR, then feat(standard-tests): check spec backward compatibility against the published version #1220.

Test plan

• pytest unit_tests/test/test_spec_lint.py: 127 passed.
• pytest unit_tests/test unit_tests/test_entrypoint.py unit_tests/destinations unit_tests/utils: 607 passed. The full unit_tests run passes apart from two unstructured-file scenarios that need a local NLTK download and a timing-flaky JWT test that passes on rerun.
• Mutation check: each of 24 mutations of the new code fails at least one test, including the four the review found uncaught (removing the guarded raise_if_errors, raising it unconditionally, removing the in-process assert_spec_is_valid, dropping stderr= in run_docker_airbyte_command) and removing each encoded search form. The source_pokeapi_w_components_py standard suite: 7 passed, 1 skipped (the read test, skipped by --read-from-streams=false), including the real Docker build+spec and build+check tests.
• ruff format --check, ruff check and mypy on the changed files: clean.
• airbyte-cdk image test --no-creds from this branch on real Python connectors from airbyte master:
  • source-recharge and source-okta (images built locally): spec and check[invalid_config] pass. Both invalid configs carry a searchable fake secret, so the leak check really ran.
  • destination-typesense with --image airbyte/destination-typesense:latest: test_docker_image_build_and_spec fails with /properties/api_key` looks like a secret but is not marked `airbyte_secret: true`.. The image built from master cannot run spec at all (No module named 'destination_typesense.run', a known pre-existing break), so the published image was used.
• Leak dry run: check against the published :latest image for all 27 searchable credential-free scenarios (24 manifest-only, 7 of them certified; 3 Python). All 27 returned FAILED as their configs expect, with 0 leaks and 0 errors.

Follow-ups

Remaining slices of airbytehq/airbyte-internal-issues#17136, in the issue's landing order:

• test_config_match_spec: validate every committed config against the connector's own spec.
• Structural lints: test_oneof_usage, test_enum_usage, test_forbidden_complex_types, test_array_type, test_object_not_empty, test_property_type_is_not_array, test_defined_refs_exist_in_json_spec_file, test_additional_properties_is_true.
• A decision per presentational lint (test_date_pattern, test_date_format, test_display_type, test_duplicate_order, test_nested_group): harness, QA checks, or drop.
• OAuth (test_oauth_flow_parameters, test_oauth_is_default_method) and image metadata (test_image_labels, test_image_environment_variables).
• Not in the issue, but cheap once this lands: the same leak check on discover and read output.

Design notes: secret names and the leak heuristic

Show/Hide Content

Opt-out. When a secret-looking name holds no secret, set airbyte_secret: false on the property. The UI treats it as unmarked, as before, and the explicit value makes the decision visible in review.

Where the flag may sit. get_secret_paths drops only properties and oneOf from the schema path of a flag and looks the rest up in the config, starting at the top-level properties. So the flag works on the property and on a oneOf variant of it, and does nothing under anyOf, allOf or a top-level combinator; the lint follows exactly that. A flag inside array items is not masked by the runtime either, but the lint does not report it, since an array of credentials has no other place to mark them. No connector in the fleet has a flag under anyOf or allOf today.

Secret names. The exact-name list comes from CAT's test_secret_is_properly_marked, with these changes.

• tenant_id, app_id and appid are removed. They identify an account or an app rather than authenticate it, and flagging them would push connectors to hide values that are not secret.
• Suffix matching is added (_token, _secret, password, api_key, apikey, secret_key, private_key, access_key). CAT matched exact names only, so it never flagged api_key, the most common secret property name in the fleet. Unmarked api_key accounts for 7 of the 13 findings.
• An *access_key next to a property named secret, *_secret or *secret_key is exempt. It is the public half of a key pair, like an AWS access key ID. A lone access_key is still flagged, because many APIs take it as their only credential.
• Names are compared in snake case: clientSecret, client-secret and Client_Secret all match client_secret.

Only properties with a type are checked. An object such as a credentials group cannot hold a secret itself, so only the fields inside it are checked.

Leak check scope. CAT's test_secret_never_in_the_output was an empty stub, so this is new coverage, not a port. The CDK log formatter and traced exceptions already mask registered secrets. A CONNECTION_STATUS message that the connector builds itself, raw stdout and stderr go through unfiltered, and that is the realistic leak class. On the Docker path all three are searched. In-process, only yielded messages and root-logger records are captured, so a raw print() is visible only on the Docker path. CONTROL messages are excluded because a connector config update carries the full config, secrets included, by design. Secrets under array items and anyOf/allOf are searched too, although the runtime log filter (get_secrets) still skips them; fixing that filter is a separate change. For a oneOf, only the variant the config selects is searched, so a value that is secret only in another variant is not reported.

Heuristic. Only secret values of at least 8 characters that contain a digit are searched for. Shorter or word-like values (123, client_secret, invalid_api_key) show up in ordinary output by coincidence: they would make the test flaky without catching anything. Most real credentials pass this filter; a purely alphabetic key is skipped. Each secret is searched in its literal, JSON-escaped, repr-escaped and URL-encoded forms, and line by line for a multi-line secret such as a PEM key.

Failure text. The failure never quotes the output. It names the config path of each leaked secret (built from spec property names and array indices only) and where it was found, for example: /credentials/client_secret in message #4 (CONNECTION_STATUS), or /api_key in stderr line 12. A stream name is shown only when it overlaps no secret. Masking an excerpt cannot be made safe, because a second secret can appear in any encoding; with no excerpt, the test output cannot print a secret in any form. The leak check also runs before the outcome assertions, which print raw messages, so it sees every outcome.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/spec-lint#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/spec-lint

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 270 tests  +132   5 258 ✅ +132   11m 29s ⏱️ + 4m 4s
    1 suites ±  0      12 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 4d000bf. ± Comparison against base commit d3a2d2b.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 273 tests   5 261 ✅  12m 51s ⏱️
    1 suites     12 💤
    1 files        0 ❌

Results for commit 4d000bf.

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Verdict: fix before merge — 0 P0 · 4 P1 · 6 P2 · 3 P3 (all inline).

Breaking change: BREAKING under criterion A.7 only — two new mandatory standard tests (spec marking lint, check secret-leak check) that every Python and manifest-only connector picks up automatically on the next release, because monorepo CI installs airbyte-cdk[dev] unpinned (poe-tasks/*-connector-tasks.toml). No runtime, API, manifest or state break; every signature change is additive and keyword-only. The call-out is in the body only (title is plain feat:, while sibling #1218 uses feat! for the same kind of gate). The rollout depends on three open monorepo PRs that this PR does not link: airbytehq/airbyte#88404, airbytehq/airbyte#88413, airbytehq/airbyte#88414 — together they cover all 12 connectors / 13 properties. Please link them under Risks and do not release this version before they merge (inline P1).

Done well

• Fleet homework before review: the marking lint was dry-run against every connector; I re-ran the head _spec_lint.py over all committed specs plus the OSS registry and got exactly the same 12 connectors / 13 properties, 0 certified.
• The failure text cannot leak: pointers from spec property names and indices only, stream name dropped on any overlap, and the leak check runs before the outcome assertions that print raw messages — with tests proving the ordering on both paths.
• Additive, keyword-only API (EntrypointOutput(stderr=), run_test_job(inspect_output=)); 90 hermetic unit tests with no CI-hiding markers; the Docker spec cache really does collapse the spec test and every check scenario into one docker run when spec succeeds.

Rollout / risk notes

• Check: destination-motherduck is red here but identically red on #1217, #1218 and #1220 (MotherDuck token rejected at _assert_check_outcome, after the new leak check passed) — baseline, not this diff. Skipped Check: source-google-drive did not run.
• The leak check has only been dry-run with credential-free configs (27 searchable scenarios). With GSM credentials every secret becomes searchable; a /prerelease + airbyte-cdk image test with creds on a few certified connectors that build their own CONNECTION_STATUS would tell us before the fleet does.
• Conflicts: #1220 conflicts in docker_base.py (both add a spec-running helper after _assert_check_outcome and both edit the import block); #1217 touches source_base.py but merges cleanly. Neither is in the body's conflict note (#1142/#1199 are).