feat(standard-tests): assert discovered catalog properties in test_discover

[Anatolii Yatsuk (tolik0)]

What

SourceTestSuiteBase.test_discover ran discover and asserted only that it raised no errors. This PR adds back the TestDiscovery checks from the retired Connector Acceptance Tests (CAT), so the standard tests check the catalog that discover returns.

The checks are in a new module, airbyte_cdk/test/standard_tests/_catalog_checks.py. Each check takes the AirbyteCatalog and returns a list of problems. Each problem names the stream and the field or JSON pointer. One run reports every problem, so fixing one problem does not reveal the next on a later run.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17134 (part of epic airbytehq/airbyte-internal-issues#17129).

Staged rollout: enforced and advisory checks

Monorepo CI installs the CDK CLI unpinned. Any new assertion in test_discover therefore runs for every connector on the next CDK release. To roll out safely, the checks are split into two groups:

• Enforced checks fail the test. Each has zero failing connectors in the fleet dry run below.
• Advisory checks emit a DiscoveredCatalogWarning and do not fail the test. Connectors in the fleet still violate them.

To make the advisory checks fail locally or in a connector's own CI, use the standard pytest filter:

pytest -W error::airbyte_cdk.test.standard_tests.DiscoveredCatalogWarning

Check	CAT origin	Status	Manifest-only connectors failing
catalog has streams	test_discover	enforced	0
stream names are unique (namespace + name)	test_discover	enforced	0
streams declare sync modes	test_streams_has_sync_modes	enforced	0
refs are resolved	test_defined_refs_exist_in_schema	enforced	0
no unsupported keywords (allOf, not)	test_defined_keyword_exist_in_schema	enforced	0
additionalProperties is not false	test_additional_properties_is_true	enforced	0
schemas are valid JSON schema (Draft 7)	test_streams_have_valid_json_schemas	advisory	6
cursor fields exist in schema	test_defined_cursors_exist_in_schema	advisory	15
primary keys exist in schema	test_primary_keys_exist_in_schema	advisory	33
primary keys have scalar types	test_primary_keys_data_type	advisory	5
supported data types	test_catalog_has_supported_data_types	advisory	6

To promote a check from advisory to enforced, flip its enforced=False to True in CATALOG_CHECKS. Do that per check, after the connectors that violate it are fixed.

test_discover also asserts that discover emits exactly one CATALOG message. If a scenario allows discover to fail (ALLOW_ANY with errors), the test returns early because there is no catalog to check.

Differences from CAT

• Only real schema keywords are walked. CAT recursed through every key, so it flagged a property named $ref, not or allOf, and $ref text inside examples or default data. The new walker descends only into keywords whose values are schemas.
• Every violation is reported. Most CAT checks stopped at the first failing stream. CAT's JSON schema check raised jsonschema's single best_match error, which picked a different error on each run for the same schema. The new checks report every error, sorted by location.
• Primary key type check. CAT built the type set with set("object"), which gives {'o', 'b', 'j', ...}. As a result, a key typed as a plain "object" string was never caught. The new check reads type as a string or a list.
• Supported data types now follows the protocol. CAT's version was pytest.mark.skipped for its whole life. The new check requires:
  • an object at the top level;
  • every airbyte_type to be one of the protocol's types and to sit on a compatible JSON type;
  • the temporal formats (date, date-time, time) to sit on a string type.
• empty_streams does not skip any discovery check. An invalid schema is invalid whether or not the stream has test data.

Declined

• CAT's format whitelist (date, date-time only). Formats such as email and uri are harmless to destinations. The whitelist also rejected time, which the type system supports. Only temporal formats are checked, and only for their JSON type.
• CAT's unknown-type-name check. The Draft 7 meta-schema already rejects unknown type names, for example int in source-bunny-inc.
• The validate_primary_keys_data_type opt-out. No connector in the monorepo sets it, and P1.5 is removing scenario fields rather than adding them. A connector that needs an exemption should fix its schema instead.
• test_backward_compatibility. It compares against the previous published image's catalog. That needs a second connector image, so it is out of scope for an in-process discover assertion.

Fleet dry run

Method. Every manifest-only connector on airbyte master (3ab60a0e19f) was built in-process with this branch. Each run used a dummy config generated from the spec: defaults, enums, examples and per-connector date overrides, plus integration_tests/sample_config.json where present. All HTTP was blocked. The checks ran against every stream whose schema could be produced without a network call.

Coverage.

• 515 connectors (42 certified) were built.
• 505 connectors evaluated fully and 3 partially, for 5,223 streams.
• source-google-analytics-data-api and source-hubspot need HTTP for some schemas. source-google-search-console and source-hubspot had dynamic_streams skipped.
• 7 connectors were not evaluated: airtable, bigcommerce, braze, google-sheets, harvest, postmarkapp and workable. Each fails to build on the current CDK for reasons unrelated to this change.

Results.

• 0 connectors fail an enforced check.
• 50 connectors fail at least one advisory check. None of them is certified.
• Only 198 of the 515 manifest-only connectors run test_discover with a config in CI today. The rest have no scenario with a config_path.

Of the 50 failing connectors, these 11 run test_discover in CI and will show the warning on the next release:

Connector	Advisory checks failing
source-aircall	primary keys exist
source-coingecko-coins	cursor fields exist, primary keys scalar
source-goldcast	primary keys exist, supported data types (top-level array)
source-google-calendar	primary keys scalar
source-primetric	supported data types (airbyte_type: big_number)
source-railz	primary keys exist, supported data types (no top-level type)
source-statuspage	primary keys exist
source-tmdb	supported data types (no top-level type)
source-tvmaze-schedule	schemas are valid JSON schema (null property schemas)
source-uptick	supported data types (airbyte_type: decimal, 82 fields)
source-zoom	supported data types (no top-level type)

All 50 connectors by check

• primary keys exist in schema (33): agilecrm, aircall, bitly, brex, buzzsprout, drip, easypromos, encharge, finnhub, freshbooks, getgist, goldcast, huntr, katana, kissmetrics, lob, mendeley, miro, newsdata-io, onfleet, pandadoc, persona, pingdom, railz, referralhero, sage-hr, simplecast, spotify-ads, statsig, statuspage, stockdata, veeqo, zoho-expense
• cursor fields exist in schema (15): bitly, brex, buzzsprout, coingecko-coins, finnhub, height, lob, mendeley, newsdata-io, ninjaone-rmm, pingdom, spotify-ads, statsig, tavus, teamwork
• schemas are valid JSON schema (6): algolia, bunny-inc, convertkit, freshbooks, track-pms, tvmaze-schedule
• supported data types (6): goldcast, primetric, railz, tmdb, uptick, zoom
• primary keys have scalar types (5): codefresh, coingecko-coins, drip, google-calendar, wordpress

Python connectors (P0.7)

Of the 106 Python connectors, only source-hardcoded-records runs the standard tests today, and it pins its CDK version, so it picks up these checks on a CDK bump rather than automatically. The others do not run the standard tests yet. A static scan of their schema files suggests what to expect once P0.7 brings them in:

• source-tplcentral sets top-level additionalProperties: false on 6 schemas. That check is enforced, so tplcentral needs a schema fix before, or together with, P0.7. Otherwise P0.7 needs a second staged step.
• source-commercetools uses format: date-time on an integer. This is an advisory check.
• source-facebook-marketing has hits in shared schema fragments. These are probably false positives, because the fragments are resolved before discover.
• The primary key and cursor checks depend on code-defined keys. They cannot be scanned statically for Python connectors.

Test fixture change

unit_tests/resources/source_pokeapi_w_components_py/manifest.yaml declared primary key id with an empty properties schema. That is the defect the new check reports. The schema now declares id and name, so the fixture's standard tests pass with the advisory warnings turned into errors.

Testing

• unit_tests/test/test_catalog_checks.py covers:
  • the passing and failing side of every check;
  • the CAT false positives listed above;
  • a check that crashes;
  • test_discover with mocked discover output: enforced failure, advisory warning, no catalog, and ALLOW_ANY;
  • an end-to-end DeclarativeSourceTestSuite run on an inline manifest.
• airbyte-cdk connector test on source-tvmaze-schedule with this branch passes and shows the warning. It fails with -W error::...DiscoveredCatalogWarning.
• mypy airbyte_cdk and ruff are clean.

Follow-ups

1. Fix the 50 connectors in a monorepo PR, then flip the advisory checks to enforced in a later CDK release.
2. 317 manifest-only connectors never reach test_discover because they have no scenario with a config. A credential-free discover path would extend these checks to them.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/discover-assertions#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/discover-assertions

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 138 tests  +104   5 126 ✅ +104   11m 22s ⏱️ +41s
    1 suites ±  0      12 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 4808b4f. ± Comparison against base commit 916d6f6.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 141 tests  +104   5 129 ✅ +104   12m 46s ⏱️ - 2m 46s
    1 suites ±  0      12 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 4808b4f. ± Comparison against base commit 916d6f6.

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Verdict: fix before merge — 0 P0 · 1 P1 · 5 P2 · 2 P3 (all inline).

Breaking change: BREAKING under criterion A.7 only — a new mandatory standard-test gate that every source connector running airbyte-cdk connector test picks up automatically on the next release (monorepo CI installs the CLI unpinned). No runtime, API, manifest or state break; feat: → minor is correct; the staged-rollout call-out in the body meets the requirement. A cdk-migrations.md entry is still worth adding (inline P2).

Done well

• Staged rollout backed by real data: fleet dry run, the exact list of 11 connectors that will warn, a one-flag promotion path.
• Keyword-aware schema walker removes CAT's false positives on properties named $ref/not/allOf and on $ref text in examples — and is tested for exactly those cases.
• Every violation reported in one run; a crashing advisory check is reported, not raised, so it cannot fail a connector by throwing on an unexpected shape. The CAT set("object") bug is fixed and pinned.

Fleet sweep (independent). Static sweep of every language:manifest-only connector on origin/master (239eafd614e) through the PR-head ManifestReferenceResolver + ManifestComponentTransformer, all offline-resolvable schemas run through CATALOG_CHECKS: 516 connectors, 5,256 streams, 0 enforced failures; plus the 4 enterprise manifest-only connectors, 38 streams, 0 enforced failures. Advisory counts for JSON-schema validity (6) and supported data types (6) match the PR body. Not evaluable offline: 9 connectors with HTTP-built catalogs (dynamic_streams in airtable, bing-ads, google-analytics-data-api, google-search-console, google-sheets, hubspot, linkedin-ads, pinterest; CustomSchemaLoader in us-census). Reading the generators, DynamicSchemaLoader hard-codes additionalProperties: True and emits only oneOf, and the custom loaders set additionalProperties: True, so an enforced violation there is not plausible — but reasoned, not run.

Rollout / risk

• CI: all required checks green on 3.10–3.14; Check: destination-motherduck fails at the same step on #1224/#1221/#1219 (baseline, and destinations do not run test_discover); Check: source-shopify still pending; source-google-drive skipped. None of the three passing connector checks exercised the advisory warning path.
• Before release: /prerelease and a pinned airbyte-cdk connector test with CI secrets on source-google-sheets, source-hubspot, source-google-analytics-data-api (HTTP-built catalogs) and source-tvmaze-schedule (warning path).
• Overlap with #1219: its inspect_output secret scan is wired into test_check only; the new test_discover call here does not pass it, so after both merge discover output is not secret-scanned unless one PR adds it. Note for #1219, not a defect here.

[Daryna Ishchenko (darynaishchenko)]

Approving. All eight round-1 threads are resolved, and I checked each fix the same way the problem was found. The mutation harness now catches every wiring mutation. The draft-04 and anyOf/oneOf schema examples that produced false alarms now pass. The docs and the cdk-migrations.md entry are in place and match the code. Re-running the fleet sweep with the round-1 code and this head gives byte-identical results, so the two fix commits change nothing for connectors.

I have only minor nits left, none blocking.