feat(standard-tests): check primary keys, state messages and error traces in read tests

[Anatolii Yatsuk (tolik0)]

Part of airbytehq/airbyte-internal-issues#17135 (P1.2 "Restore read assertions", epic airbytehq/airbyte-internal-issues#17129). Follows #1218.

Fleet impact: of the 88 manifest-only connectors whose basic read passes today, 2 go red, both community connectors and both on the primary key check. The state check and the two trace tests turn no connector red. Details under "Fleet impact".

Merge order: merge the connector opt-outs in airbytehq/airbyte#88937 first, then this PR.

What

#1218 restored per-stream record presence and schema validation in test_basic_read. Three more checks of CAT's basic read test were still missing from the Standard Tests. This PR adds them, with CAT's config flags as the opt-outs:

1. Primary keys (CAT _validate_primary_keys_data_type). For every config under basic_read, every record of a stream with a source_defined_primary_key must have a usable key: no key field holds an object or an array, and not every key field is missing or null. Opt out with validate_primary_keys_data_type: false.
2. State messages (CAT _validate_state_messages). Every stream the read covers must be named by a STREAM state message (or a GLOBAL state's stream_states); no state uses the deprecated LEGACY type; every state carries sourceStats.recordCount. Opt out with validate_state_messages: false.
3. A failing read reports a well-formed error trace (CAT test_airbyte_trace_message_on_failure). New per-connector test test_fail_read_with_invalid_config, and test_fail_read_with_bad_catalog now checks its error traces the same way. Opt out with expect_trace_message_on_failure: false.

Failure messages name the stream, the record number and the key path, never a value:

Records have invalid primary keys. Destinations deduplicate on the primary key, ...
- Stream 'issues' (primary key `$.id`): 2 of 40 records have an invalid primary key.
    - record #3: key field `$.id` is an object; a primary key field must hold a scalar (a string, a number or a boolean)

The read's state messages are incomplete. To skip this check for a config, set `validate_state_messages: false` ...
- 1 of 3 configured streams emitted no `STREAM` or `GLOBAL` state message: 'events'. ...

How

• _read_assertions.py: _StreamPrimaryKeyChecker (runs in the same single pass over the records as the schema check), _find_state_message_failures, assert_error_trace_messages, make_invalid_config. assert_read_records gains validate_primary_keys and state_messages, both default off, so the Docker read test, which P1.4 (airbytehq/airbyte-internal-issues#17137) is changing in parallel, is not touched. It takes record messages plus the configured catalog, so P1.4 can pass the new arguments as is.
• ConnectorTestScenario: validate_primary_keys_data_type, validate_state_messages, expect_trace_message_on_failure, with CAT's names and defaults (true). _dedup_scenarios keeps an opt-out declared by any entry for the config, as it already did for validate_schema.
• test_basic_read passes the new arguments for basic_read configs. A config read only because of full_refresh keeps the "some records" floor, as in feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218.
• cdk-migrations.md: a section under "Upgrading to the next minor release (unreleased)".

Decisions

Primary keys follow CAT exactly. A composite key may have null parts as long as one part is present, and booleans count as scalars (the message calls them "boolean", not "integer"). The stricter "every key part present" rule was measured only (see below), not enforced. As in CAT, validate_primary_keys_data_type: false is allowed at test_strictness_level: high; only validate_schema: false is refused there.

State: the per-stream version, not CAT's count. CAT asserted len(state_messages) >= len(streams). That is not vacuous, but one stream that checkpoints many times hides another stream that emitted no state at all. This PR checks coverage per stream (matched on namespace and name). The LEGACY check is kept; CAT's version crashed on GLOBAL states, which this one handles. The sourceStats check is kept, but it is vacuous for CDK connectors, because the CDK entrypoint sets sourceStats.recordCount on every state it emits; it guards non-CDK sources and the Docker path. The CDK emits a final {"__ab_no_cursor_state_message": true} state for every full-refresh stream it completes, so a CDK connector fails only when a stream ends abnormally or custom code drops the state.

The failure input for the trace test is a config the connector's spec rejects. CAT's trace test used the valid config plus a broken catalog, and FAST's test_fail_read_with_bad_catalog already is that test. It now uses the same well-formedness check instead of only requiring some trace message, so there is no duplicate. The new test needs a different failure, one that reaches the connector:

• The config holds only a marker key (__airbyte_standard_tests_invalid_config) and none of the spec's required top-level properties. The CDK entrypoint rejects it against the spec with a config_error trace, before any request is sent.
• It needs no secret and runs once per connector, so connectors whose basic_read configs are all secret-backed are covered too.
• Invalid real-looking credentials were rejected: they send failed logins to vendor test accounts, which some vendors answer with lockouts or CAPTCHAs.
• A missing stream was rejected: AbstractSource reports it as INCOMPLETE without an error.
• The test skips when the spec requires no top-level property (no config is certainly invalid) and when any basic_read entry sets expect_trace_message_on_failure: false, as CAT did.
• In-process, entrypoint_wrapper turns any uncaught exception into a well-formed trace message, so this check mostly guards the CDK and the harness. The same helper is meant for the Docker path (P1.4), where a crashing image can emit nothing.

source-s3's hand-ported CAT harness (integration_tests/test_acceptance.py), named as prior art in the issue, has none of these three checks, so they are ported from CAT's semantics directly.

Fleet impact

I ran this branch's checks in-process (as airbyte-cdk connector test does) against airbytehq/airbyte master at 1e22e3c689c. The population was every manifest-only source with a basic_read config: 185 connectors. 16 were skipped, because their test secret holds a single-use rotating refresh token, and a local run would invalidate the copy CI uses. Each basic_read scenario was read once (20-minute timeout, and 45 minutes for a second try of the 6 that timed out), and the verdict on main (presence and schema, from #1218) and the new checks were taken from the same read. test_fail_read_with_invalid_config ran once per connector, and test_fail_read_with_bad_catalog on the first read scenario.

Connectors	Count
Run	169
No verdict for the read: the read did not finish within 45 minutes locally (both trace tests passed)	4
Already red before this change	77
Green before and after	86
Green before, red after	2 (none certified)

Already red before this change: 41 have expired, revoked or under-scoped test credentials, or hit plan limits; 8 have no test secret in GSM; 3 fail to load with the current CDK or have an acceptance-test-config.yml the harness cannot parse; 25 fail on other API or connector errors, or on #1218's checks. None of the reads that reached the checks failed the new ones, so they hide nothing that is known.

Connector	Check	Stream	Problem
source-google-calendar	primary key	colors	The key fields calendar and event hold objects.
source-statuspage	primary key	subscribers_histogram_by_state	The records have no id, the declared key.

Both are real key problems, not false positives, and CAT would have failed them too. Changing a primary key is a breaking change, so airbytehq/airbyte#88937 opts the two configs out with validate_primary_keys_data_type: false instead, and the key fixes are left for separate PRs.

Per check:

• Primary keys: 81 of 97 scenarios that reached the checks read at least one stream with a primary key; 2 failed. The stricter "every key field present" rule, measured as an advisory only, flags no other stream.
• State: 96 scenarios passed and 1 is opted out (source-chargebee, which already declares validate_state_messages: false). No connector fails.
• test_fail_read_with_invalid_config: 161 passed and 6 skipped because the spec requires no top-level property. 2 failed, both on connectors whose every test already fails (one cannot load its custom components with the current CDK, one has an acceptance-test-config.yml the harness cannot parse).
• test_fail_read_with_bad_catalog: 159 passed. The 9 failures are connectors that are red before: 8 have no test secret, and 1 cannot load its custom components.

The CDK's Python test connectors, run in-process with this branch: source-hardcoded-records passes (the new trace test skips, as its spec requires nothing). source-shopify's test_fail_read_with_invalid_config passes; its two test_basic_read scenarios fail before any new check, because the test store's Europe/Kyiv time zone is missing from the pendulum 2 time zone data, which this PR does not touch. source-google-drive cannot be created by the harness (its constructor needs catalog, config and state), and the new trace test skips on its existing expect_trace_message_on_failure: false.

Rollout

The connector monorepo installs the airbyte-cdk CLI unpinned, so connector CI picks the new checks up with the release, without a per-connector CDK bump. That reaches the manifest-only connectors, which run the in-process suite (airbyte-cdk connector test). Python connectors run only the Docker suite (airbyte-cdk image test) or pytest against their own pinned CDK, so they get these checks through P1.4's Docker read test (airbytehq/airbyte-internal-issues#17137) or a CDK bump. source-hardcoded-records is the only Python connector that subclasses the suite. This ships as a minor (feat, no !), as standard-test changes do.

Merge order:

1. test(connectors): opt google-calendar and statuspage out of the primary key check airbyte#88937: primary key opt-outs for source-google-calendar and source-statuspage (test-only, no connector release).
2. This PR.

The monorepo picks the release up through the unpinned CLI, so the connector opt-outs must be on master before this PR is released.

Landing order for airbytehq/airbyte-internal-issues#17135 (updated)

1. feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218 (merged, 7.36.0): per-stream record presence, record-vs-schema validation, failure messages that name the stream and the record path.
2. This PR: primary-key data types, per-stream state messages, and a well-formed error trace on a failing read.
3. Later, after a measurement pass: strict extra columns (fail_on_extra_columns), stream status completeness (validate_stream_statuses), and data points (validate_data_points). CAT never applied fail_on_extra_columns and defaulted validate_data_points to false, so those two ship behind flags. CAT required stream statuses by default for certified connectors only, so that check needs its own measurement before it can be enforced. Not in this PR.

The Docker read test (test_docker_image_build_and_read) is not changed here; P1.4 (airbytehq/airbyte-internal-issues#17137) owns it.

Test plan

• poetry run pytest unit_tests/test/ -p no:cacheprovider: 848 passed, 9 skipped. New tests in unit_tests/test/test_read_keys_state_and_traces.py cover each check directly and through SourceTestSuiteBase (fake sources that emit a null key, no state, a malformed trace, or a read that succeeds on the invalid config), the opt-outs, high strictness, and opt-outs surviving _dedup_scenarios in both orders.
• ruff format, ruff check, mypy airbyte_cdk/test: clean.
• source_pokeapi_w_components_py standard tests (the CDK fixture): 12 passed, 10 skipped; test_fail_read_with_invalid_config passes.
• CDK Test Connectors matrix, in-process with this branch: source-hardcoded-records green; source-shopify, source-google-drive and source-intercom as described under "Fleet impact" (source-intercom is manifest-only and is green in the fleet run).
• Fleet run described above.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/read-assertions-slice-2#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/read-assertions-slice-2

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 813 tests  +65   5 782 ✅ +64   11m 32s ⏱️ -38s
    1 suites ± 0      31 💤 + 1 
    1 files   ± 0       0 ❌ ± 0 

Results for commit cc674f5. ± Comparison against base commit 2082c2d.

This pull request skips 1 test.

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_spec_backward_compatibility

[github-actions]

PyTest Results (Full)

5 816 tests  +65   5 785 ✅ +64   16m 31s ⏱️ -29s
    1 suites ± 0      31 💤 + 1 
    1 files   ± 0       0 ❌ ± 0 

Results for commit cc674f5. ± Comparison against base commit 2082c2d.

This pull request skips 1 test.

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_spec_backward_compatibility