feat(standard-tests): assert per-stream records and record schemas in the basic read test

[Anatolii Yatsuk (tolik0)]

Part of airbytehq/airbyte-internal-issues#17135 (P1.2 "Restore read assertions", epic #17129).

Stacked on #1142. This branch builds on #1142 (merged up to its head e105226c), because the record checks are gated on the sections that #1142 adds to ConnectorTestScenario and run inside its section-to-command mapping. The diff against #1142's branch shows only this PR. Retarget to main once #1142 merges.

What

SourceTestSuiteBase.test_basic_read, the in-process read test that airbyte-cdk connector test runs, only failed when the whole read returned no records. One stream with records passed the test for every stream, and no record was ever checked against its schema. For configs listed under basic_read and expected to succeed (status: succeed, or no status, which #1142 makes mean succeed), this PR restores the two record checks of CAT's TestBasicRead.test_read:

1. Every stream returns records. Every configured stream must return at least one record. empty_streams stays the list of exceptions, with the same meaning as in CAT: those streams are not read.
2. Records match their schema. Every record is validated against its stream's JSON schema with the validator CAT used: Draft 7, strict integers, a lenient date-time check, and no failure on extra columns. Like CAT, a record that shares no field with a schema that declares properties also fails. validate_schema: false in the basic_read entry turns the check off for that config. As in CAT, it is rejected for a connector at test_strictness_level: high: the test fails before reading, and points at fixing the schema or lowering the strictness level.

Both checks run in one pass over the records, which also enforces the existing "Expected records but got none" floor, and all failures are reported in a single AssertionError. The message names each stream and each distinct schema error once (up to 5 per stream), with the record number, the path inside the record, the failing schema keyword and the schema rule:

Records do not match their stream's JSON schema. Fix the schema or the records; to skip this check for a config, set `validate_schema: false` (with a reason) in its `basic_read` entry of `acceptance-test-config.yml`.
- Stream 'orders': 3 of 40 records do not match the stream's JSON schema.
    - record #7 at `$.items[1].price`: expected type 'number', got string (schema rule: `properties/items/items/properties/price/type`)

Record values are never printed, for any kind of schema error, so record contents (often whole nested objects) stay out of public CI logs. The message names the value's JSON type and the schema's own scalar constraint (a format name, a pattern, a length), and for an anyOf/oneOf the closest option's failure, described the same way:

- record #0 at `$.user`: fails `anyOf`, got object; closest option fails at `$.user.inner.x`: fails `enum`, got string (schema rule: `properties/user/anyOf`)

A stream schema the validator cannot apply (an unknown type, an unresolvable $ref, an invalid pattern) fails that stream with the schema problem, instead of aborting the check with a raw jsonschema exception (whose text prints the record).

Landing order for #17135

The issue covers more than one PR. This PR is the first:

1. This PR: per-stream record presence, record-vs-schema validation, and failure messages that name the stream and the record path.
2. Next: primary-key data types, state messages, and the test that a failing read emits an AirbyteTraceMessage.
3. Later, opt-in: strict extra columns, stream statuses and data points. These ship behind flags after a measurement pass, because CAT never enforced them by default.

Which scenarios get which check

With #1142, test_basic_read runs only for configs listed under basic_read or full_refresh.

Scenario	"Some records" floor	Per-stream presence	Schema validation
listed under basic_read (or built by hand, so sections is empty), status: succeed or none	yes	yes	yes
listed under full_refresh but not basic_read	yes	no	no
status: failed / exception	no (the read is expected to fail)	no	no

A config read only because of full_refresh keeps the old floor: CAT's full refresh test did not run these checks, and the config has no basic_read entry to declare empty_streams or validate_schema: false in.

At test_strictness_level: high, a basic_read config with validate_schema: false fails outright (CAT's test_core.py did the same). get_scenarios() now carries the top-level test_strictness_level into each scenario.

_dedup_scenarios merges entries for the same config_path. A validate_schema: false from any entry is kept.

Deviations from CAT

• Booleans are not integers. Python's bool is an int subclass, so CAT's strict-integer checker accepted true for an integer field. JSON does not, and neither do destinations.
• Non-string date-time values pass the format check. As in jsonschema itself, a format only constrains strings; a wrong type is the type keyword's job. CAT's checker also skipped them.
• pendulum is replaced by ab_datetime_try_parse. The shape regex is CAT's.
• Local $refs are resolved when collecting a schema's field paths, with a cycle guard, so recursive schemas do not loop.

Not in this PR

• The Docker read test (test_docker_image_build_and_read) is not changed. It does not run by default, and P1.4 (#17137) owns it. assert_read_records takes the record messages and the configured catalog, not a scenario, so P1.4 can call it as is.

Impact

The connector monorepo installs the airbyte-cdk CLI unpinned, so connector CI picks this up with the release, without a per-connector bump. Python connectors' in-process tests are deselected until P0.7, so the change reaches manifest-only connectors first.

The fleet count below covers manifest-only connectors only. Python connectors were not measured. They are deselected in monorepo CI until P0.7, but the CDK's own Test Connectors matrix runs their in-process suites today:

Matrix connector	Language	basic_read in-process	Result on this PR
source-shopify	python	2 configs	red, genuine (see the rollout list)
source-hardcoded-records	python	1 config	green
source-google-drive	python	2 configs	not measured: the job only runs when file_based changes, so it was skipped here and will first run on the next file-based CDK PR
destination-motherduck	python (destination)	none	red before this PR (pre-existing)
source-intercom, source-pokeapi	manifest-only	yes	green

Before P0.7 turns Python in-process tests back on, re-run this measurement on every Python connector with a credentialed basic_read config. They will go red the same way, and shopify shows one extra failure class that the Docker-based CAT never saw: class-level mutable state shared between configs read in the same process (see countries below).

Measured on the manifest-only fleet

I ran test_basic_read from this branch (on #1142 at d64a79c7) locally against airbytehq/airbyte master. The population was every manifest-only connector whose acceptance-test-config.yml has a credentialed config that reaches the read test: 188 connectors. 16 were skipped, because their test secret holds a single-use rotating refresh token, and a local run would invalidate the copy CI uses. Each scenario was read once, and the old verdict and the new checks came from the same read. 11 connectors whose manifests need CDK features newer than #1142's base were re-run on a local merge of this branch with current main. The later #1142 commit (e105226c) does not change the set below: the record checks only apply to basic_read configs, and the only basic_read config without a status (now succeed) returned records from every stream and matched its schemas.

Connectors	Count
Run	172
No verdict (20-minute local timeout, or a pre-existing config parse error)	7
Already red before this change: expired or missing test credentials, custom components incompatible with the current CDK, upstream API errors	77
Green before and after	65
Green before, red after	23 (10 of them certified)

So 23 of the 88 connectors whose read passes today (26%) go red. These are 24 scenarios: 16 with streams that return no records and 12 with schema errors; some have both.

I checked each failure by hand, and none is a false positive of the validator:

• Streams with no records: the streams are, with one exception, in the connector's CAT configured_catalog.json, so CAT would have caught them too. The test accounts' data has drifted since CAT was removed: for example GitHub classic projects and workflow runs, PagerDuty incidents, and LinkedIn member analytics.
• Schema errors: genuine drift between the schema and the API. Examples are strings in number/integer fields, nulls in non-nullable fields, date values under format: date-time, '' under format: email, and objects where the schema says string. The only failure caused by a deviation from CAT is bool-as-integer in source-sentry's project_detail, and source-sentry also fails the per-stream check.

Connectors that go red (stream names only)

Connector	Config	Certified	Failing check
source-aircall	config		no records: calls, numbers
source-amazon-seller-partner	config	yes	no records: GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL
source-ashby	config		schema: applications, job_postings
source-clickup-api	config		no records: list_comments, list_custom_fields, space_tags, team, team_custom_fields, team_goals, time_tracking, time_tracking_tags
source-customer-io	config		schema: campaigns_actions
source-fleetio	config		schema: submitted_inspection_forms
source-freshdesk	config	yes	no records: ticket_activities; schema: sla_policies
source-github	config	yes	no records: project_cards, project_columns, projects, workflow_jobs, workflow_runs
source-jira	service_account_config	yes	no records: filter_sharing, filters, issue_comments, issue_properties, issue_remote_links, issue_worklogs, labels, project_categories, project_components, project_versions
source-jira	config	yes	schema: issues
source-klaviyo	config	yes	no records: segments
source-linkedin-ads	config_oauth	yes	no records: ad_member_company_analytics, ad_member_company_size_analytics, ad_member_country_analytics, ad_member_industry_analytics, ad_member_job_function_analytics, ad_member_job_title_analytics, ad_member_region_analytics, ad_member_seniority_analytics
source-merge	config		no records: job_interview_stages, users
source-pagerduty	config		no records: incident_logs, incidents
source-paypal-transaction	config	yes	no records: list_payments
source-pendo	config		schema: Report, Visitor Metadata
source-pexels-api	config		schema: videos_search
source-railz	config		no records: bank_accounts, bank_transfers, commerce_disputes, commerce_orders, commerce_products, commerce_transactions
source-recurly	config		no records: plans; schema: accounts, subscriptions
source-sentry	config	yes	no records: releases; schema: project_detail, all_projects_detail
source-snapchat-marketing	config	yes	no records: segments
source-tmdb	config		schema: movies_latest, movies_reviews
source-uptick	config		no records: promptanswergroups; schema: clients, branches, products, contractors, clientcontacts, propertycontacts
source-woocommerce	config	yes	schema: product_variations, shipping_zone_methods

Merge order

Connector CI installs the CDK CLI unpinned, so these checks apply to every connector PR from the day the CDK is released. Merging into main does not publish anything; the release is the gate. All connector PRs below are green on the current released CDK, so they can land first, in any order:

• Connector PRs for this PR's checks, all merged: test(connectors): declare empty streams and schema opt-outs for basic read assertions airbyte#88406 (empty streams and schema opt-outs), test(source-amazon-seller-partner): declare an empty stream for basic read assertions airbyte#88477 (amazon-seller-partner), fix(source-freshdesk): emit sla_policies ids as integers airbyte#88407 (freshdesk), fix(source-sentry): type the sentry:replay_rage_click_issues project option as boolean airbyte#88408 (sentry), fix(source-jira): type rendered comment bodies in issues as strings airbyte#88409 (jira), fix(source-shopify): keep the countries dedup list per stream instance airbyte#88405 (shopify), fix(source-customer-io)!: 🚨🚨 declare integer from_id, reply_to_id and sent_at and typed array items 🚨🚨 airbyte#88159 (customer-io).
• Connector PRs for feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142's stricter checks, all merged: test(source-sentry, source-hubspot): fix limited-scope full_refresh configs airbyte#88412 (sentry, hubspot), test(source-zoho-crm, source-microsoft-lists, source-mixpanel): fix configs for stricter standard-test checks airbyte#88480 (zoho-crm, microsoft-lists, mixpanel).
• feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142, then this PR (it builds on feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142)
• Before merging this PR: confirm the draft release is still v7.36.0 and unpublished. This PR titles the shared migration-guide section ## Upgrading to 7.36.0, which also covers feat(standard-tests): assert discovered catalog properties in test_discover #1217, feat(standard-tests): lint airbyte_secret marking and secret leaks #1219, feat(standard-tests): check spec backward compatibility against the published version #1220 and feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142, so all five must ship in that one release.
• Re-run the manifest-only sweep on current master (test-account data drifts)
• Cut the CDK release

Rollout

The 23 connectors go red on their next PR after the release. Proposal: before the release, land one monorepo PR that declares the empty streams in empty_streams and sets validate_schema: false for these configs, each with a bypass_reason linking a follow-up issue, so the release turns nothing red, and then fix the connectors one by one. The other option is to release as is and fix each connector when it is next touched.

The opt-out PR also needs:

• source-shopify (certified, test_strictness_level: high; CI job 113491220276). deleted_products returns no records for both basic_read configs, and countries returns none for secrets/config.json.

  • deleted_products, both configs: declare it empty (or seed a product deletion in the test shop). It is in neither expected_records*.jsonl; the stream reads action:destroy product events.
  • countries, secrets/config.json: a connector bug, not an empty stream. Both expected-records files contain the same countries records, so the shop has data. Countries deduplicates through a class attribute, _synced_countries_ids = [] (source_shopify/streams/streams.py:773), which every instance shares. The in-process suite reads config_transactions_with_user_id first, which fills the list, so the second config's read skips every country. Production syncs run one config per process and are not affected; CAT ran each config in its own container. The fix is a one-line connector change (initialise the list per instance) with a patch bump. Until it lands, a stopgap empty_streams entry with a reason that names the bug keeps CI green.

    basic_read:
      tests:
        - config_path: "secrets/config_transactions_with_user_id.json"
          # ...unchanged keys...
          empty_streams:
            # ...existing four entries...
            - name: deleted_products
              bypass_reason: The test shop has no product deletion events for the Events API to return.
        - config_path: "secrets/config.json"
          # ...unchanged keys...
          empty_streams:
            # ...existing four entries...
            - name: deleted_products
              bypass_reason: The test shop has no product deletion events for the Events API to return.
            - name: countries
              bypass_reason: >-
                Not empty: the Countries stream dedups through a class-level list shared across
                instances, so when the standard tests read this config after
                config_transactions_with_user_id in the same process, every country is skipped.
                Remove once the stream keeps the list per instance (follow-up issue).

• No validate_schema: false for high-strictness connectors. Three of the twelve connectors with schema failures declare test_strictness_level: high: source-freshdesk (sla_policies), source-jira (issues) and source-sentry (project_detail, all_projects_detail). The opt-out PR cannot set validate_schema: false for them (the test now fails on it), so their schemas must be fixed before the release, or the release moves after those fixes. The other nine (ashby, customer-io, fleetio, pendo, pexels-api, recurly, tmdb, uptick, woocommerce) declare no strictness level, which CAT treated as low, so the opt-out stays available to them. woocommerce is certified but declares no strictness level. source-azure-table already sets validate_schema: False on master (no strictness level), so it keeps working.

cdk-migrations.md has a "### Every stream returns records that match its schema (basic_read)" subsection with the migration steps: add data or declare empty_streams with a bypass_reason; fix the schema or set validate_schema: false (not available at test_strictness_level: high). It sits under the "Upgrading to the next minor release (unreleased)" heading shared with #1142, #1217, #1219 and #1220; rename the heading to the released version when the release is cut. This ships as a minor release: only the test harness changes.

Tests

• unit_tests/test/test_read_assertions.py (77 tests) covers:
  • the helper on its own: presence, the "no records" floor, a lazily consumed iterator; type and format errors; float and bool as integer; date-time shapes; nested and non-identifier record paths; the structure check; the per-stream error cap; dedup per schema rule; recursive $ref;
  • that no failure message prints a record value: nested anyOf/oneOf, not, allOf, enum, const, pattern, format, uniqueItems, required, each with a sentinel value that must not appear;
  • invalid schemas: unknown type, unresolvable local and remote $ref, invalid pattern;
  • a fake source run through SourceTestSuiteBase.test_basic_read for every combination of outcome and sections, including status: failed and exception reads that raise, a failed read that succeeds, and test_strictness_level: high and low with and without validate_schema: false;
  • the dedup of validate_schema (with a connection entry, and two basic_read entries in both orders) and the parsing of test_strictness_level.
• Results:
  • unit_tests/test plus unit_tests/resources (including the pokeapi Docker image tests): 380 passed, 12 skipped. This includes feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records #1142's test_scenario_sections.py.
  • ruff check, ruff format --check and mypy airbyte_cdk/test are clean.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/basic-read-assertions#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/basic-read-assertions

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 326 tests  +81   5 305 ✅ +81   11m 25s ⏱️ -12s
    1 suites ± 0      21 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 71efdfd. ± Comparison against base commit a07a090.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 329 tests  +81   5 308 ✅ +81   16m 21s ⏱️ + 3m 10s
    1 suites ± 0      21 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 71efdfd. ± Comparison against base commit a07a090.

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Approving the change itself. The assertions are a single lazy pass over the records, every branch of test_basic_read and _dedup_scenarios is pinned by a test (all 13 wiring mutants caught, 30/32 overall), and date-time verdicts match CAT's on the shapes I tried. This is BREAKING under A.7 only (a new mandatory fleet-wide test gate). That is called out in the body and in cdk-migrations.md, and feat: (minor) matches #1217.

One thing to settle before merge: the release order, inline. CI is green apart from Check: destination-motherduck, which is red on every recent PR (expired token).

[Daryna Ishchenko (darynaishchenko)]

🟠 medium: nothing enforces the rollout order, and 22 connectors depend on five open monorepo PRs.

Connector CI installs the CDK CLI unpinned (connector-ci-checks.yml:592, which the migration entry also says). So this check applies to every connector PR from the day of the release. Companion PR state today: #88405 (shopify) and #88159 (customer-io) are merged. Still open: airbytehq/airbyte#88406 (18 connectors), #88477 (amazon-seller-partner), #88407 (freshdesk), #88408 (sentry), #88409 (jira). freshdesk, jira and sentry are at test_strictness_level: high, so validate_schema: false isn't an escape for them (_read_assertions.py:350–358). Only their schema fixes clear the check.

Suggest adding a merge-order checklist to the PR body (#1142, then the five PRs above, then this PR, then release) and holding merge until it's done. Re-run the manifest-only sweep on current master just before release, since test-account data drifts. Optionally list the 16 rotating-token connectors that weren't measured, so their owners expect a possible red run.

[Anatolii Yatsuk (tolik0)]

Added a merge-order checklist to the PR body: the connector PRs (airbytehq/airbyte#88406, airbytehq/airbyte#88477, airbytehq/airbyte#88407, airbytehq/airbyte#88408, airbytehq/airbyte#88409 for this PR; airbytehq/airbyte#88412, airbytehq/airbyte#88480 for #1142) land first, then #1142 and this PR, then a fresh manifest-only sweep on current master, then the release. Merging into main doesn't publish, so the release is the gate.