feat(standard-tests): check spec backward compatibility against the published version

[Anatolii Yatsuk (tolik0)]

New mandatory standard test. On the next CDK release, every connector that runs the Docker suite (airbyte-cdk image test), about 619 non-JVM connectors in the monorepo, is held to test_docker_image_spec_backward_compatibility without opting in, because connector CI installs airbyte-cdk[dev] unpinned. A connector PR whose spec rejects configs that the published version accepted fails until the change is made compatible, declared as a breaking change, or waived with disable_for_version. There is no runtime, API, manifest, state or packaging change. cdk-migrations.md documents the test and both waivers.

What

Adds test_docker_image_spec_backward_compatibility to DockerConnectorTestSuite. The test fails when a connector's spec rejects configs that the published version accepted. Users' saved configs are validated against the new spec when a release rolls out, so this kind of change breaks existing connections at upgrade time.

This is P3.1 of the CAT → FAST coverage epic (airbytehq/airbyte-internal-issues#17144, under #17129). It replaces CAT's test_backward_compatibility for specs. The discovery/catalog half is P3.2 and is not in this PR.

How it works

1. The test fetches the published latest spec from the public registry, once per registry: https://connectors.airbyte.com/files/metadata/<dockerRepository>/latest/{oss,cloud}.json. Registry latest is what most users run. During a progressive rollout it stays on the previous GA release, so a patch on top of a release candidate is compared against the version users actually have.

2. The test runs spec in the image under test once per registry. It sets the same environment the registry publisher sets (DEPLOYMENT_MODE plus AIRBYTE_EDITION), so cloud-only spec differences are compared like for like. Both sides are compared as raw JSON, not through the protocol model, because the registry stores the raw output.

3. compare_specs classifies every difference as breaking or compatible. Only JSON Schema validation keywords and the protocol keys that locate values in a saved config (path_in_connector_config, path_in_oauth_response, predicate_key/predicate_value, the OAuth input and output schemas) can be breaking. Every other key is an annotation (title, description, order, group, airbyte_hidden, multiline, $schema, x-*, …), and changing it is compatible.

   • Removed properties are breaking.
   • Newly required properties are breaking, including new properties and a node's first required list.
   • Added constraints (enum, pattern, const, format, maxLength, additionalProperties: false, …) are breaking.
   • Tightened bounds, narrowed type sets, narrowed enums and removed destination sync modes are breaking.
   • Moved OAuth output paths are breaking.
   • Unmarking a secret (airbyte_secret becoming false or removed) is breaking.
   • Validation keywords the comparator does not model (not, if/then/else, dependencies, …) are breaking when added or changed. None appears in the fleet today.
   • Annotations, defaults, relaxed constraints, null added to type, integer widened to number, new optional properties and newly secret fields are compatible. airbyte_secret: "true" rewritten as true is compatible too.
   • A keyword added or removed at its default value (additionalProperties: true, uniqueItems: false, minLength: 0, required: [], …) is compatible, because it accepts the same configs.
   • Under oneOf, a config must match exactly one branch. An added oneOf branch is compatible only if it provably rejects the configs of every previous branch: it requires a field they do not declare, or pins a discriminator to a different value (required by one of the two, or left out only by configs that another branch already matched). A matched branch that loses that proof, for example by dropping its discriminating const, is breaking. Under anyOf, an added branch only widens and is compatible.
   • An added patternProperties, or a pattern added to one, is breaking: it applies to every field whose name matches, so it can reject values that saved configs already set.
   • Equality is type-strict, as in JSON: enum [1] becoming [true], or const 0 becoming false, is a change. additionalProperties or items relaxed from false to a schema, and a multipleOf relaxed to a divisor of the old value, are compatible.
   • oneOf/anyOf branches are matched by a ranked match: identical branches first, then shared discriminating consts, then the same title, then shared fields and types, then position. Reordering auth methods, adding an optional discriminator, or renaming the title of a branch is therefore not a change. Two branches that disagree on a discriminator are never paired, so a renamed auth_type const is still one branch removed and one added.

   The rules are ported from the spec comparator in the ops regression tests (airbyte-ops-mcp#1201). Because this test blocks merges, it differs from that comparator in two ways:

   • Only connectionSpecification, advanced_auth and supported_destination_sync_modes are compared. Nothing else in a spec affects existing connections.
   • advanced_auth.oauth_config_specification.oauth_connector_input_specification (declarative OAuth consent URLs, token URLs, scopes, extract_output) is compatible. It only affects new authorizations. In the fleet history it was the single largest source of findings that cannot break a saved config.

When the test passes, fails or is skipped

Situation	Result
No breaking differences	pass. The compatible changes are reported as a warning, which pytest shows in the warnings summary by default, up to 20 per list.
Breaking differences	fail. The message lists every finding and points at the breaking-change process.
Breaking differences, with a releases.breakingChanges entry in metadata.yaml for a version in (published latest, dockerImageTag]	skip, and the waived findings are reported as a warning. A pre-release tag such as 2.0.0-rc.1 counts as 2.0.0.
Breaking differences, with backward_compatibility_tests_config.disable_for_version in the spec section of acceptance-test-config.yml equal to the published latest version	skip, and the waived findings are reported as a warning. Both the spec.tests layout and the legacy list layout are read. An empty or malformed file has no waivers, and the file is only read once a comparison is breaking.
No entry in a registry (never published, or disabled there)	that registry is not compared. If neither registry has an entry, the test is skipped.
dockerImageTag older than the published version (a branch that has not merged the latest release)	skip. The published spec contains changes the branch has never seen. The version-increment QA check already forces a bump past it before merge.
Registry unreachable after retries (connection, proxy or timeout errors, or 5xx/429 after 2 retries)	skip within about 30 seconds (5 s connect and 10 s read timeouts), with a warning that the spec was not compared. A registry outage does not fail every connector PR, nor this repo's CI.
Registry answers with a client error, invalid JSON, or an entry without spec or dockerImageTag	fail, naming the URL
spec exits non-zero or emits no SPEC message	fail, with the exit code, the connector's last error TRACE message, its last log lines and its stderr

On the waivers:

• disable_for_version is the same key CAT honored, so existing configs keep working. A connector without acceptance-test-config.yml, such as the Java connectors that the base-image workflow rebuilds without a version bump, can add a file that holds only this entry. Because it must equal the published latest version, the waiver expires on its own once the next release ships. A stale value cannot silently disable the check. This is the documented escape for config migrations and false positives such as a widened regex.
• A declared breaking change covers every later version until the release that carries it reaches latest. A patch on top of a major that is still rolling out is waived too.

The failure message tells the author to do one of the following:

1. Make the change compatible.
2. Bump the major version, declare it under releases.breakingChanges, and write a migration guide. The message links the breaking-changes docs.
3. Waive the check with disable_for_version. The message gives the exact YAML.

Acceptance criteria (#17144)

• Spec diff against the previously published version, run in connector CI. Done. The test is in the Docker suite (@pytest.mark.image_tests), so it runs under airbyte-cdk image test. That is the only path Python connectors use in monorepo CI, and manifest-only connectors run it too. It reaches every connector on the next CDK release, because monorepo CI installs airbyte-cdk[dev] unpinned. P0.1 and P0.7 are not prerequisites for this reason.
• disable_for_version (or a replacement escape) is honored. Done, plus the declared-breakingChanges waiver. See above.
• Interaction with the breaking-change process is documented, and a failing diff points at breaking-change evaluation and the migration-guide requirement. Done in the failure message, in the test docstring, which pdoc renders, and in cdk-migrations.md. The comparator module is private (_spec_compatibility.py), so its helpers are not published as CDK API. A follow-up docs PR in the monorepo should update connector-acceptance-tests-reference.md.
• validate_previous_configs is ported or explicitly dropped. Dropped. CAT generated configs from the previous spec with hypothesis-jsonschema and validated them against the new one. Porting it would add two heavy dependencies to the CDK and make a merge-blocking test nondeterministic. The structural rules here catch the same classes of breakage deterministically, and the failure message names the exact field. CAT's previous_connector_version is not ported either: one connector sets it, and that entry also sets disable_for_version.

Validation

• Unit tests (unit_tests/test/test_spec_compatibility.py, 213 tests) cover:

  • the comparator rules, including the ops comparator's cases, annotations, airbyte_secret direction, keywords at their default value, integer → number, branch matching, overlapping oneOf branches, patternProperties and type-strict equality
  • waiver ranges, including release candidates and unparsable versions
  • disable_for_version parsing, in both layouts and for empty or malformed configs
  • registry fetch: 200, 404, error and malformed responses through requests_mock, plus the retry policy against a local HTTP server (recovery, give-up after 2 retries, no retry on 4xx, fast failure on a closed port) and on both schemes
  • test orchestration (both registries, registries on different versions, a waiver that covers one registry only, cloud-only breaks, missing registry entries, older versions, each waiver and its expiry, the reported warnings, the skip on an unreachable registry, the image build path)
  • the exact docker run command per deployment mode, and the diagnostics of a failing spec

  A skip of the test under test is turned into an error in these tests, so a test that expects a failure cannot pass on a skip. The reviewer's mutation probes (any newer registry skipping all registries, skip before fail, no https retry mount) each fail at least one test.

• End to end against published images. I ran the real test (registry fetch plus docker run … spec per mode) with --connector-image set to the published latest of six connectors. All six pass:

  • source-faker 7.2.1 (Python)
  • source-file 0.6.0 (Python, with a Cloud-specific spec)
  • source-linear 1.0.3 (manifest-only)
  • source-jira 6.0.3
  • source-postgres 3.8.5 (Java)
  • source-oracle 0.5.8 (Java, with a Cloud-specific spec)

  For source-file and source-oracle, the spec emitted in Cloud mode differs from the one emitted in OSS mode. Compared against the published OSS spec, it reports the expected Cloud-only removals (the local-file provider, and encryption becoming required). This confirms that the environment variables reach both the legacy Python CDK and the Java CDK.

• CDK's own fixture. The pokeapi fixture in unit_tests/resources (source_pokeapi_w_components_py) runs the Docker suite in this repo's Pytest jobs. It was an older copy (0.3.24) of source-pokeapi, which is published at 0.3.74, so the test was skipped by the older-version rule and its Docker path never ran in CDK CI. This PR bumps the fixture's dockerImageTag to 0.3.74 and aligns its pokemon_name enum with the published spec (nidoran-f, nidoran-m; the old spellings dropped two published values). The test now builds the fixture image, runs spec in both deployment modes and compares it against the registry on every CDK PR. It passes, and with the old enum spellings it fails on exactly those two values. Once source-pokeapi publishes a version past 0.3.74, the fixture falls back to the skip until it is bumped again.

• Fleet dry run. I ran the comparator over every connector in the monorepo with a published registry entry: 619 connectors, and every consecutive pair of published GA versions (4,541 pairs).

  • Impact at rollout: 8 connectors (16 connector × registry pairs) currently have a master version ahead of latest. None of them has a breaking finding, so releasing this does not turn any connector red by itself.
  • History: 17 of 4,541 consecutive releases (0.4%) would have been flagged.

    • 1 was a declared major: facebook-marketing 6.0.0 narrowed an enum. It would have been waived.

    • 16 were undeclared:

      • Removed or newly required fields: circleci 0.1.0, convertkit 0.3.0, google-tasks 0.0.71, reddit 0.0.67, ip2whois 0.2.69, sigma-computing 0.0.63, zoho-crm 0.3.0.
      • Narrowed values: zoho-campaign 0.0.69 (enum), zendesk-support 5.6.0 (num_workers minimum 1 → 2), sentry 1.0.10 (new pattern/minLength).
      • A oneOf shape change: destination-google-sheets 0.3.0.
      • Pattern rewrites: coingecko-coins, ramp, shopify. These are flagged because two regexes cannot be compared. disable_for_version is the escape.
      • Items added to arrays: asana, cimis.

      Some of these may have shipped with a config migration. Under this test they would have needed disable_for_version.

  • Before the two scoping rules above, 31 pairs were flagged. The difference was 13 OAuth consent-flow template changes and one non-protocol top-level key.

• Re-run after the review fixes, over every non-prerelease, non-yanked version in each connector's versions.json that has a per-version registry entry. This is a wider history than the 4,541 pairs above:

  Registry	Connectors	Pairs	Flagged before the fixes	Flagged after
  OSS	644	34,005	310	265
  Cloud	617	31,906	276	246

  No pair went from compatible to breaking. The pairs that are no longer flagged break down as follows:

  • root or branch additionalProperties: true added (the manifest-only migration shape)
  • $schema edits
  • airbyte_hidden removed
  • a non-standard max key
  • one oneOf branch mismatch

  facebook-marketing 6.0.0, the pattern rewrites (coingecko-coins, ramp, shopify) and zendesk-support 5.6.0 are still flagged. The fixed comparator is stricter only on keywords that occur 0 times in current specs, so the impact at rollout is unchanged: 0 connectors red.

• Re-run after the second review round (overlapping oneOf branches, patternProperties, type-strict equality), over the same history: OSS 266 of 34,005 pairs flagged (was 265), Cloud 247 of 31,906 (was 246). The one new pair is a oneOf branch that stopped requiring the field that set it apart from the other branch, so configs of the other branch without the discriminator match both. No pair went from breaking to compatible. At rollout, all 516 manifest-only connectors on master compared against the published OSS and Cloud latest (1,004 connector × registry pairs): 0 breaking.

Rollout

1. Land this PR, then release the CDK. Connector CI picks the test up from the release, so no connector PR is needed.
   There is no warn-only release: the test blocks connector PRs from the first CDK release that ships it.
2. Rollout impact measured before landing: 0 of the 516 manifest-only connectors on master is red. The two connectors with the legacy list layout of acceptance_tests.spec (source-pardot, source-netsuite) no longer crash the test.
3. A registry outage skips the test with a warning instead of failing it, so it cannot block the fleet. The skip reason says the spec was not compared.
4. For the first week after the release, watch connector PRs for failures of this test. A false positive is waived with disable_for_version in the connector, and the comparator fixed here.

Coordination with #1219

#1219 adds _run_docker_spec(connector_image, *, connector_name) beside _run_spec_in_image in docker_base.py, so whichever lands second conflicts in the import block and has two spec runners. The second one to land should fold both into one helper keyed by (image, deployment mode) that returns the raw spec, with #1219 parsing it into ConnectorSpecification.

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/spec-backward-compatibility#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/spec-backward-compatibility

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 485 tests  +215   5 473 ✅ +215   11m 42s ⏱️ +20s
    1 suites ±  0      12 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 109a6ed. ± Comparison against base commit 6ce4a40.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 488 tests  +215   5 476 ✅ +215   16m 11s ⏱️ -18s
    1 suites ±  0      12 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 109a6ed. ± Comparison against base commit 6ce4a40.

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Verdict: fix before merge — P0: 0 · P1: 2 · P2: 4 · P3: 6. Inline comments carry F1–F12.

Breaking change: BREAKING under criterion 7 only — a new mandatory standard test that ~619 non-JVM monorepo connectors adopt automatically on the next CDK release, because connector CI installs airbyte-cdk[dev] unpinned. No runtime, API, manifest, state or packaging break. This needs an unmissable call-out and a rollout plan (F6), not a major bump.

Done well

• The comparator models JSON Schema semantics rather than diffing text: keyword defaults, bound direction, type as a set with the integer → number widening, and oneOf branches matched by discriminator. Nine tightening probes all landed on the correct side.
• The waivers are self-expiring: disable_for_version must equal the published latest, and the declared-breakingChanges range handles release candidates and progressive rollouts. All 42 existing disable_for_version entries are inert today, so nothing is silently waived at rollout.
• 156 hermetic unit tests in under 6 s (local HTTP server for the retry policy, no real docker or network); 7 of 12 mutation probes caught. The failure message names the field, the three resolution paths and the exact waiver YAML.

Rollout / risk

• Fleet sweep: every manifest-only connector's manifest.yaml spec on origin/master compared against the published OSS and Cloud latest with this PR's compare_specs — 516 connectors, 0 breaking. 12 manifest-only connectors rebuilt on the current SDM base emitted specs identical to the registry. The 105 Python connectors are covered only by the author's six end-to-end runs; the two confirmed day-one reds are the F1 crashes.
• The Docker half of the new test never executes in CDK CI: the pokeapi fixture (0.3.24) is skipped by the older-version rule after a live registry fetch, and the six Check: <connector> jobs run connector unit tests, not image test. Consider bumping the fixture so the real path runs once per CDK PR.
• CI: Check: destination-motherduck is red here but also on #1217, #1218, #1219, #1221, #1223 and #1224 — infrastructure, not this diff. Skipped Check: source-google-drive did not run (bucket untouched).
• #1219 adds a different spec runner at the same place in docker_base.py; expect a conflict with whichever lands second (F10).