Skip to content

test: hold the Postman collection to the OpenAPI spec - #275

Merged
dmccoystephenson merged 1 commit into
mainfrom
feature/postman-spec-drift-guard
Nov 20, 2026
Merged

dmccoystephenson merged 1 commit into
mainfrom
feature/postman-spec-drift-guard

Conversation

@dmccoystephenson

Copy link
Copy Markdown
Member

Summary

  • postman/Viron.postman_collection.json describes itself as "generated from docs/openapi/viron-api.json", but nothing regenerated or checked it, so it could drift silently. OpenApiSpecDriftTest already holds the spec to the controllers; the new PostmanCollectionDriftTest holds the collection to the spec, closing the chain.
  • The test (plain JUnit, no Spring context) fails the build when:
    • the collection's VERB /path set differs from the spec's operations (reported as missing= / undocumented=), or a route appears twice;
    • a request's name or url.raw disagrees with the method, path and query it actually sends;
    • a query parameter is sent that the operation does not declare, or a required one is omitted;
    • a body is sent where none is declared (or omitted where one is), names a property outside the schema, omits a required one, or lacks Content-Type: application/json;
    • the collection-level auth is not bearer from {{authToken}}, or any request overrides it.
  • README.md's project-structure line for postman/ names the test.

No tracking issue — gap found during triage. The committed collection currently agrees with the spec on all 36 operations, so no collection change was needed.

Test plan

  • mvn -B test — 752 tests, 0 failures (5 new in PostmanCollectionDriftTest).
  • Mutation check: against a collection with a renamed path (environmentz), an undeclared body property (bogus), an undeclared query parameter (nope) and a request-level noauth override, all 5 new tests fail, each naming the offending request.
  • CI green on the PR head.
  • The Python client is untouched, so pytest is not affected by this change.

Deferred issues

This PR description was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).

🤖 Generated with Claude Code


drafted by Claude on behalf of Daniel Stephenson

The collection says it is generated from docs/openapi/viron-api.json,
but nothing regenerates or checks it. PostmanCollectionDriftTest fails
the build when the collection's operations, request names, raw URLs,
query parameters, JSON bodies or bearer auth disagree with the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dmccoystephenson

Copy link
Copy Markdown
Member Author

Self-review rubric (head 7f81bb7):

  • Scope: PASS — two files: the new src/test/java/preponderous/viron/PostmanCollectionDriftTest.java and the one README line describing postman/. No production code, spec or collection change.
  • Tests-new: PASS — the PR adds no public production method; the five new tests ran in CI (Tests run: 5, Failures: 0 in the build job log).
  • Tests-fix: N/A (no bug fix). In its place, a mutation check: against a collection with a renamed path, an undeclared body property, an undeclared query parameter and a request-level noauth override, all five tests failed locally, each naming the offending request.
  • Sibling structure: PASS — placed beside OpenApiSpecDriftTest, same licence header, Jackson JsonNode reading of the checked-in files, and TreeSet problem lists asserted against Set.of() as in its siblings.
  • Sibling renames: N/A — nothing renamed.
  • Docs: PASS — README's postman/ line names the guard. docs/MVP.md, docs/PLANNING.md and tickets.md refer to Postman only as a manual testing aid, which remains accurate. Spec and collection unchanged.
  • Issue resolution: N/A — no Closes; the gap was found during triage and is recorded as such in the PR body.
  • CI: PASS — build, python-client (3.8) and python-client (3.12) green on the head.
  • DTO boundary / Spec alignment / Java-Python parallelism / Override correctness: N/A — no controller, endpoint, model or client change.

Judgment notes (non-blocking):

  • src/test/java/preponderous/viron/PostmanCollectionDriftTest.java:76 — requestNamesAndRawUrlsAgreeWithTheRequestSent makes the VERB /path naming convention a hard rule. Every one of the 36 current requests follows it, and it is what makes a renamed path visible in Postman's sidebar, but a contributor renaming a request for readability would now see a failure.
  • operationOf yields a missing node for an undocumented route, so such a route also surfaces in the body/query checks. That double report only occurs alongside the route-set failure, which names the cause.

Do-not-auto-merge paths: none matched (git diff --name-only lists only the test file and README.md; no file loses more than 50 lines).

This review comment was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

@dmccoystephenson
dmccoystephenson merged commit c7b2c72 into main Nov 20, 2026
3 checks passed
@dmccoystephenson
dmccoystephenson deleted the feature/postman-spec-drift-guard branch November 20, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant