Skip to content

chore(demo): check the stub against the API's OpenAPI spec - #73

Merged
alukach merged 1 commit into
mainfrom
chore/check-stub
Nov 6, 2026
Merged

alukach merged 1 commit into
mainfrom
chore/check-stub

Conversation

@alukach

@alukach alukach commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Checks demo/stub.py against the API's OpenAPI spec, so the stub can't drift from the API unnoticed, and fixes the drift the check found.

uv run --with jsonschema python demo/check_stub.py            # against https://staging.source.coop/api/openapi
uv run --with jsonschema python demo/check_stub.py spec.json  # or a file

How it works

  • demo/check_stub.py starts the stub, makes a request to each of its routes (filled in from fixtures.json, including a 404 and a 409), and validates each answer against the spec's schema for that path, method and status. It fails if an answer doesn't match, or if the stub answers something the spec doesn't have. It prints one line per request.
  • CI (.github/workflows/demo-stub.yml) runs it on pull requests that change demo/, and by hand from the Actions tab. Demos are recorded in PRs that touch demo/, so that's when drift would show up in one.
  • No weekly run. GitHub sends a scheduled workflow's failure only to whoever last edited its cron line, so a weekly failure would most likely go unseen. If the API changes, the next PR that touches demo/ fails the check.
  • demo/README.md and AGENTS.md say to run it and to add a request for each new route. AGENTS.md now points at staging's spec: https://source.coop/api/openapi currently answers 500 with an empty body.

Drift it found, fixed here

Route Problem Fix
/whoami The account lacked created_at, updated_at, identity_id and metadata_public Fixtures
All product routes The embedded account lacked disabled and metadata_public. demo-user wasn't in accounts, so its products embedded a bare {"account_id"} with_account() gives the full public view; demo-user added to accounts
/data-connections details was just {"provider": "s3"}; the spec requires bucket, base_prefix and region Fixtures
GET /memberships/{id} The spec only has PATCH, and the CLI only sends PATCH Other methods get the stub's 404
/products/featured Answered a page; the spec says {"products": [...]}. The CLI uses /products?featured=true instead Route removed

Stack

On top of #72, whose RFC 9457 errors are what the 404 and 409 checks expect. Merge #72 first.

Testing

🤖 Generated with Claude Code

@alukach
alukach added this pull request to stack #74 November 6, 2026 22:44
@claude

claude Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Claude finished @alukach's task in 11s —— View job


✅ No blocking issues, safe to merge.

I read the diffs of check_stub.py, stub.py, the workflow and AGENTS.md. I didn't run anything. I didn't review the fixtures.json changes.

Non-blocking notes:

  • demo/check_stub.py:63: the response lookup only matches an exact status code or default. If the spec lists a range such as 2XX, the check reports "the spec doesn't list N" even though the stub is fine. That is correct for today's spec, so it only matters if the spec changes.
  • .github/workflows/demo-stub.yml: the check fetches the spec from staging on every run. A staging outage fails unrelated demo PRs. Re-running the job is enough in that case. Pinning a spec file would defeat the point of the check.
  • demo/stub.py: dropping GET /memberships/{id} and /products/featured matches the spec and the CLI, which only sends PATCH and /products?featured=true.

Simplify (ponytail)

  • demo/check_stub.py:58: pointer() is only a JSON Pointer escape. This is fine, but if it grows, use a library rather than extending it.
  • Nothing else to cut. A single script plus one job is already minimal.

💰 Estimated review cost: $0.12 · 0m11s · 4 turns

Base automatically changed from fix/demo-stub to main November 6, 2026 22:44
demo/check_stub.py starts the stub, requests each of its routes and
validates the answers against staging's OpenAPI spec. CI runs it on PRs
that change demo/.

It found drift, fixed here: accounts lacked fields the spec requires
(and demo-user wasn't in accounts, so its products embedded a bare ID),
data connection details lacked bucket, base_prefix and region, and the
stub answered GET /memberships/{id} and /products/featured, which the
CLI doesn't use (the latter in a shape the API doesn't).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alukach
alukach merged commit 6b169f6 into main Nov 6, 2026
11 checks passed
@alukach
alukach deleted the chore/check-stub branch November 6, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant