diff --git a/.claude/skills/stackql-provider-development/SKILL.md b/.claude/skills/stackql-provider-development/SKILL.md index 10c57c8..13a7f5f 100644 --- a/.claude/skills/stackql-provider-development/SKILL.md +++ b/.claude/skills/stackql-provider-development/SKILL.md @@ -20,7 +20,7 @@ The detail lives in `references/`; `scripts/` holds two generic helpers: - **`all_services.csv` is the contract.** Committed and diffed on every regeneration; a method moving resource or a resource renamed is a breaking change to review, not noise. - **Environment.** Linux, macOS or WSL (on Windows prefer WSL and the latest `stackql` release on its PATH). GNU make + bash. `stackql` resolves as `$STACKQL`, `./stackql`, then PATH. - **Never touch production accounts.** Live tests run against a dedicated dev tenant, pace under the vendor's rate limit, name everything `stackql-smoke-`, sweep breadcrumbs first, restore anything toggled. -- **Writing.** Applies to every piece of text the build generates, whoever or whatever reads it: README, docs headers and examples, NOTES, CLAUDE.md, code comments, commit messages, console output, inventory and CSV columns, mock fixtures, hand-over notes. No em dashes (use `-`), no `--` as a dash, no characters that are not on a QWERTY keyboard (no emoji, no typographic quotes, no arrows other than `->`), no stacked headings, measured copy, no hyperbole. Text the vendor owns - operation summaries, schema and parameter descriptions, examples carried from the upstream spec into the snapshot, `provider-dev/source`, the generated provider and `website/docs` - is passed through as the vendor wrote it; the only alteration is the MDX/JSX escaping `sanitize-docs` applies so the docs site builds. +- **Writing.** Applies to every piece of text the build generates, whoever or whatever reads it: README, docs headers and examples, NOTES, AGENTS.md, code comments, commit messages, console output, inventory and CSV columns, mock fixtures, hand-over notes. No em dashes (use `-`), no `--` as a dash, no characters that are not on a QWERTY keyboard (no emoji, no typographic quotes, no arrows other than `->`), no stacked headings, measured copy, no hyperbole. Text the vendor owns - operation summaries, schema and parameter descriptions, examples carried from the upstream spec into the snapshot, `provider-dev/source`, the generated provider and `website/docs` - is passed through as the vendor wrote it; the only alteration is the MDX/JSX escaping `sanitize-docs` applies so the docs site builds. ## Workflow @@ -35,7 +35,7 @@ Work the steps in order. **Do not execute a step without having read its referen 7. **Wrap it in the Makefile** - [references/repo-layout-and-makefile.md](references/repo-layout-and-makefile.md). `make help` lists every step; `make build`, `make test`, `make all` need no credentials; `make smoke*` sources `.env`. 8. **Test in three credential-free layers, then live** - [references/testing.md](references/testing.md). Offline `SHOW`/`DESCRIBE`, a mock API with row-level and wire-level assertions, the meta-route walk; then the pystackql smoke suite with `--live` and a budget. 9. **Docs and CI** - [references/docs-and-ci.md](references/docs-and-ci.md). Docusaurus 3.10 microsite on the shared config with `showLastUpdateTime`, a getting-started page that leads with the queries the provider exists for, `sanitize-docs`, and the build-and-test / spec-drift workflows. -10. **Record and hand over** - [references/handover.md](references/handover.md). CLAUDE.md (settled decisions), NOTES.md (findings with evidence, plus the open questions the first live run must answer), README.md (numbered build guide with counts that match the artifacts), `.env.example`. +10. **Record and hand over** - [references/handover.md](references/handover.md). AGENTS.md (settled decisions; `CLAUDE.md` and `.github/copilot-instructions.md` are short entry points to it), NOTES.md (findings with evidence, plus the open questions the first live run must answer), README.md (numbered build guide with counts that match the artifacts), `.env.example`. For an existing provider, start from [references/uplift-checklist.md](references/uplift-checklist.md) instead of step 1; it routes back into the steps above. @@ -54,4 +54,4 @@ Verified against stackql v0.10.605 / any-sdk v0.5.4-alpha01 (2026-08). These cha ## Definition of done -`make all` green from a clean checkout (deps, pipeline, offline + integration + meta-route tests, docs, site build); `all_services.csv`, `provider-dev/source`, `provider-dev/openapi` and `website/docs` committed; CLAUDE.md, NOTES.md, README.md and `.env.example` current; the smoke suite runnable with `make smoke` / `make smoke-live` by someone with credentials, with its open questions listed. Publishing is the next skill. +`make all` green from a clean checkout (deps, pipeline, offline + integration + meta-route tests, docs, site build); `all_services.csv`, `provider-dev/source`, `provider-dev/openapi` and `website/docs` committed; AGENTS.md, NOTES.md, README.md and `.env.example` current; the smoke suite runnable with `make smoke` / `make smoke-live` by someone with credentials, with its open questions listed. Publishing is the next skill. diff --git a/.claude/skills/stackql-provider-development/references/handover.md b/.claude/skills/stackql-provider-development/references/handover.md index 7f03aa0..15e0d51 100644 --- a/.claude/skills/stackql-provider-development/references/handover.md +++ b/.claude/skills/stackql-provider-development/references/handover.md @@ -2,7 +2,7 @@ ## What to record where -- **CLAUDE.md** - the settled decisions (archetype, scoping variable, auth, the flagship mapping, skip codes, casing policy), the toolchain rules, where the findings live, the non-negotiables. Short; it is read every session. +- **AGENTS.md** - the settled decisions (archetype, scoping variable, auth, the flagship mapping, skip codes, casing policy), the toolchain rules, where the findings live, the non-negotiables. Short; it is read every session. It is the one tool-neutral guidance file: Codex reads it directly, `CLAUDE.md` imports it for Claude Code (an `@AGENTS.md` line) and `.github/copilot-instructions.md` points GitHub Copilot to it. Project rules go in AGENTS.md, never in the two entry points; those carry only what applies to their tool alone. - **NOTES.md** - numbered findings with evidence (what was measured, against what, what was decided and why), a Blockers list of what only a live run can establish, and the testing requirements. It is the memory of the build; a future refresh reads it before touching a rule. - **README.md** - the numbered build guide (steps 0-8: pin, inventory, split, mappings, normalize, generate, test, publish, docs) with counts that match the committed artifacts, the Makefile targets, credentials, and the CI description. - **.env.example** - every live-test variable with a one-line comment; never a real value. diff --git a/.claude/skills/stackql-provider-development/references/reference-repos.md b/.claude/skills/stackql-provider-development/references/reference-repos.md index 3458c0d..7c94cc9 100644 --- a/.claude/skills/stackql-provider-development/references/reference-repos.md +++ b/.claude/skills/stackql-provider-development/references/reference-repos.md @@ -1,6 +1,6 @@ # Reference repositories -Clone these next to the provider repo (shallow clones are fine) and grep them for any extension key to find a real, shipped example. Sibling build repos carry a NOTES.md (or CLAUDE.md) of findings - reuse them, do not re-derive. +Clone these next to the provider repo (shallow clones are fine) and grep them for any extension key to find a real, shipped example. Sibling build repos carry a NOTES.md (or AGENTS.md / CLAUDE.md) of findings - reuse them, do not re-derive. | Repo | What to read | |---|---| diff --git a/.claude/skills/stackql-provider-development/references/repo-layout-and-makefile.md b/.claude/skills/stackql-provider-development/references/repo-layout-and-makefile.md index 3cfc9af..eadfe7d 100644 --- a/.claude/skills/stackql-provider-development/references/repo-layout-and-makefile.md +++ b/.claude/skills/stackql-provider-development/references/repo-layout-and-makefile.md @@ -18,11 +18,13 @@ tests/ smoke_test.py website/ # Docusaurus microsite .github/workflows/ # build-and-test.yml, prod-web-deploy.yml, test-web-deploy.yml -CLAUDE.md NOTES.md README.md SECURITY.md LICENSE .env.example +.github/copilot-instructions.md # GitHub Copilot entry point to AGENTS.md +AGENTS.md CLAUDE.md # the shared agent guidance and the Claude Code entry point that imports it +NOTES.md README.md SECURITY.md LICENSE .env.example ``` Makefile targets (bash shell, `.DEFAULT_GOAL := help`, `##` help comments): `deps`, `fetch-spec` (verify pin), `refresh-spec` (`--update`), `inventory`, `split`, `mappings` (rm CSV + analyze + map), `pre-normalize`, `normalize`, `generate` (rm output + generate + post-process), `post-process`, `build` (the chain), `test-offline`, `test-integration`, `test-meta` (start server; run; stop; preserve exit status), `test`, `venv`, `smoke`, `smoke-live`, `smoke-read-only`, `smoke-`, `smoke-cleanup`, `docs` (passes `--snake-case-aliases` and `--source-project $(SOURCE_PROJECT)`, the provider repository URL, `?=` so it can be overridden), `website`, `website-start`, `clean`, `all` = `deps build test docs website`. `make all` never needs credentials and never bills. The smoke targets source `.env` when present (`set -a; source <(tr -d '\r' < .env); set +a`). package.json scripts wrap the two CLI entry points through `node ./node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs ` and `docgen-utils.mjs generate-docs` (never `.bin` shims); pass flags after `--`. -CLAUDE.md records the settled decisions and where the findings live; NOTES.md records the findings with evidence; README.md is the numbered build guide (steps 0-8) with counts that match the artifacts. +AGENTS.md records the settled decisions and where the findings live (`CLAUDE.md` and `.github/copilot-instructions.md` are short entry points to it, see handover.md); NOTES.md records the findings with evidence; README.md is the numbered build guide (steps 0-8) with counts that match the artifacts. diff --git a/.claude/skills/stackql-provider-development/references/uplift-checklist.md b/.claude/skills/stackql-provider-development/references/uplift-checklist.md index 4b2c109..213810c 100644 --- a/.claude/skills/stackql-provider-development/references/uplift-checklist.md +++ b/.claude/skills/stackql-provider-development/references/uplift-checklist.md @@ -6,5 +6,5 @@ 4. Add what is missing against this list: env-var scoping, naive bodies, snake surface, pagination, pushdown, transforms, objectKeys, lifecycle EXECs attached to their resources, views, GraphQL merge, skip codes for what is not mappable. 5. Convert ad-hoc scripts into Makefile targets; add `make all`. 6. Write or extend the three test layers until they cover every archetype present; add the smoke suite with `--live`. -7. Refresh CLAUDE.md, NOTES.md, README.md, `.env.example`, the docs headers; bring `website/sidebars.js` and the `.sidebar-back-link` rules in `website/src/css/global.css` in line with the template (the `Back to StackQL Docs` link, see docs-and-ci.md); `make docs && make website`. +7. Refresh AGENTS.md (moving the guidance out of an existing CLAUDE.md and leaving `CLAUDE.md` and `.github/copilot-instructions.md` as the short entry points, see handover.md), NOTES.md, README.md, `.env.example`, the docs headers; bring `website/sidebars.js` and the `.sidebar-back-link` rules in `website/src/css/global.css` in line with the template (the `Back to StackQL Docs` link, see docs-and-ci.md); `make docs && make website`. 8. `make all` green from a clean checkout before handing over. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..195579e --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,3 @@ +# GitHub Copilot instructions + +GitHub Copilot entry point. The project guidance is in [AGENTS.md](../AGENTS.md) at the repository root; read it before starting any task in this repository and follow it. Project rules are maintained there, not here - add to this file only what applies to GitHub Copilot alone. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..77db790 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,106 @@ +# AGENTS.md + +The project guidance for every coding agent and contributor working in this repository, and the only place project rules are maintained. Codex reads this file directly; `CLAUDE.md` (Claude Code) and `.github/copilot-instructions.md` (GitHub Copilot) are short entry points that load or point to it and carry nothing else unless it applies to that tool alone. + +## Project + +This repository builds and documents the `myprovider` provider for [StackQL](https://github.com/stackql/stackql), enabling SQL-based query and provisioning operations against the My Provider API. + +TODO(template): replace the paragraph above with what the provider covers (the API surface, the services in plain words) and the scope notes - what is deliberately out of scope and why, and any sibling provider reserved for the other surface - so they are never relitigated. + +## How to work in this repository + +The build procedure is the `stackql-provider-development` skill in `.claude/skills/stackql-provider-development/` (`SKILL.md` is the map; `references/*.md` are the procedures; `scripts/` holds `spec_diff.mjs` and `find_extension_examples.sh`). Read the reference file for a step before executing it. A tool that does not load skills from `.claude/skills/` opens `SKILL.md` and the reference files as plain files; the procedure is the same. Work the steps in order: archetype -> spec pin -> inventory and split -> mappings -> pre-normalize / normalize / generate / post-process -> GraphQL and views -> Makefile -> the three credential-free test layers -> smoke suite -> docs and CI -> hand-over. + +Starting from a fresh clone of the template: `bin/init-provider.sh "" <api-base>` rewrites the placeholders (`myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com`), then fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `SCOPE_PREFIX`, `ROOT_PATHS`, `WIRE_CASING`) and `provider-dev/config/*.json`, confirm `SOURCE_PROJECT` in the Makefile (the repository URL docgen links from the docs landing page; the default is the `stackql-registry/stackql-provider-<name>` convention), and `make fetch-spec`. Every file carrying a `TODO(template)` marker is a decision the build must make; `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is left. + +## Archetype + +TODO(template): direct (the vendor publishes OpenAPI / Swagger / a discovery document - this template's pipeline) or derived (the SDK is the truth; the provider is built in a fork of the SDK repository - see the skill's derived-spec-pipeline.md, which reuses everything here from the inventory step on). Record the spec source URL, how it is served (versioned or not, authenticated or not) and the refresh cadence. + +## Design decisions (settled - see NOTES.md for the evidence) + +TODO(template): one bullet each, as they are decided: + +- **Authentication** - the `auth.type` and the env var(s), named as the vendor's Terraform provider names them (unless Terraform only offers `TF_VAR_*`, then the vendor CLI's). Fixed API base or a server template. +- **Scoping** - the server variable and its `x-stackQL-envVar` (or "none - fixed host"); which paths are root paths pinned to the API base; the JOIN-cannot-fan-out consequence and the `UNION ALL` pattern the docs teach. +- **The flagship mapping** - the capability the provider exists for and how it is bound. +- **Casing** - `snake_case_aliases` + `request.nativeCasing` when the wire is camelCase; the accepted column-rename break. +- **Pagination** - the vendor scheme per endpoint (confirmed in the inventory), placed at the narrowest true level; what is not expressible and is left as parameter-driven windowing. +- **Update semantics** - PATCH presumed partial; PUT is not REPLACE until proven live. +- **Skip codes** - the reason codes in use and their counts. +- **Rate limit** - the documented limit, the harness pacing constant; a 429 in CI is a harness bug. +- **Labels** - how beta / alpha / deprecated flow into the docs. +- **Engine typing** - UPDATE marshals every SET value as a string; INSERT and EXEC send typed JSON; EXEC cannot carry a boolean. Documented, not worked around. + +## Toolchain rules + +- Latest `@stackql/provider-utils`, `@stackql/pgwire-lite` and `@apidevtools/swagger-parser` (check npm before starting: `npm view @stackql/provider-utils version`); Docusaurus `^3.10.x`. Node >= 22.19 (Node 20 reached end of life in April 2026; swagger-parser 13 needs 22.19), `type: module`. `js-yaml` stays on 4.x: provider-utils emits with js-yaml 4, and v5 drops the default export, removes the `quotingType` dump option and changes scalar quoting, so that bump is a coordinated change with provider-utils. `package-lock.json` pins what CI installs; a toolchain bump is an explicit commit that regenerates the artifacts. +- Linux, macOS or WSL: GNU make + bash, a `stackql` binary (`$STACKQL`, `./stackql`, then PATH; `bin/start-server.sh` downloads one if none is found), Python 3 and yarn. The Makefile is the operator surface (`make help`). +- The two provider-utils CLI entry points are npm scripts invoked through `node` (never `.bin` shims); flags go after `--`. + +## Repository layout + +``` +Makefile # make help / build / test / smoke* / docs / website / all +bin/ # fetch-spec.sh, split.mjs, init-provider.sh, server lifecycle, test-meta-routes.cjs +provider-dev/ + downloaded/ # pinned spec snapshot (committed) + config/ # spec_pin.json, service_names.json, servers.json, provider_config.json, + # endpoint_inventory.csv, all_services.csv (the mapping contract) + scripts/ # record_spec_pin, build_inventory, map_operations, pre_normalize, post_process, + # graphql_merge, lib/spec_helpers (the single source of provider constants) + source/ # split + normalized service specs (committed build artifacts) + source-graphql/ # manifest.yaml + ops/*.yaml GraphQL fragments (optional) + openapi/src/<name>/ # generated provider (committed) + docgen/provider-data # headerContent1.txt / headerContent2.txt (landing + getting-started page; + # examples close the page under "## Example Queries" - intro sentence, then + # H3 + lead-in + sql block per example, see docs-and-ci.md) +views/<service>/views.yaml # optional provider views (spliced at generate time) +tests/ + offline_validation.mjs + integration/ # mock_<name>_server.mjs, run_integration_tests.mjs, probe.mjs + smoke_test.py # pystackql live suite (--live, --read-only, --with-gated-lifecycle, --cleanup-only) +website/ # Docusaurus microsite (shared stackql/docusaurus-config vendored at build) +.github/workflows/ # build-and-test.yml (pin check, build, drift check, 3 test layers, gated smoke, + # weekly spec-drift); web deploys (.disabled until GitHub Pages is configured) +.github/copilot-instructions.md # GitHub Copilot entry point to this file +AGENTS.md CLAUDE.md # this file (the shared guidance) and the Claude Code entry point that imports it +NOTES.md README.md SECURITY.md LICENSE .env.example +``` + +## Build pipeline + +`make all` = `deps build test docs website`: fetch-spec (pin verify) -> inventory -> split -> mappings -> pre-normalize -> normalize -> generate (+ post-process, views, GraphQL merge) -> test-offline -> test-integration -> test-meta -> docs -> website. It never needs credentials and never bills. Every step is deterministic and re-runnable; manual decisions are rules in scripts (`RESOURCE_RULES` / `METHOD_RULES` in `map_operations.mjs`, `SKIP_RULES` in `lib/spec_helpers.mjs`, fix classes in `record_spec_pin.mjs`, passes in `pre_normalize.mjs`, numbered rules in `post_process.mjs`), never hand-edits to CSVs or specs. Validate-and-fail-without-writing is the standard for every script. A regeneration reproduces the committed artifacts byte-for-byte (CI checks this). + +`provider-dev/config/all_services.csv` is committed as the durable record of every operation -> resource.method mapping; a diff there on a regeneration is a breaking-change review (a method moving resource, a resource renamed), not noise. + +## Tests + +1. `make test-offline` - `SHOW` / `DESCRIBE` against the local file registry (services, resources, verbs, env-var behaviour, snake aliases, views, the flagship). +2. `make test-integration` - the mock API (`tests/integration/mock_<name>_server.mjs`, real wire shapes, auth enforced, request log) with row-level and wire-level assertions per archetype. `npm run probe -- "<sql>"` prints stackql output and the wire calls for ad-hoc binding checks. +3. `make test-meta` - the meta-route walk over a local server. A resource with no columns or a service with no resources fails it - fix the mapping, never the test. +4. `make smoke` / `make smoke-live` / `make smoke-read-only` / `make smoke-gated-lifecycle` / `make smoke-cleanup` - live, from `.env` (see `.env.example`), against a dedicated dev account. Budget under $5 (aim under $1); the expensive lifecycle only in its gated target. + +Never run tests against a production account. + +## Docs and publish + +`make docs` (generate with `--snake-case-aliases` and `--source-project $(SOURCE_PROJECT)`, then `sanitize-docs`) and `make website`; `website/docs` is committed after every regeneration so pages stamp with their regeneration date. Publishing to the registry is a separate, human-in-the-loop step (push the generated provider dir to `providers/src` in a feature branch of `stackql-provider-registry`, follow the release flow, verify with `make smoke-live`). + +## Writing conventions + +These apply to every piece of text the build generates, whoever or whatever reads it: README, docs headers and examples, NOTES, this file, code comments, commit messages, console output, inventory and CSV columns, mock fixtures, hand-over notes. Text the vendor owns (operation summaries, schema and parameter descriptions, examples carried from the upstream spec into the snapshot, `provider-dev/source`, the generated provider and `website/docs`) is passed through as the vendor wrote it; the only alteration is the MDX escaping `sanitize-docs` applies so the docs site builds. + +- Measured, precise copy, no hyperbole. Comparisons with the vendor's Terraform provider are capability statements and runnable examples, never editorializing. +- No em dashes and no `--` as a dash; use `-`. No characters that are not on a QWERTY keyboard: no emoji, no typographic quotes, `->` for arrows. No stacked headings. +- Sample queries are realistic and runnable against the generated surface: snake_case names, `json_extract` for nested fields, parser keywords such as `"database"` quoted. + +## Non-negotiables + +1. Latest `@stackql/provider-utils`, always +2. The skill is the procedure; sibling-build NOTES.md findings (the repos in the skill's reference-repos.md) are reused, not re-derived - deviate only with a documented reason in NOTES.md +3. Deterministic scripts, never hand-edits to derived artifacts +4. Every regeneration is followed by `make test` before commit +5. Test harnesses pace under the rate limit - a 429 in CI is a harness bug +6. Smoke tests clean up everything they create and restore anything they toggle diff --git a/CLAUDE.md b/CLAUDE.md index a4d5170..b483c5a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,102 +1,5 @@ # CLAUDE.md -## Project +Claude Code entry point. The project guidance is in [AGENTS.md](AGENTS.md), imported by the line below; read it and follow it. Project rules are maintained there, not here - add to this file only what applies to Claude Code alone. -This repository builds and documents the `myprovider` provider for [StackQL](https://github.com/stackql/stackql), enabling SQL-based query and provisioning operations against the My Provider API. - -TODO(template): replace the paragraph above with what the provider covers (the API surface, the services in plain words) and the scope notes - what is deliberately out of scope and why, and any sibling provider reserved for the other surface - so they are never relitigated. - -## How to work in this repository - -The build procedure is the `stackql-provider-development` skill in `.claude/skills/stackql-provider-development/` (`SKILL.md` is the map; `references/*.md` are the procedures; `scripts/` holds `spec_diff.mjs` and `find_extension_examples.sh`). Read the reference file for a step before executing it. Work the steps in order: archetype -> spec pin -> inventory and split -> mappings -> pre-normalize / normalize / generate / post-process -> GraphQL and views -> Makefile -> the three credential-free test layers -> smoke suite -> docs and CI -> hand-over. - -Starting from a fresh clone of the template: `bin/init-provider.sh <name> "<Title>" <api-base>` rewrites the placeholders (`myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com`), then fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `SCOPE_PREFIX`, `ROOT_PATHS`, `WIRE_CASING`) and `provider-dev/config/*.json`, confirm `SOURCE_PROJECT` in the Makefile (the repository URL docgen links from the docs landing page; the default is the `stackql-registry/stackql-provider-<name>` convention), and `make fetch-spec`. Every file carrying a `TODO(template)` marker is a decision the build must make; `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is left. - -## Archetype - -TODO(template): direct (the vendor publishes OpenAPI / Swagger / a discovery document - this template's pipeline) or derived (the SDK is the truth; the provider is built in a fork of the SDK repository - see the skill's derived-spec-pipeline.md, which reuses everything here from the inventory step on). Record the spec source URL, how it is served (versioned or not, authenticated or not) and the refresh cadence. - -## Design decisions (settled - see NOTES.md for the evidence) - -TODO(template): one bullet each, as they are decided: - -- **Authentication** - the `auth.type` and the env var(s), named as the vendor's Terraform provider names them (unless Terraform only offers `TF_VAR_*`, then the vendor CLI's). Fixed API base or a server template. -- **Scoping** - the server variable and its `x-stackQL-envVar` (or "none - fixed host"); which paths are root paths pinned to the API base; the JOIN-cannot-fan-out consequence and the `UNION ALL` pattern the docs teach. -- **The flagship mapping** - the capability the provider exists for and how it is bound. -- **Casing** - `snake_case_aliases` + `request.nativeCasing` when the wire is camelCase; the accepted column-rename break. -- **Pagination** - the vendor scheme per endpoint (confirmed in the inventory), placed at the narrowest true level; what is not expressible and is left as parameter-driven windowing. -- **Update semantics** - PATCH presumed partial; PUT is not REPLACE until proven live. -- **Skip codes** - the reason codes in use and their counts. -- **Rate limit** - the documented limit, the harness pacing constant; a 429 in CI is a harness bug. -- **Labels** - how beta / alpha / deprecated flow into the docs. -- **Engine typing** - UPDATE marshals every SET value as a string; INSERT and EXEC send typed JSON; EXEC cannot carry a boolean. Documented, not worked around. - -## Toolchain rules - -- Latest `@stackql/provider-utils`, `@stackql/pgwire-lite` and `@apidevtools/swagger-parser` (check npm before starting: `npm view @stackql/provider-utils version`); Docusaurus `^3.10.x`. Node >= 22.19 (Node 20 reached end of life in April 2026; swagger-parser 13 needs 22.19), `type: module`. `js-yaml` stays on 4.x: provider-utils emits with js-yaml 4, and v5 drops the default export, removes the `quotingType` dump option and changes scalar quoting, so that bump is a coordinated change with provider-utils. `package-lock.json` pins what CI installs; a toolchain bump is an explicit commit that regenerates the artifacts. -- Linux, macOS or WSL: GNU make + bash, a `stackql` binary (`$STACKQL`, `./stackql`, then PATH; `bin/start-server.sh` downloads one if none is found), Python 3 and yarn. The Makefile is the operator surface (`make help`). -- The two provider-utils CLI entry points are npm scripts invoked through `node` (never `.bin` shims); flags go after `--`. - -## Repository layout - -``` -Makefile # make help / build / test / smoke* / docs / website / all -bin/ # fetch-spec.sh, split.mjs, init-provider.sh, server lifecycle, test-meta-routes.cjs -provider-dev/ - downloaded/ # pinned spec snapshot (committed) - config/ # spec_pin.json, service_names.json, servers.json, provider_config.json, - # endpoint_inventory.csv, all_services.csv (the mapping contract) - scripts/ # record_spec_pin, build_inventory, map_operations, pre_normalize, post_process, - # graphql_merge, lib/spec_helpers (the single source of provider constants) - source/ # split + normalized service specs (committed build artifacts) - source-graphql/ # manifest.yaml + ops/*.yaml GraphQL fragments (optional) - openapi/src/<name>/ # generated provider (committed) - docgen/provider-data # headerContent1.txt / headerContent2.txt (landing + getting-started page; - # examples close the page under "## Example Queries" - intro sentence, then - # H3 + lead-in + sql block per example, see docs-and-ci.md) -views/<service>/views.yaml # optional provider views (spliced at generate time) -tests/ - offline_validation.mjs - integration/ # mock_<name>_server.mjs, run_integration_tests.mjs, probe.mjs - smoke_test.py # pystackql live suite (--live, --read-only, --with-gated-lifecycle, --cleanup-only) -website/ # Docusaurus microsite (shared stackql/docusaurus-config vendored at build) -.github/workflows/ # build-and-test.yml (pin check, build, drift check, 3 test layers, gated smoke, - # weekly spec-drift); web deploys (.disabled until GitHub Pages is configured) -CLAUDE.md NOTES.md README.md SECURITY.md LICENSE .env.example -``` - -## Build pipeline - -`make all` = `deps build test docs website`: fetch-spec (pin verify) -> inventory -> split -> mappings -> pre-normalize -> normalize -> generate (+ post-process, views, GraphQL merge) -> test-offline -> test-integration -> test-meta -> docs -> website. It never needs credentials and never bills. Every step is deterministic and re-runnable; manual decisions are rules in scripts (`RESOURCE_RULES` / `METHOD_RULES` in `map_operations.mjs`, `SKIP_RULES` in `lib/spec_helpers.mjs`, fix classes in `record_spec_pin.mjs`, passes in `pre_normalize.mjs`, numbered rules in `post_process.mjs`), never hand-edits to CSVs or specs. Validate-and-fail-without-writing is the standard for every script. A regeneration reproduces the committed artifacts byte-for-byte (CI checks this). - -`provider-dev/config/all_services.csv` is committed as the durable record of every operation -> resource.method mapping; a diff there on a regeneration is a breaking-change review (a method moving resource, a resource renamed), not noise. - -## Tests - -1. `make test-offline` - `SHOW` / `DESCRIBE` against the local file registry (services, resources, verbs, env-var behaviour, snake aliases, views, the flagship). -2. `make test-integration` - the mock API (`tests/integration/mock_<name>_server.mjs`, real wire shapes, auth enforced, request log) with row-level and wire-level assertions per archetype. `npm run probe -- "<sql>"` prints stackql output and the wire calls for ad-hoc binding checks. -3. `make test-meta` - the meta-route walk over a local server. A resource with no columns or a service with no resources fails it - fix the mapping, never the test. -4. `make smoke` / `make smoke-live` / `make smoke-read-only` / `make smoke-gated-lifecycle` / `make smoke-cleanup` - live, from `.env` (see `.env.example`), against a dedicated dev account. Budget under $5 (aim under $1); the expensive lifecycle only in its gated target. - -Never run tests against a production account. - -## Docs and publish - -`make docs` (generate with `--snake-case-aliases` and `--source-project $(SOURCE_PROJECT)`, then `sanitize-docs`) and `make website`; `website/docs` is committed after every regeneration so pages stamp with their regeneration date. Publishing to the registry is a separate, human-in-the-loop step (push the generated provider dir to `providers/src` in a feature branch of `stackql-provider-registry`, follow the release flow, verify with `make smoke-live`). - -## Writing conventions - -These apply to every piece of text the build generates, whoever or whatever reads it: README, docs headers and examples, NOTES, this file, code comments, commit messages, console output, inventory and CSV columns, mock fixtures, hand-over notes. Text the vendor owns (operation summaries, schema and parameter descriptions, examples carried from the upstream spec into the snapshot, `provider-dev/source`, the generated provider and `website/docs`) is passed through as the vendor wrote it; the only alteration is the MDX escaping `sanitize-docs` applies so the docs site builds. - -- Measured, precise copy, no hyperbole. Comparisons with the vendor's Terraform provider are capability statements and runnable examples, never editorializing. -- No em dashes and no `--` as a dash; use `-`. No characters that are not on a QWERTY keyboard: no emoji, no typographic quotes, `->` for arrows. No stacked headings. -- Sample queries are realistic and runnable against the generated surface: snake_case names, `json_extract` for nested fields, parser keywords such as `"database"` quoted. - -## Non-negotiables - -1. Latest `@stackql/provider-utils`, always -2. The skill is the procedure; sibling-build NOTES.md findings (the repos in the skill's reference-repos.md) are reused, not re-derived - deviate only with a documented reason in NOTES.md -3. Deterministic scripts, never hand-edits to derived artifacts -4. Every regeneration is followed by `make test` before commit -5. Test harnesses pace under the rate limit - a 429 in CI is a harness bug -6. Smoke tests clean up everything they create and restore anything they toggle +@AGENTS.md diff --git a/README.md b/README.md index 803110b..1a83381 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ A template repository for building a [StackQL](https://github.com/stackql/stackql) provider to the current standard: a deterministic `make` pipeline from a pinned upstream spec to a generated provider, every any-sdk primitive available where the API warrants it (snake_case surface, request and response transforms, pagination, query-parameter pushdown, `x-stackQL-envVar` scoping, `objectKey`, lifecycle `EXEC` methods, GraphQL merge, provider views), three credential-free test layers, a budgeted live smoke suite, a Docusaurus microsite and CI. -The procedure is the bundled Claude Code skill, [`.claude/skills/stackql-provider-development`](.claude/skills/stackql-provider-development/SKILL.md). The repository is laid out so a Claude Code session (or a person) can complete it for a given provider by working the skill's steps in order; every decision left open is marked `TODO(template)`. The skill also uplifts an existing provider repository - copy the `.claude/skills` directory there and start from its `references/uplift-checklist.md`. +The procedure is the bundled Claude Code skill, [`.claude/skills/stackql-provider-development`](.claude/skills/stackql-provider-development/SKILL.md). The repository is laid out so a coding agent session (or a person) can complete it for a given provider by working the skill's steps in order; every decision left open is marked `TODO(template)`. The skill also uplifts an existing provider repository - copy the `.claude/skills` directory there and start from its `references/uplift-checklist.md`. The rest of this file is the shape of the finished provider README (numbered build guide, steps 0-8, with counts that match the committed artifacts). Replace the sections as the build settles them. @@ -11,7 +11,7 @@ The rest of this file is the shape of the finished provider README (numbered bui 1. Create the repository from this template (GitHub "Use this template", or clone and re-init) and run `npm install`. 2. Rewrite the placeholders: `bin/init-provider.sh <name> "<Title>" [https://api.vendor.com]` replaces `myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com` and the `stackql-provider-myprovider` repository slug across the files that carry them (`grep -rn myprovider .` afterwards should find nothing outside `.claude/`). 3. Fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `PATH_VERSION_PREFIX`, `SCOPE_PREFIX` / `ROOT_PATHS` for a scoped API, `WIRE_CASING`) and `provider-dev/config/` (`servers.json`, `provider_config.json`, `service_names.json`). `SOURCE_PROJECT` in the Makefile defaults to `https://github.com/stackql-registry/stackql-provider-<name>`; set it to the repository's own URL when it lives elsewhere. -4. Open a Claude Code session in the repository and ask it to build the provider - it picks up `CLAUDE.md` and the skill. Or work the skill's steps by hand: `make fetch-spec`, `make inventory`, add service rules, `make split`, `make mappings-report` / `make mappings`, and so on to `make all`. +4. Open a coding agent session in the repository (Claude Code, GitHub Copilot or Codex) and ask it to build the provider - it picks up [AGENTS.md](AGENTS.md), the shared project guidance (Codex directly, Claude Code through `CLAUDE.md`, GitHub Copilot through `.github/copilot-instructions.md`), and the skill it points to. Or work the skill's steps by hand: `make fetch-spec`, `make inventory`, add service rules, `make split`, `make mappings-report` / `make mappings`, and so on to `make all`. 5. `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is still open. What the template is not: a publisher. Publishing to the public registry is a separate, human-in-the-loop step (section 7). diff --git a/bin/init-provider.sh b/bin/init-provider.sh index 16445c8..c66f9d2 100644 --- a/bin/init-provider.sh +++ b/bin/init-provider.sh @@ -11,7 +11,7 @@ # the Makefile SOURCE_PROJECT default, the SECURITY.md link) # # Everything else (auth type, scoping variable, service rules, mapping rules, -# spec URL) is a decision the build makes afterwards - see CLAUDE.md and the +# spec URL) is a decision the build makes afterwards - see AGENTS.md and the # stackql-provider-development skill. Review `git diff` after running. # # Usage: bin/init-provider.sh <name> "<Title>" [https://api.vendor.com] @@ -37,7 +37,7 @@ REPO_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )" cd "$REPO_ROOT" FILES=( - Makefile package.json package-lock.json README.md CLAUDE.md NOTES.md SECURITY.md .env.example + Makefile package.json package-lock.json README.md AGENTS.md NOTES.md SECURITY.md .env.example bin/fetch-spec.sh provider-dev/scripts/lib/spec_helpers.mjs provider-dev/scripts/record_spec_pin.mjs