Repository navigation
feat(migrations): retired columns and labels, step class and expand-only, migrate new --replace, foreign defaults as drift (#623) - #636
Conversation
…or, allow_ahead="expand-only" (#626) (#632) Every step has a step class, expand or contract, decided once in the core from the planner's op verdicts and recorded in the step record at apply time. `require_applied(allow_ahead="expand-only")`, `up(...)` and `ferro migrate up --allow-ahead=expand-only` admit a database ahead only by expand steps; `DatabaseAheadError.contract_steps` names the rest. The tracking table gains an additive `minor` (format stays 1), so the upgrade rolls back to 0.22. A hand-written SQL step is expand only with the `-- ferro: expand` header. Closes #626 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…nd stays in the snapshot (#625) (#631) `Field(retired=True)`, `FerroField(retired=True)` and `ForeignKey(retired=True)` take a column out of the running model while it stays in the schema snapshot, nullable, with every artifact that names it. No statement the model issues names the column: the root fetch names the registration's mapped columns instead of `*`. Retiring a required column generates the planner's `DROP NOT NULL`; deleting the declaration next release generates the destructive `DROP COLUMN` and the orphaned enum type's `DROP TYPE`. Contradictions are refused at class creation with a `TypeError` naming the fix. Closes #625 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…627) (#633) `__ferro_retired_labels__ = frozenset({"canceled"})` on a `StrEnum` keeps the member, so rows holding the label still hydrate and save, while assigning the label at construction or by attribute set is refused with an error naming it as retired in this release. The label stays in the Postgres enum type and in any `db_check` check; retiring renders no DDL and writes no migration. The guard is a per-field validator attached only to enum fields whose enum retires a label; models without retired labels gain nothing. Also carries the `retired` field a `SchemaColumn` test literal in the step-class golden table was missing after #625 and #626 merged independently. Closes #627 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…migration carrying hand-written steps (#629) (#635) `ferro migrate new <name> --replace NNNN_<name>` regenerates the newest migration against the current newest snapshot below it: a new snapshot, regenerated DDL steps, and every data step and hand-written SQL step carried across by name with its body intact, renumbered when the regenerated steps move it. The number is kept when free and the next taken when not. The directory loader admits the duplicate at the replaced number for this verb only; `check` and the loader name both duplicates in full and print one `--replace` line each; the run's snapshot-mismatch and applied-missing refusals name a replaced migration and its revert recipe. The migrations guide gains "Two branches, one number" and documents the `-- ferro: expand` header. Closes #629 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…efaults and DropDefault runs on every door (#628) (#634) `ferro migrate drift` and `ferro migrate baseline` see column defaults. The live read carries each column's server default on both dialects, and the one planner plans a `DropDefault` op for every foreign default, a live default ferro did not leave. `drift` prints `<table>.<column> has default <literal> in the database, none declared`; `baseline` refuses it and its success sentence now says only that objects ferro does not own were not compared; the Postgres pass drops it under `migrate_destructive` and reports it; the SQLite pass warns naming the migrations door (the pass never rebuilds a table); the Alembic bridge writes `alter_column(server_default=None)` with an irreversible downgrade. Ferro's own defaults are never drift: the serial sequence a Postgres column owns, and a literal on a `NOT NULL` SQLite column. A casebook case pins every door. Closes #628 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
Review panel: PR #636 (
|
| Fix PR | Findings | Re-review |
|---|---|---|
| #637 | F6 | ADDRESSED (#637 (review)) |
| #638 | F4, F5, F8 | all ADDRESSED (#638 (review)) |
| #639 | F1, F2, F3, F7, F9, C1 | all ADDRESSED after two rounds; round 2 narrowed F2's unclassifiable refusal to up and baseline while down, rerecord and baseline --remove proceed with the class left NULL and a warning (#639 (review), #639 (review)) |
This PR's head is now 6246f83. The epic's aggregate Verify (every ticket's block, deduplicated) passed there with each command's exit status checked: maturin develop; cargo check --all-targets; cargo test -p ferro-schema-ir, -p ferro-migrate, --no-default-features --features testing (354); the IR vector contract (14); pytest --db-backends=sqlite,postgres over 31 files: 1637 passed, 33 skipped, 1 xfailed; prek run --all-files clean; tree unchanged.
Revised verdict: MERGE. No Critical, no open Important, Spec PASS, Coherence resolved, Verify green. Minors F10–F17, one wording nit on the new refusal ("none of its steps was applied" reads better as "this run applied nothing"), the Candidate ADRs and the terminology drift go to close-epic's Learnings. The merge to main is the maintainer's.
Deepening pass (2026-10-12)
After the fix round, improve-pr-architecture ran over this PR and the maintainer asked for its recommended candidates to be implemented and merged here rather than as follow-ups. Two refactor PRs, each gated by review-pr (Spec PASS, Quality PASS, no Important findings) and checked to merge cleanly together:
| PR | What | Review |
|---|---|---|
| #641 | One step_class entry point deciding from the laid-out step, with constrains derived from the step's own ops through the staging code's own function; Carried names its carry index so the Python side maps files instead of re-matching by name. Behaviour unchanged at every former call site. |
#641 (review) |
| #640 | One retired_enum_label error type on every write path (construction and validated assignment used to report value_error); allow_ahead parsed once, in the core, before any connection opens. |
#640 (review) |
Three further candidates (a retired-column object on the model, a live-only server-default field in the IR, RunRefused payload typed by kind) go to close-epic's Learnings as follow-ups.
This PR's head is now 36448ff. The epic's aggregate Verify passed there with each exit status checked: all crates (--features testing 354), the IR vector contract (14), pytest --db-backends=sqlite,postgres over 31 files: 1661 passed, 33 skipped, 1 xfailed; prek clean; tree unchanged. Verdict stands: MERGE.
…ver deleted blindly (panel F6) (#637) Panel finding F6 on PR #636: `ferro migrate new --replace` set the old directory aside under a hidden name before writing the new one, and a hard kill in between left the only copy where the directory reader skipped it and a later `--replace` deleted it first. Every directory read now refuses a ferro-named leftover (`interrupted_replace`, `interrupted_write`) naming the recovery, a renumbered replacement's aside carries the new directory's name so the reader can tell whether it completed, and nothing deletes an existing aside. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
… the primary-key refusal names its fix (panel F4, F5, F8) (#638) Panel findings F4, F5 and F8 on PR #636. The retired-label guard now compares the value that would be written, so a plain `Enum` member carrying a retired label is refused like a `StrEnum` one, at construction, on attribute assignment under either `validate_assignment` setting, and in keyword and recipe `update()`; hydration still bypasses it and an unchanged label is still written back by `save()`. The retired primary-key refusal names its fix, and the migrations guide's sentence matches. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
… after upgrade succeeds (panel F1, F2, F3, F7, F9, C1) (#639) Panel findings F1, F2, F3, F7, F9 and Coherence C1 on PR #636. A generated step's class is the regenerated one only when the regenerated rendering matches the applied file's checksum, otherwise the file's own declaration decides (F1). A migration that fails to regenerate makes `up` and `baseline` refuse with `unclassifiable`, naming the fix, before any tracking-table change; `down`, `rerecord` and `baseline --remove` proceed with the class left NULL and a warning (F2). The class backfill trusts the directory's files only when snapshot checksum, step file name and step checksum match the record, otherwise `contract` (F3). The first `down` after upgrading from 0.22 no longer refuses on the backfilled class (F7). The baseline class test pins a contract step (F9). `adds_labels_only` ignores fields that render no storage, so a label addition beside a retired label is `expand` (C1). Candidate ADR recorded in the PR: when a class may be decided from files. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…ch carried step's move (architecture pass on #636) (#641) Architecture pass on PR #636, cards 1 and 4; behaviour unchanged. The step class has one entry point: the generator describes each laid-out step and the run planner's module decides, deriving the `constrains` flag from the step's own ops through the same function the staging code uses; the duplicated unique-index rule is gone; the apply-time path classes data and hand steps through the same entry point. `Carried` names which carry entry a step came from, so the Python side maps files by index instead of re-matching steps by name. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
… allow_ahead parsed once (architecture pass on #636) (#640) Architecture pass on PR #636, cards 2 and 6. One function raises the retired-label refusal as a `retired_enum_label` error on every write path, construction and validated assignment included, which used to surface as `value_error`; hydration still bypasses it. The `allow_ahead` vocabulary is parsed once, in the core, exposed over the FFI and called before any connection opens; the Python duplicate is gone, and the interface test now proves the refusal arrives before connecting. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
migrate new --replace, foreign defaults as drift (#623)
…ver deleted blindly (panel F6) (#637) Panel finding F6 on PR #636: `ferro migrate new --replace` set the old directory aside under a hidden name before writing the new one, and a hard kill in between left the only copy where the directory reader skipped it and a later `--replace` deleted it first. Every directory read now refuses a ferro-named leftover (`interrupted_replace`, `interrupted_write`) naming the recovery, a renumbered replacement's aside carries the new directory's name so the reader can tell whether it completed, and nothing deletes an existing aside. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
… the primary-key refusal names its fix (panel F4, F5, F8) (#638) Panel findings F4, F5 and F8 on PR #636. The retired-label guard now compares the value that would be written, so a plain `Enum` member carrying a retired label is refused like a `StrEnum` one, at construction, on attribute assignment under either `validate_assignment` setting, and in keyword and recipe `update()`; hydration still bypasses it and an unchanged label is still written back by `save()`. The retired primary-key refusal names its fix, and the migrations guide's sentence matches. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
… after upgrade succeeds (panel F1, F2, F3, F7, F9, C1) (#639) Panel findings F1, F2, F3, F7, F9 and Coherence C1 on PR #636. A generated step's class is the regenerated one only when the regenerated rendering matches the applied file's checksum, otherwise the file's own declaration decides (F1). A migration that fails to regenerate makes `up` and `baseline` refuse with `unclassifiable`, naming the fix, before any tracking-table change; `down`, `rerecord` and `baseline --remove` proceed with the class left NULL and a warning (F2). The class backfill trusts the directory's files only when snapshot checksum, step file name and step checksum match the record, otherwise `contract` (F3). The first `down` after upgrading from 0.22 no longer refuses on the backfilled class (F7). The baseline class test pins a contract step (F9). `adds_labels_only` ignores fields that render no storage, so a label addition beside a retired label is `expand` (C1). Candidate ADR recorded in the PR: when a class may be decided from files. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
…ch carried step's move (architecture pass on #636) (#641) Architecture pass on PR #636, cards 1 and 4; behaviour unchanged. The step class has one entry point: the generator describes each laid-out step and the run planner's module decides, deriving the `constrains` flag from the step's own ops through the same function the staging code uses; the duplicated unique-index rule is gone; the apply-time path classes data and hand steps through the same entry point. `Carried` names which carry entry a step came from, so the Python side maps files by index instead of re-matching steps by name. Part of #623 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR
Closes #623
Pinch does rolling deploys against ferro-managed databases and asked for five things the migration doors could not do (#618–#622). #618 shipped alone as the patch release in #630. This PR carries the other four PRDs, built as five slices on
feat/pinch-migration-needs, each gated byreview-prand re-verified on the integration branch after merge.What a user sees
Retire a column before dropping it (#619, #625). Release B declares
bio: str | None = Field(retired=True). The field leaves the running model: noSELECT,INSERTorUPDATEnames the column, so the code works whether the column is still there or the next release has dropped it. The column stays in the schema snapshot, nullable, with every index, check and foreign key that names it. Retiring a required column generates one change,ALTER COLUMN … DROP NOT NULL. Release C deletes the declaration and gets today's destructiveDROP COLUMN, plus theDROP TYPEfor an orphaned enum type. Contradictions (retired=Trueon a non-optional field, a primary key, a liverenamed_from,nullable=False, aBackRefover it) are refused at class creation with aTypeErrornaming the fix.Retire an enum label before removing it (#619, #627).
__ferro_retired_labels__ = frozenset({"canceled"})keeps the member, so rows holding the label still hydrate and save, while assigning the label is refused with an error naming it as retired. The label stays in the Postgres type and in any check; nothing renders DDL.Boot the previous release beside an expand-only database (#620, #626). Every step has a step class,
expandorcontract, decided once in the core from the planner's op verdicts and recorded in the step record when it is applied.require_applied(allow_ahead="expand-only"),up(...)andferro migrate up --allow-ahead=expand-onlyadmit a database ahead only by expand steps;DatabaseAheadError.contract_stepsnames the rest. The tracking table gains an additiveminorcolumn;formatstays 1, so the release that upgrades ferro still rolls back to 0.22. A hand-written SQL step isexpandonly with the-- ferro: expandheader.Regenerate a migration after a rebase (#621, #629). Two branches both generated
0007.ferro migrate new add_x --replace 0007_add_xregenerates it against the newest snapshot below it, carrying every data step and hand-written step by name with its body intact.checkand the directory loader print the--replaceline for each duplicate; the run's refusals name a replaced migration and its revert recipe.Hand-set defaults are drift (#622, #628). The live read carries each column's server default. A default ferro did not leave is a foreign default:
driftprintsauthor.slug has default '' in the database, none declared,baselinerefuses it, the Postgres pass drops it undermigrate_destructive, the Alembic bridge writesalter_column(server_default=None). Ferro's own defaults (a serial key's sequence, SQLite's kept literal on aNOT NULLcolumn) are never drift.Design record
Already on
mainvia #630: CONTEXT.md entries for Retired column, Retired label, Step class, Expand-only, Replaced migration, Foreign default, Tracked database; ADR-0053 (a column is retired one release before it is dropped); ADR-0054 (a step's class is recorded when it is applied); amendments to ADR-0027, 0030, 0031, 0034.Narrowed (approved in the plan)
On SQLite the reconciliation pass never rebuilds a table, so for a foreign default it raises its existing rebuild
UserWarningnaming the fix instead of dropping the default;driftandbaselinebehave as on Postgres.Slices
ferro migrate new --replacePlan and wave log: #623 (comment) and #623 (comment)
Exit steps
Closes #623so the epic closes on mergereview-panelreport posted on this PR (full-epic mode)🤖 Generated with Claude Code
https://claude.ai/code/session_014ERLk8yFhJy7rdDyU7rGdR