[IAmJSD]

Root cause

GET /api/ladder/tranches and GET /api/ladder/runs returned 100-row pages, and the web client's completeTimeline follows next_cursor until the last page. This caused two problems that are still open on #668:

1. Read scaling. Every page statement materialized and aggregated the whole account scope to compute total_count, total_usd_hr and the on-demand total. A tranches page also shipped every scoped row's amount triple to Go. Loading n rows therefore took ceil(n/100) statements, each scanning all n rows: O(n²/100).
2. Mid-load changes. Each page was a separate statement with its own snapshot. completeTimeline checks that total_count, total_usd_hr and total_od_usd_hr match on every page, that no ID repeats, and that the rows add up to the totals. So any amendment, new tranche or new run committed between two page requests failed the whole load with "The ladder timeline changed while loading. Reload to try again."

Fix

The fix is backend only (internal/config/store_postgres_ladder_timeline.go, internal/config/ladder_timeline.go):

• Both queries now return every scoped row, plus the count and totals, from one statement in one aggregate pass. There is no LIMIT and no second CTE scan. One statement means one snapshot: a concurrent write lands wholly before or wholly after the read.
• No next_cursor is issued (NextCursor and LadderTimelinePageSize are removed). The unchanged web client therefore finishes after its first request, and its cross-page checks have nothing to disagree about.
• The after_created_at/after_id parameters are still accepted, as a lower bound on the rows only (never on the totals). An old browser bundle that is part-way through a paginated load during the deploy still gets a consistent tail. The handler, the store interface and the mocks are unchanged.
• The on-demand total keeps exactly the same computation (fillLadderTimelineOnDemand is untouched), so the results match the old stitched-together pages. The integration test checks this against an independent SQL oracle.

Response size: the whole history now travels in one response instead of ceil(n/100) responses. The client already held all of it in memory. Large JSON responses are gzipped by internal/server/lambda_compress.go (#723), which keeps the Lambda 6 MB limit far above realistic ladder histories (daily cadence, at most max_actions_per_run tranches per run).

Closes #668

Verification

All of this is local evidence: real Postgres 16 through the shared testcontainers DB (testhelpers.RequireMigratedDatabase) for the integration tests, and pgxmock for the unit tests. Nothing ran against a deployed environment or a real cloud account.

New integration tests, in internal/config/store_postgres_ladder_timeline_snapshot_integration_test.go. They drive the store exactly like completeTimeline: they marshal each response to JSON, follow next_cursor, and apply the same cross-page checks.

• TestLadderTimelineLoadsWholeScopeInOneRead: 250 tranches and 250 runs. It asserts the client load takes 1 request per endpoint, and that the rows (in (created_at, id) order) and total_usd_hr equal a direct SQL oracle.
• TestLadderTimelineLoadToleratesMidLoadChanges: commits an amendment (through AmendLadderTranche), a new tranche or a new run between two page requests, and asserts the load succeeds and the next load sees the change.
• TestLadderTimelineLoadIsSnapshotConsistentUnderConcurrentWrites: one goroutine keeps amending the timeline and adding tranches and runs, while 40 rounds of client loads (tranches and runs, 450 rows each) run concurrently. Every load must be internally consistent, and the loads must have observed more than four distinct states, so the writes really did interleave.

Fail before (origin/main a14dc8f2 plus only the new test file, in a temporary worktree):

go test -tags integration -count=1 -timeout 60m -run 'TestLadderTimelineLoad' -v ./internal/config/
--- FAIL: TestLadderTimelineLoadsWholeScopeInOneRead     expected: 1  actual: 3   (requests per load)
--- FAIL: TestLadderTimelineLoadToleratesMidLoadChanges/amendment     the ladder timeline changed while loading: totals moved on page 2
--- FAIL: TestLadderTimelineLoadToleratesMidLoadChanges/new_tranche   the ladder timeline changed while loading: totals moved on page 2
--- FAIL: TestLadderTimelineLoadToleratesMidLoadChanges/new_run       the ladder timeline changed while loading: totals moved on page 2
--- FAIL: TestLadderTimelineLoadIsSnapshotConsistentUnderConcurrentWrites   the ladder timeline changed while loading: totals moved on page 2

The concurrent test failed on main in 4 of 4 runs. With the one-read assertion turned into a log line, main's 3-page stitched load matched the same SQL oracle (same rows, order and total). The new single read returns identical results.

Pass after (this branch):

go test -tags integration -count=1 -timeout 60m -run 'TestLadderTimeline' -v ./internal/config/   # all 5 timeline integration tests PASS
go test -tags integration -count=1 -timeout 60m -run 'TestLadderTimelineLoad' ./internal/config/  # 5 consecutive runs: ok

Other checks:

go build ./... && go vet ./... && go vet -tags integration ./...          # clean
go test -count=1 ./internal/config/                                       # ok (pgxmock: one statement returns all 250 rows, no next_cursor; legacy cursor passed through)
go test -count=1 -run Ladder ./internal/api/                              # ok
go test -count=1 ./internal/mocks/                                        # ok
golangci-lint v2.10.1 run --allow-parallel-runners --build-tags integration ./internal/config/...   # 0 issues
gocyclo -over 10 -ignore "_test\.go" internal/config/store_postgres_ladder_timeline.go internal/config/ladder_timeline.go   # clean

The existing TestLadderTimelineScopedPagesAndSnapshots was updated: one read now returns all 101 rows, and a legacy cursor narrows the rows but not the totals.

Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree: queue/platform @ 936996ad = origin/main a14dc8f2 plus these heads, merged in this order:

Branch	Tested head
fix/issue-337	ff3420c
fix/issue-169	7d75eb5
fix/issue-342	66b221f
fix/issue-310	5e11c21
fix/issue-326	5f0ea59
fix/issue-243	365e959
fix/issue-343	6c94036
fix/issue-312	d2848a8
fix/issue-316	acd60dd
fix/issue-339	75ea9b9
fix/issue-339-commitment-type	73bdb19
fix/issue-353	5d25fcd
fix/issue-313	3156bc4
fix/issue-357	53d9e58
fix/issue-368	0ba727d
fix/issue-419	61648c9
fix/issue-111	b04c372
fix/issue-686	0ad90b7
fix/issue-674	793095d
fix/issue-1055	4ec96f1
fix/issue-559	dfa9525
fix/issue-558	034eed4
fix/issue-874	1720539
fix/issue-666	1e120da
fix/issue-668	fb26345
fix/issue-568	ce04173
fix/issue-311	e54f250
fix/issue-177	0cbb88a
fix/issue-151	c21377c
fix/issue-166	41fb77b
fix/issue-27	2d67622
fix/issue-323	69288ae
fix/issue-59	36463e8
fix/issue-43	35e4069
fix/issue-158	6f76f8a
fix/issue-104	bb00491
fix/issue-55	971ef1b
fix/issue-361	89ac220
fix/issue-120	b458719
fix/issue-36	42bac6e
fix/issue-525	57c97dd
fix/issue-712	e5bcda0
fix/issue-72	6cc63ed

fix/issue-72 and fix/issue-312 conflict on one line in internal/api/handler_dashboard.go (both change the calculateCommitmentMetrics call). The tree used #72's side with #312's validated provider variable; whichever PR merges second takes that same resolution.

Linux, Go 1.26.9, AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers:

• Passed at 936996ad: go build ./..., go vet ./..., go vet -tags integration ./... and the full go test -race -short -count=1 ./....
• Static checks at 936996ad: golangci-lint v2.10.1 found 0 issues on every changed package, with and without --build-tags integration. tests/e2e is a separate module, so it was vetted under its own e2e tag. gocyclo -over 10 is silent on the changed non-test files.
• Integration tests, every changed package, one package at a time (-tags integration -count=1, 60-minute local timeout because the VM was shared):
  • At 936996ad, these pass: internal/server, internal/credentials, internal/scheduler, internal/database/postgres/migrations, internal/purchase and internal/api.
  • internal/config passes except TestPostgresStoreDB_ResolveStaleRIExchange_ValidatesCloudCheck, which fails the same way on unmodified main (fixed by fix(config): compare resolve checked_at at the database's microsecond precision #1252).
  • These were run on the first build of this tree (26795376, same branches at earlier heads): internal/analytics, internal/auth, internal/commitmentopts, internal/database and cmd/lambda/clear-rate-limit all pass. The later fixes don't touch those packages.
• What the combined run caught, each fixed on its own branch before this PR opened:
  • a compile break between fix/issue-104's test and fix/issue-326's httpToLambdaRequest signature change;
  • fix/issue-316 rejecting legacy-keyed plans;
  • fix/issue-120 versus main's global-filter test.
• Secrets: git-secrets, using the repo's scripts/setup-git-secrets.sh patterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged. scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.
• Nothing was applied to any cloud.

Out of scope (per the maintainer's comment on #668)

• ErrLadderAmendInvalid mapping: already done in fix(config): map invalid ladder amendments to ErrLadderAmendInvalid #694.
• FOR NO KEY UPDATE and an explicit ON DELETE RESTRICT / editing migration 000107: deliberately not done, as decided on the issue. This PR touches neither the amend path nor any migration.
• The tranches and runs endpoints are still two requests, loaded in parallel by the UI, so they are two snapshots. completeTimeline does not cross-check them, so this cannot fail a load. A run and its tranches committed between the two requests would just show up on the next reload.

Frontend follow-up for the UI owner

No UI change is needed: the current completeTimeline works unchanged and now makes one request per endpoint. Optionally, once this ships, its pagination loop and cross-page "changed while loading" checks could be simplified, because the server no longer sends next_cursor.

Maintainer: please mirror the issue's triage labels onto this PR: type/chore, triaged, priority/p3, severity/low, urgency/eventually, impact/few, effort/m.

🤖 Generated with Claude Code

[coderabbitai]

Caution

Review failed

The pull request is closed.

Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[IAmJSD]

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 940813e6548e68e13a6175c2e93a5d2a61b4ec82

Blocking:

• none

Non-blocking:

• Response size: the timeline endpoints no longer have any page bound. Response size now grows linearly with ladder history, and the only guard is gzip under the 6 MB Lambda cap (internal/server/lambda_compress.go). That is fine at realistic sizes. A soft row cap, or an explicit error past N rows, would turn a far-future overflow into a clear message instead of a Lambda payload error. This is a judgment call, not a defect.
• Frontend: completeTimeline's pagination loop in frontend/src/api/ladder.ts is now dead code against this server. It is kept on purpose for old bundles during deploy. It can be simplified in a later frontend change.

Attacked:

• Single snapshot: both queries are now one statement. That gives one snapshot for COUNT, SUM, the on-demand triples and the rows. The scoped CTE is referenced once, so dropping MATERIALIZED changes nothing. jsonb_agg(... ORDER BY ...) FILTER (...) applies the legacy cursor to the rows only, never to the totals.

• Legacy client across the deploy: an old bundle that loaded page 1 from the old server and sends after_created_at/after_id to the new one gets the full tail, the same totals and no next_cursor. Its sum and count checks then pass. next_cursor appears nowhere else on the Go side or in openapi.yaml.

• Fail-before: reverted ladder_timeline.go and store_postgres_ladder_timeline.go to the merge-base and kept the new tests. On real Postgres:

  • TestLadderTimelineLoadsWholeScopeInOneRead failed (expected 1 request, got 3).
  • All three TestLadderTimelineLoadToleratesMidLoadChanges subtests failed with "totals moved on page 2".
  • ...UnderConcurrentWrites failed.

  With the PR code restored, all 10 TestLadderTimeline* tests pass under -race -tags integration.

• Checks run: go build ./..., then go vet with and without -tags integration on internal/config and internal/api. Then go test -race -short -count=1 ./internal/config/ ./internal/api/. Then golangci-lint v2.10.1 with and without --build-tags integration: 0 issues.

• Acceptance criteria: this meets chore(ladder): amend/timeline store lock mode, validation mapping and read scaling #668's two remaining items, as cristim's comment scopes them: read scaling, and tolerance of mid-load changes.

Body-stale:

• none

_{Reviewer: rB5. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.}

[IAmJSD]

Local verification at 940813e6548e68e13a6175c2e93a5d2a61b4ec82 merged onto main (5aa3df5b), together with #1155, #1219, #1221, #1223, #1224, #1225, #1226, #1227, #1256, #1262, #1266, #1270, #1272, #1275, #1283, #1286, #1293, #1297, #1299, #1300, #1301, #1304, #1307, #1311, #1313, #1325 in that order (combined tree f3a3331f, Linux, Go 1.26.10): go build ./..., go vet with and without -tags integration, the full go test -race -short -count=1 ./..., and go test -race -tags integration -count=1 on every changed package, against Docker Postgres. All pass. CI Success and pre-commit are green on this head, with 1 approving independent review(s) on it. Nothing was run against any cloud.

[IAmJSD]

Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 1 approving independent review(s) on 940813e6, green CI, and the combined-tree verification above. Squash-merged at exactly 940813e6548e68e13a6175c2e93a5d2a61b4ec82.