Skip to content

Postgres: create organization schemas before their pools connect - #89

Merged
bjacobso merged 1 commit into
mainfrom
fix/postgres-org-setup-race
Nov 12, 2026
Merged

bjacobso merged 1 commit into
mainfrom
fix/postgres-org-setup-race

Conversation

@bjacobso

Copy link
Copy Markdown
Owner

Fixes the intermittent postgres-integration failure seen on #88 ("permission denied to create pg_catalog.triplex_schema_migrations") before the 0.3.0 release (ask 0007). It also fixes a connection leak that turned up while stress-testing it.

The race

Each organization database gets its own schema, reached through two pools (adapter and query). Each pool binds search_path=<schema>,pg_catalog at connection startup. The pool used to open first and then create the schema.

I reproduced the CI error locally at about 1 failure in 60 rounds of 16 concurrent organization creations under CPU load, then captured one with Postgres statement logging:

  • The failing connection opened 36 ms before the other pool committed CREATE SCHEMA.
  • In its own later transaction, that connection ran CREATE SCHEMA IF NOT EXISTS, which saw the schema and skipped. Its very next unqualified CREATE TABLE still resolved to pg_catalog.

Postgres 17's recomputeNamespacePath marks the search path valid after computing it (namespace.c). A schema-change invalidation that arrives during that computation can therefore be lost, and the stale "schema missing" path sticks.

Fix: create the schema on a separate, short-lived connection that is not bound to it, under the existing advisory lock. Only then open the schema-bound pool. Every schema-bound connection now starts after the schema is committed, so it can never compute a path that lacks it.

The leak

DatabaseManagerLive's finalizer stopped its cleanup fiber but never closed cached databases' scopes. Every organization's pools stayed open until the process exited, and the stress loop ran Postgres out of connections. The finalizer now calls closeAllConnections().

Tests

packages/postgres/test/integration/organization-setup-stress.test.ts:

  • Opens no schema-bound connection until the schema exists. This test is deterministic. It holds the schema's creation lock and starts setup, then counts open backends while setup is blocked.
    • Old code: 3 (the pool's minimum connections, bound to a schema that doesn't exist yet).
    • Fixed code: 1 (only the unbound creator).
  • Creates many organizations at once and releases their pools. It runs concurrent manager setup in rounds, then checks that connections return to baseline.
    • Without the manager fix: 192 leaked connections.
    • With it: 0.
    • PG_SETUP_STRESS_ROUNDS and PG_SETUP_STRESS_ORGS raise the load.

The race itself is too rare to catch on demand. The old code also passed one 240-round run, so the stress loop is not the proof of the fix; the ordering test is. The fixed code passed 240 rounds × 16 organizations (3,840 creations) under CPU load, with no failures and no leaked connections.

Local: pnpm check passes; pnpm test:postgres:integration 79/79.

🤖 Generated with Claude Code

A pooled connection that opened before another pool committed an
organization's schema could keep a search path without it, so migrations
tried to create pg_catalog.triplex_schema_migrations and failed. Create the
schema on a separate unbound connection first, so every schema-bound
connection starts after the schema exists.

Closing DatabaseManager now also closes each cached database's pools, which
previously stayed open until the process exited.

Adds an ordering test that fails on the old setup and a concurrent setup
stress test that checks for leaked connections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Documentation preview removed (PR closed).

@bjacobso
bjacobso merged commit 177e448 into main Nov 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant