Skip to content

Proposals from the rollout (fetch_one/fetch_scalar/execute, ComplexHelper cache, readonly_transaction, ...) - #52

Merged
aersam merged 12 commits into
mainfrom
issue-51-proposals
Oct 9, 2026
Merged

aersam merged 12 commits into
mainfrom
issue-51-proposals

Conversation

@aersam

@aersam aersam commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Refs #51 (not Closes: the lazy default pool, where_and, paginate and fetch_all(read_only=True) are proposed below, not implemented)
Fixes #53

Version 0.14.2 -> 0.15.0 (additive; behaviour notes under "Decisions").

Implemented

  1. fetch_one, fetch_scalar, execute in pgdevkit/db/fetch.py, exported from pgdevkit.db. They share _acquire/_render/_statement_timeout and the cancel/abort machinery with fetch_all through one internal _run (only what is read from the cursor differs), same keywords (con=/pool=/cancel=/statement_timeout=, model=/row_mapper= on fetch_one), t-string overloads without params. execute returns the rowcount; on a pooled connection it commits when the call returns, on a given con nothing is committed.
  2. ComplexHelper cache (plus complex_helper="auto" on every pg_* helper, from the OneSales comment: a default helper per call, which the cache makes cheap; opt-in, so existing reads stay plain SELECT *): catalog lookups, column types and "registered on this connection" state are cached per connection (WeakKeyDictionary, so it dies with the connection and pooled connections keep theirs). A helper created per call now costs 1 query (the SELECT) instead of ~5 after the first use of a connection; asserted in tests by counting statements. helper.clear_cache() and ComplexHelper(con, cache=False) for schema changes. Also fixes _get_complex_type re-fetching (and re-registering) enums on every call.
  3. pg_retrieve_many: where= (t-string, or psycopg.sql composable + params=), order_by= (column / (column, "asc"|"desc") / list / composable), limit=. Backwards compatible. Smallest design that fits; the select-list builder was not made public (where/order_by/limit cover the MDMApp case; say if you want it exported too).
  4. (partly) set_default_pool() also accepts a plain Callable[[], AsyncContextManager[AsyncConnection]] (PoolLike).
  5. readonly_transaction(con): set_read_only(True), a con.transaction() block, and a finally that always puts read_only back (to its previous value, i.e. None normally) (skipped for a closed/broken connection so it cannot mask the original error). Tested on a 1-connection pool, with errors, a refused write, and an autocommit connection.
  6. statement_timeout=at_most(seconds): opt-in "never loosen" (min(current, requested), current 0 = unlimited), computed server-side from pg_settings. at_most returns a float subclass, so no signature changed; works for fetch_*, execute and PostgresJsonResponse. Plain values keep replacing the setting in both directions; documented.
  7. % in sqlglot expressions: with params is not None, _render doubles every literal % (string literals, quoted identifiers, % modulo) but not the %(name)s placeholders (rendered as unique tokens first, then restored), on a copy of the tree. Also applied to PostgresJsonResponse. Docstring/README/SKILL no longer document it as a limitation.
  8. PgPool (OneSales): min_size, timeout, configure, check pass through to the psycopg pool (min_size now defaults to min(4, max_size), so max_size < 4 no longer raises). PgPool._dsn doesn't quote password/dsn_params values (spaces, backslashes break the connection) #53: the conninfo is built with make_conninfo, so passwords / dsn_params with spaces, backslashes, quotes work; round-trip tests incl. Azure tokens.
  9. Docs: pool exhaustion with streamed responses, {"error"} vs {"detail"} (plus a small opt-in error_key = "detail" class attribute on PostgresJsonResponse), NaN/Infinity as strings, the BaseHTTPMiddleware truncation caveat, +00:00 vs Z, statement_timeout semantics. README and skills/pgdevkit/SKILL.md updated for everything above.

Decisions for a human

  • readonly_transaction is not a sandbox (found in review, documented in the docstring/README/SKILL): SQL without bound params uses psycopg's simple protocol, so COMMIT; INSERT ... leaves the read-only transaction (TEMP tables stay writable, SET sticks). Untrusted SQL needs a SELECT-only DB role in addition. A stricter helper (e.g. rejecting multi-statement text) could be built on top; not done here. This also argues against fetch_all(read_only=True) (OneSales): it would suggest more safety than it gives, so it is proposed only (use readonly_transaction).
  • Lazy default pool (item 4): not implemented, a lifecycle design question. PgPool.open() is already idempotent. Proposal: PgPool(..., lazy=True) whose connection() returns an async context manager that does await self.open() (guarded by an asyncio.Lock) before borrowing, plus the existing explicit close() at shutdown. Open points: who calls close() if nobody opened at startup, and whether a failed first open (bad credentials, DB down) is retried on the next use or cached.
  • where_and(conditions) / paginate(query, limit, offset, order_by=) (OneSales): not done. They are optional and pg_retrieve_many(where=, order_by=, limit=) already covers the table-bound case; for free-form queries a t-string with {x:q} fragments does the job. Happy to add if wanted, but I'd want the target API agreed first (the returned fragment type: psycopg.sql.Composed vs Template).
  • ComplexHelper cache default on, as the issue asks. Risk (review finding): DDL rolled back after it was cached (e.g. a test fixture that creates types in a rolled-back transaction on a reused connection) leaves stale OIDs (cache lookup failed for type ...); documented, helper.clear_cache() / cache=False handle it. Making it opt-in would avoid the hazard but defeat the per-call-helper use case; your call. An explicit complex_fields= shortcut was not added.
  • Item 7 and callers that worked around the limitation: someone who wrote LIKE '50%%' in a sqlglot expression together with params (the old documented workaround) would now send 50%% to Postgres (harmless for LIKE wildcards, but different for =/to_char formats); CCMT2's FixedPostgres dialect, which already doubles %, would double twice. Plain strings and psycopg.sql composables are untouched.
  • readonly_transaction restores the connection's previous read_only value instead of hard-resetting to None (identical in the normal case; keeps a replica connection's True).
  • Anonymous (exp.Placeholder() / %s) placeholders in sqlglot expressions remain rejected by the existing name check; unchanged.

Test notes

Live-Postgres tests were run against a private throwaway cluster (they skip without a container runtime): tests/db/test_fetch.py, test_fetch_statement_timeout.py, test_transaction.py, test_complex_types.py, test_crud.py, tests/test_pg_json_response.py, tests/test_pg_json_statement_timeout.py. uv run ty check pgdevkit is clean; import pgdevkit.db still does not load sqlglot.

🤖 Generated with Claude Code

aersam and others added 12 commits October 9, 2026 16:17
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…_most() statement_timeout

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…er connection

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ostgresJsonResponse

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, cache=False semantics, order_by ambiguity)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t/configure/check to the psycopg pool

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…middleware notes

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PgPool._dsn doesn't quote password/dsn_params values (spaces, backslashes break the connection)

1 participant