Skip to content

Store oversized step results using mutation file storage - #310

Draft
reeceyang wants to merge 1 commit into
mainfrom
storage-in-mutations
Draft

reeceyang wants to merge 1 commit into
mainfrom
storage-in-mutations

Conversation

@reeceyang

@reeceyang reeceyang commented Oct 21, 2026 •

Copy link
Copy Markdown

Preserve oversized queued query, mutation, and action step results in file storage before recording the step failure. The blob, oversized-value record, and failed step commit together, so preservation cannot race with workflow cleanup. Results still fail above the existing 800 KiB threshold; the error includes the step ID and a mutation call for retrieving the full value.

Create the blob and compare Blob.size with ctx.meta.getTransactionMetrics().fileWriteBytes.remaining, also checking filesWritten.remaining. These are server-reported budgets, with no hardcoded storage limits. If the value cannot fit, record a terminal step failure explaining that it was not preserved. There is no scheduled preservation or pending state, and unrelated storage errors propagate.

Oversized events retain their existing failure-and-preview behavior without file preservation, whether sent before or after the workflow starts waiting. Inline results, workflow final returns, and oversized arguments remain outside this feature. Add paginated list and mutation read/remove APIs, cleanup of stored files, and README documentation.

This PR is standalone on main and incorporates the storage work from #242. Requires released Convex 1.48 (convex@1.48.0 in development, ^1.48.0 as a peer). Tests use released convex-test@0.0.67 with native mutation-storage metrics and limit enforcement. Budget tests configure actual transaction limits, without metrics mocks or storage syscall adapters. Live probes also validate backend accounting and limits.

Validation:

  • All 145 tests, build, and TypeScript checks pass. Lint has no errors and three existing warnings.
  • Unit coverage includes exact byte-budget boundaries, UTF-8 and Convex-value round trips, exhausted byte/file budgets with no writes or scheduling, unrelated storage errors, 11 oversized pre-sent events failing without preservation, workflow cleanup and restart deleting preserved records and blobs, and removal rollback preserving a readable value.
  • Manual development-deployment checks verified 900 KB result retrieval, removal rollback, idempotent removal, cleanup, and both event timings without preservation. A 3 MB result that serializes to an 18 MB blob fails cleanly through the real queued completion path.
  • Manual backend validation confirmed that backend file-byte usage equals Blob.size. It also verified a storage-limit rejection can be caught without consuming storage budget or preventing a smaller write from committing. HTTP downloads return 200; test storage is empty after cleanup.

@pkg-pr-new

pkg-pr-new Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@convex-dev/workflow@310

commit: 458fbb3

@reeceyang
reeceyang force-pushed the storage-in-mutations branch from b374fd7 to 158c664 Compare October 21, 2026 18:06
@reeceyang reeceyang changed the title Use mutation file storage for oversized values Store oversized step results using mutation file storage Oct 21, 2026
@reeceyang
reeceyang changed the base branch from ian/store-oversized-values to main October 21, 2026 18:06
@reeceyang
reeceyang force-pushed the storage-in-mutations branch 3 times, most recently from 62dfe7d to f62c50c Compare November 4, 2026 00:40
@reeceyang
reeceyang force-pushed the storage-in-mutations branch from f62c50c to 458fbb3 Compare November 6, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant