[gjtorikian]

Summary

• A per-parent backfill (sessions per user, memberships per organization, directory users per directory) checkpointed the full list of finished parent ids after every parent. The list grew with the parent stream and was rewritten at each boundary, so the bytes written were quadratic in the number of parents, and a document store's size limit ends the backfill outright: MongoDB's 16 MiB document cap falls at roughly half a million users on sessions, before the final state that would have cleared the list. Surfaced by Greptile on feat(mongodb): add the MongoDB destination package #41.
• Progress is now one id, doneThrough: parents are listed in id order and every parent sorting at or before the watermark is done. WorkOS ids are time-ordered, so a parent created after a slice sorts past the watermark and the next slice lists it.
• The engine sorts the parent ids itself rather than trusting a database collation to agree with the comparison; Postgres's order by id under a non-C collation need not.
• An in-flight sliced backfill survives the upgrade. A checkpoint written by an earlier release, with each finished parent recorded by id, is still honored on resume; its ids are carried until the watermark passes them, then dropped, and doneParents is never written afresh.
• Covered by a new engine-level suite (packages/sync-engine/test/backfill.test.ts) over a stubbed destination: the watermark replaces the list, in-parent checkpoints carry it, the sort is the engine's, a late parent is picked up, and a legacy checkpoint is honored and shrinks. The Postgres and MySQL suites pass unchanged against real servers.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Replaces the growing list of finished parents with one doneThrough watermark.

• Parent backfills resume from one sorted progress marker.

No actionable issues were found. Tests were not run during this review.

_{Reviews (1) · Last reviewed commit: "fix(engine): record per-parent backfill ..." · Reviewed by Greptile}