Skip to content

feat(mongodb): add the MongoDB destination package - #41

Open
gjtorikian wants to merge 11 commits into
mainfrom
feat/sync-mongodb
Open

gjtorikian wants to merge 11 commits into
mainfrom
feat/sync-mongodb

Conversation

@gjtorikian

Copy link
Copy Markdown
Collaborator

Summary

  • Adds @workos/sync-mongodb, a MongoDB 5.0+ destination with the same CLI, container image, and Destination semantics as the SQL packages: workos_* collections in the application's database, _live views that hide tombstones, the single-writer lease, the fenced checkpoint, server-side reconcile, and the quarantine store.
  • Every write is one update pipeline with upsert: true, so the staleness gate runs in the same operation that writes; payloads go in as $literal so a string like "$100 plan" is never read as a field path. $mergeObjects gives the shallow merge the contract asks for.
  • The lease and every cursor for an environment share one document, which makes setStateIfHeld a single conditional updateOne with no transaction, so a standalone mongod, a replica set, and Atlas all behave the same. Cursors are stored as canonical JSON because document equality in MongoDB is field-order sensitive and the engine's is not.
  • Catalog columns become indexes on _raw.<path>; there are no generated fields and so no column drift. Timestamps go through a strict ISO-8601 parser because new Date() accepts 2026-02-30.

Testing

158 tests on an embedded mongod 8.0.4 (mongodb-memory-server, binary downloaded lazily and cached in CI), spot-checked against a mongo:8 container. CI gains a MongoDB job, the Docker workflow gains the mongodb matrix entry, and release-please gains the package. The image builds and streams runs from it.

A MongoDB 5.0+ destination with the same CLI, container image and
Destination semantics as the SQL packages: workos_* collections in the
application's database, _live views over them, the single-writer lease,
the fenced checkpoint, server-side reconcile and the quarantine store.

Every write is one update pipeline with upsert, so the staleness gate runs
in the operation that writes, and payloads go in as $literal so a string
like "$100 plan" is never read as a field path. The lease and every cursor
for an environment share one document, which makes setStateIfHeld a single
conditional update with no transaction: a standalone mongod, a replica set
and Atlas all behave the same. Cursors are stored as canonical JSON because
document equality in MongoDB is field-order sensitive and the engine's is
not. Catalog columns become indexes on _raw.<path>; there are no generated
fields and so no column drift.

The suite runs on an embedded mongod (mongodb-memory-server), whose
postinstall download is turned off in pnpm-workspace.yaml so install and
the Docker build stay network-light; the binary downloads on first test
run and CI caches it. The Docker workflow gains the mongodb matrix entry
and release-please gains the package.
.claude/worktrees/docs-neon-guide belongs to another checkout and was
swept into the previous commit as an embedded repository.
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues remain.

Summary

Adds @workos/sync-mongodb with MongoDB writes, generated indexes and live views, checkpoints, leases, reconciliation, and quarantine storage.

  • MongoDB stores each WorkOS record with its sync metadata.
  • MongoDB keeps each listener's lease beside its cursors.
  • MongoDB compares mirrored records with the records returned by WorkOS.
  • Failed events stay available for review and replay.
  • Operators can run and monitor the MongoDB sync from one CLI.
  • The MongoDB package is ready to install, test, build, and release.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  E[Sync engine] --> D[MongoDB destination]
  D --> M[Mirror collections]
  M --> V[Live views]
  D --> S[Lease and cursor document]
  D --> Q[Quarantined events]
  D --> R[Reconcile staging collections]
  R --> M
Loading

Reviews (7) · Last reviewed commit: "ci(mongodb): publish sync-mongodb releas..." · Reviewed by Greptile

Comment thread packages/sync-mongodb/src/cli.ts Resolved
Comment thread packages/sync-mongodb/src/db/lease.ts Resolved
Comment thread packages/sync-mongodb/src/db/lease.ts Resolved
Comment thread packages/sync-mongodb/src/db/upsert.ts Resolved
Comment thread packages/sync-mongodb/README.md Resolved
Comment thread packages/sync-mongodb/src/db/upsert.ts Resolved
# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/docker.yml
#	.release-please-manifest.json
#	README.md
#	packages/sync-engine/README.md
#	pnpm-lock.yaml
#	release-please-config.json
Comment thread packages/sync-mongodb/src/db/upsert.ts Resolved
… configured strings as text

Six review findings on the MongoDB destination, each with a test:

- The entrypoint check compared `process.argv[1]` against `cli.js`, but npm
  installs the `bin` as a symlink named `workos-sync-mongodb`, so the
  installed command imported `main()` and exited. Both sides are now
  resolved to real paths and compared.
- `migrate` returned early on a recorded fingerprint before reaching
  `ensureLiveView`, so `init --repair` with unchanged streams never
  redefined a view changed by hand. A repair now always makes the (still
  idempotent) full pass.
- In the lease's update pipeline and the mirror writes' pipelines a sync
  id, environment label, holder or `_env` that starts with `$` was read as
  a field path. They are wrapped in `$literal`.
- `keyFilter` copied a row's key value straight into the query, so a
  malformed row with `id: { $ne: null }` could match another row. The
  filter is an explicit `$eq`, and `keyValues` treats an object as no key.
- `chunk` looped forever on a batch size of zero or less; it now refuses
  anything but a positive integer.
- The README's `$lookup` example joins on `_env` as well as the user id.
Two more review findings:

- `dedupeByKey` compared `updatedAt` as text, so a batch mixing offset and
  UTC forms could keep the older row. It now compares the parsed instants,
  and a timestamp that is not an instant fails there with the same error
  the write would have raised.
- `parseIso8601` range-checked the clock but not the offset, so `+00:99`
  was accepted and applied. An offset's hours and minutes are now checked
  like the clock's.
A failed exit-code assertion shows only the code; checking the diff first
shows the whole result when something is off.
A stream whose diff throws is reported with a skippedReason, not thrown;
asserting it is absent shows the failure's text when it is not.
…eams

Main's #45 taught the engine to abandon a stream whose listing alone
passes the reconcile page budget, and gave the Postgres CLI
`--exclude-streams`. This brings the MongoDB CLI, its README and its
tests to the same place: the reconcile help and table describe the
oversized verdict, `--exclude-streams` is parsed the same way, and the
drift test budgets the three pages users needs rather than two.
release-please knows the package, but the workflow that publishes a
component once its release is cut listed only the engine, postgres and
(since #47) mysql. The MongoDB package gets the same job, so a release
tag reaches npm and the image registry rather than stopping at GitHub.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant