Repository navigation
feat(mongodb): add the MongoDB destination package - #41
Open
gjtorikian wants to merge 11 commits into
Open
gjtorikian wants to merge 11 commits into
gjtorikian wants to merge 11 commits into
Conversation
A MongoDB 5.0+ destination with the same CLI, container image and Destination semantics as the SQL packages: workos_* collections in the application's database, _live views over them, the single-writer lease, the fenced checkpoint, server-side reconcile and the quarantine store. Every write is one update pipeline with upsert, so the staleness gate runs in the operation that writes, and payloads go in as $literal so a string like "$100 plan" is never read as a field path. The lease and every cursor for an environment share one document, which makes setStateIfHeld a single conditional update with no transaction: a standalone mongod, a replica set and Atlas all behave the same. Cursors are stored as canonical JSON because document equality in MongoDB is field-order sensitive and the engine's is not. Catalog columns become indexes on _raw.<path>; there are no generated fields and so no column drift. The suite runs on an embedded mongod (mongodb-memory-server), whose postinstall download is turned off in pnpm-workspace.yaml so install and the Docker build stay network-light; the binary downloads on first test run and CI caches it. The Docker workflow gains the mongodb matrix entry and release-please gains the package.
.claude/worktrees/docs-neon-guide belongs to another checkout and was swept into the previous commit as an embedded repository.
[Medium impact] The PR appears safe to merge; no new actionable issues remain. SummaryAdds
|
# Conflicts: # .github/workflows/ci.yml # .github/workflows/docker.yml # .release-please-manifest.json # README.md # packages/sync-engine/README.md # pnpm-lock.yaml # release-please-config.json
… configured strings as text
Six review findings on the MongoDB destination, each with a test:
- The entrypoint check compared `process.argv[1]` against `cli.js`, but npm
installs the `bin` as a symlink named `workos-sync-mongodb`, so the
installed command imported `main()` and exited. Both sides are now
resolved to real paths and compared.
- `migrate` returned early on a recorded fingerprint before reaching
`ensureLiveView`, so `init --repair` with unchanged streams never
redefined a view changed by hand. A repair now always makes the (still
idempotent) full pass.
- In the lease's update pipeline and the mirror writes' pipelines a sync
id, environment label, holder or `_env` that starts with `$` was read as
a field path. They are wrapped in `$literal`.
- `keyFilter` copied a row's key value straight into the query, so a
malformed row with `id: { $ne: null }` could match another row. The
filter is an explicit `$eq`, and `keyValues` treats an object as no key.
- `chunk` looped forever on a batch size of zero or less; it now refuses
anything but a positive integer.
- The README's `$lookup` example joins on `_env` as well as the user id.
Two more review findings: - `dedupeByKey` compared `updatedAt` as text, so a batch mixing offset and UTC forms could keep the older row. It now compares the parsed instants, and a timestamp that is not an instant fails there with the same error the write would have raised. - `parseIso8601` range-checked the clock but not the offset, so `+00:99` was accepted and applied. An offset's hours and minutes are now checked like the clock's.
A failed exit-code assertion shows only the code; checking the diff first shows the whole result when something is off.
A stream whose diff throws is reported with a skippedReason, not thrown; asserting it is absent shows the failure's text when it is not.
…eams Main's #45 taught the engine to abandon a stream whose listing alone passes the reconcile page budget, and gave the Postgres CLI `--exclude-streams`. This brings the MongoDB CLI, its README and its tests to the same place: the reconcile help and table describe the oversized verdict, `--exclude-streams` is parsed the same way, and the drift test budgets the three pages users needs rather than two.
This was referenced Nov 5, 2026
release-please knows the package, but the workflow that publishes a component once its release is cut listed only the engine, postgres and (since #47) mysql. The MongoDB package gets the same job, so a release tag reaches npm and the image registry rather than stopping at GitHub.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@workos/sync-mongodb, a MongoDB 5.0+ destination with the same CLI, container image, andDestinationsemantics as the SQL packages:workos_*collections in the application's database,_liveviews that hide tombstones, the single-writer lease, the fenced checkpoint, server-side reconcile, and the quarantine store.upsert: true, so the staleness gate runs in the same operation that writes; payloads go in as$literalso a string like"$100 plan"is never read as a field path.$mergeObjectsgives the shallow merge the contract asks for.setStateIfHelda single conditionalupdateOnewith no transaction, so a standalonemongod, a replica set, and Atlas all behave the same. Cursors are stored as canonical JSON because document equality in MongoDB is field-order sensitive and the engine's is not._raw.<path>; there are no generated fields and so no column drift. Timestamps go through a strict ISO-8601 parser becausenew Date()accepts2026-02-30.Testing
158 tests on an embedded
mongod8.0.4 (mongodb-memory-server, binary downloaded lazily and cached in CI), spot-checked against amongo:8container. CI gains a MongoDB job, the Docker workflow gains themongodbmatrix entry, and release-please gains the package. The image builds andstreamsruns from it.