Skip to content

Record tracking results in a dedicated Git repository - #1275

Merged
Ndpnt merged 40 commits into
mainfrom
tracking-results
Sep 23, 2026
Merged

Ndpnt merged 40 commits into
mainfrom
tracking-results

Conversation

@Ndpnt

@Ndpnt Ndpnt commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Implements the first part of RFC #1241: a tracking-results Git repository recording, for each terms, its latest tracking status, and for each run, its lifecycle (start, per-terms outcomes, completion). Its history is tamper-evident, so it shows which terms were actually tracked, when, and why they failed.

The recorder's definition of a safe path segment moves verbatim to
src/git so it stays single: hostile pathspecs are a git-level hazard,
and other modules persisting through git need the exact same
predicate. The recorder keeps its canMatchRecordFilePath policy
wrapper on top.
Reads the declarations exactly as they were at a given commit of
their repository, listing { serviceId, termsType } pairs without
loading full Service objects.
Orchestrates a run from the storage perspective: a run-start commit
with the skips known upfront, per-terms outcomes feeding counters
and transitions, a run-completion commit, and additive finalization
of a crashed run with its coverage and tracked counts derived from
the per-terms commits. The run only becomes current once its
run-start commit landed, and every commit of a run carries its runId
as a Git trailer.
The engine-facing entry point: translates the tracking lifecycle and
domain objects into recorder operations and owns the run policy,
namely crash recovery gating with retry, declarations commit
resolution, coverage of partial runs and categorisation of failure
reasons. Emits warn events instead of logging.

Creating it requires the collection id that identifies the collection
in every persisted run. Failures of the audit trail never fail the
tracking itself: a failed run-start commit disables tracking-results
for the run, and a failed push only warns, since every record is
already committed locally and the next successful finalize pushes all
accumulated commits.
Errors were pushed from concurrent callbacks, so their order depended
on completion timing; consumers treating the failure reasons as an
ordered value would see spurious changes between identical runs.
The error listener ran its asynchronous cleanup on every emission:
two fatal errors in close succession ran two concurrent cleanups,
racing the recorder finalize pushes and process.exit. The sequence is
memoised behind a promise, and its force-exit timer is cleared once
the cleanup completes.
The async.queue error callback is fire-and-forget, so drain() could
resolve while a failure was still being handled; the worker now
awaits the handler.
The Archivist starts and completes a tracking-results run around its
queue, records each outcome as it settles, and relays the module's
warnings. A failure to complete the run routes to the fatal shutdown,
and recording happens before the fatal emission since nothing may
write to the repositories once the shutdown started. When the
collection provides no id, tracking-results is disabled with a
warning. Technical upgrades skip the lifecycle entirely.
@Ndpnt
Ndpnt requested a review from clementbiron September 23, 2026 08:03
@clementbiron

Copy link
Copy Markdown
Member

I made these changes after our pairing session with @Ndpnt:

  • b7f7f5e1 Clarify the skip reason of crash recovery: terms without a committed outcome are skipped as no outcome recorded before the engine crashed instead of engine crashed, since unchanged terms leave no trace and may well have been processed.
  • 58ca9987 Omit snapshot links for unrecorded documents: tracking issues no longer link a nonexistent .null file for source documents that were never recorded as snapshots.
  • e62b7b8a Unquote file names listed from the git log: crash recovery no longer loses terms whose service ID contains a double quote, which git C-quotes in its log output.
  • 568c4125 Reuse the previous content read on save: the content read to detect changes doubles as the rollback backup, sparing a second read per recorded outcome.
  • 4887ceeb Point operators to the repository on failed setup: the recurring warning now names the repository path to inspect or delete.

@Ndpnt
Ndpnt merged commit 2804802 into main Sep 23, 2026
8 checks passed
@Ndpnt
Ndpnt deleted the tracking-results branch September 23, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants