Repository navigation
docs(lore-0344,lore-0345,lore-0346,lore-0347): backlog monitoring review findings - #420
Open
stkrolikiewicz wants to merge 1 commit into
Open
stkrolikiewicz wants to merge 1 commit into
stkrolikiewicz wants to merge 1 commit into
Conversation
…iew findings Record the verified findings of the 2026-11-13 read-only CloudWatch review of the shared account, each re-checked against origin/develop and live CloudWatch: - 0344: CoarseSweepTableFailures has no alarm, and a failed table still returns Ok. - 0345: two new Comet pools have dropped trades since 2026-11-10; the unregistered-pool alarm has the wrong shape; six unclassified sweep emitters are untracked. - 0346: remove three alarms that a kept alarm always beats. - 0347: stop publishing 15 unread or duplicate metric series and trim the overview dashboard to 50 metrics. - 0093: confirm the deployed push-freshness state; sdex has had no datapoint since 2026-07-27. No code or infra changes; infra work is agreed with the other teams first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backlog tasks from the 2026-11-13 read-only review of CloudWatch metrics and alarms in the shared account (eu-central-1). Docs only: no code, no infra. Infra changes are agreed with the other teams on the account before they ship.
Protection gaps
CoarseSweepTableFailures, and a table that errors still returnsOkfrom the sweep (repair.rs:685-694,coarse-sweep-worker/src/main.rs:160-187). The runbook says to alarm on it.STATIC_POOLShave dropped trades since 2026-11-10 14:17 UTC. This is the third time in five weeks, after 0334 and 0337. The task also covers:ledger-processor-unregistered-pool's shape: an ALARM/OK pair per dropped trade, 61 + 62 messages in 30 days;coverage-sweep-unclassified, in ALARM since 2026-11-09. They are not these pools.describe-alarmsoutput.sdex-push-freshnesshas had no datapoint since 2026-07-27.Fewer pages and metrics
ledger-processor-no-invocations, which fired afterrollup-freshness-1min 4 of 4 stalls. Also removeoracle-no-invocations(dark-feed covers it) andmtls-notafter-probe-duration-near-timeout(-errorspages a hang a day earlier).cleanup-errorswas considered and rejected, with the reason recorded.prices-production-overviewfrom 59 metrics to at most 50.Checked
origin/develop6672466. The review had readmaster, which is 165 commits behind.soroban-readonly, using list/describe/get calls only: alarm definitions and history, metric statistics, dashboards, and the ledger-processor and coverage-sweep Lambda logs.