Skip to content

feat(api): silence and unsilence alerts in the external API - #3504

Merged
kodiakhq[bot] merged 8 commits into
ClickHouse:mainfrom
andrewdunndev:ai/v2-alert-silence
Nov 19, 2026
Merged

kodiakhq[bot] merged 8 commits into
ClickHouse:mainfrom
andrewdunndev:ai/v2-alert-silence

Conversation

@andrewdunndev

Copy link
Copy Markdown
Contributor

Summary

Closes #3469

Adds POST and DELETE /api/v2/alerts/{id}/silenced, so an API key can mute an alert across a maintenance window and clear the silence afterwards. Until now silencing was only reachable through the session-auth internal route, and v2 could read silenced but not set it.

  • POST takes { "mutedUntil": "<ISO 8601, in the future>" }; DELETE clears the silence. Both return the updated alert in the v2 envelope, with { "message": ... } errors for 400 and 404 like the other v2 alert routes.
  • Same semantics as the internal route: the checker skips the alert while silenced.until is ahead of now, and silenced.by records the key's user.
  • The set and clear logic moves into controllers/alerts.ts, so the internal and external routes share one path.
  • PUT /api/v2/alerts/{id} keeps an existing silence, since the silence isn't part of the alert's definition.
  • OpenAPI entries and a changeset are included.

How to reproduce

Starting point: yarn dev, an alert, and a personal API key.

On main:

  1. There is no v2 route to silence the alert; PUT /api/v2/alerts/<id> has no silenced field.

On this branch:

  1. POST /api/v2/alerts/<id>/silenced with { "mutedUntil": "<an hour from now>" }. The response shows silenced.until, and the alert shows as silenced in the UI.
  2. DELETE /api/v2/alerts/<id>/silenced. The silence is gone from the response and the UI.

make dev-int FILE=external-api/__tests__/alerts runs the new cases: silence, unsilence, past and malformed mutedUntil, a UTC-offset timestamp, invalid id, 404s, the read path, and PUT keeping the silence.

How to test on Vercel preview

N/A - API-only change

References

  • Linear Issue:
  • Related PRs: replaces feat(api): silence and unsilence alerts in the external API #3480, which the open-PR limit closed before the vouch went through and which I can't reopen from my side. Its first review round is addressed here: the saved alert is populated in place instead of re-fetched, mutedUntil uses one shared schema (so the internal route now accepts UTC offsets too), and the DELETE 400 is in the OpenAPI spec.

Add POST and DELETE /api/v2/alerts/{id}/silenced. Both return the updated
alert in the v2 envelope so clients see the new silenced state, with
{message} errors.

The set and clear logic moves into controllers/alerts.ts so the internal
router and the external one share a single path.
@andrewdunndev
andrewdunndev requested a review from a team as a code owner October 30, 2026 22:25
@changeset-bot

changeset-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5c83455

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 30, 2026

Copy link
Copy Markdown

@andrewdunndev is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This is a focused, well-tested addition: the silence/unsilence logic is extracted into shared controllers, tenancy is enforced through the team-scoped getAlertById, and the new external routes mirror the existing v2 envelope and error conventions. The security pass confirmed cross-team isolation, auth guards, input validation (objectIdSchema, mutedUntilSchema), and rate-limiter coverage all hold. The remaining items are recommendations and nits.

🟡 P2 — recommended

  • packages/api/src/routers/external-api/__tests__/alerts.int.test.ts:500 — the new external silence/unsilence suite covers the happy, 404, cross-team, validation, and read paths, but never exercises the teamId == null || userId == null 403 guard branch introduced on these routes.
    • Fix: add a case that authenticates as a key whose user resolves no team/userId and asserts a 403 { message } response for both POST and DELETE.

🟡 P2 — recommended

  • packages/api/src/controllers/alerts.ts:225 — silenceAlert and unsilenceAlert do a read-then-save() across two round-trips; while mongoose dirty-tracking limits the write to the silenced path (so this is benign last-writer-wins, not cross-field clobbering), an atomic update is simpler and removes the window entirely.
    • Fix: replace the read-modify-write with a single Alert.findOneAndUpdate({ _id, team }, { $set/$unset: { silenced } }, { new: true }), returning null when no document matches.
🔵 P3 nitpicks (3)
  • packages/api/src/controllers/alerts.ts:242 — unsilenceAlert always calls save() even when the alert carries no silence, issuing a redundant write on a no-op DELETE.
    • Fix: return early when alert.silenced is already undefined, or move to the atomic $unset suggested above.
  • packages/api/src/controllers/alerts.ts:230 — alert.save() runs full-document mongoose validation, so silencing or unsilencing an alert document that predates a stricter schema would surface as a 500 rather than a scoped update.
    • Fix: use findOneAndUpdate (which bypasses full-document validation) or pass validateModifiedOnly on save.
  • packages/api/openapi.json:7187 — the silence/unsilence paths are specified both in openapi.json and in inline @openapi JSDoc in v2/alerts.ts, creating two sources of truth that can drift.
    • Fix: confirm which artifact is authoritative (regenerated vs hand-maintained) and keep the other in sync, per the repo's OpenAPI convention.

Reviewers (1 returned): security. The correctness, adversarial, api-contract, reliability, testing, maintainability, project-standards, kieran-typescript, and previous-comments personas were dispatched but had not returned at synthesis time; the findings above combine the security pass with orchestrator diff analysis across those lenses.

Testing gaps:

  • No coverage of the 403 (missing team/userId) guard on the new external silence/unsilence routes.
  • No concurrency test for simultaneous silence + update, and no integration test asserting the checker actually skips a silenced alert.
  • Prior-review items (cross-team 404 on POST/DELETE, malformed-id 400 on unsilence, shared mutedUntil schema, in-place populate) appear addressed by the new test cases in this diff.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@andrewdunndev

Copy link
Copy Markdown
Contributor Author

Thanks. Both tests are in 3d1bd8b: a key from another team gets a 404 with { message: 'Alert not found' } on POST and DELETE and the silence is left as it was, and DELETE with a malformed id returns 400.

I've left the read-modify-write in silenceAlert and unsilenceAlert as it is. It's the internal route's logic moved over unchanged, so a targeted update felt like a separate change for both routes.

try {
const teamId = req.user?.team;
const userId = req.user?._id;
if (teamId == null || userId == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — userId == null check can never be true on its own

Drop || userId == null and pass req.user._id after the teamId check, as the internal route does at packages/api/src/routers/api/alerts.ts:428. req.user?._id can only be nullish when req.user is missing, and in that case teamId is already nullish and returns 403. The extra check never changes the result.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This follows the existing v2 create route in this file, which guards teamId == null || userId == null the same way, so I would rather keep the two consistent than change one.

kodiakhq Bot pushed a commit that referenced this pull request Nov 3, 2026
)

## Summary

Closes #3468

Adds optional `tag`, `search`, `source` and `state` query parameters to `GET /api/v2/alerts`, so a client that wants the alerts for one service or one state gets just those from the server. Until now the route returned every alert on the team and the client filtered locally.

- Filters combine with AND. `tag`, `source` and `state` are repeatable and match any of the values given. `search` is a case-insensitive substring match on the display name, taken literally.
- `meta.total` and `X-Total-Count` count the filtered set, and `limit`/`offset` slice within it.
- Filters match stored tags and displayName, the same as the internal alerts page, since both use the shared filter builder. Tags inherited from a dashboard or saved search are not matched.
- An unrecognized `state` or `source` value now returns 400.
- With no filters the response is the same as today.

The issue isn't marked Accepting PR yet, so this is a draft and the parameter names are open to change.

### How to reproduce

**On `main`:** `curl -H "Authorization: Bearer <key>" "http://localhost:8000/api/v2/alerts?tag=prod&state=ALERT"` returns every alert; the parameters are ignored.

**On this branch:** the same request returns only alerts tagged `prod` in the `ALERT` state, and `meta.total` equals that count.

`make dev-int FILE=external-api/__tests__/alerts` runs the new cases.

### How to test on Vercel preview

N/A, API-only change

### References

- Linear Issue:
- Related PRs: #3504 touches the same router, controller and OpenAPI file; I'll resolve the overlap once one of them lands.
@pulpdrew
pulpdrew requested a review from a team as a code owner November 19, 2026 16:06
pulpdrew
pulpdrew previously approved these changes Nov 19, 2026

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution!

@kodiakhq
kodiakhq Bot merged commit 7905721 into ClickHouse:main Nov 19, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silence and unsilence alerts through the external API v2

2 participants