Skip to content

feat(audit): ingest rate-limit circuit breaker (#831) - #98

Merged
nycomp merged 1 commit into
mainfrom
feat/831-rate-breaker
Oct 6, 2026
Merged

nycomp merged 1 commit into
mainfrom
feat/831-rate-breaker

Conversation

@nycomp

@nycomp nycomp commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Producer side of the audit ingest rate limit — the circuit breaker half of campus#831 (Phase 3 of campus#538). The audit-side enforcement landed in nyjc-computing/campus#835: POST /traces/ 429s carry a Retry-After header and the tripped bucket key in error.details.bucket. Design (ratified): Phase 3 design on campus#538.

What

  • New module campus_python.audit.ratelimit — a thread-safe, per-process circuit breaker:
    • observe(status, headers, body): a 429 trips the bucket named in the body (floor 30s, ×2 backoff on consecutive trips, cap 5 min; audit's Retry-After honored up to the cap). A 2xx clears trips whose cooldown has expired — the ratified two-stage retraction: cooldown expiry opens the gate (verifying) without clearing, and the first 2xx ingest fully clears. Availability errors (timeouts, connection refused, 5xx) neither set nor clear — an audit outage must not take producers down.
    • apikey= trips are remembered as this producer's own fallback bucket (an apikey-bucket 429 seen by this process's client is necessarily its own), so identity-less requests can be gated.
    • check_request(client_id, user_id): per-identity gate key (same priority encoding as the server: pair → user → client), degrading to the learned fallback bucket or the wildcard when identity isn't resolvable at request entry.
  • Traces.ingest feeds the breaker before raising, with the observation wrapped in contextlib.suppress — a breaker failure can never break ingestion.

Consumers

The follow-on campus PR wires this into the producer middleware (campus/audit/middleware): a flag-gated (AUDIT_TRACING_FAIL_CLOSED) before_request gate that 503s matching requests with Retry-After while a bucket is tripped. campus-classroom inherits via its dependency on the campus package. This PR is self-contained (no behavior change for existing callers — the breaker only records; nothing gates until a consumer opts in).

Testing

  • New tests/unit/test_ratelimit.py (19 tests): trip/check/backoff/cap math, two-stage retraction, availability neutrality, fallback learning, check_request key routing, and the Traces.ingest hook (429 trips + raises, 201 doesn't trip).
  • Full unit suite green (268 tests); ruff clean on the touched files (remaining repo baseline findings are pre-existing on main).

Phase 3 of campus#538. The breaker is the producer side of the audit
ingest rate limit (campus PR #835): audit 429s carry Retry-After and
the tripped bucket key; this module records them and exposes a gate
check for producers.

- campus_python.audit.ratelimit: thread-safe per-process breaker —
  429 trips the bucket (floor 30s, x2 backoff on consecutive trips,
  cap 5min), 2xx clears expired trips (two-stage retraction: cooldown
  expiry opens the gate, first 2xx clears), availability errors
  (timeouts/refused/5xx) neither set nor clear.
- apikey= trips are remembered as the producer's own fallback bucket
  so identity-less requests can be gated coarsely.
- Traces.ingest feeds the module breaker before raising; observation
  is guarded and can never break ingestion.
- check_request(): per-identity gate key when identity is resolvable,
  degrading to the learned fallback bucket or wildcard.
@nycomp
nycomp merged commit 9caf58b into main Oct 6, 2026
2 checks passed
@nycomp
nycomp deleted the feat/831-rate-breaker branch October 6, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants