fix(limits): enforce route caps and close the places limits leaked

CHANGELOG.md

@@ -11,6 +11,78 @@ target.
 
 ## [Unreleased]
 
+Limits hold where they used to leak: route caps are enforced, a limits
+store that cannot be read follows `security.rate_limit_fail_closed`, a
+refused model or tool no longer spends a request, a team-scoped role no
+longer limits the gateway it does not open, and cached answers count their
+tokens. The legacy monthly token quota is gone.
+
+### Read before upgrading
+
+- **Route caps are enforced.** A route's RPM cap and TPM cap
+  (`model_routes.rpm_cap` / `tpm_cap`, set in the route editor) were saved
+  but never applied. They now are, per route over a sliding minute, the TPM
+  cap in weighted tokens like every other token limit (counted after the
+  response). A route at either cap is skipped and the model's next route
+  serves the request; when every route left is at its cap, the request gets
+  429 with `Retry-After`, labelled `route:requests/1m` or
+  `route:tokens/1m`. Review the caps already set on your routes: a cap that
+  did nothing until now takes effect on upgrade. Routes with a cap write new
+  Redis keys, `ratelimit:{route:<route_id>}:ai_gateway:route:…`.
+- **A limits store that cannot be read follows
+  `security.rate_limit_fail_closed`.** When the database failed while a
+  request's rate limits and budgets were being loaded, the request ran with
+  no limits at all; when Redis failed while a budget was being read, the
+  budget read as unspent. Both now follow the setting, as a Redis failure
+  in the rate limiter always has, and so do the route caps: off (the
+  default), the request goes on and a warning with the error is logged
+  (`gateway_limits_load_fail_open_total`,
+  `lifecycle_budget_fail_open_total`); on, the request is refused with 429,
+  `Retry-After: 30`, labelled `limits_unavailable`, `budget_unavailable` or
+  `rate_limiter_unavailable`.
+- **A role granted at team scope no longer limits gateway requests.** Such
+  a grant administers that team and has never opened a model or an MCP tool
+  at the gateways, but its rate limits and budgets were applied to every
+  gateway request of the user who held it. A role's limits now apply
+  exactly where its gateway grant does: roles assigned globally, and roles
+  attached to a team the user is a member of. A user who held a limiting
+  role only at team scope is no longer held to its limits; assign the role
+  globally, attach it to the team, or set a user limit if they were meant.
+- **Answers from the response cache count their tokens.** A cache hit
+  counted toward request limits but added no tokens to token limits and
+  budgets, so repeating a cached request got round them. It now adds the
+  tokens the cached answer records, weighted like any answer (all of its
+  input as plain input). The cost a cache hit reports stays 0. Callers that
+  lean on the cache reach their token limits and budgets sooner.
+
+### Fixed
+
+- **A request the key may not make no longer spends a request limit.** The
+  rate-limit check, which counts the request, ran before the model check on
+  the AI gateway and before the tool check on the MCP gateway, so a refused
+  model or tool used up the caller's `requests` limits. Access is now
+  checked first (after the budget check, which counts nothing).
+- **Route caps are applied**, **limits and budgets that cannot be read
+  follow the fail-closed setting**, **team-scoped roles no longer limit**
+  and **cached answers count their tokens**: see Read before upgrading.
+- Comments that described a limits cache, a `limits:changed` subscriber and
+  team-level budget caps, none of which exist, now say what happens: limits
+  are read from the database on every request, nothing subscribes to
+  `limits:changed`, and teams carry no limits of their own.
+
+### Changed
+
+- **The legacy monthly token quota is removed.** Every request counted its
+  tokens into `quota:{<user or key>:<model>}:used:<YYYY-MM>` in Redis
+  against a limit nothing could set, so it never refused anything. The
+  gateway no longer reads or writes these keys; existing `quota:*` keys can
+  be deleted. Monthly limits are budgets. Its metric,
+  `gateway_quota_overflow_total`, is gone too: the Grafana overview panel
+  that plotted it shows routes at their cap (`gateway_route_capped_total`)
+  instead.
+- The `security.rate_limit_fail_closed` hint in Settings names everything
+  the setting now covers.
+
 ## [3.4.0] — 2026-10-11
 
 Conversations with reasoning models behind Chat-format upstreams keep their

crates/auth/src/rbac.rs

@@ -15,6 +15,10 @@ use uuid::Uuid;
 // (a role granted at team scope administers that team only).
 // Rate limits: per (metric, window) take MIN MaxCount — most restrictive.
 // Budgets: per Period take MIN MaxTokens — most restrictive.
+// Rate limits and budgets come from the same roles as the model / tool
+// scope (`load_gateway_policy_documents`), and from the same statements
+// (the `Constraints` of the Allow for `*_gateway:use`): a role's limits
+// apply exactly when its gateway grant does.
 // Deny statements: win over Allow across all roles.
 //
 // `compute_user_permissions` is the single source of truth for the
@@ -49,8 +53,9 @@ pub async fn compute_user_permissions(
 }
 
 /// Load all `policy_document` JSONB values from roles assigned to
-/// `user_id` (direct + team-inherited). Shared helper for permission
-/// extraction, resource limit computation, and constraint aggregation.
+/// `user_id` (direct at any scope + team-inherited). The console's
+/// permission checks start from these; the gateways do not (see
+/// [`load_gateway_policy_documents`]).
 async fn load_user_policy_documents(
     pool: &PgPool,
     user_id: Uuid,
@@ -72,14 +77,18 @@ async fn load_user_policy_documents(
     Ok(rows.into_iter().map(|(v,)| v).collect())
 }
 
-/// The policy documents that decide gateway access for `user_id`:
-/// roles assigned at global scope, plus roles attached to a team the
-/// user is a member of (a team's roles are its members' working roles).
+/// The policy documents that apply to `user_id`'s gateway requests —
+/// the one place that decides it, for access (models, MCP tools) and
+/// for the role-level rate limits and budgets alike: roles assigned at
+/// global scope, plus roles attached to a team the user is a member of
+/// (a team's roles are its members' working roles).
 ///
 /// Roles granted at `scope_kind = 'team'` are left out. Such a grant
 /// lets the holder administer that team from the console; gateway
 /// requests carry no team, so honouring it here would turn a team
-/// grant into platform-wide model and tool access.
+/// grant into platform-wide model and tool access — or, for its limits,
+/// hold the holder's every request to a role that grants them nothing
+/// at the gateway.
 async fn load_gateway_policy_documents(
     pool: &PgPool,
     user_id: Uuid,
@@ -128,26 +137,6 @@ pub async fn load_user_role_names(
     Ok(rows.into_iter().map(|(n,)| n).collect())
 }
 
-/// Load the distinct role IDs assigned to `user_id`, including
-/// roles inherited through team membership.
-/// Used by the rate-limit engine so role-level quotas apply to the user.
-pub async fn load_user_role_ids(pool: &PgPool, user_id: Uuid) -> Result<Vec<Uuid>, sqlx::Error> {
-    let rows: Vec<(Uuid,)> = sqlx::query_as(
-        "SELECT DISTINCT role_id FROM ( \
-           SELECT ra.role_id FROM rbac_role_assignments ra WHERE ra.user_id = $1 \
-           UNION \
-           SELECT tra.role_id \
-             FROM team_members tm \
-             JOIN team_role_assignments tra ON tra.team_id = tm.team_id \
-            WHERE tm.user_id = $1 \
-         ) combined",
-    )
-    .bind(user_id)
-    .fetch_all(pool)
-    .await?;
-    Ok(rows.into_iter().map(|(id,)| id).collect())
-}
-
 /// Load every `(role_id, scope_kind, scope_id)` row for `user_id`,
 /// including roles inherited through team membership (surfaced as
 /// global scope since team roles grant permissions platform-wide).
@@ -315,11 +304,16 @@ fn resource_limits_from_documents(docs: &[serde_json::Value]) -> UserResourceLim
 /// `compute_user_surface_constraints` so callers that need both the
 /// pre-override and post-override view (the admin dashboard) can avoid
 /// re-running the role document load.
+///
+/// From the roles that decide gateway access
+/// ([`load_gateway_policy_documents`]), the same set
+/// [`compute_user_resource_limits`] reads: a role granted at team scope
+/// neither opens the gateway nor limits it.
 pub async fn compute_user_role_constraints(
     pool: &PgPool,
     user_id: Uuid,
 ) -> Result<think_watch_common::limits::SurfaceConstraints, sqlx::Error> {
-    let docs = load_user_policy_documents(pool, user_id).await?;
+    let docs = load_gateway_policy_documents(pool, user_id).await?;
     let per_role: Vec<think_watch_common::limits::SurfaceConstraints> = docs
         .iter()
         .map(think_watch_common::limits::extract_surface_constraints)
@@ -329,9 +323,9 @@ pub async fn compute_user_role_constraints(
     ))
 }
 
-/// Aggregate surface constraints from policy_documents across every
-/// role assigned to `user_id` (direct + team-inherited) using "most
-/// restrictive wins": per `(surface, metric, window_secs)` take the
+/// Aggregate surface constraints from the policy_documents of the roles
+/// that apply at the gateway (see [`compute_user_role_constraints`])
+/// using "most restrictive wins": per `(surface, metric, window_secs)` take the
 /// MIN `max_count`; per `(surface, period)` take the MIN
 /// `limit_tokens`. Disabled or non-positive entries are ignored, then
 /// any active side-table overrides for the user are applied on top.

crates/common/src/lifecycle/mod.rs

@@ -9,8 +9,8 @@
 //! ```text
 //! Raw<S>
 //!   → check_budget      (pre-call budget peek; charges nothing)
+//!   → check_access      (allowed_models / allowed_tools; charges nothing)
 //!   → check_limits      (rate-limit gate; charges only when it passes)
-//!   → check_access      (allowed_models / allowed_tools)
 //!   → surface-specific  (cache lookup, breaker, credential resolution,
 //!                        invoke_upstream → Invocation<S>)
 //!   → run_post_invoke
@@ -29,7 +29,7 @@
 //!   `record_outcome` / `write_cache` / `record_usage` /
 //!   `emit_audit` and short-circuit response factories.
 //! - [`state`] — per-stage state structs ([`state::Raw`],
-//!   [`state::LimitsChecked`], [`state::Authorized`],
+//!   [`state::AccessChecked`], [`state::Authorized`],
 //!   [`state::Invoked`], [`state::Emitted`]) plus the
 //!   [`state::Invocation`] / [`state::CapturedView`] enums that
 //!   model the buffered/streaming fork at `invoke_upstream`.

crates/common/src/lifecycle/stages/check_access.rs

@@ -4,14 +4,18 @@
 //! decision in the same audit + short-circuit pattern
 //! [`super::check_limits`] uses, so the deny path is uniform
 //! regardless of which surface invoked the stage.
+//!
+//! Runs before `check_limits`, which charges the `requests` counters:
+//! a request the identity may not make is refused without counting
+//! against any limit. Charges nothing itself.
 
 use crate::audit::AuditLogger;
 
 use super::super::Surface;
-use super::super::state::{Authorized, LimitsChecked};
+use super::super::state::{AccessChecked, Raw};
 
 /// Run the access check. On allow, transition to
-/// [`Authorized`] carrying the `candidate` string for downstream
+/// [`AccessChecked`] carrying the `candidate` string for downstream
 /// audit. On deny, emit a `"access_denied"` audit row tagged with
 /// the subject and short-circuit with
 /// [`Surface::access_denied_response`].
@@ -20,10 +24,10 @@ use super::super::state::{Authorized, LimitsChecked};
     fields(trace_id = %state.trace_id, candidate = %candidate),
 )]
 pub async fn check_access<S: Surface>(
-    state: LimitsChecked<S>,
+    state: Raw<S>,
     candidate: &str,
     audit: &AuditLogger,
-) -> Result<Authorized<S>, S::Response> {
+) -> Result<AccessChecked<S>, S::Response> {
     if !S::is_access_allowed(&state.identity, candidate) {
         metrics::counter!("lifecycle_access_denied_total").increment(1);
         tracing::warn!(
@@ -38,42 +42,27 @@ pub async fn check_access<S: Surface>(
         return Err(S::access_denied_response(candidate));
     }
 
-    Ok(Authorized {
+    Ok(AccessChecked {
         identity: state.identity,
         trace_id: state.trace_id,
         started_at: state.started_at,
         client_ip: state.client_ip,
-        limit_check: state.limit_check,
         access_candidate: candidate.to_owned(),
     })
 }
 
 #[cfg(test)]
 mod tests {
-    use super::super::super::state::{LimitCheckRecord, LimitsChecked};
     use super::super::super::test_surface::{TestResponse, TestSurface, make_raw};
     use super::*;
     use uuid::Uuid;
 
-    fn make_limits_checked(user_id: Uuid) -> LimitsChecked<TestSurface> {
-        let raw = make_raw(user_id);
-        LimitsChecked {
-            identity: raw.identity,
-            trace_id: raw.trace_id,
-            started_at: raw.started_at,
-            client_ip: raw.client_ip,
-            limit_check: LimitCheckRecord {
-                currents: Vec::new(),
-            },
-        }
-    }
-
     #[tokio::test]
     async fn allow_passes_through() {
         // `TestSurface::is_access_allowed` returns true iff
         // candidate starts with "allowed_" — see test_surface.rs.
         let user_id = Uuid::new_v4();
-        let state = make_limits_checked(user_id);
+        let state = make_raw(user_id);
         let result = check_access::<TestSurface>(
             state,
             "allowed_tool",
@@ -92,7 +81,7 @@ mod tests {
     #[tokio::test]
     async fn deny_short_circuits_with_response() {
         let user_id = Uuid::new_v4();
-        let state = make_limits_checked(user_id);
+        let state = make_raw(user_id);
         let result = check_access::<TestSurface>(
             state,
             "blocked_tool",

crates/common/src/lifecycle/stages/check_budget.rs

@@ -14,13 +14,16 @@
 //!
 //! No mutation, so failure of the Redis read defaults to fail-open
 //! (request allowed) unless the caller passes `fail_closed = true`
-//! — matching the [`super::check_limits`] semantics.
+//! — matching the [`super::check_limits`] semantics. A failed read is
+//! never taken for an unspent budget (`budget::current_spend` returns
+//! the error).
 //!
-//! Ordering note: this stage runs BEFORE `check_limits`, on the
-//! [`Raw`] state. `check_limits` charges the `requests` counters, so
-//! running it first would charge a request this stage then refuses.
-//! Since this stage charges nothing, a request refused by either
-//! charges nothing.
+//! Ordering note: this stage runs first, on the [`Raw`] state, then
+//! `check_access`, then `check_limits`. `check_limits` charges the
+//! `requests` counters, so running it before either of the others
+//! would charge a request they then refuse. Neither of them charges
+//! anything, so a request refused before `check_limits` charges
+//! nothing.
 
 use chrono::Utc;
 use fred::clients::Client;
@@ -141,8 +144,16 @@ mod tests {
     use uuid::Uuid;
 
     fn dummy_redis() -> fred::clients::Client {
+        // Never connected, so a command waits for a connection that
+        // never comes; the timeout turns that into the error an outage
+        // gives.
         let cfg = RedisConfig::from_url("redis://127.0.0.1:6379").expect("parse url");
-        Builder::from_config(cfg).build().expect("build client")
+        Builder::from_config(cfg)
+            .with_performance_config(|c| {
+                c.default_command_timeout = std::time::Duration::from_millis(50)
+            })
+            .build()
+            .expect("build client")
     }
 
     fn dummy_audit() -> crate::audit::AuditLogger {
@@ -168,6 +179,50 @@ mod tests {
         assert_eq!(passed.started_at, started_at);
     }
 
+    fn one_cap() -> BudgetCap {
+        BudgetCap {
+            id: Uuid::nil(),
+            subject_kind: BudgetSubject::User,
+            subject_id: Uuid::nil(),
+            period: BudgetPeriod::Daily,
+            limit_tokens: 1000,
+            enabled: true,
+            expires_at: None,
+            reason: None,
+            created_by: None,
+        }
+    }
+
+    /// The budget can't be read and the gateway fails closed: refused
+    /// as unavailable. (The disconnected client fails every command.)
+    #[tokio::test]
+    async fn an_unreadable_budget_refuses_when_failing_closed() {
+        let result = check_budget::<TestSurface>(
+            make_raw(Uuid::new_v4()),
+            &[one_cap()],
+            &dummy_redis(),
+            true,
+            &dummy_audit(),
+        )
+        .await;
+        assert_eq!(result.err(), Some(TestResponse::BudgetUnavailable));
+    }
+
+    /// The budget can't be read and the gateway fails open: the request
+    /// goes on.
+    #[tokio::test]
+    async fn an_unreadable_budget_passes_when_failing_open() {
+        let result = check_budget::<TestSurface>(
+            make_raw(Uuid::new_v4()),
+            &[one_cap()],
+            &dummy_redis(),
+            false,
+            &dummy_audit(),
+        )
+        .await;
+        assert!(result.is_ok());
+    }
+
     #[test]
     fn label_format_is_subject_budget_period() {
         let cap = BudgetCap {