Skip to content

feat(llm): alert when Venice credits run low - #398

Merged
spalen0 merged 4 commits into
mainfrom
feat/llm-credit-alert
Sep 30, 2026
Merged

spalen0 merged 4 commits into
mainfrom
feat/llm-credit-alert

Conversation

@spalen0

@spalen0 spalen0 commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The Venice account ran out of credits. Every AI explanation failed, and the only trace was a log line. Safe and timelock alerts went out without an AI summary, and nothing reported it.

What

  • New utils/llm/credits.py. It reads the key's balance from GET /api_keys/rate_limits because /billing/balance needs an admin key.
  • Venice reports what the key can still spend: the lower of the account balance and what's left of the key's daily spend limit (currently $5). A low reading can mean the account is nearly empty or the daily cap is nearly used. The alert names both causes and shows the 00:00 UTC reset time. The inference key can't tell them apart, because /billing/balance needs an admin key.
  • Called at the start of main() in protocols/safe/main.py (every 20 min) and protocols/timelock/timelock_alerts.py (hourly).
  • Alerts when DIEM + bundled credits + USD (keys spend all three, in that order) is below LLM_CREDIT_ALERT_THRESHOLD_USD (default $1), or when Venice reports accessPermitted: false.
  • Sends to the ops errors channel via send_error_message with notification sound on. It's labelled yearn and stored under yearn-internal, so it stays off public pages.
  • Sends at most one alert every 24h while the balance stays low. A top-up clears the cooldown.
  • Does nothing for providers other than Venice or when no key is set. It never raises: a failed balance read, cooldown write or alert send is logged and the monitor carries on. A failed send retries on the next run.

Testing

  • tests/test_llm_credits.py: 12 cases covering the threshold, DIEM, bundled credits, blocked access, the env override, cooldown, top-up reset, fetch failure, send failure, a bad threshold value and the skip conditions.
  • tests/test_safe_main.py: the quota tests that run main() now stub the check so they don't make real calls.
  • Ran the balance read once against the live API and got usd=2.83. The account held $24.75 at the time, so that figure is what was left of the key's $5 daily limit.
  • 4 tests in test_safe_main.py fail locally both on this branch and on unchanged main, so this PR didn't introduce them.

🤖 Generated with Claude Code

spalen0 and others added 4 commits September 30, 2026 16:37
The Venice account ran out of credits and every AI explanation silently
degraded to no summary. Check the key's balance at the start of each Safe
and timelock run and send an ops alert when USD + DIEM drops below $1
(LLM_CREDIT_ALERT_THRESHOLD_USD) or access is blocked. Deduped to once per
24h while low; a top-up resets the cooldown.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Venice's per-key balance is the lower of the account balance and what is
left of the key's daily spend limit, so a low reading can also mean the
daily cap is nearly used. Say so in the alert and show when it resets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Venice keys spend DIEM, then bundled credits, then USD, so bundled credits
belong in the spendable balance. Catch every failure in the check (alert
send, cooldown state, threshold parsing) so a diagnostic can never stop
the Safe/timelock monitors from running; a failed send retries next run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fixes GHSA-8988-9cw3-xx77, GHSA-gh4c-6fx4-qh6g and GHSA-vxq7-64xx-v4gw,
which fail the audit job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spalen0
spalen0 marked this pull request as ready for review September 30, 2026 17:36
@spalen0
spalen0 merged commit f99f71f into main Sep 30, 2026
3 checks passed
@spalen0
spalen0 deleted the feat/llm-credit-alert branch September 30, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant