Skip to content

feat(voice): one voice, Crow, on the phone overlay, kiosk and glasses, with its own card and editor in Bot Builder (kiosk 0.4.0, meta-glasses 0.3.0, artifacts 0.6.4) - #533

Merged
kh0pper merged 13 commits into
mainfrom
feat/crow-voice-pr1-bot
Oct 20, 2026

Conversation

@kh0pper

@kh0pper kh0pper commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Why

Each voice surface answered as a bot chosen on that device: the phone overlay and kiosk as the dashboard bot, the glasses as their own bound bot. This PR replaces that with one voice, Crow. Crow is a reserved bot (crow.voice) with its own settings, answering the same way on all three surfaces. Hand-offs to other bots come in a later PR. This PR ships the voice bot itself, so it is useful on its own.

What changes

The voice bot is created once, and nothing else is edited (servers/shared/voice-bot.js, servers/gateway/voice/ensure-voice-bot.js, servers/gateway/boot/post-listen.js)

  • At boot, the gateway inserts crow.voice (display name "Crow") if no such row exists, using INSERT … ON CONFLICT DO NOTHING. If a row already exists, whether enabled, disabled or unreadable, it is never rewritten. An unreadable row gets a log warning instead.
  • The old voice bindings are recorded in migrated_from and nothing else is touched. That covers the effective dashboard bot ("Automatic" resolves as the kiosk does) and each device's bound bot. Device records, settings and other bots' definitions are left as they are.
  • The fast model carries over from the old dashboard bot when its provider row is enabled and lists that model. Otherwise the router's fast voice key is used. A missing key refuses to create the row.
  • Tool carry-over is allowlist-based. An old voice bot's tool selection is carried only when all of these hold:
    • no source bot denies it;
    • it only reads (search, list, get);
    • it is not one of the delegation or bot-schedule tools;
    • it does not open a Crow core tool category that Crow's preset doesn't already open.
  • Source confirm entries for carried tools come along, so the voice bot never loosens an approval. Skipped tools are recorded in migrated_from.skipped_tools with the reason. Write tools (e.g. a calendar create) are not carried; Crow's own routine tools arrive in the next PR.

Every voice turn answers as Crow (servers/gateway/voice/turn.js)

  • The shared voice turn resolves crow.voice instead of the device's bound bot.
  • opts.botId still overrides it, for the turn check and the evaluation harness.
  • The fast paths (pause, stop, clock and so on) still run before the bot loads.
  • A missing, disabled or unreadable voice bot gives the surface's no-assistant line and never throws.
  • The bot cache is dropped when Bot Builder saves.
  • The turn records its surface (phone, kiosk or glasses, with unknown treated as kiosk).
  • With escalation off, the turn never escalates.

One set of voice rules, and a yes before changes (servers/gateway/voice/crow-voice.js, voice-lines.js, action-claims.js, turn-helpers.js)

  • The kiosk and the glasses now merge the same turn options: tool families on intent, the shared deny list (delegation and bot-schedule tools) and the action lines. The kiosk keeps its own localized lines.
  • For the voice bot, the turn asks for a spoken yes before any tool whose first verb deletes, removes, sends, posts, publishes, shares, moves, renames, updates, edits, replaces, writes, uploads, applies, revokes, cancels and the like. The same holds for schedule and crow.md section changes and notification settings, called directly or through the memory category tool.
  • A schedule for background bot work (pipeline:) is refused from voice, and so is re-enabling a schedule without naming its task.
  • The confirm question names what will actually run. For the proxy tools (the Crow category tools and crow_tools), it reads the arguments the executor passes on. The proxy rules now live in one shared module (servers/gateway/ai/proxy-args.js), which the executor and the voice gate both import. On voice, a proxied call is refused when its top-level and executed values disagree, and so is a crow_tools call that needs a yes but puts its arguments outside params.
  • Spanish yes and no words work ("sí", "claro", "dale"; "no", "espera", "cancela"). A negation in the first words is never taken as a yes ("claro que no", "of course not").

Surfaces, Crow's speaking voice and speech recognition (bundles/kiosk, bundles/meta-glasses)

  • The kiosk's session display answers on surface "phone" and a paired display on "kiosk". The glasses answer as Crow with or without a bound bot.
  • Crow's own TTS and STT profiles, when set, are applied in memory for each turn. Device records are never rewritten. The glasses fall back to the device's own voice when Crow's isn't playable there.
  • A cloud fast model turns off tool-family gating.
  • On a look turn, every one of Crow's tools is withheld except the camera.
  • The kiosk "Automatic" pick never resolves to Crow's row.

Bot Builder: Crow's card and editor (servers/gateway/dashboard/panels/bot-builder/voice-bot-card.js, voice-bot-editor.js, i18n.js)

  • Crow's card is listed first, with a "Voice" chip, its model, what it can do by voice, and a Customise button. It has no delete, board, give or use-for-voice items.
  • The editor covers:
    • on/off;
    • preset (Fast & local or Cloud), with "Reset to preset";
    • model;
    • persona and name;
    • speaking voice;
    • speech recognition;
    • Memory tools;
    • the abilities line.
  • With a cloud model, the editor warns that turns are slower and that what you say is sent to the provider.
  • Saves are compare-and-swap, including the on/off switch, so a stale page writes nothing.
  • A preset switch keeps the operator's Memory-tools choice.
  • The generic Bot Builder actions still refuse Crow's row; its refusals now show en/es text.
  • The "Use for voice…" item and the Kiosk/glasses assistant pickers are replaced by "Answers as Crow · Customise in Bot Builder". Pairing a display no longer asks for an assistant, and stored bindings are left untouched.

Guards carried from #528

  • The Artifacts house-style route refuses Crow's row.
  • crow_create_schedule refuses pipeline:botcron:crow.voice, returning an error.

No schema change, no SCHEMA_GENERATION bump, no DELETE or DROP. The AI Companion, which is being retired, is not touched.

Tests

  • New test files:
    • voice-bot-ensure: create once, never overwrite, the migration record, the carry-over rules, and two gateways at once.
    • voice-turn-crow: answers as Crow; fast path with no bot; missing, disabled or unreadable bot; escalation off; surface; spoken yes; Spanish; refusals; a byte-stable system message.
    • crow-voice-options.
    • bot-builder-voice-card.
  • Existing files gained tests: kiosk session mode, kiosk panel, glasses session, action claims, artifacts house style and the voice-bot guards.
  • Full suite (rebased on current main): 10369 tests, 10334 pass, 0 fail, 35 skipped.
  • A gateway started from this branch in a scratch home listens cleanly and creates crow.voice once.

Deploy

#528 is already merged. Pull main on each host and restart the gateway(s). The kiosk, meta-glasses and artifacts installed copies refresh because of the version bumps.

After the restart:

  • check that ~/.crow/bundles/kiosk/manifest.json reads 0.4.0;
  • open Talk to Crow, ask one question and confirm Crow answers;
  • read crow.voice's migrated_from.tools and migrated_from.skipped_tools to see which old voice tool selections were carried and which were not.

Rollback

Revert this PR only, never #528. Then disable the row; this is a disable, never a delete:

UPDATE pi_bot_defs SET enabled = 0 WHERE bot_id = 'crow.voice'

Devices still hold their old bound_bot_id and the old setting is untouched, so the previous behaviour returns.

https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp

The shared voice turn resolves opts.botId, then deps.voiceBotId (the Crow voice bot), then the device's bound bot, after the fast paths. A missing Crow voice bot is ensured once per runner and answers the same turn. Escalation off acts as noEscalate, timings.surface is recorded, and a bot-defs change drops the cached bot. The boot and turn ensures share one builder. The glasses session tests now pin the voice bot.

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
… as Crow, in Crow's own voice (kiosk 0.4.0, meta-glasses 0.3.0)

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
… a preset switch keeps Memory tools off, and the no-assistant link opens Crow's editor

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
… never confirms, and a corrupt voice row says so

- Memory tools off drops every crow-memory/ entry (a carried read-only one kept the whole
  crow_memory proxy); the switch and a preset switch read any memory entry; on adds the three back.
- A negation word in the first four words ("claro que no", "of course not", "yes, no wait")
  is never a yes and always a no.
- A corrupt crow.voice row warns once from the boot and the turn-time ensure and is left
  untouched; the voice editor says the settings can't be read (en + es) instead of
  "not set up yet", and a save on it is refused with voice_corrupt.
- Kiosk and glasses no-assistant lines point at Crow's voice in Bot Builder; the room hint
  names Reset to preset.
- Hello warms Crow's STT profile, not the device's own.
- The memory server's voice-bot botcron refusal is isError.
- A voice bot's notification settings change needs a spoken yes (direct or proxied; get stays free).
- The confirm question reads a proxied call's params and says "delete schedule N" for a delete.
- A test pins the voice system message byte-identical across turns.

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
…roxied call whose top-level and params values disagree is refused

- crow_notification_settings: the deciding action field comes from the tool (direct: args.action,
  crow_memory proxy: params.action), and any other present action field that is not "get" also asks.
- The confirm question describes only what the executor passes on (params for a category proxy,
  the call's own args for a direct tool).
- A voice bot's proxied call whose top-level copy of a described field (id, delete, section_key,
  task, ...) disagrees with params is refused outright, before any question.

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
…uestion names what runs, and a disagreeing or flat-only voice delete is refused

- One shared module (servers/gateway/ai/proxy-args.js) holds the proxy matcher and crow_tools'
  parameter normalization; the executor and the voice gate both use it, so they cannot drift.
- The confirm question reads the object the executor runs (params, the arguments unwrap, the
  flat fallback) for every proxy, crow_tools included.
- For a voice bot, a proxied call whose top level, params or params.arguments disagrees with what
  runs is refused; so is a crow_tools call that needs a yes and relies on the flat fallback (the
  remote route never runs those values). "3" and 3 are the same id.

Claude-Session: https://claude.ai/code/session_01N9fXzvNy8rCEArj1y5p6fp
@kh0pper
kh0pper merged commit 9211829 into main Oct 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant