From 3d86313dfe5ce59b1f3c176ca1fcd642de3c2fcc Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:55:20 -0600 Subject: [PATCH 01/17] feat(sessions): a weight edit shows as a stat, a tag and a readable line in the CLI, MCP and viewer A weights.edit notification was only stored: show printed a blank notification row, MCP's session_get left it out of the timeline, and no stat or tag said a session had edited weights. Sessions now carry stats.weightEdits (events, tensors, weightsChanged) and the weights:edited tag, with --where keys weights.edits and weights.changed. show, MCP (sessions_list, session_get, the session resource) and the viewer write one line per edit from one formatter. A session that holds only a weight edit is no longer hidden as empty. SESSION_FOLD_VERSION 20 refolds stored sessions. --- CHANGELOG.md | 1 + docs/reference/api.md | 12 +- docs/reference/cli.md | 2 +- docs/reference/query.md | 3 + packages/coding.sh/src/cli/commands/show.ts | 3 + packages/coding.sh/src/core/sessions.ts | 19 +++- packages/coding.sh/src/core/weights.ts | 106 ++++++++++++++++++ packages/coding.sh/src/mcp/data.ts | 5 +- packages/coding.sh/src/mcp/digest.ts | 2 + packages/coding.sh/src/mcp/tools.ts | 2 +- packages/coding.sh/src/query/where.ts | 2 + packages/coding.sh/src/refine/taxonomy.ts | 2 + packages/coding.sh/src/schema/index.ts | 15 +++ .../coding.sh/src/schema/trace.schema.json | 27 +++++ packages/coding.sh/src/viewer/model.ts | 9 +- packages/coding.sh/src/viewer/ui/index.html | 26 +++++ .../coding.sh/test/cli/weights-show.test.ts | 101 +++++++++++++++++ packages/coding.sh/test/core/reparse.test.ts | 18 +++ packages/coding.sh/test/core/weights.test.ts | 81 +++++++++++++ packages/coding.sh/test/mcp/weights.test.ts | 78 +++++++++++++ .../coding.sh/test/viewer/ui-browser.test.ts | 15 +++ .../coding.sh/test/viewer/weights-row.test.ts | 63 +++++++++++ 22 files changed, 579 insertions(+), 13 deletions(-) create mode 100644 packages/coding.sh/src/core/weights.ts create mode 100644 packages/coding.sh/test/cli/weights-show.test.ts create mode 100644 packages/coding.sh/test/core/weights.test.ts create mode 100644 packages/coding.sh/test/mcp/weights.test.ts create mode 100644 packages/coding.sh/test/viewer/weights-row.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d0c7ba07..f451bf07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ grouped by area, with links to the docs for the details. - **Lint.** New rules: vendor-literal, site-voice, universal-transition, reduced-motion, global-install and personal-data. `slop.mjs --strict-allowlist` fails on allowlist entries that no longer match anything. [Contributing](https://github.com/jackulau/coding.sh/blob/main/CONTRIBUTING.md) - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) +- **Sessions.** A session that recorded weight edits gets `stats.weightEdits` (edits, distinct tensors, weights changed) and the `weights:edited` tag, and is listed even when it holds nothing else. `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit instead of a blank notification (MCP dropped them), and `coding.sh sessions --where 'weights.changed>1000'` filters by size. Stored sessions are folded again. [Docs](https://coding.sh/docs/api) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/reference/api.md b/docs/reference/api.md index 220898e7..f1261277 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -718,11 +718,19 @@ Three `notification` kinds carry the events behind these stats: | `fork` | `toolUseId` (the parent's Agent call) and `inheritedTokens` | | `hook.context` | `hookEvent`, `hookName`, `hookSource`, `command`, `chars`, `tokens` and `text` (clipped). Metadata mode keeps only `hookEvent`, `hookSource`, `chars` and `tokens` | -One more `notification` kind records weight edits. The Python SDK's `coding_sh.weights_edit()` sends it, and any client can post it to `/v1/ingest`. It adds no stat and no flag; the event shows in the session's events. Redaction runs on `tensor` and `note` like on any payload. +One more `notification` kind records weight edits. The Python SDK's `coding_sh.weights_edit()` sends it, and any client can post it to `/v1/ingest`. Redaction runs on `tensor` and `note` like on any payload. | `payload.kind` | Fields | |---|---| -| `weights.edit` | `tensor` (string, the tensor's name), `shape` (array of non-negative integers, at most 16), `changed` (integer, the weights that differ, at most the product of `shape`), `l2` (number >= 0, the L2 norm of edited minus base), `maxAbs` (number >= 0, the largest absolute value of that change), `baseSha256` and `editedSha256` (64 lowercase hex characters, the tensor's hash before and after) and `note` (string, optional, left out in metadata mode) | +| `weights.edit` | `tensor` (string, the tensor's name), `shape` (array of non-negative integers, at most 16), `changed` (integer, the weights that differ, at most the product of `shape`), `l2` (number >= 0, the L2 norm of edited minus base), `maxAbs` (number >= 0, the largest absolute value of that change), `baseSha256` and `editedSha256` (64 lowercase hex characters, the tensor's hash before and after) and `note` (string, optional) | + +| Field | Meaning | +|---|---| +| `stats.weightEdits` (`events`, `tensors`, `weightsChanged`) | Weight edits the session recorded: how many, how many distinct tensor names, and their `changed` counts added up (a weight two edits changed counts twice). Absent when the session recorded none. `tensors` is absent when no edit kept its name | + +- A session with a weight edit gets the `weights:edited` tag. `coding.sh sessions --flag weights:edited` lists them, and `--where 'weights.changed>1000'` filters by size. +- `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. +- Metadata mode: the SDK leaves out `note`. A server with `capture.mode = "metadata"` also drops `tensor`, `shape` and both hashes, and keeps `changed`, `l2` and `maxAbs`. - The caller computes the hashes and norms. The SDK checks the values, drops an invalid edit with one warning, and never raises into the app. Numpy and torch scalars count as numbers. - `weights_edit()` returns `True` only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index b847b3d5..4da08c63 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -508,7 +508,7 @@ coding.sh sessions --agent codex --since 7d ``` - --where: key:value clauses, AND by default; OR between clauses; ( ) to group; - to negate; a,b for either value; \* as a wildcard in text; > >= < <= and a..b on numbers. Bare words search prompts, replies and tool calls (every word in one event). -- Session keys: agent, model, provider, repo, launch, branch, dir, title, id, run, parent, flag, skill, hook, hook.tokens, since, until, duration, turns, tools, tool.errors, llm.errors, files, events, tests.passed, tests.failed, tokens, tokens.in, tokens.out, cost, cache.breaks, cache.lost, cache.cost, unapproved, has, is. Event keys (the session has a matching event; event keys in one group match the same event): kind, tool, ok, mcp, error, event.duration. Also sort:-cost and limit:N. Full list with examples: https://coding.sh/docs/query. +- Session keys: agent, model, provider, repo, launch, branch, dir, title, id, run, parent, flag, skill, hook, hook.tokens, weights.edits, weights.changed, since, until, duration, turns, tools, tool.errors, llm.errors, files, events, tests.passed, tests.failed, tokens, tokens.in, tokens.out, cost, cache.breaks, cache.lost, cache.cost, unapproved, has, is. Event keys (the session has a matching event; event keys in one group match the same event): kind, tool, ok, mcp, error, event.duration. Also sort:-cost and limit:N. Full list with examples: https://coding.sh/docs/query. - --facets counts each key over the query without that key's own clauses, so every value of a picked key still shows. --limit sets the values per key (default 10). - With --json each session is a digest: its stats without stats.phases, and each derail without the event it started at (eventId, index). show <id> --json has both, and --fields stats.phases or --fields stats.derails prints them here. - With --json, an answer given before every capture was indexed carries "pending": {"files": N, "indexing": true\|false} (files still to ingest; indexing: an ingest is still running). The field is absent once all are indexed. diff --git a/docs/reference/query.md b/docs/reference/query.md index 63458e45..e1691acb 100644 --- a/docs/reference/query.md +++ b/docs/reference/query.md @@ -64,6 +64,8 @@ test each hit. | `skill` | session | Skill tag (`skill:`) | `skill:pdf` | | `hook` | session | Whose hooks added text to the context: `plugin:`, `plugin`, `settings` or `unknown` | `hook:plugin:terse` | | `hook.tokens` | session | Tokens hooks added to the context (characters / 4) | `hook.tokens>500` | +| `weights.edits` | session | Weight edits the session recorded | `weights.edits>0` | +| `weights.changed` | session | Weights those edits changed, added up | `weights.changed>1000` | | `since` | session | Started at or after: `7d`, `12h`, `today`, `yesterday` or a date | `since:7d` | | `until` | session | Started before (exclusive): `7d`, `12h`, `today` or a date | `until:2026-11-01` | | `duration` | session | Wall time of an ended session | `duration>10m` | @@ -220,6 +222,7 @@ tags out. | `secret:exposed` | Had a vendor token replaced by redaction. The agent and its provider saw it, so rotate it. | | `model:switch` | Changed model mid-session. | | `model:fallback` | Recorded a fallback to another model, such as Claude Code's `--fallback-model`. | +| `weights:edited` | Recorded a weight edit (`stats.weightEdits`). | | `cache:low` | Wrote a prompt cache of at least 4,096 tokens over three or more turns, and read back less than it wrote. | | `fold:error` | Could not be folded into stats. `coding.sh ingest` tries again. | diff --git a/packages/coding.sh/src/cli/commands/show.ts b/packages/coding.sh/src/cli/commands/show.ts index 6adda68a..9643b785 100644 --- a/packages/coding.sh/src/cli/commands/show.ts +++ b/packages/coding.sh/src/cli/commands/show.ts @@ -11,6 +11,7 @@ import { costNote } from "../../core/pricing.ts"; import { failureFlags } from "../../core/sessions.ts"; import { SKILL_EVENT_KINDS, SKILL_TAG_PREFIX, skillUse, type SkillUseReport } from "../../core/skills-use.ts"; import { pageTree, subagentsOf, type TreeNode } from "../../core/tree.ts"; +import { WEIGHTS_EDIT_KIND, weightEditsText, weightsEditText } from "../../core/weights.ts"; import type { AgentEvent, Session } from "../../core/types.ts"; import type { DurableStats } from "../../schema/index.ts"; import { positiveInt } from "../args.ts"; @@ -148,6 +149,7 @@ function describe(e: AgentEvent, starts: Map): string | unde case "context.compact": case "notification": if (p.kind === "harness.loaded") return `harness ${oneLine(clean(harnessText(p)), 90)}`; + if (p.kind === WEIGHTS_EDIT_KIND) return `weights ${oneLine(clean(weightsEditText(p)), 140)}`; if (p.kind === "hook.context") return `hook ${str(p.hookEvent) ?? ""} ${str(p.hookSource) ?? "unknown"} ~${tokens(typeof p.tokens === "number" ? p.tokens : undefined)} tok ${oneLine(clean(str(p.text) ?? ""), 70)}`; return `${`${e.kind} `.padEnd(12)}${oneLine(clean(str(p.text) ?? str(p.message) ?? str(p.reason) ?? ""), 80)}`; default: @@ -475,6 +477,7 @@ export async function run(c: CommandContext): Promise { if (red) rows.push(red); const hooks = hookContextRow(s); if (hooks) rows.push(hooks); + if (s.stats.weightEdits) rows.push(["Weight edits", weightEditsText(s.stats.weightEdits)]); if (skills?.used.length) rows.push(["Skills", skillsText(skills)]); if (derails.length) rows.push(["Derails", derails.slice(0, 4).map((d) => `${d.flag} at +${duration(Math.max(0, d.ts - s.startedAt))}`).join(", ") + (derails.length > 4 ? ` +${derails.length - 4} more` : "")]); for (const [k, v] of rows) out.print(` ${k.padEnd(14)}${v}`); diff --git a/packages/coding.sh/src/core/sessions.ts b/packages/coding.sh/src/core/sessions.ts index 9815f1e8..de946f8d 100644 --- a/packages/coding.sh/src/core/sessions.ts +++ b/packages/coding.sh/src/core/sessions.ts @@ -21,6 +21,7 @@ import { foldWaste, wasteState, wasteStatsOf, type WasteState } from "./waste.ts import { foldModels, mainModelOf, MODEL_FALLBACK, MODEL_SWITCH, modelFlags, modelState, modelSwitchOf, modelsOf, REFUSAL, type ModelState } from "./models.ts"; import { exposesSecret, foldRedactions, redactionsOf, redactionState, SECRET_EXPOSED, type RedactionState } from "./redaction-stats.ts"; import { foldUnapproved, unapprovedOf, unapprovedState, type UnapprovedState } from "./unapproved.ts"; +import { foldWeightEdits, weightEditFlags, weightEditsOf, weightEditState, WEIGHTS_EDITED, type WeightEditState } from "./weights.ts"; export { REOPEN_SLACK_MS }; @@ -55,6 +56,8 @@ export const FLAGS = { unapprovedAction: "unapproved-action", /** A tag, not a failure: the session wrote a prompt cache it barely read back (core/cache-breaks.ts cacheUnderused). */ cacheLow: "cache:low", + /** A tag, not a failure: the session recorded a weight edit (core/weights.ts, Session.stats.weightEdits). */ + weightsEdited: WEIGHTS_EDITED, /** A tag, not a failure: coding.sh could not fold the session's events into its stats (Session.stats.foldError, core/ingest.ts recomputeSession). */ foldError: "fold:error", /** Derails (core/derail.ts): where the session went off the rails. */ @@ -455,7 +458,7 @@ function turnCount(t: { live: number; backfill: number; liveIdless: number; back * new derivation. A new or retyped sub-state changes the stamp by itself; a * new optional field or a change of meaning still needs the bump. */ -export const SESSION_FOLD_VERSION = 19; // 3: TamperState tracks shell calls, own lines and source edits since the last green run. 4: skip markers inside string literals are not skips. 5: proxy calls (sansProxy, otherCalls). 6: hook context (stats.hookContext, hook: tags). 7: stats.launch. 8: premature-completion reads what the agent said as it stopped (PrematureState.said, reported). 9: models and proxyModels (stats.models, stats.modelSwitch, model:switch, refusal; model = the main thread's most used). 10: text-tool-call (FlagState.textToolCall). 11: cache:low tag (core/cache-breaks.ts cacheUnderused). 12: model switches per scope (recorded fallbacks, subagents; ModelSwitch count, switchedCalls, extraUsd; model:fallback). 13: idle cache breaks split by where the wait was (CacheBreak.idle, CacheBreakStats.byIdle). 14: unapproved-action (FlagState.unapproved, stats.unapprovedActions). 15: stats.inheritedTokens (a fork subagent's inherited context). 16: tool failures classified (core/failures.ts): toolErrors counts real ones, stats.benignToolErrors and stats.interruptedToolCalls the rest, repeated-tool-failure and phase toolErrors skip them; a REPL cell's bash()/subprocess commands count as verification (core/cells.ts). 17: a session.start an older ingest stamped with a harness marker names no parent (isSpawnMarker), so a ghost harness parent goes; a gateway record with no cost bills nothing (billedCallOf), so the call's cost stays in usage.estimatedCostUsd. 18: stats.memoryUse, the memory and instruction files a session's events name (core/memory-use.ts, SessionState.memory). 19: a session with transcript model calls counts them, not the copies its OTLP logs carry (sansOtel, transcriptCalls). +export const SESSION_FOLD_VERSION = 20; // 3: TamperState tracks shell calls, own lines and source edits since the last green run. 4: skip markers inside string literals are not skips. 5: proxy calls (sansProxy, otherCalls). 6: hook context (stats.hookContext, hook: tags). 7: stats.launch. 8: premature-completion reads what the agent said as it stopped (PrematureState.said, reported). 9: models and proxyModels (stats.models, stats.modelSwitch, model:switch, refusal; model = the main thread's most used). 10: text-tool-call (FlagState.textToolCall). 11: cache:low tag (core/cache-breaks.ts cacheUnderused). 12: model switches per scope (recorded fallbacks, subagents; ModelSwitch count, switchedCalls, extraUsd; model:fallback). 13: idle cache breaks split by where the wait was (CacheBreak.idle, CacheBreakStats.byIdle). 14: unapproved-action (FlagState.unapproved, stats.unapprovedActions). 15: stats.inheritedTokens (a fork subagent's inherited context). 16: tool failures classified (core/failures.ts): toolErrors counts real ones, stats.benignToolErrors and stats.interruptedToolCalls the rest, repeated-tool-failure and phase toolErrors skip them; a REPL cell's bash()/subprocess commands count as verification (core/cells.ts). 17: a session.start an older ingest stamped with a harness marker names no parent (isSpawnMarker), so a ghost harness parent goes; a gateway record with no cost bills nothing (billedCallOf), so the call's cost stays in usage.estimatedCostUsd. 18: stats.memoryUse, the memory and instruction files a session's events name (core/memory-use.ts, SessionState.memory). 19: a session with transcript model calls counts them, not the copies its OTLP logs carry (sansOtel, transcriptCalls). 20: weight edits (stats.weightEdits, the weights:edited tag, SessionState.weightEdits). /** * A session's events folded so far: everything buildSession needs from them, @@ -559,6 +562,7 @@ export interface SessionState { memory: MemoryUseState; redactions: RedactionState; hookContext: HookContextState; + weightEdits: WeightEditState; /** * Models of the agent's own model calls (core/models.ts), and of the * proxy's: the proxy's count only in a session with no other model calls. @@ -605,6 +609,7 @@ export function sessionState(): SessionState { memory: memoryUseState(), redactions: redactionState(), hookContext: hookContextState(), + weightEdits: weightEditState(), models: modelState(), proxyModels: modelState(), }; @@ -752,6 +757,7 @@ export function foldSession(s: SessionState, e: AgentEvent): void { foldMemoryUse(s.memory, e, s.cwd); foldRedactions(s.redactions, e); foldHookContext(s.hookContext, e); + foldWeightEdits(s.weightEdits, e); if (isEnrichment(e)) return; s.captured++; @@ -912,6 +918,9 @@ export function sessionOf(s: SessionState, prev?: Session, sorted?: readonly Age const memoryUse = memoryUseOf(s.memory, s.cwd ?? prev?.cwd); if (memoryUse) session.stats.memoryUse = memoryUse; session.flags.push(...hookContextFlags(s.hookContext)); + const weightEdits = weightEditsOf(s.weightEdits); + if (weightEdits) session.stats.weightEdits = weightEdits; + session.flags.push(...weightEditFlags(s.weightEdits)); const redactions = redactionsOf(s.redactions); if (redactions) session.stats.redactions = redactions; if (exposesSecret(redactions)) session.flags.push(FLAGS.secretExposed); @@ -1184,13 +1193,13 @@ function flagsOf(s: FlagState, derails: readonly Derail[], sorted?: readonly Age /** * A session with nothing to look at: no turn, no tool call and no model call * (a hook that fired with no agent work behind it, such as a wrapper process - * or a subagent that never ran). A changed file or a start snapshot (what - * `coding.sh run` takes) is something to look at. `coding.sh sessions` hides + * or a subagent that never ran). A changed file, a start snapshot (what + * `coding.sh run` takes) or a recorded weight edit is something to look at. `coding.sh sessions` hides * the empty ones unless --empty. Applied when listing, not when folding. */ export function isEmptySession(s: Pick & Partial>): boolean { - const { turns, toolCalls, usage, latency, filesChanged } = s.stats; - if (filesChanged > 0 || s.repo?.snapshot) return false; + const { turns, toolCalls, usage, latency, filesChanged, weightEdits } = s.stats; + if (filesChanged > 0 || s.repo?.snapshot || weightEdits) return false; return !turns && !toolCalls && !latency?.llmCalls && !usage?.inputTokens && !usage?.outputTokens && !usage?.costUsd; } diff --git a/packages/coding.sh/src/core/weights.ts b/packages/coding.sh/src/core/weights.ts new file mode 100644 index 00000000..6b1fa34f --- /dev/null +++ b/packages/coding.sh/src/core/weights.ts @@ -0,0 +1,106 @@ +// Weight edits: `notification` events with `payload.kind: "weights.edit"` (the SDKs' +// weights_edit helpers, or any client of /v1/ingest), folded into +// Session.stats.weightEdits and the `weights:edited` session tag, and read +// into one line of text for the CLI, the MCP server and the viewer's rows. +import type { AgentEvent, WeightEditStats } from "../schema/index.ts"; + +export const WEIGHTS_EDIT_KIND = "weights.edit"; +/** A tag, not a failure: the session recorded at least one weight edit. */ +export const WEIGHTS_EDITED = "weights:edited"; +/** Distinct tensor names kept per session; `tensors` stops counting there. */ +const TENSORS_MAX = 10_000; + +/** What the fold keeps. Serializable (node:v8). */ +export interface WeightEditState { + events: number; + weightsChanged: number; + tensors: Set; +} + +export const weightEditState = (): WeightEditState => ({ events: 0, weightsChanged: 0, tensors: new Set() }); + +/** True for a weights.edit event. */ +export const isWeightsEdit = (e: AgentEvent): boolean => e.kind === "notification" && e.payload.kind === WEIGHTS_EDIT_KIND; + +const count = (v: unknown): number | undefined => (typeof v === "number" && Number.isSafeInteger(v) && v >= 0 ? v : undefined); +const norm = (v: unknown): number | undefined => (typeof v === "number" && Number.isFinite(v) && v >= 0 ? v : undefined); +const text = (v: unknown): string | undefined => (typeof v === "string" && v ? v : undefined); + +/** The fields of a weights.edit payload that are present and well-formed. Metadata capture keeps only the numbers. */ +export interface WeightsEdit { + tensor?: string; + shape?: number[]; + changed?: number; + l2?: number; + maxAbs?: number; + baseSha256?: string; + editedSha256?: string; + note?: string; +} + +export function weightsEditOf(p: Record): WeightsEdit { + const out: WeightsEdit = {}; + const set = (k: K, v: WeightsEdit[K] | undefined) => { + if (v !== undefined) out[k] = v; + }; + set("tensor", text(p.tensor)); + if (Array.isArray(p.shape) && p.shape.every((d) => count(d) !== undefined)) out.shape = p.shape as number[]; + set("changed", count(p.changed)); + set("l2", norm(p.l2)); + set("maxAbs", norm(p.maxAbs)); + set("baseSha256", text(p.baseSha256)); + set("editedSha256", text(p.editedSha256)); + set("note", text(p.note)); + return out; +} + +export function foldWeightEdits(s: WeightEditState, e: AgentEvent): void { + if (!isWeightsEdit(e)) return; + const w = weightsEditOf(e.payload); + s.events++; + s.weightsChanged += w.changed ?? 0; + if (w.tensor && s.tensors.size < TENSORS_MAX) s.tensors.add(w.tensor); +} + +/** Session.stats.weightEdits, or undefined when the session recorded no weight edit. */ +export function weightEditsOf(s: WeightEditState | undefined): WeightEditStats | undefined { + if (!s?.events) return undefined; + const out: WeightEditStats = { events: s.events, weightsChanged: s.weightsChanged }; + if (s.tensors.size) out.tensors = s.tensors.size; + return out; +} + +export const weightEditFlags = (s: WeightEditState | undefined): string[] => (s?.events ? [WEIGHTS_EDITED] : []); + +const int = (n: number): string => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ","); +const real = (n: number): string => String(Number(n.toPrecision(4))); +const short = (sha: string): string => sha.slice(0, 8); + +/** + * One line for a weights.edit payload, most useful part first: + * `layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · 9f2c1a7e → 4b1d0c33`. + * A part whose field is missing is left out, so an event captured in metadata mode reads + * `1,204 weights · L2 0.0183 · max 0.004`. + */ +export function weightsEditText(p: Record): string { + const w = weightsEditOf(p); + const parts: string[] = []; + const dims = w.shape?.length ? `[${w.shape.join("×")}]` : ""; + if (w.tensor || dims) parts.push([w.tensor, dims].filter(Boolean).join(" ")); + if (w.changed !== undefined) { + const total = w.shape?.length ? w.shape.reduce((a, b) => a * b, 1) : undefined; + parts.push(`${int(w.changed)}${total !== undefined && Number.isSafeInteger(total) ? ` of ${int(total)}` : ""} ${w.changed === 1 && total === undefined ? "weight" : "weights"}`); + } + if (w.l2 !== undefined) parts.push(`L2 ${real(w.l2)}`); + if (w.maxAbs !== undefined) parts.push(`max ${real(w.maxAbs)}`); + if (w.baseSha256 && w.editedSha256) parts.push(`${short(w.baseSha256)} → ${short(w.editedSha256)}`); + return parts.join(" · "); +} + +/** The session line: `2 edits, 2 tensors, 66,560 weights changed`. */ +export function weightEditsText(w: WeightEditStats): string { + const parts = [`${int(w.events)} ${w.events === 1 ? "edit" : "edits"}`]; + if (w.tensors) parts.push(`${int(w.tensors)} ${w.tensors === 1 ? "tensor" : "tensors"}`); + parts.push(`${int(w.weightsChanged)} ${w.weightsChanged === 1 ? "weight" : "weights"} changed`); + return parts.join(", "); +} diff --git a/packages/coding.sh/src/mcp/data.ts b/packages/coding.sh/src/mcp/data.ts index f5689ea6..1dac8ec5 100644 --- a/packages/coding.sh/src/mcp/data.ts +++ b/packages/coding.sh/src/mcp/data.ts @@ -9,6 +9,7 @@ import { CACHE_BREAK_FIX, CACHE_IDLE_FIX } from "../core/cache-breaks.ts"; import { cacheBaseline, CACHE_BASELINE_NOTE } from "../core/cache-baselines.ts"; import { mergeCallCopies } from "../core/call-copies.ts"; import { failureFlags, percentile } from "../core/sessions.ts"; +import { WEIGHTS_EDIT_KIND, weightsEditText } from "../core/weights.ts"; import { sessionMatcher } from "../query/run.ts"; import type { Node, Query } from "../query/where.ts"; @@ -109,6 +110,7 @@ export function sessionSummary(s: Session, now: number = Date.now()): SessionSum derails: s.stats.derails?.length ? s.stats.derails.map((d) => ({ flag: d.flag, at: iso(d.ts), index: d.index, eventId: d.eventId, detail: d.detail })) : undefined, waste: s.stats.waste, cacheBreaks: s.stats.cacheBreaks, + weightEdits: s.stats.weightEdits, parentSessionId: s.parent?.sessionId, subagentType: s.parent?.type, }; @@ -330,7 +332,8 @@ export function timeline(events: AgentEvent[], startedAt: number): TimelineEntry push("compact", `context compacted (${str(p.trigger) ?? "auto"})`); break; case "notification": - push("notification", clip(str(p.message) ?? str(p.text), 160)); + if (p.kind === WEIGHTS_EDIT_KIND) push("weights", clip(weightsEditText(p), 240)); + else push("notification", clip(str(p.message) ?? str(p.text), 160)); break; default: break; diff --git a/packages/coding.sh/src/mcp/digest.ts b/packages/coding.sh/src/mcp/digest.ts index 6b1a39ce..9c325781 100644 --- a/packages/coding.sh/src/mcp/digest.ts +++ b/packages/coding.sh/src/mcp/digest.ts @@ -3,6 +3,7 @@ import { costNote } from "../core/pricing.ts"; import { SKILL_EVENT_KINDS, SKILL_TAG_PREFIX, skillUse } from "../core/skills-use.ts"; import type { IndexStore, Session } from "../core/types.ts"; +import { weightEditsText } from "../core/weights.ts"; import { clip, countNodes, fmtMs, isRunning, iso, lastActivity, sessionSummary, splitTurns, subagentTree, timeline, turnIndex, type SubagentNode, type TimelineEntry, type TurnInfo } from "./data.ts"; type Obj = Record; @@ -112,6 +113,7 @@ export async function sessionMarkdown(index: IndexStore, s: Session, maxEntries if (s.stats.waste) row("Re-fetches", `${s.stats.waste.duplicateFetches} of ${s.stats.waste.readOnlyCalls} read-only calls repeated with the same result (~${s.stats.waste.duplicateFetchTokens} tokens)`); const cb = s.stats.cacheBreaks; if (cb) row("Cache breaks", `${cb.count}, ${cb.lostTokens} prompt tokens billed again${cb.extraUsd !== undefined ? `, $${cb.extraUsd.toFixed(4)} extra` : ""} (${Object.entries(cb.byCause).map(([k, c]) => `${k} ${c!.count}${k === "idle" && cb.byIdle ? ` (${Object.entries(cb.byIdle).map(([w, i]) => `${w} ${i!.count}`).join(", ")})` : ""}`).join(", ")})`); + if (s.stats.weightEdits) row("Weight edits", weightEditsText(s.stats.weightEdits)); for (const d of s.stats.derails ?? []) row("Derail", `${d.flag} at +${fmtMs(d.ts - s.startedAt)} (event ${d.index + 1}): ${d.detail}`); if (s.parent) row("Parent", `\`${s.parent.sessionId}\``); if (turns.length > 1) { diff --git a/packages/coding.sh/src/mcp/tools.ts b/packages/coding.sh/src/mcp/tools.ts index 64f2b145..6fba3609 100644 --- a/packages/coding.sh/src/mcp/tools.ts +++ b/packages/coding.sh/src/mcp/tools.ts @@ -407,7 +407,7 @@ function toolDefs(): ToolDef[] { name: "session_get", title: "Get a session", description: - "One session as a digest: summary (with waste: repeated reads and prompt-cache breaks), skills used, per-turn index, and a timeline of prompts, replies, tool calls (with failures and durations), model calls, file changes and tests. Pass turn to zoom into one turn; page with cursor.", + "One session as a digest: summary (with waste: repeated reads and prompt-cache breaks), skills used, per-turn index, and a timeline of prompts, replies, tool calls (with failures and durations), model calls, file changes, tests and weight edits. Pass turn to zoom into one turn; page with cursor.", inputSchema: input( { sessionId: SESSION_ID, diff --git a/packages/coding.sh/src/query/where.ts b/packages/coding.sh/src/query/where.ts index ec288c96..a20822ba 100644 --- a/packages/coding.sh/src/query/where.ts +++ b/packages/coding.sh/src/query/where.ts @@ -201,6 +201,8 @@ const KEY_LIST: KeyDef[] = [ facet: {}, }, { name: "hook.tokens", level: "session", type: "number", unit: "count", doc: "Tokens hooks added to the context (characters / 4)", example: "hook.tokens>500", sql: [stat("hookContext.tokens")], get: (s) => [num(s.stats?.hookContext?.tokens)] }, + { name: "weights.edits", level: "session", type: "number", unit: "count", doc: "Weight edits the session recorded", example: "weights.edits>0", sql: [stat("weightEdits.events")], get: (s) => [num(s.stats?.weightEdits?.events)] }, + { name: "weights.changed", level: "session", type: "number", unit: "count", doc: "Weights those edits changed, added up", example: "weights.changed>1000", sql: [stat("weightEdits.weightsChanged")], get: (s) => [num(s.stats?.weightEdits?.weightsChanged)] }, { name: "since", level: "session", type: "time", bound: "since", doc: "Started at or after: 7d, 12h, today, yesterday or a date. A date without a zone is local time: 2026-11-09 is local midnight", example: "since:7d", sql: ["s.started_at"], notNull: true, get: (s) => [s.startedAt] }, { name: "until", level: "session", type: "time", bound: "until", doc: "Started before (exclusive): 7d, 12h, today or a date in local time. until:2026-11-01 leaves that day out", example: "until:2026-11-01", sql: ["s.started_at"], notNull: true, get: (s) => [s.startedAt] }, { name: "duration", level: "session", type: "number", unit: "ms", doc: "Wall time of an ended session (30s, 10m, 2h)", example: "duration>10m", sql: ["(s.ended_at - s.started_at)"], get: (s) => [s.endedAt !== undefined ? s.endedAt - s.startedAt : undefined] }, diff --git a/packages/coding.sh/src/refine/taxonomy.ts b/packages/coding.sh/src/refine/taxonomy.ts index 9a304ea9..708f5e08 100644 --- a/packages/coding.sh/src/refine/taxonomy.ts +++ b/packages/coding.sh/src/refine/taxonomy.ts @@ -206,6 +206,8 @@ export const FLAG_DIAGNOSIS: Readonly; } +/** + * Session.stats.weightEdits: weight edits the session recorded (`notification` + * events with `payload.kind: "weights.edit"`). + */ +export interface WeightEditStats { + /** Edits recorded. */ + events: number; + /** Distinct tensor names edited. Unset when no edit kept its name, as in metadata capture. */ + tensors?: number; + /** The edits' `changed` counts added up: a weight two edits changed counts twice. */ + weightsChanged: number; +} + /** Why a prompt cache broke: a compaction rewrote the context, the model changed, the cache expired while idle, or something else. */ export type CacheBreakCause = "compaction" | "model-switch" | "idle" | "other"; @@ -376,6 +389,8 @@ export interface Session { inheritedTokens?: number; /** Text hooks added to the model's context, by source (`plugin:`, `settings`, …). Unset when no hook added any. */ hookContext?: HookContextStats; + /** Weight edits the session recorded; it also tags the session `weights:edited`. Unset when it recorded none. */ + weightEdits?: WeightEditStats; /** * Memory and instruction files (AGENTS.md, rules, skills, Claude auto-memory) the * session's events name, in the order first seen: what `harness audit` counts diff --git a/packages/coding.sh/src/schema/trace.schema.json b/packages/coding.sh/src/schema/trace.schema.json index 600f06a9..70a3c8ac 100644 --- a/packages/coding.sh/src/schema/trace.schema.json +++ b/packages/coding.sh/src/schema/trace.schema.json @@ -930,6 +930,10 @@ "description": "Text hooks added to the model's context, by source (`plugin:`, `settings`, …). Unset when no hook added any.", "$ref": "#/$defs/HookContextStats" }, + "weightEdits": { + "description": "Weight edits the session recorded; it also tags the session `weights:edited`. Unset when it recorded none.", + "$ref": "#/$defs/WeightEditStats" + }, "memoryUse": { "description": "Memory and instruction files (AGENTS.md, rules, skills, Claude auto-memory) the\nsession's events name, in the order first seen: what `harness audit` counts\n(core/memory-use.ts). Unset when its events name none.", "type": "array", @@ -1302,6 +1306,29 @@ "readOnlyCalls" ], "additionalProperties": false + }, + "WeightEditStats": { + "description": "Session.stats.weightEdits: weight edits the session recorded (`notification`\nevents with `payload.kind: \"weights.edit\"`).", + "type": "object", + "properties": { + "events": { + "description": "Edits recorded.", + "type": "number" + }, + "tensors": { + "description": "Distinct tensor names edited. Unset when no edit kept its name, as in metadata capture.", + "type": "number" + }, + "weightsChanged": { + "description": "The edits' `changed` counts added up: a weight two edits changed counts twice.", + "type": "number" + } + }, + "required": [ + "events", + "weightsChanged" + ], + "additionalProperties": false } } } diff --git a/packages/coding.sh/src/viewer/model.ts b/packages/coding.sh/src/viewer/model.ts index 3dd055c9..5bd80fdd 100644 --- a/packages/coding.sh/src/viewer/model.ts +++ b/packages/coding.sh/src/viewer/model.ts @@ -10,11 +10,12 @@ import { failureState, foldFailure } from "../core/failures.ts"; import type { IndexStore } from "../core/types.ts"; import type { AgentEvent, Session } from "../schema/index.ts"; +import { isWeightsEdit, weightsEditText } from "../core/weights.ts"; /** Bump when the model's shape or its rules change: cached models are rebuilt. */ -export const MODEL_VERSION = 1; +export const MODEL_VERSION = 2; // 2: a weight edit has its own code, so a fold never hides it. -const KINDS = ["other", "session.start", "session.end", "user.message", "assistant.message", "llm.call", "tool.start", "tool.end", "context.compact", "turn.end", "subagent.start", "subagent.end", "file.change", "test.result", "notification", "span", "error"] as const; +const KINDS = ["other", "session.start", "session.end", "user.message", "assistant.message", "llm.call", "tool.start", "tool.end", "context.compact", "turn.end", "subagent.start", "subagent.end", "file.change", "test.result", "notification", "span", "error", "weights.edit"] as const; const CODE = new Map(KINDS.map((k, i) => [k, i])); const C = Object.fromEntries(KINDS.map((k, i) => [k, i])) as Record<(typeof KINDS)[number], number>; /** Kinds a fold may hide: routine work. Prompts, replies, compactions, subagent edges, file changes and tests stay rows. */ @@ -127,7 +128,7 @@ export function modelBuilder(key = ""): { add(batch: readonly AgentEvent[]): voi grown.set(codes); codes = grown; } - const code = CODE.get(e.kind) ?? C.other; + const code = isWeightsEdit(e) ? C["weights.edit"] : (CODE.get(e.kind) ?? C.other); codes[i] = code; const toolFailed = foldFailure(fs, e) !== undefined; // A turn opens at the main agent's prompt, or at a turn id not seen before (an agent @@ -310,6 +311,8 @@ export function rowLabel(e: AgentEvent): string { return cut(text(p.message), 200); case "span": return text(p.name); + case "notification": + return isWeightsEdit(e) ? cut(weightsEditText(p), 200) : e.kind; default: return e.kind; } diff --git a/packages/coding.sh/src/viewer/ui/index.html b/packages/coding.sh/src/viewer/ui/index.html index 882306b6..a8173d9d 100644 --- a/packages/coding.sh/src/viewer/ui/index.html +++ b/packages/coding.sh/src/viewer/ui/index.html @@ -3780,6 +3780,27 @@ function subagentText(p) { return oneLine([p.type, p.description || p.prompt].filter((x) => typeof x === "string" && x).join(" · ")); } +// One line for a weights.edit payload; the same text as weightsEditText in core/weights.ts +// (test/viewer/weights-row.test.ts runs both on the same payloads). +function weightsEditText(p) { + const count = (v) => typeof v === "number" && Number.isSafeInteger(v) && v >= 0; + const norm = (v) => typeof v === "number" && Number.isFinite(v) && v >= 0; + const text = (v) => (typeof v === "string" && v ? v : ""); + const int = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ","); + const real = (n) => String(Number(n.toPrecision(4))); + const shape = Array.isArray(p.shape) && p.shape.every(count) ? p.shape : undefined; + const parts = []; + const dims = shape && shape.length ? "[" + shape.join("×") + "]" : ""; + if (text(p.tensor) || dims) parts.push([text(p.tensor), dims].filter(Boolean).join(" ")); + if (count(p.changed)) { + const total = shape && shape.length ? shape.reduce((a, b) => a * b, 1) : undefined; + parts.push(int(p.changed) + (total !== undefined && Number.isSafeInteger(total) ? " of " + int(total) : "") + (p.changed === 1 && total === undefined ? " weight" : " weights")); + } + if (norm(p.l2)) parts.push("L2 " + real(p.l2)); + if (norm(p.maxAbs)) parts.push("max " + real(p.maxAbs)); + if (text(p.baseSha256) && text(p.editedSha256)) parts.push(p.baseSha256.slice(0, 8) + " → " + p.editedSha256.slice(0, 8)); + return parts.join(" · "); +} function eventView(ev) { const p = ev.payload || {}; switch (ev.kind) { @@ -3814,6 +3835,10 @@ case "error": return { name: "Error", arg: clip(oneLine(p.message || p.text), 160), label: clip(oneLine(p.message || p.text) || "Error", 90), tone: "fail", kindText: ev.kind }; case "context.compact": return { name: "Context compacted", arg: p.trigger ? String(p.trigger) : "", label: "Context compacted" + (p.trigger ? " (" + p.trigger + ")" : ""), tone: "muted", kindText: ev.kind }; case "notification": { + if (p.kind === "weights.edit") { + const t = weightsEditText(p); + return { name: "Edited weights", arg: clip(t, 200), label: "Edited weights" + (t ? " · " + clip(t, 90) : ""), tone: "text", kindText: "weights.edit" }; + } // Claude Code loading a nested CLAUDE.md or a skill mid-session. if (p.kind !== "harness.loaded") return { name: KIND_LABEL[ev.kind], arg: "", label: KIND_LABEL[ev.kind], tone: "muted", kindText: ev.kind }; const names = (v) => (Array.isArray(v) ? v.filter((x) => typeof x === "string") : []); @@ -5031,6 +5056,7 @@ : [st.toolCalls ? h("span", { class: "badge" }, plural(st.toolCalls, "tool call")) : null, st.toolErrors ? h("span", { class: "badge fail" }, plural(st.toolErrors, "tool error")) : null], st.filesChanged ? h("span", { class: "badge" }, plural(st.filesChanged, "file") + " changed") : null, + st.weightEdits ? h("span", { class: "badge", title: fmtNum(st.weightEdits.weightsChanged) + " weights changed" + (st.weightEdits.tensors ? " in " + plural(st.weightEdits.tensors, "tensor") : "") }, plural(st.weightEdits.events, "weight edit")) : null, st.turns && !api_ ? h("span", { class: "badge" }, plural(st.turns, "turn")) : null, kids.length ? h("span", { class: "badge" }, plural(kids.length, "subagent")) : null, s.parent && s.parent.type ? h("span", { class: "badge" }, "subagent · " + s.parent.type) : null, diff --git a/packages/coding.sh/test/cli/weights-show.test.ts b/packages/coding.sh/test/cli/weights-show.test.ts new file mode 100644 index 00000000..58724308 --- /dev/null +++ b/packages/coding.sh/test/cli/weights-show.test.ts @@ -0,0 +1,101 @@ +// Weight edits in the CLI: `sessions` lists a session that holds nothing but an edit and finds +// it by tag and by count, and `show` prints the Weight edits row and one line per edit. +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { after, before, describe, it } from "node:test"; +import { fileURLToPath } from "node:url"; +import { openContext } from "../../src/core/context.ts"; +import { buildSession } from "../../src/core/sessions.ts"; +import type { AgentEvent } from "../../src/schema/index.ts"; + +const CLI = fileURLToPath(new URL("../../src/cli/main.ts", import.meta.url)); +const A = "a".repeat(64); +const B = "b".repeat(64); +const EDITS = "sess-edits"; +const PLAIN = "sess-plain"; + +describe("weight edits in the CLI", () => { + let root = ""; + let env: Record; + const cli = (...args: string[]) => { + const r = spawnSync(process.execPath, ["--no-warnings", CLI, ...args], { env, encoding: "utf8", cwd: root }); + assert.equal(r.status, 0, `${args.join(" ")}: ${r.stderr}`); + return r.stdout; + }; + const ids = (...args: string[]) => (JSON.parse(cli("sessions", ...args, "--json")) as { sessions: { id: string }[] }).sessions.map((s) => s.id).sort(); + + before(async () => { + root = mkdtempSync(join(tmpdir(), "csh-weights-cli-")); + const home = join(root, "home"); + mkdirSync(home, { recursive: true }); + env = { + PATH: process.env.PATH ?? "", + ["HO" + "ME"]: home, + USERPROFILE: home, + CODING_SH_HOME: join(home, ".coding.sh"), + CLAUDE_CONFIG_DIR: join(root, "claude"), + CODEX_HOME: join(root, "codex"), + XDG_CONFIG_HOME: join(root, "xdg-config"), + XDG_DATA_HOME: join(root, "xdg-data"), + CURSOR_CONFIG_DIR: join(root, "cursor"), + OPENCODE_CONFIG_DIR: join(root, "opencode"), + OPENCODE_DB: join(root, "opencode.db"), + CODING_SH_NO_AUTO_INGEST: "1", + CODING_SH_NO_UPDATE_CHECK: "1", + NO_COLOR: "1", + TZ: "UTC", + }; + const ctx = await openContext({ env, cwd: root }); + const t0 = Date.now() - 3600e3; + const put = async (evs: AgentEvent[]) => { + await ctx.index.putEvents(evs); + await ctx.index.upsertSession(buildSession(evs)); + }; + const e = { v: 0 as const, sessionId: EDITS, agent: "weights-run" }; + // What a client that posts only the edit leaves behind: no turn, no tool call, no model call. + await put([ + { ...e, id: "w-0", ts: t0, kind: "session.start", payload: { name: "weights.edit" } }, + { ...e, id: "w-1", ts: t0 + 100, kind: "notification", payload: { kind: "weights.edit", tensor: "model.layers.15.mlp.down_proj.weight", shape: [4096, 11008], changed: 65536, l2: 0.0183, maxAbs: 0.004, baseSha256: A, editedSha256: B } }, + { ...e, id: "w-2", ts: t0 + 200, kind: "notification", payload: { kind: "weights.edit", changed: 1024, l2: 0.002, maxAbs: 0.001 } }, + { ...e, id: "w-3", ts: t0 + 300, kind: "session.end", payload: {} }, + ]); + const p = { v: 0 as const, sessionId: PLAIN, agent: "codex" }; + await put([ + { ...p, id: "p-0", ts: t0, kind: "session.start", payload: {} }, + { ...p, id: "p-1", ts: t0 + 1, kind: "user.message", payload: { text: "hello" } }, + { ...p, id: "p-2", ts: t0 + 2, kind: "turn.end", payload: {} }, + ]); + await ctx.close(); + }); + after(() => rmSync(root, { recursive: true, force: true })); + + it("sessions lists a session that holds only weight edits, and finds it by tag and by count", () => { + assert.deepEqual(ids(), [EDITS, PLAIN].sort(), "not hidden as an empty session"); + assert.deepEqual(ids("--flag", "weights:edited"), [EDITS]); + assert.deepEqual(ids("--where", "flag:weights:edited"), [EDITS]); + assert.deepEqual(ids("--where", "weights.edits>0"), [EDITS]); + assert.deepEqual(ids("--where", "weights.changed>=66560"), [EDITS]); + assert.deepEqual(ids("--where", "weights.changed>66560"), []); + assert.deepEqual(ids("--where", "-flag:weights:edited"), [PLAIN]); + }); + + it("show prints the Weight edits row and one line per edit", () => { + const text = cli("show", EDITS); + assert.match(text, /Weight edits\s+2 edits, 1 tensor, 66,560 weights changed/); + assert.match(text, /weights\s+model\.layers\.15\.mlp\.down_proj\.weight \[4096×11008\] · 65,536 of 45,088,768 weights · L2 0\.0183 · max 0\.004 · aaaaaaaa → bbbbbbbb/); + assert.match(text, /weights\s+1,024 weights · L2 0\.002 · max 0\.001/, "an edit captured without its name still says what changed"); + assert.doesNotMatch(text, /^\s*\+\S+\s+notification\s*$/m, "no blank notification line"); + const j = JSON.parse(cli("show", EDITS, "--json")) as { session: { flags: string[]; stats: { weightEdits?: unknown } } }; + assert.deepEqual(j.session.stats.weightEdits, { events: 2, tensors: 1, weightsChanged: 66560 }); + assert.ok(j.session.flags.includes("weights:edited")); + }); + + it("a session with no weight edit has no row", () => { + assert.doesNotMatch(cli("show", PLAIN), /Weight edits/); + const j = JSON.parse(cli("show", PLAIN, "--json")) as { session: { stats: Record } }; + assert.equal(Object.hasOwn(j.session.stats, "weightEdits"), false); + }); +}); diff --git a/packages/coding.sh/test/core/reparse.test.ts b/packages/coding.sh/test/core/reparse.test.ts index f1a771b3..753bef70 100644 --- a/packages/coding.sh/test/core/reparse.test.ts +++ b/packages/coding.sh/test/core/reparse.test.ts @@ -311,6 +311,24 @@ describe("reparse", () => { await t.index.close(); }); + it("a session stored before weight edits were folded gains the stat and the tag on the refold", async () => { + const t = await setup(); + const edit = { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4, 8], changed: 12, l2: 0.25, maxAbs: 0.1, baseSha256: "a".repeat(64), editedSha256: "b".repeat(64) }; + await t.index.replaceSessionEvents!("sdk", [], [ + { id: "w0", v: 0, sessionId: "sdk", ts: T0, agent: "sdk", kind: "session.start", payload: {} }, + { id: "w1", v: 0, sessionId: "sdk", ts: T0 + 1, agent: "sdk", kind: "notification", payload: edit }, + ]); + // What fold version 19 stored for it: no stat, no tag. + await t.index.upsertSession({ id: "sdk", agent: "sdk", startedAt: T0, stats: { events: 2, turns: 0, toolCalls: 0, toolErrors: 0, filesChanged: 0 }, flags: [] }); + await t.index.setCursor(REFOLD_CURSOR, 19 * 1e9); + assert.equal(await t.due(), true); + assert.equal((await t.reparse()).refolded, 1); + const s = (await t.index.getSession("sdk"))!; + assert.deepEqual(s.stats.weightEdits, { events: 1, tensors: 1, weightsChanged: 12 }); + assert.deepEqual(s.flags, ["weights:edited"]); + await t.index.close(); + }); + it("one re-parse at a time", async () => { const t = await setup(); const lock = await acquireLock(paths(t.env).reparseLock, { waitMs: 0 }); diff --git a/packages/coding.sh/test/core/weights.test.ts b/packages/coding.sh/test/core/weights.test.ts new file mode 100644 index 00000000..f35f56b4 --- /dev/null +++ b/packages/coding.sh/test/core/weights.test.ts @@ -0,0 +1,81 @@ +import assert from "node:assert/strict"; +import { describe, it } from "node:test"; +import type { AgentEvent, EventKind } from "../../src/schema/index.ts"; +import { toMetadata } from "../../src/core/redact.ts"; +import { buildSession, failureFlags, isEmptySession } from "../../src/core/sessions.ts"; +import { weightEditsText, weightsEditText } from "../../src/core/weights.ts"; + +const A = "a".repeat(64); +const B = "b".repeat(64); +let t = 0; +const ev = (kind: EventKind, payload: Record = {}, o: Partial = {}): AgentEvent => ({ + id: `e${++t}`, v: 0, sessionId: "s", ts: 1000 + t * 10, agent: "sdk", kind, payload, ...o, +}); +const edit = (o: Record = {}): AgentEvent => + ev("notification", { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4096, 11008], changed: 1204, l2: 0.0183, maxAbs: 0.004, baseSha256: A, editedSha256: B, ...o }); + +describe("weight edits in a session", () => { + it("two edits give the stat and the tag, and the tag is not a failure", () => { + const s = buildSession([ev("session.start"), edit(), edit({ tensor: "layers.4.mlp.down_proj.weight", changed: 96 }), edit({ changed: 1 }), ev("session.end")]); + assert.deepEqual(s.stats.weightEdits, { events: 3, tensors: 2, weightsChanged: 1301 }); + assert.ok(s.flags.includes("weights:edited")); + assert.deepEqual(failureFlags(s.flags), []); + }); + + it("a session with no weight edit has neither the stat nor the tag", () => { + const s = buildSession([ev("session.start"), ev("notification", { kind: "hook.context", text: "x", tokens: 1 }), ev("notification", { message: "weights.edit" }), ev("session.end")]); + assert.equal(Object.hasOwn(s.stats, "weightEdits"), false); + assert.equal(s.flags.some((f) => f.startsWith("weights:")), false); + }); + + it("metadata capture keeps the counts and drops the tensor names", () => { + const stripped = toMetadata(edit()); + assert.equal(Object.hasOwn(stripped.payload, "tensor"), false, "the test's premise: metadata mode drops the name"); + const s = buildSession([ev("session.start"), stripped, ev("session.end")]); + assert.deepEqual(s.stats.weightEdits, { events: 1, weightsChanged: 1204 }); + assert.equal(Object.hasOwn(s.stats.weightEdits!, "tensors"), false); + assert.equal(weightsEditText(stripped.payload), "1,204 weights · L2 0.0183 · max 0.004"); + }); + + it("an edit whose numbers are not numbers still counts as an edit", () => { + const s = buildSession([ev("session.start"), edit({ changed: "many", tensor: 7 }), edit({ changed: -3 }), edit({ changed: 1.5 })]); + assert.deepEqual(s.stats.weightEdits, { events: 3, tensors: 1, weightsChanged: 0 }); + }); + + it("a session that holds only a weight edit is not hidden as empty", () => { + const own = buildSession([ev("session.start", { name: "weights.edit" }), edit(), ev("session.end")]); + assert.equal(own.stats.turns, 0, "the test's premise: nothing else makes this session count"); + assert.equal(isEmptySession(own), false); + assert.equal(isEmptySession(buildSession([ev("session.start"), ev("session.end")])), true); + }); +}); + +describe("weightsEditText", () => { + it("names the tensor, its shape, the share of weights changed, both norms and both hashes", () => { + assert.equal( + weightsEditText(edit().payload), + "layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb", + ); + }); + + it("leaves out what the payload does not have", () => { + assert.equal(weightsEditText({ kind: "weights.edit" }), ""); + assert.equal(weightsEditText({ tensor: "w", changed: 1 }), "w · 1 weight"); + assert.equal(weightsEditText({ tensor: "w", shape: [2, 3], changed: 1 }), "w [2×3] · 1 of 6 weights"); + assert.equal(weightsEditText({ tensor: "bias", shape: [], changed: 0, l2: 0, maxAbs: 0 }), "bias · 0 weights · L2 0 · max 0"); + assert.equal(weightsEditText({ tensor: "w", baseSha256: A }), "w", "one hash alone says nothing about the change"); + assert.equal(weightsEditText({ tensor: "w", shape: [2, "x"], changed: 2 }), "w · 2 weights", "a shape that is not all counts is not shown"); + }); + + it("keeps four significant digits of a norm", () => { + assert.equal(weightsEditText({ l2: 0.018300000000000004, maxAbs: 1234.5678 }), "L2 0.0183 · max 1235"); + assert.equal(weightsEditText({ l2: 1e-9 }), "L2 1e-9"); + }); +}); + +describe("weightEditsText", () => { + it("counts edits, tensors and weights", () => { + assert.equal(weightEditsText({ events: 2, tensors: 2, weightsChanged: 66560 }), "2 edits, 2 tensors, 66,560 weights changed"); + assert.equal(weightEditsText({ events: 1, weightsChanged: 1 }), "1 edit, 1 weight changed"); + }); +}); diff --git a/packages/coding.sh/test/mcp/weights.test.ts b/packages/coding.sh/test/mcp/weights.test.ts new file mode 100644 index 00000000..16a1ce65 --- /dev/null +++ b/packages/coding.sh/test/mcp/weights.test.ts @@ -0,0 +1,78 @@ +// Weight edits over MCP: sessions_list finds the session by its tag and carries the stat, +// session_get's timeline has one line per edit, and the session resource has the row. +import assert from "node:assert/strict"; +import { after, before, describe, it } from "node:test"; +import { openContext } from "../../src/core/context.ts"; +import { buildSession } from "../../src/core/sessions.ts"; +import type { AgentEvent } from "../../src/schema/index.ts"; +import { RawClient, cleanup, makeWorld, seed, type World } from "./helpers.ts"; + +const SID = "sess-weights"; +const A = "a".repeat(64); +const B = "b".repeat(64); +const C = "c".repeat(64); + +let w: World; +let c: RawClient; + +before(async () => { + w = makeWorld(); + await seed(w); + const ctx = await openContext({ env: w.env, cwd: w.root }); + const t0 = w.now - 3600e3; + const base = { v: 0 as const, sessionId: SID, agent: "weights-run" }; + const edit = (tensor: string, changed: number, from: string, to: string) => ({ kind: "weights.edit", tensor, shape: [4096, 11008], changed, l2: 0.0183, maxAbs: 0.004, baseSha256: from, editedSha256: to }); + const evs: AgentEvent[] = [ + { ...base, id: "w-0", ts: t0, kind: "session.start", payload: { name: "edit layer 15" } }, + { ...base, id: "w-1", ts: t0 + 100, kind: "notification", payload: edit("model.layers.15.mlp.down_proj.weight", 65536, A, B) }, + { ...base, id: "w-2", ts: t0 + 200, kind: "notification", payload: edit("model.layers.16.mlp.down_proj.weight", 1024, B, C) }, + { ...base, id: "w-3", ts: t0 + 300, kind: "session.end", payload: {} }, + ]; + await ctx.index.putEvents(evs); + await ctx.index.upsertSession(buildSession(evs)); + await ctx.close(); + c = new RawClient(w.env, w.repo); + await c.initialize(); +}); + +after(async () => { + await c?.close(); + cleanup(); +}); + +describe("weight edits over MCP", () => { + it("sessions_list finds the session by its tag, with the stat", async () => { + const r = await c.call("sessions_list", { flag: "weights:edited" }); + const sessions = r.structuredContent!.sessions as { id: string; flags?: string[]; weightEdits?: unknown }[]; + assert.deepEqual(sessions.map((s) => s.id), [SID]); + assert.deepEqual(sessions[0]!.weightEdits, { events: 2, tensors: 2, weightsChanged: 66560 }); + assert.ok(sessions[0]!.flags!.includes("weights:edited")); + const where = await c.call("sessions_list", { where: "weights.changed>1000" }); + assert.deepEqual((where.structuredContent!.sessions as { id: string }[]).map((s) => s.id), [SID]); + const all = await c.call("sessions_list", {}); + const other = (all.structuredContent!.sessions as { id: string; weightEdits?: unknown }[]).filter((s) => s.id !== SID); + assert.ok(other.length > 0 && other.every((s) => !Object.hasOwn(s, "weightEdits")), "sessions without edits carry no weightEdits key"); + }); + + it("session_get has one timeline line per edit", async () => { + const r = await c.call("session_get", { sessionId: SID }); + const d = r.structuredContent as { session: { weightEdits?: unknown }; timeline: { kind: string; text: string }[]; timelineTotal: number }; + assert.deepEqual(d.session.weightEdits, { events: 2, tensors: 2, weightsChanged: 66560 }); + assert.deepEqual( + d.timeline.map((t) => [t.kind, t.text]), + [ + ["weights", "model.layers.15.mlp.down_proj.weight [4096×11008] · 65,536 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"], + ["weights", "model.layers.16.mlp.down_proj.weight [4096×11008] · 1,024 of 45,088,768 weights · L2 0.0183 · max 0.004 · bbbbbbbb → cccccccc"], + ], + ); + assert.equal(d.timelineTotal, 2); + }); + + it("the session resource has the weight edits row and both lines", async () => { + const r = await c.request("resources/read", { uri: `codingsh://session/${SID}` }); + const text = (r.result!.contents as { text: string }[])[0]!.text; + assert.match(text, /Weight edits.*2 edits, 2 tensors, 66,560 weights changed/); + assert.match(text, /model\.layers\.15\.mlp\.down_proj\.weight \[4096×11008\]/); + assert.match(text, /model\.layers\.16\.mlp\.down_proj\.weight \[4096×11008\]/); + }); +}); diff --git a/packages/coding.sh/test/viewer/ui-browser.test.ts b/packages/coding.sh/test/viewer/ui-browser.test.ts index 4f0e55c2..87518d27 100644 --- a/packages/coding.sh/test/viewer/ui-browser.test.ts +++ b/packages/coding.sh/test/viewer/ui-browser.test.ts @@ -749,6 +749,21 @@ describe("viewer UI: timeline, inspector and compare", { skip: chrome ? false : } }); + test("a weight edit reads as a row of its own, not a blank Notification", async () => { + await open("#/"); + try { + const payload = { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4096, 11008], changed: 1204, l2: 0.0183, maxAbs: 0.004, baseSha256: "a".repeat(64), editedSha256: "b".repeat(64) }; + const v = await page.eval(`eventView({ kind: "notification", payload: ${JSON.stringify(payload)} })`); + assert.equal(v.name, "Edited weights"); + assert.equal(v.arg, "layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"); + assert.equal(v.kindText, "weights.edit"); + const other = await page.eval(`eventView({ kind: "notification", payload: { kind: "hook.context", text: "x" } })`); + assert.equal(other.name, "Notification"); + } finally { + await page.close(); + } + }); + test("session header: tokens line, cost dash and two columns from 1200px", async () => { await open("#/s/" + WORK, 1280); try { diff --git a/packages/coding.sh/test/viewer/weights-row.test.ts b/packages/coding.sh/test/viewer/weights-row.test.ts new file mode 100644 index 00000000..362c12f2 --- /dev/null +++ b/packages/coding.sh/test/viewer/weights-row.test.ts @@ -0,0 +1,63 @@ +// A weight edit in the viewer: the page's own weightsEditText (run straight from index.html) gives +// the same line as core/weights.ts, the rows route labels the event with it, and a fold of routine +// events never hides one. +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; +import { test } from "node:test"; +import { weightsEditText } from "../../src/core/weights.ts"; +import type { AgentEvent } from "../../src/schema/index.ts"; +import { buildModel, FOLD_MIN, layoutRows, rowLabel } from "../../src/viewer/model.ts"; + +const html = readFileSync(new URL("../../src/viewer/ui/index.html", import.meta.url), "utf8"); +const from = html.indexOf("function weightsEditText("); +const to = html.indexOf("function eventView(", from); +assert.ok(from >= 0 && to > from, "the page defines weightsEditText just before eventView"); +const page: (p: Record) => string = runInNewContext(`${html.slice(from, to)}\nweightsEditText`, { Number, String, Array }); + +const A = "a".repeat(64); +const B = "b".repeat(64); +const FULL = { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4096, 11008], changed: 1204, l2: 0.0183, maxAbs: 0.004, baseSha256: A, editedSha256: B }; + +test("the page and the CLI write the same line for a weight edit", () => { + const payloads: Record[] = [ + FULL, + { ...FULL, note: "rank-1 edit" }, + { kind: "weights.edit", changed: 1204, l2: 0.0183, maxAbs: 0.004 }, + { kind: "weights.edit" }, + { tensor: "w", changed: 1 }, + { tensor: "w", shape: [2, 3], changed: 1 }, + { tensor: "bias", shape: [], changed: 0, l2: 0, maxAbs: 0 }, + { tensor: "w", baseSha256: A }, + { tensor: "w", shape: [2, "x"], changed: 2 }, + { tensor: 7, shape: "no", changed: -1, l2: Number.NaN, maxAbs: "big", baseSha256: 1, editedSha256: null }, + { tensor: "w", shape: [1e9, 1e9], changed: 5 }, + { l2: 0.018300000000000004, maxAbs: 1234.5678 }, + { l2: 1e-9, changed: 1234567 }, + ]; + for (const p of payloads) assert.equal(page(p), weightsEditText(p), JSON.stringify(p)); + assert.equal(page(FULL), "layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"); +}); + +const base = { v: 0, sessionId: "w-1", agent: "sdk" } as const; +const ev = (i: number, kind: AgentEvent["kind"], payload: Record = {}): AgentEvent => ({ ...base, id: `e${i}`, ts: 1000 + i, kind, payload }); + +test("the rows route labels a weight edit with that line", () => { + assert.equal(rowLabel(ev(0, "notification", FULL)), weightsEditText(FULL)); + assert.equal(rowLabel(ev(1, "notification", { kind: "hook.context", text: "x" })), "notification"); +}); + +test("a fold of routine events never hides a weight edit", () => { + // One turn: a long run of plain notifications with one weight edit in the middle. + const n = FOLD_MIN * 6; + const mid = n / 2; + const evs = [ev(0, "session.start"), ev(1, "user.message", { text: "edit the weights" })]; + for (let i = 0; i < n; i++) evs.push(ev(2 + i, "notification", i === mid ? FULL : { message: `tick ${i}` })); + const m = buildModel(evs); + const { slots } = layoutRows(m, m.turns[0]!, m.turns[0]!.start, 10_000); + const folds = slots.filter((s): s is { from: number; to: number } => "from" in s); + assert.ok(folds.length >= 2, `the runs on both sides fold: ${JSON.stringify(slots)}`); + const at = 2 + mid; + assert.ok(slots.some((s) => "i" in s && s.i === at), "the weight edit is a row of its own"); + assert.ok(folds.every((f) => at < f.from || at > f.to)); +}); From 9b61ad21cac5eff576a1bca39968b3114644b890 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:04:49 -0600 Subject: [PATCH 02/17] refactor(sdk): the TypeScript queue reports whether it took an event Transport.enqueue(), the client's enqueue() and emit() return true when the item was queued and false when it was dropped or the SDK is off or shut down. Nothing reads the result yet; weightsEdit() will. --- packages/sdk/src/client.ts | 5 +++-- packages/sdk/src/context.ts | 6 +++--- packages/sdk/src/transport.ts | 12 +++++++----- packages/sdk/test/transport.test.ts | 25 +++++++++++++++++++++++++ 4 files changed, 38 insertions(+), 10 deletions(-) diff --git a/packages/sdk/src/client.ts b/packages/sdk/src/client.ts index e4bb62c6..dbdc5e6e 100644 --- a/packages/sdk/src/client.ts +++ b/packages/sdk/src/client.ts @@ -61,8 +61,9 @@ export function current(): State { export const config = (): Config => current().cfg; -export function enqueue(kind: ItemKind, value: unknown): void { - current().transport.enqueue(kind, value); +/** Queues one item. True when it was queued; false when it was dropped or the SDK is off or shut down. */ +export function enqueue(kind: ItemKind, value: unknown): boolean { + return current().transport.enqueue(kind, value); } /** True when events would be sent anywhere (a URL is configured and the SDK is on). */ diff --git a/packages/sdk/src/context.ts b/packages/sdk/src/context.ts index 2e3e49ba..5b671b5e 100644 --- a/packages/sdk/src/context.ts +++ b/packages/sdk/src/context.ts @@ -129,7 +129,8 @@ function clean(p: Record): Record { return p; } -function emit(span: SpanState, kind: EventKind, ts: number, payload: Record, extra: Partial = {}): SdkEvent { +/** Queues one event. True when it was queued. */ +function emit(span: SpanState, kind: EventKind, ts: number, payload: Record, extra: Partial = {}): boolean { const e: SdkEvent = { v: 0, id: extra.id ?? uuid(), @@ -145,8 +146,7 @@ function emit(span: SpanState, kind: EventKind, ts: number, payload: Record)[k] = v; if (e.durationMs !== undefined) e.durationMs = Math.max(0, Math.round(e.durationMs * 1000) / 1000); - enqueue("event", e); - return e; + return enqueue("event", e); } // ---------------------------------------------------------------- traces diff --git a/packages/sdk/src/transport.ts b/packages/sdk/src/transport.ts index 80716cf3..ce6cd245 100644 --- a/packages/sdk/src/transport.ts +++ b/packages/sdk/src/transport.ts @@ -101,8 +101,9 @@ export class Transport { return { queued: this.q.length, sent: this.sent, dropped: this.dropped, failed: this.failed }; } - enqueue(kind: ItemKind, value: unknown): void { - if (!this.enabled) return; + /** True when the item was queued; false when it was dropped or the SDK is off or shut down. */ + enqueue(kind: ItemKind, value: unknown): boolean { + if (!this.enabled) return false; if (this.q.length >= this.cfg.maxQueue) { this.dropped++; const now = Date.now(); @@ -110,24 +111,25 @@ export class Transport { this.lastDropWarn = now; console.warn(`[coding.sh] queue full (${this.cfg.maxQueue} events), dropping new events`); } - return; + return false; } let json: string; try { json = JSON.stringify(value); } catch { this.dropped++; - return; + return false; } const bytes = enc.encode(json).length; if (bytes > MAX_BATCH_BYTES - 1024) { this.dropped++; this.warnOnce("too-large", "[coding.sh] dropped an event larger than 4 MiB"); - return; + return false; } this.q.push({ kind, json, bytes }); if (this.q.length >= this.cfg.maxBatch) this.kick(); else this.schedule(); + return true; } private schedule(): void { diff --git a/packages/sdk/test/transport.test.ts b/packages/sdk/test/transport.test.ts index e0a1e110..0af2ff64 100644 --- a/packages/sdk/test/transport.test.ts +++ b/packages/sdk/test/transport.test.ts @@ -146,6 +146,31 @@ test("a full queue drops new events, counts them and warns at most once a minute assert.equal(stats().sent, 10); }); +test("enqueue says whether the item was queued", async () => { + const ev = (id: string, extra: Record = {}) => ({ v: 0, id, sessionId: "s", kind: "span", ts: 1, payload: extra }); + const warns: string[] = []; + const orig = console.warn; + console.warn = (m: string) => void warns.push(m); + try { + const off = new Transport(resolveConfig({}, {})); + assert.equal(off.enqueue("event", ev("off")), false, "no URL"); + assert.equal(new Transport(resolveConfig({ url: serve.url, disabled: true }, {})).enqueue("event", ev("disabled")), false, "disabled"); + + const t = new Transport(resolveConfig({ url: serve.url, key: KEY, flushIntervalMs: 60_000, maxQueue: 2 }, {})); + assert.equal(t.enqueue("event", ev("a")), true); + assert.equal(t.enqueue("event", ev("big", { n: 1n })), false, "not serializable"); + assert.equal(t.enqueue("event", ev("huge", { text: "x".repeat(4 * 1024 * 1024) })), false, "larger than a batch"); + assert.equal(t.enqueue("event", ev("b")), true); + assert.equal(t.enqueue("event", ev("c")), false, "the queue is full"); + assert.deepEqual(t.stats(), { queued: 2, sent: 0, dropped: 3, failed: 0 }); + assert.equal(await t.close(2000), true); + assert.equal(t.enqueue("event", ev("d")), false, "closed"); + assert.equal(t.stats().dropped, 3, "an event after close() is not counted as dropped"); + } finally { + console.warn = orig; + } +}); + test("server down: flush and close are bounded, the batch is counted failed once, never throws", async () => { const t = new Transport(resolveConfig({ url: "http://127.0.0.1:9", flushIntervalMs: 10 }, {})); const warns: string[] = []; From 7bb2b2d09764d907c80427216a8c5ae270479d2d Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:04:50 -0600 Subject: [PATCH 03/17] feat(sdk): weightsEdit() and weightsDelta() in the TypeScript SDK record the same weight edit as Python weightsEdit() sends one notification event with payload.kind "weights.edit", with the same keys, values, validation and session shape as the Python SDK's weights_edit(). It never throws and returns true only when the event was queued. weightsDelta() computes shape, changed, l2 and maxAbs for nested arrays and typed arrays. packages/sdk/test/fixtures/weights-edit.json holds the events an edit must send, recorded from the Python SDK. Both test suites read it. --- CHANGELOG.md | 1 + docs/reference/api.md | 2 +- docs/reference/sdk.md | 99 +++- packages/coding.sh/scripts/sdk-reference.ts | 5 +- packages/sdk-python/tests/test_weights.py | 129 ++++- packages/sdk/README.md | 5 +- packages/sdk/src/client.ts | 11 + packages/sdk/src/context.ts | 28 ++ packages/sdk/src/index.ts | 4 + packages/sdk/src/weights.ts | 213 ++++++++ packages/sdk/test/fixtures/weights-edit.json | 100 ++++ packages/sdk/test/weights.test.ts | 504 +++++++++++++++++++ 12 files changed, 1094 insertions(+), 7 deletions(-) create mode 100644 packages/sdk/src/weights.ts create mode 100644 packages/sdk/test/fixtures/weights-edit.json create mode 100644 packages/sdk/test/weights.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d0c7ba07..31257429 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ grouped by area, with links to the docs for the details. - **Lint.** New rules: vendor-literal, site-voice, universal-transition, reduced-motion, global-install and personal-data. `slop.mjs --strict-allowlist` fails on allowlist entries that no longer match anything. [Contributing](https://github.com/jackulau/coding.sh/blob/main/CONTRIBUTING.md) - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) +- **SDK.** `weightsEdit()` in the TypeScript SDK records a weight edit as the same `notification` event the Python SDK's `weights_edit()` sends, and returns `true` only when the event was queued. `weightsDelta()` computes the shape, count and norms for nested arrays and typed arrays. Both SDKs' tests check one fixture of the events an edit must send. [Docs](https://coding.sh/docs/sdk) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/reference/api.md b/docs/reference/api.md index 220898e7..672fadf8 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -718,7 +718,7 @@ Three `notification` kinds carry the events behind these stats: | `fork` | `toolUseId` (the parent's Agent call) and `inheritedTokens` | | `hook.context` | `hookEvent`, `hookName`, `hookSource`, `command`, `chars`, `tokens` and `text` (clipped). Metadata mode keeps only `hookEvent`, `hookSource`, `chars` and `tokens` | -One more `notification` kind records weight edits. The Python SDK's `coding_sh.weights_edit()` sends it, and any client can post it to `/v1/ingest`. It adds no stat and no flag; the event shows in the session's events. Redaction runs on `tensor` and `note` like on any payload. +One more `notification` kind records weight edits. The Python SDK's `coding_sh.weights_edit()` and the TypeScript SDK's `weightsEdit()` send it, and any client can post it to `/v1/ingest`. It adds no stat and no flag; the event shows in the session's events. Redaction runs on `tensor` and `note` like on any payload. | `payload.kind` | Fields | |---|---| diff --git a/docs/reference/sdk.md b/docs/reference/sdk.md index 8608e351..45d65e32 100644 --- a/docs/reference/sdk.md +++ b/docs/reference/sdk.md @@ -6,7 +6,7 @@ Site version: [coding.sh/docs/sdk](https://coding.sh/docs/sdk). -Both SDKs send to a server you run with `coding.sh serve` ([production guide](https://coding.sh/docs/production), [serverless apps](https://coding.sh/docs/deploy)). Without `CODING_SH_URL` they do nothing. They never change return values, errors or stream behavior, and never throw into your app. The SDKs do not price calls; the server does. +Both SDKs send to a server you run with `coding.sh serve` ([production guide](https://coding.sh/docs/production), [serverless apps](https://coding.sh/docs/deploy)). Without `CODING_SH_URL` they do nothing. They never change return values, errors or stream behavior, and never throw into your app. The one exception is `weightsDelta()` (`coding_sh.weights_delta()` in Python), a pure helper you call on purpose: it throws on bad input. The SDKs do not price calls; the server does. ## Environment variables @@ -113,6 +113,11 @@ That is AI SDK 7. On AI SDK 6 and older, pass `experimental_telemetry: { isEnabl | [`TraceOptions`](#traceoptions) | interface | | [`TraceHandle`](#tracehandle) | interface | | [`SpanOptions`](#spanoptions) | interface | +| [`weightsEdit`](#weightseditedit) | function | +| [`weightsDelta`](#weightsdeltabase-edited) | function | +| [`WeightsEdit`](#weightsedit) | interface | +| [`WeightsDelta`](#weightsdelta) | interface | +| [`WeightsArray`](#weightsarray) | type | | [`aiSdkTracer`](#aisdktraceropts) | function | | [`AiSdkTracer`](#aisdktracer) | interface | | [`AiSdkTracerOptions`](#aisdktraceroptions) | interface | @@ -228,6 +233,39 @@ function withSession(id: string, fn: () => T): T | `id` | `string` | required | | `fn` | `() => T` | required | +### `weightsEdit(edit)` + +Records one weight edit as a `notification` event with `payload.kind` "weights.edit". The Python SDK's `weights_edit()` sends the same event. + +Returns true only when the event was queued. Returns false when a value is invalid, with one warning per process. Returns false without a warning when the SDK is off or shut down or the queue is full. Never throws. + +A `note` that is not a string is dropped with one warning; the edit is still recorded. Inside `trace()` or `span()` the edit joins that trace. Outside them it gets a session of its own, or a turn of the `withSession()` session, so group a run of edits inside `trace()`. + +```ts +function weightsEdit(edit: WeightsEdit): boolean +``` + +| Parameter | Type | Default | +|---|---|---| +| `edit` | [`WeightsEdit`](#weightsedit) | required | + +### `weightsDelta(base, edited)` + +Computes `{ shape, changed, l2, maxAbs }` for two arrays of the same shape, ready to spread into `weightsEdit()`. The Python SDK's `weights_delta()` gives the same numbers, with `max_abs` for `maxAbs`. + +Takes nested arrays of numbers, typed arrays such as `Float32Array`, or arrays of typed arrays, up to 16 levels deep, holding finite numbers. It reads every value twice on the calling thread and copies nothing. + +A pure helper you call on purpose, unlike the never-throw `weightsEdit()`: it throws on bad input. A `TypeError` for an argument that is not an array or a value that is not a number (null and BigInt included). A `RangeError` for a shape mismatch, ragged or too deep nesting, NaN or Infinity. + +```ts +function weightsDelta(base: WeightsArray, edited: WeightsArray): WeightsDelta +``` + +| Parameter | Type | Default | +|---|---|---| +| `base` | [`WeightsArray`](#weightsarray) | required | +| `edited` | [`WeightsArray`](#weightsarray) | required | + ### `aiSdkTracer(opts?)` An OpenTelemetry Tracer-shaped object that sends AI SDK spans to coding.sh. @@ -324,6 +362,62 @@ interface SpanOptions { |---|---|---| | `input` | `unknown` | Tool arguments (tool spans) or step input. | +### `WeightsEdit` + +One weight edit, as `weightsEdit()` records it. + +```ts +interface WeightsEdit { + tensor: string; + shape: readonly number[]; + changed: number; + l2: number; + maxAbs: number; + baseSha256: string; + editedSha256: string; + note?: string; +} +``` + +| Field | Type | Description | +|---|---|---| +| `tensor` | `string` | The tensor's name. | +| `shape` | `readonly number[]` | Its dimensions: at most 16 non-negative integers. | +| `changed` | `number` | How many weights differ: an integer from 0 to the product of `shape`. | +| `l2` | `number` | L2 norm of the change (edited minus base): a finite number >= 0. | +| `maxAbs` | `number` | Largest absolute value of the change: a finite number >= 0. | +| `baseSha256` | `string` | sha256 of the tensor before the edit: 64 hex characters, sent in lowercase. | +| `editedSha256` | `string` | sha256 of the tensor after the edit: 64 hex characters, sent in lowercase. | +| `note` | `string` | Free text, kept only in full capture mode. | + +### `WeightsDelta` + +What `weightsDelta()` returns. It spreads into `weightsEdit()`. + +```ts +interface WeightsDelta { + shape: number[]; + changed: number; + l2: number; + maxAbs: number; +} +``` + +| Field | Type | Description | +|---|---|---| +| `shape` | `number[]` | Dimensions of both arrays. | +| `changed` | `number` | How many values differ. | +| `l2` | `number` | L2 norm of edited minus base. | +| `maxAbs` | `number` | Largest absolute value of edited minus base. | + +### `WeightsArray` + +What `weightsDelta()` reads: nested arrays of numbers, typed arrays such as `Float32Array`, or arrays of typed arrays. + +```ts +type WeightsArray = readonly number[] | (ArrayBufferView & ArrayLike) | readonly WeightsArray[]; +``` + ### `AiSdkTracer` What `aiSdkTracer()` returns: the subset of the OpenTelemetry `Tracer` interface the AI SDK calls. @@ -705,3 +799,6 @@ A constant: `"gen_ai.conversation.id"`. - TypeScript times are in milliseconds (`timeoutMs`, `flushIntervalMs`). Python timeouts are in seconds; only `flush_interval_ms` is in milliseconds. - In TypeScript, `trace()`, `span()` and `withSession()` take the function to run. In Python, they are context managers and decorators. - `aiSdkTracer()` and `observeFetch()` are TypeScript only. `coding_sh.otel` is Python only. +- `weightsEdit()` takes one object in TypeScript; `weights_edit()` takes arguments in Python. Both record the same event. `weightsDelta()` returns `maxAbs` where `weights_delta()` returns `max_abs`, and throws a `TypeError` or `RangeError` where Python raises `ValueError`. +- A weight count or dimension must be an integer below 2^53 in TypeScript; Python takes any integer. Python also takes numpy scalars as numbers, and `weights_delta()` reads anything with `.tolist()`; in TypeScript `weightsDelta()` reads arrays and typed arrays. +- A long tensor name or note is cut. TypeScript cuts at 64 KiB of UTF-8 and sets `payload.truncated`; Python cuts at 65,536 characters and appends `…[truncated]`. diff --git a/packages/coding.sh/scripts/sdk-reference.ts b/packages/coding.sh/scripts/sdk-reference.ts index 3be0a745..067a30ff 100644 --- a/packages/coding.sh/scripts/sdk-reference.ts +++ b/packages/coding.sh/scripts/sdk-reference.ts @@ -732,7 +732,7 @@ export function renderSdkReference(): string { "", "", "", - "Both SDKs send to a server you run with `coding.sh serve` ([production guide](https://coding.sh/docs/production), [serverless apps](https://coding.sh/docs/deploy)). Without `CODING_SH_URL` they do nothing. They never change return values, errors or stream behavior, and never throw into your app. The SDKs do not price calls; the server does.", + "Both SDKs send to a server you run with `coding.sh serve` ([production guide](https://coding.sh/docs/production), [serverless apps](https://coding.sh/docs/deploy)). Without `CODING_SH_URL` they do nothing. They never change return values, errors or stream behavior, and never throw into your app. The one exception is `weightsDelta()` (`coding_sh.weights_delta()` in Python), a pure helper you call on purpose: it throws on bad input. The SDKs do not price calls; the server does.", "", "## Environment variables", "", @@ -789,6 +789,9 @@ export function renderSdkReference(): string { "- TypeScript times are in milliseconds (`timeoutMs`, `flushIntervalMs`). Python timeouts are in seconds; only `flush_interval_ms` is in milliseconds.", "- In TypeScript, `trace()`, `span()` and `withSession()` take the function to run. In Python, they are context managers and decorators.", "- `aiSdkTracer()` and `observeFetch()` are TypeScript only. `coding_sh.otel` is Python only.", + "- `weightsEdit()` takes one object in TypeScript; `weights_edit()` takes arguments in Python. Both record the same event. `weightsDelta()` returns `maxAbs` where `weights_delta()` returns `max_abs`, and throws a `TypeError` or `RangeError` where Python raises `ValueError`.", + "- A weight count or dimension must be an integer below 2^53 in TypeScript; Python takes any integer. Python also takes numpy scalars as numbers, and `weights_delta()` reads anything with `.tolist()`; in TypeScript `weightsDelta()` reads arrays and typed arrays.", + "- A long tensor name or note is cut. TypeScript cuts at 64 KiB of UTF-8 and sets `payload.truncated`; Python cuts at 65,536 characters and appends `…[truncated]`.", "", ); return out.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd() + "\n"; diff --git a/packages/sdk-python/tests/test_weights.py b/packages/sdk-python/tests/test_weights.py index de3ea499..b7a1a5e4 100644 --- a/packages/sdk-python/tests/test_weights.py +++ b/packages/sdk-python/tests/test_weights.py @@ -1,7 +1,9 @@ """weights_edit: one `notification` event with payload.kind "weights.edit" (docs/reference/api.md). -The unit tests read the batch the SDK posts to a fake ingest endpoint. The last test starts a real -`coding.sh serve` on a temp CODING_SH_HOME and reads the event back, redacted. +The unit tests read the batch the SDK posts to a fake ingest endpoint. The parity tests check +those batches against packages/sdk/test/fixtures/weights-edit.json, which the TypeScript SDK's +weights.test.ts checks too. The last test starts a real `coding.sh serve` on a temp +CODING_SH_HOME and reads the event back, redacted. """ from __future__ import annotations @@ -15,7 +17,7 @@ import time import unittest import urllib.request -from typing import Any, Dict +from typing import Any, Callable, Dict, List, Optional import support from support import IngestServer, by_kind, collect, start_sdk @@ -25,6 +27,8 @@ A = "a" * 64 B = "b" * 64 CLI = os.path.join(support.ROOT, "..", "coding.sh", "src", "cli", "main.ts") +# What both SDKs must send for the same edit. The TypeScript SDK's tests read the same file. +FIXTURE = os.path.join(support.ROOT, "..", "sdk", "test", "fixtures", "weights-edit.json") NODE = shutil.which("node") # Built in pieces so push protection does not flag the test file. FAKE_TOKEN = "".join(["ghp_", "0123456789", "abcdefghij", "ABCDEFGHIJ", "klmnop"]) @@ -305,6 +309,125 @@ def test_weights_delta_feeds_weights_edit(self) -> None: self.assertTrue(coding_sh.weights_edit("w", base_sha256=A, edited_sha256=B, **d)) +def _alone(edit: Callable[[], bool], sid: Optional[str]) -> bool: + return edit() + + +def _with_session(edit: Callable[[], bool], sid: Optional[str]) -> bool: + with coding_sh.with_session(str(sid)): + return edit() + + +def _trace(edit: Callable[[], bool], sid: Optional[str]) -> bool: + with coding_sh.trace("run", session_id=sid): + return edit() + + +def _tool_span(edit: Callable[[], bool], sid: Optional[str]) -> bool: + with coding_sh.trace("run"): + with coding_sh.span("tool", "patch"): + return edit() + + +def _step_span(edit: Callable[[], bool], sid: Optional[str]) -> bool: + with coding_sh.trace("run"): + with coding_sh.span("step", "patch"): + return edit() + + +# One entry per scenario in the fixture; weights.test.ts has the same list. +SCENARIOS: Dict[str, Callable[[Callable[[], bool], Optional[str]], bool]] = { + "alone": _alone, + "withSession": _with_session, + "trace": _trace, + "traceWithSessionId": _trace, + "toolSpan": _tool_span, + "stepSpan": _step_span, +} + + +class WeightsEditParityTest(unittest.TestCase): + """weights_edit() here and weightsEdit() in the TypeScript SDK send the same events. + + The fixture holds, for one edit in each place it can be recorded, the events in the order + they are posted: each event's kind, its keys and its payload. A missing fixture is a failure. + """ + + fx: Dict[str, Any] = {} + + @classmethod + def setUpClass(cls) -> None: + with open(FIXTURE, encoding="utf-8") as f: + cls.fx = json.load(f) + + def tearDown(self) -> None: + coding_sh.shutdown(1.0) + + def _edit(self) -> bool: + e = self.fx["edit"] + return coding_sh.weights_edit( + e["tensor"], + shape=e["shape"], + changed=e["changed"], + l2=e["l2"], + max_abs=e["maxAbs"], + base_sha256=e["baseSha256"], + edited_sha256=e["editedSha256"], + note=e["note"], + ) + + def _posted(self, name: str) -> List[Dict[str, Any]]: + server = IngestServer() + try: + start_sdk(server) + self.assertIs(SCENARIOS[name](self._edit, self.fx["scenarios"][name]["sessionId"]), True) + return collect(server) + finally: + coding_sh.shutdown(1.0) + server.close() + + def test_every_scenario_in_the_fixture_runs_here(self) -> None: + self.assertEqual(sorted(SCENARIOS), sorted(self.fx["scenarios"])) + + def test_the_same_events_as_the_typescript_sdk(self) -> None: + volatile = set(self.fx["volatilePayloadKeys"]) + ignored = set(self.fx["ignoredEventKeys"]) + for name, sc in self.fx["scenarios"].items(): + with self.subTest(name): + events = self._posted(name) + got = [ + { + "kind": e["kind"], + "keys": sorted(k for k in e if k not in ignored), + "payload": {k: "" if k in volatile else v for k, v in e["payload"].items()}, + } + for e in events + ] + want = [dict(e, payload=self.fx["payload"]) if e["kind"] == "notification" else e for e in sc["events"]] + self.assertEqual(got, want) + + # One trace, one session, and the edit hangs off the event the fixture names. + first = events[0] + (note,) = by_kind(events, "notification") + self.assertEqual({e["traceId"] for e in events}, {first["traceId"]}) + self.assertEqual({e["sessionId"] for e in events}, {sc["sessionId"] or first["sessionId"]}) + if sc["parent"] is None: + self.assertEqual(note["spanId"], first["spanId"]) + else: + (parent,) = by_kind(events, sc["parent"]) + self.assertEqual(note["parentId"], parent["id"]) + self.assertEqual(note["parentSpanId"], parent["spanId"]) + self.assertNotEqual(note["spanId"], parent["spanId"]) + if "turnId" in note: + self.assertEqual(note["turnId"], note["traceId"]) + + def test_the_same_delta_as_the_typescript_sdk(self) -> None: + d = self.fx["delta"] + got = coding_sh.weights_delta(d["base"], d["edited"]) + got["maxAbs"] = got.pop("max_abs") # the TypeScript SDK's name for it + self.assertEqual(got, d["result"]) + + def _node_ok() -> bool: if not NODE or not os.path.exists(CLI): return False diff --git a/packages/sdk/README.md b/packages/sdk/README.md index c8dbc249..5a2e0810 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -18,7 +18,8 @@ env vars at a `coding.sh serve` instance (backed by SQLite, or Postgres plus S3) shows up as a session with tokens, cost, latency and errors. - Zero runtime dependencies, ESM with types, Node ≥ 18 (Vercel's Node runtime too; edge is best effort). -- Never changes your return values, errors or streams, and never throws into your app. +- Never changes your return values, errors or streams, and never throws into your app. The one exception is + `weightsDelta()`, a pure helper you call on purpose: it throws on bad input. - Without `CODING_SH_URL` it does nothing at all. ```sh @@ -63,6 +64,8 @@ app.post("/chat", async (req, res) => { | `span("tool" \| "step", name, fn, {input?})` | A tool call or a named step inside the current trace. | | `withSession(id, fn)` | Every trace and untraced call inside `fn` goes to session `id`. | | `aiSdkTracer({sessionId?})` | An OpenTelemetry `Tracer` for the Vercel AI SDK (see below). | +| `weightsEdit({tensor, shape, changed, l2, maxAbs, baseSha256, editedSha256, note?})` | Records one weight edit as a `notification` event with `payload.kind: "weights.edit"`, the same event the Python SDK's `weights_edit()` sends. Returns `true` only when the event was queued. An invalid value drops the event with one warning per process. Inside `trace()` or `span()` it joins that trace; outside it gets a session of its own. | +| `weightsDelta(base, edited)` | Computes `{shape, changed, l2, maxAbs}` for two nested arrays or typed arrays of the same shape, ready to spread into `weightsEdit()`. Throws a `TypeError` or `RangeError` on bad input. | | `init(opts?)` | Optional. The SDK reads env on first use. Options: `url`, `key`, `service`, `capture: "full" \| "metadata"`, `flushIntervalMs` (1000), `maxBatch` (500), `maxQueue` (10000), `disabled`, `debug`, `fetch`. | | `flush(timeoutMs?)` / `shutdown(timeoutMs?)` | Send what is queued. `flush` resolves `true` when the queue is drained. `shutdown` also stops the SDK. | | `stats()` | `{queued, sent, dropped, failed}`. | diff --git a/packages/sdk/src/client.ts b/packages/sdk/src/client.ts index dbdc5e6e..66ea89af 100644 --- a/packages/sdk/src/client.ts +++ b/packages/sdk/src/client.ts @@ -66,6 +66,16 @@ export function enqueue(kind: ItemKind, value: unknown): boolean { return current().transport.enqueue(kind, value); } +/** Problems already reported through warnOnce(): one warning per key per process. */ +const warnedKeys = new Set(); + +/** Warns once per process for each key, however often init() is called. */ +export function warnOnce(key: string, msg: string): void { + if (warnedKeys.has(key)) return; + warnedKeys.add(key); + console.warn(msg); +} + /** True when events would be sent anywhere (a URL is configured and the SDK is on). */ export const isEnabled = (): boolean => current().transport.enabled; @@ -116,4 +126,5 @@ export function _reset(): void { state = undefined; carried.sent = carried.dropped = carried.failed = 0; warnedUrls.clear(); + warnedKeys.clear(); } diff --git a/packages/sdk/src/context.ts b/packages/sdk/src/context.ts index 5b671b5e..445f56f0 100644 --- a/packages/sdk/src/context.ts +++ b/packages/sdk/src/context.ts @@ -10,6 +10,7 @@ // model call llm.call (ts = start, durationMs, usage, model) · assistant.message (ts = end) // tool span tool.start · tool.end // step span span { name, ok } +// notification notification { kind, … } (weightsEdit: one event, a child of the current span) // // A model call or tool span outside any trace is its own root (a one-call // session), unless withSession() names the session it belongs to. @@ -29,6 +30,7 @@ export type EventKind = | "tool.end" | "turn.end" | "span" + | "notification" | "error"; export interface Usage { @@ -407,3 +409,29 @@ function textPayload(s: string): Record { text(p, "text", s); return p; } + +// ---------------------------------------------------------------- notifications + +/** + * Records one `notification` event. Inside a trace or span it is a child of + * the current span. Outside one it gets a session of its own, named `name` + * (or a turn of the withSession() session). True when the notification itself + * was queued. + */ +export function recordNotification(name: string, payload: Record): boolean { + const ctx = als.getStore(); + const parent = ctx?.span; + const start = nowMs(); + if (parent) { + const s: SpanState = { trace: parent.trace, spanId: newSpanId(), parentSpanId: parent.spanId, anchorId: uuid() }; + return emit(s, "notification", start, payload, { parentId: parent.anchorId }); + } + const root = openRoot(ctx?.sessionId); + emit(root, "session.start", start, { name, traceId: root.trace.traceId }, { id: root.anchorId }); + const queued = emit(root, "notification", start, payload); + const end = nowMs(); + const durationMs = end - start; + emit(root, "turn.end", end, { ok: true, durationMs }, { durationMs }); + if (!root.trace.conversation) emit(root, "session.end", end, { reason: "completed" }, { durationMs }); + return queued; +} diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts index ce7b4501..435b1d81 100644 --- a/packages/sdk/src/index.ts +++ b/packages/sdk/src/index.ts @@ -6,6 +6,8 @@ // Set CODING_SH_URL (and CODING_SH_INGEST_KEY) and every model call is sent to // your coding.sh server. Without a URL the SDK does nothing. It never changes // return values, errors or stream semantics, and never throws into the app. +// The one exception is weightsDelta(), a pure helper you call on purpose: it +// throws on bad input. // Contract: docs/PRODUCTION.md §4 in the coding.sh repo. export { init, flush, shutdown, stats } from "./client.ts"; export type { InitOptions, CaptureMode } from "./config.ts"; @@ -13,6 +15,8 @@ export { observe } from "./observe.ts"; export { observeFetch } from "./fetch.ts"; export { trace, span, withSession } from "./context.ts"; export type { TraceOptions, TraceHandle, SpanOptions } from "./context.ts"; +export { weightsEdit, weightsDelta } from "./weights.ts"; +export type { WeightsEdit, WeightsDelta, WeightsArray } from "./weights.ts"; export { aiSdkTracer } from "./ai-sdk.ts"; export type { AiSdkTracer, AiSdkTracerOptions } from "./ai-sdk.ts"; export type { TransportStats } from "./transport.ts"; diff --git a/packages/sdk/src/weights.ts b/packages/sdk/src/weights.ts new file mode 100644 index 00000000..f836fe1f --- /dev/null +++ b/packages/sdk/src/weights.ts @@ -0,0 +1,213 @@ +// Weight edits: one `notification` event per edit, `payload.kind` "weights.edit". +// +// weightsEdit({ +// tensor: "model.layers.3.mlp.down_proj.weight", +// shape: [4096, 11008], +// changed: 1204, +// l2: 0.0183, +// maxAbs: 0.004, +// baseSha256: shaBefore, +// editedSha256: shaAfter, +// }); +// +// The caller's script computes the hashes and norms; this module only checks +// them and sends them. weightsEdit() never throws into the app: a bad value +// drops the event with a warning (once per process) and it returns false. +// weightsDelta() is a helper you call on purpose, so it throws on bad input. +// The Python SDK's weights_edit() sends the same event (packages/sdk-python, +// _weights.py); test/fixtures/weights-edit.json holds what both must send. +// Payload fields are in docs/reference/api.md. +import { config, warnOnce } from "./client.ts"; +import { recordNotification } from "./context.ts"; +import { errorMessage, quiet, truncate } from "./util.ts"; + +const KIND = "weights.edit"; +const SHA256 = /^[0-9a-fA-F]{64}$/; +const MAX_DIMS = 16; + +/** One weight edit, as `weightsEdit()` records it. */ +export interface WeightsEdit { + /** The tensor's name. */ + tensor: string; + /** Its dimensions: at most 16 non-negative integers. */ + shape: readonly number[]; + /** How many weights differ: an integer from 0 to the product of `shape`. */ + changed: number; + /** L2 norm of the change (edited minus base): a finite number >= 0. */ + l2: number; + /** Largest absolute value of the change: a finite number >= 0. */ + maxAbs: number; + /** sha256 of the tensor before the edit: 64 hex characters, sent in lowercase. */ + baseSha256: string; + /** sha256 of the tensor after the edit: 64 hex characters, sent in lowercase. */ + editedSha256: string; + /** Free text, kept only in full capture mode. */ + note?: string; +} + +/** What `weightsDelta()` returns. It spreads into `weightsEdit()`. */ +export interface WeightsDelta { + /** Dimensions of both arrays. */ + shape: number[]; + /** How many values differ. */ + changed: number; + /** L2 norm of edited minus base. */ + l2: number; + /** Largest absolute value of edited minus base. */ + maxAbs: number; +} + +/** What `weightsDelta()` reads: nested arrays of numbers, typed arrays such as `Float32Array`, or arrays of typed arrays. */ +export type WeightsArray = readonly number[] | (ArrayBufferView & ArrayLike) | readonly WeightsArray[]; + +const shown = (v: unknown): string => truncate(typeof v === "string" ? JSON.stringify(v) : String(v), 80).text; + +function count(v: unknown, what: string): number { + if (typeof v !== "number" || !Number.isSafeInteger(v) || v < 0) throw new Error(`${what} must be a non-negative integer, got ${shown(v)}`); + return v; +} + +function norm(v: unknown, what: string): number { + if (typeof v !== "number" || !Number.isFinite(v) || v < 0) throw new Error(`${what} must be a finite number >= 0, got ${shown(v)}`); + return v; +} + +function sha(v: unknown, what: string): string { + if (typeof v !== "string" || !SHA256.test(v)) throw new Error(`${what} must be 64 hex characters`); + return v.toLowerCase(); +} + +/** A string field, cut like every other SDK string. */ +function put(p: Record, key: string, s: string): void { + const t = truncate(s); + p[key] = t.text; + if (t.truncated) p.truncated = true; +} + +/** The event payload for an edit. Throws an Error that says which value is invalid. */ +function payloadOf(edit: WeightsEdit): Record { + if (typeof edit !== "object" || edit === null) throw new Error("expected an object with tensor, shape, changed, l2, maxAbs, baseSha256 and editedSha256"); + const { tensor, shape, changed, l2, maxAbs, baseSha256, editedSha256, note } = edit as { [K in keyof WeightsEdit]: unknown }; + if (typeof tensor !== "string" || !tensor) throw new Error("tensor must be a non-empty string"); + if (!Array.isArray(shape) || shape.length > MAX_DIMS) throw new Error(`shape must be an array of at most ${MAX_DIMS} integers`); + // Array.from visits the holes of a sparse array, so they are rejected like any other non-integer. + const dims = Array.from(shape, (d) => count(d, "shape dimension")); + const n = count(changed, "changed"); + // Exact for this comparison: a product past 2^53 only rounds to a value still above any `changed`. + const total = dims.reduce((a, d) => a * d, 1); + if (n > total) throw new Error(`changed (${n}) is larger than the tensor (${total} weights)`); + const p: Record = { kind: KIND }; + put(p, "tensor", tensor); + p.shape = dims; + p.changed = n; + p.l2 = norm(l2, "l2"); + p.maxAbs = norm(maxAbs, "maxAbs"); + p.baseSha256 = sha(baseSha256, "baseSha256"); + p.editedSha256 = sha(editedSha256, "editedSha256"); + if (note !== undefined && note !== null) { + // The edit is still recorded; only the note is dropped. + if (typeof note !== "string") quiet(warnOnce)("weights-edit-note", "[coding.sh] weightsEdit ignored a note that is not a string"); + else if (note && config().capture === "full") put(p, "note", note); + } + return p; +} + +/** + * Records one weight edit as a `notification` event with `payload.kind` "weights.edit". + * The Python SDK's `weights_edit()` sends the same event. + * + * Returns true only when the event was queued. Returns false when a value is + * invalid, with one warning per process. Returns false without a warning when + * the SDK is off or shut down or the queue is full. Never throws. + * + * A `note` that is not a string is dropped with one warning; the edit is still + * recorded. Inside `trace()` or `span()` the edit joins that trace. Outside + * them it gets a session of its own, or a turn of the `withSession()` session, + * so group a run of edits inside `trace()`. + */ +export function weightsEdit(edit: WeightsEdit): boolean { + try { + let payload: Record; + try { + payload = payloadOf(edit); + } catch (err) { + quiet(warnOnce)("weights-edit", `[coding.sh] weightsEdit dropped an event: ${errorMessage(err)}`); + return false; + } + return recordNotification(KIND, payload); + } catch { + return false; + } +} + +const isTyped = (x: unknown): x is ArrayBufferView & ArrayLike => ArrayBuffer.isView(x) && !(x instanceof DataView); + +const isArrayish = (x: unknown): x is ArrayLike => Array.isArray(x) || isTyped(x); + +/** Checks that x is a rectangular array of finite numbers and returns its shape. */ +function shapeOf(x: unknown): number[] { + if (!isArrayish(x)) throw new TypeError(`expected an array or a typed array, got ${x === null ? "null" : typeof x}`); + const shape: number[] = []; + let leafDepth = -1; + const walk = (v: unknown, depth: number): void => { + if (isArrayish(v)) { + if (depth >= MAX_DIMS) throw new RangeError(`nested more than ${MAX_DIMS} levels deep`); + if (shape.length === depth) shape.push(v.length); + else if (shape[depth] !== v.length) throw new RangeError("ragged array"); + for (let i = 0; i < v.length; i++) walk(v[i], depth + 1); + return; + } + if (leafDepth === -1) leafDepth = depth; + else if (leafDepth !== depth) throw new RangeError("ragged array"); + if (typeof v !== "number") throw new TypeError(`values must be numbers, got ${shown(v)}`); + if (!Number.isFinite(v)) throw new RangeError(`values must be finite, got ${v}`); + }; + walk(x, 0); + if (leafDepth !== -1 && leafDepth !== shape.length) throw new RangeError("ragged array"); + return shape; +} + +interface Sums { + changed: number; + sq: number; + peak: number; +} + +/** Adds up edited minus base over two arrays that shapeOf() found to have the same shape. */ +function diff(a: ArrayLike, b: ArrayLike, sums: Sums): void { + for (let i = 0; i < a.length; i++) { + const x = a[i]; + if (typeof x !== "number") { + diff(x as ArrayLike, b[i] as ArrayLike, sums); + continue; + } + const d = Math.abs((b[i] as number) - x); + if (d === 0) continue; + sums.changed++; + sums.sq += d * d; + if (d > sums.peak) sums.peak = d; + } +} + +/** + * Computes `{ shape, changed, l2, maxAbs }` for two arrays of the same shape, + * ready to spread into `weightsEdit()`. The Python SDK's `weights_delta()` + * gives the same numbers, with `max_abs` for `maxAbs`. + * + * Takes nested arrays of numbers, typed arrays such as `Float32Array`, or + * arrays of typed arrays, up to 16 levels deep, holding finite numbers. It + * reads every value twice on the calling thread and copies nothing. + * + * A pure helper you call on purpose, unlike the never-throw `weightsEdit()`: + * it throws on bad input. A `TypeError` for an argument that is not an array + * or a value that is not a number (null and BigInt included). A `RangeError` + * for a shape mismatch, ragged or too deep nesting, NaN or Infinity. + */ +export function weightsDelta(base: WeightsArray, edited: WeightsArray): WeightsDelta { + const shape = shapeOf(base); + const other = shapeOf(edited); + if (shape.length !== other.length || shape.some((d, i) => d !== other[i])) throw new RangeError(`shapes differ: [${shape.join(", ")}] and [${other.join(", ")}]`); + const sums: Sums = { changed: 0, sq: 0, peak: 0 }; + diff(base as ArrayLike, edited as ArrayLike, sums); + return { shape, changed: sums.changed, l2: Math.sqrt(sums.sq), maxAbs: sums.peak }; +} diff --git a/packages/sdk/test/fixtures/weights-edit.json b/packages/sdk/test/fixtures/weights-edit.json new file mode 100644 index 00000000..1a324190 --- /dev/null +++ b/packages/sdk/test/fixtures/weights-edit.json @@ -0,0 +1,100 @@ +{ + "about": "What weights_edit() in the Python SDK and weightsEdit() in the TypeScript SDK must both send for the same edit. packages/sdk-python/tests/test_weights.py and packages/sdk/test/weights.test.ts both assert against this file. Recorded from the batches the Python SDK posts to /v1/ingest.", + "edit": { + "tensor": "layers.3.mlp.down_proj.weight", + "shape": [4, 8], + "changed": 12, + "l2": 0.25, + "maxAbs": 0.1, + "baseSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "editedSha256": "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", + "note": "rank-1 patch" + }, + "payload": { + "kind": "weights.edit", + "tensor": "layers.3.mlp.down_proj.weight", + "shape": [4, 8], + "changed": 12, + "l2": 0.25, + "maxAbs": 0.1, + "baseSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "editedSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "note": "rank-1 patch" + }, + "volatilePayloadKeys": ["traceId", "durationMs", "toolUseId"], + "ignoredEventKeys": ["source"], + "scenarios": { + "alone": { + "how": "One edit outside trace(), span() and withSession().", + "sessionId": null, + "parent": null, + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"name": "weights.edit", "traceId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"]}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"ok": true, "durationMs": ""}}, + {"kind": "session.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"reason": "completed"}} + ] + }, + "withSession": { + "how": "One edit inside withSession(sessionId).", + "sessionId": "run-7", + "parent": null, + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"], "payload": {"name": "weights.edit", "traceId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"]}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"], "payload": {"ok": true, "durationMs": ""}} + ] + }, + "trace": { + "how": "One edit inside trace(\"run\").", + "sessionId": null, + "parent": "session.start", + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"name": "run", "traceId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"]}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"ok": true, "durationMs": ""}}, + {"kind": "session.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"reason": "completed"}} + ] + }, + "traceWithSessionId": { + "how": "One edit inside trace(\"run\") with sessionId.", + "sessionId": "conv-1", + "parent": "session.start", + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"], "payload": {"name": "run", "traceId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"]}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "turnId", "v"], "payload": {"ok": true, "durationMs": ""}} + ] + }, + "toolSpan": { + "how": "One edit inside span(\"tool\", \"patch\") inside trace(\"run\").", + "sessionId": null, + "parent": "tool.start", + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"name": "run", "traceId": ""}}, + {"kind": "tool.start", "keys": ["id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"tool": "patch", "toolUseId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"]}, + {"kind": "tool.end", "keys": ["durationMs", "id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"tool": "patch", "toolUseId": "", "ok": true, "durationMs": ""}}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"ok": true, "durationMs": ""}}, + {"kind": "session.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"reason": "completed"}} + ] + }, + "stepSpan": { + "how": "One edit inside span(\"step\", \"patch\") inside trace(\"run\").", + "sessionId": null, + "parent": "span", + "events": [ + {"kind": "session.start", "keys": ["id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"name": "run", "traceId": ""}}, + {"kind": "notification", "keys": ["id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"]}, + {"kind": "span", "keys": ["durationMs", "id", "kind", "parentId", "parentSpanId", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"name": "patch", "ok": true}}, + {"kind": "turn.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"ok": true, "durationMs": ""}}, + {"kind": "session.end", "keys": ["durationMs", "id", "kind", "payload", "sessionId", "spanId", "traceId", "ts", "v"], "payload": {"reason": "completed"}} + ] + } + }, + "delta": { + "base": [[0.1, 0.2, 0.3], [1.5, -2.0, 4.0]], + "edited": [[0.1, 0.25, 0.3], [1.0, -2.0, 4.75]], + "result": {"shape": [2, 3], "changed": 3, "l2": 0.9027735042633894, "maxAbs": 0.75} + } +} diff --git a/packages/sdk/test/weights.test.ts b/packages/sdk/test/weights.test.ts new file mode 100644 index 00000000..6a8a8b3c --- /dev/null +++ b/packages/sdk/test/weights.test.ts @@ -0,0 +1,504 @@ +// weightsEdit: one `notification` event with payload.kind "weights.edit" (docs/reference/api.md). +// +// These mirror packages/sdk-python/tests/test_weights.py case for case. Most read the batch the +// SDK posts to a fake ingest endpoint. The parity tests check those batches against +// test/fixtures/weights-edit.json, which the Python SDK's tests check too. The last test starts a +// real `coding.sh serve` on a temp CODING_SH_HOME and reads the event back, redacted. +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { createInterface } from "node:readline"; +import { after, afterEach, before, beforeEach, test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { flush, init, shutdown, span, stats, trace, weightsDelta, weightsEdit, withSession, type WeightsArray, type WeightsEdit } from "../src/index.ts"; +import { _reset } from "../src/client.ts"; +import { MAX_TEXT_BYTES } from "../src/util.ts"; +import { type FakeServe, startFakeServe } from "./helpers/fake-serve.ts"; + +const here = dirname(fileURLToPath(import.meta.url)); +const CLI = join(here, "..", "..", "coding.sh", "src", "cli", "main.ts"); +const A = "a".repeat(64); +const B = "b".repeat(64); +// Built in pieces so push protection does not flag the test file. +const FAKE_TOKEN = ["ghp_", "0123456789", "abcdefghij", "ABCDEFGHIJ", "klmnop"].join(""); + +/** An event as the SDK posted it, before the server parsed it. */ +interface Wire { + id: string; + kind: string; + sessionId: string; + traceId: string; + spanId: string; + parentId?: string; + parentSpanId?: string; + turnId?: string; + payload: Record; +} + +interface Fixture { + edit: WeightsEdit; + payload: Record; + volatilePayloadKeys: string[]; + ignoredEventKeys: string[]; + scenarios: Record }[] }>; + delta: { base: WeightsArray; edited: WeightsArray; result: unknown }; +} + +// What both SDKs must send for the same edit. A missing fixture fails every test in this file. +const fx = JSON.parse(readFileSync(join(here, "fixtures", "weights-edit.json"), "utf8")) as Fixture; + +let serve: FakeServe; +let mark = 0; +before(async () => { + serve = await startFakeServe(); +}); +after(async () => { + assert.deepEqual(serve.rejected, [], "the real server parser accepts every SDK event"); + await serve.close(); +}); +beforeEach(() => start()); +afterEach(async () => { + await shutdown(1000); +}); + +function start(opts: Parameters[0] = {}): void { + _reset(); + init({ url: serve.url, flushIntervalMs: 5, ...opts }); + mark = serve.requests.length; +} + +/** The events posted since start(), in the order they were sent. */ +async function posted(): Promise { + assert.equal(await flush(), true, "flush timed out"); + return serve.requests + .slice(mark) + .filter((r) => r.path === "/v1/ingest" && r.status === 200) + .flatMap((r) => (r.body as { events: Wire[] }).events); +} + +const byKind = (events: Wire[], kind: string) => events.filter((e) => e.kind === kind); + +function one(list: T[]): T { + assert.equal(list.length, 1); + return list[0]!; +} + +const edit = (over: Record = {}): boolean => + weightsEdit({ tensor: "layers.3.mlp.down_proj.weight", shape: [4, 8], changed: 12, l2: 0.25, maxAbs: 0.1, baseSha256: A, editedSha256: B, ...over } as WeightsEdit); + +/** Runs fn and returns what it sent to console.warn. */ +async function warnings(fn: () => unknown): Promise { + const warns: string[] = []; + const orig = console.warn; + console.warn = (...a: unknown[]) => void warns.push(a.join(" ")); + try { + await fn(); + } finally { + console.warn = orig; + } + return warns; +} + +test("one notification with the payload", async () => { + assert.equal(edit({ note: "rank-1 patch" }), true); + const events = await posted(); + assert.deepEqual(one(byKind(events, "notification")).payload, { + kind: "weights.edit", + tensor: "layers.3.mlp.down_proj.weight", + shape: [4, 8], + changed: 12, + l2: 0.25, + maxAbs: 0.1, + baseSha256: A, + editedSha256: B, + note: "rank-1 patch", + }); + assert.equal(byKind(events, "session.start").length, 1, "a session of its own"); + // The Python SDK sends `source`; here the server fills it in. Both are stored as "sdk". + assert.equal(one(serve.events().filter((e) => e.kind === "notification" && e.sessionId === events[0]!.sessionId)).source, "sdk"); +}); + +test("joins the current trace", async () => { + trace( + "edit-run", + () => { + edit(); + edit({ tensor: "lm_head.weight" }); + }, + { sessionId: "conv-1" }, + ); + const events = await posted(); + assert.deepEqual(byKind(events, "notification").map((n) => n.sessionId), ["conv-1", "conv-1"]); + assert.equal(byKind(events, "session.start").length, 1); +}); + +test("metadata mode drops the note only", async () => { + start({ capture: "metadata" }); + edit({ note: "free text" }); + const p = one(byKind(await posted(), "notification")).payload; + assert.ok(!("note" in p)); + assert.equal(p.changed, 12); + assert.equal(p.tensor, "layers.3.mlp.down_proj.weight", "the tensor's name is not free text"); +}); + +test("invalid values are dropped, not thrown", async () => { + const bad: Record[] = [ + { changed: -1 }, + { changed: true }, + { changed: 2.5 }, + { changed: 33 }, // more than 4 * 8 weights + { changed: "12" }, + { changed: 12n }, + { changed: Number.NaN }, + { changed: 2 ** 53, shape: [2 ** 53] }, // not an integer JavaScript can hold exactly + { l2: -0.1 }, + { l2: Number.NaN }, + { l2: "0.25" }, + { maxAbs: Number.POSITIVE_INFINITY }, + { maxAbs: undefined }, + { baseSha256: "abc" }, + { editedSha256: "g".repeat(64) }, + { shape: [4, -1] }, + { shape: [4, 8.5] }, + { shape: "4x8" }, + { shape: new Int32Array([4, 8]) }, + { shape: [4, , 8] }, // a hole is not an integer + { shape: Array.from({ length: 17 }, () => 1) }, + { tensor: "" }, + { tensor: null }, + { baseSha256: A + "\n" }, // a trailing newline is not 64 hex characters + ]; + const warns = await warnings(() => { + for (const over of bad) assert.equal(edit(over), false, JSON.stringify(Object.keys(over))); + // A call from plain JavaScript can pass anything. + const untyped = weightsEdit as (edit: unknown) => boolean; + for (const arg of [undefined, null, "edit", 5]) assert.equal(untyped(arg), false, String(arg)); + }); + assert.equal(warns.length, 1); + assert.deepEqual(byKind(await posted(), "notification"), []); +}); + +test("an empty tensor and a scalar are valid shapes", async () => { + assert.equal(edit({ shape: [], changed: 1 }), true, "no dimensions: one weight"); + assert.equal(edit({ shape: [0, 3], changed: 0 }), true); + assert.equal(edit({ shape: [4, 8], changed: 32 }), true, "every weight changed"); + await warnings(() => assert.equal(edit({ shape: [0, 3], changed: 1 }), false)); + assert.deepEqual(byKind(await posted(), "notification").map((n) => n.payload.shape), [[], [0, 3], [4, 8]]); +}); + +test("uppercase sha is lowercased", async () => { + assert.equal(edit({ baseSha256: "A".repeat(64) }), true); + assert.equal(one(byKind(await posted(), "notification")).payload.baseSha256, A); +}); + +test("returns false when disabled", () => { + _reset(); + init({ url: serve.url, disabled: true }); + assert.equal(edit(), false); +}); + +test("a valid edit is silent when disabled; an invalid one still warns, as in Python", async () => { + _reset(); + init({ url: serve.url, disabled: true }); + assert.deepEqual(await warnings(() => assert.equal(edit(), false)), []); + const warns = await warnings(() => assert.equal(edit({ tensor: "" }), false)); + assert.equal(warns.length, 1); + assert.match(warns[0]!, /tensor must be a non-empty string/); +}); + +test("returns false without a URL", () => { + _reset(); + init({ url: "" }); + assert.equal(edit(), false); +}); + +test("returns false after shutdown", async () => { + assert.equal(edit(), true); + await shutdown(2000); + const warns = await warnings(() => assert.equal(edit(), false)); + assert.deepEqual(warns, [], "no warning: nothing is wrong with the edit"); +}); + +test("returns false when the queue is full", async () => { + start({ maxQueue: 1, flushIntervalMs: 60_000 }); + // The trace's session.start takes the one slot. + await warnings(() => trace("full", () => assert.equal(edit(), false))); + assert.equal(stats().queued, 1); +}); + +test("warns once per process", async () => { + const warns = await warnings(() => { + edit({ changed: -1 }); + edit({ changed: -2 }); + edit({ l2: -1 }); + init({ url: serve.url, flushIntervalMs: 5 }); // a second init() is the same process + edit({ changed: -3 }); + }); + assert.equal(warns.filter((w) => w.includes("weightsEdit dropped")).length, 1); + assert.match(warns[0]!, /^\[coding\.sh\] weightsEdit dropped an event: changed must be a non-negative integer, got -1$/); +}); + +test("a note that is not a string is dropped with a warning", async () => { + const warns = await warnings(() => { + assert.equal(edit({ note: 5 }), true); + assert.equal(edit({ note: { text: "x" } }), true); + }); + assert.equal(warns.length, 1); + assert.match(warns[0]!, /note/); + const notes = byKind(await posted(), "notification"); + assert.equal(notes.length, 2); + for (const n of notes) { + assert.ok(!("note" in n.payload)); + assert.equal(n.payload.changed, 12); + } +}); + +test("an empty or missing note is left out without a warning", async () => { + const warns = await warnings(() => { + for (const note of ["", null, undefined]) assert.equal(edit({ note }), true); + }); + assert.deepEqual(warns, []); + for (const n of byKind(await posted(), "notification")) assert.ok(!("note" in n.payload)); +}); + +test("a long tensor name or note is cut like every other SDK string", async () => { + const long = "x".repeat(MAX_TEXT_BYTES + 10); + assert.equal(edit({ tensor: long, note: long }), true); + const p = one(byKind(await posted(), "notification")).payload; + assert.equal((p.tensor as string).length, MAX_TEXT_BYTES); + assert.equal((p.note as string).length, MAX_TEXT_BYTES); + assert.equal(p.truncated, true); +}); + +test("inside withSession", async () => { + withSession("run-7", () => { + edit(); + edit({ tensor: "b" }); + }); + assert.deepEqual(byKind(await posted(), "notification").map((n) => n.sessionId), ["run-7", "run-7"]); +}); + +test("inside a span links to it", async () => { + trace("run", () => span("tool", "patch", () => void edit())); + const events = await posted(); + const note = one(byKind(events, "notification")); + const tool = one(byKind(events, "tool.start")); + assert.equal(note.parentId, tool.id); + assert.equal(note.parentSpanId, tool.spanId); +}); + +test("an edit after an await still joins the trace", async () => { + await trace("run", async () => { + await new Promise((r) => setTimeout(r, 1)); + edit(); + }); + const events = await posted(); + assert.equal(one(byKind(events, "notification")).parentId, one(byKind(events, "session.start")).id); +}); + +test("server down never throws", async () => { + const dead = await startFakeServe(); + const url = dead.url; + await dead.close(); + _reset(); + init({ url, flushIntervalMs: 5 }); + await warnings(async () => { + assert.equal(edit(), true, "queued; delivery fails in the background"); + assert.equal(await flush(200), false); + await shutdown(200); + }); +}); + +// ---------------------------------------------------------------- weightsDelta + +test("weightsDelta", () => { + assert.deepEqual(weightsDelta([[1, 2], [3, 4]], [[1, 2], [3, 7]]), { shape: [2, 2], changed: 1, l2: 3, maxAbs: 3 }); + assert.deepEqual(weightsDelta([0, 0, 0], [3, 0, -4]), { shape: [3], changed: 2, l2: 5, maxAbs: 4 }); + assert.throws(() => weightsDelta([1, 2], [[1, 2]]), RangeError); +}); + +test("weightsDelta edges", () => { + assert.deepEqual(weightsDelta([], []), { shape: [0], changed: 0, l2: 0, maxAbs: 0 }); + assert.deepEqual(weightsDelta([[], []], [[], []]).shape, [2, 0]); + assert.equal(weightsDelta([0, 1, 2], [0, 1, 5]).changed, 1); + assert.deepEqual(weightsDelta(new Float32Array(0), []).shape, [0]); + let deep: WeightsArray = [1]; + for (let i = 1; i < 16; i++) deep = [deep]; + assert.deepEqual(weightsDelta(deep, deep), { shape: Array.from({ length: 16 }, () => 1), changed: 0, l2: 0, maxAbs: 0 }); +}); + +test("weightsDelta reads typed arrays", () => { + assert.deepEqual(weightsDelta(new Float32Array([0, 0, 0]), new Float32Array([3, 0, -4])), { shape: [3], changed: 2, l2: 5, maxAbs: 4 }); + assert.deepEqual(weightsDelta([new Float64Array([1, 2]), new Float64Array([3, 4])], [[1, 2], [3, 7]]), { shape: [2, 2], changed: 1, l2: 3, maxAbs: 3 }); + assert.deepEqual(weightsDelta(new Int8Array([1, 2]), new Uint16Array([1, 4])), { shape: [2], changed: 1, l2: 2, maxAbs: 2 }); + assert.throws(() => weightsDelta(new Float32Array([Number.NaN]), new Float32Array([1])), RangeError); +}); + +test("weightsDelta throws on bad input", () => { + let deep: unknown = [1]; + for (let i = 0; i < 16; i++) deep = [deep]; // 17 levels + const gen = (function* () { + yield 1; + })(); + const bad: [unknown, unknown, typeof TypeError | typeof RangeError][] = [ + [[1, [2]], [1, [2]], RangeError], + [[[1], 2], [[1], 2], RangeError], + [[[], [1]], [[], [1]], RangeError], + [["a"], ["a"], TypeError], + [[null], [null], TypeError], + [[true], [true], TypeError], + [[1n], [1n], TypeError], + [[1, , 3], [1, 2, 3], TypeError], // a hole is not a number + [new BigInt64Array(1), new BigInt64Array(1), TypeError], + [null, null, TypeError], + [undefined, [1], TypeError], + [5, 5, TypeError], + ["ab", "ab", TypeError], + [gen, [1], TypeError], + [{ length: 1, 0: 1 }, [1], TypeError], + [new DataView(new ArrayBuffer(8)), [1], TypeError], + [[Number.NaN], [1], RangeError], + [[1], [Number.POSITIVE_INFINITY], RangeError], + [deep, deep, RangeError], + [[1, 2], [1], RangeError], + ]; + for (const [a, b, kind] of bad) assert.throws(() => weightsDelta(a as WeightsArray, b as WeightsArray), kind, JSON.stringify(a, (_k, v) => (typeof v === "bigint" ? `${v}n` : v))?.slice(0, 40)); +}); + +test("weightsDelta feeds weightsEdit", async () => { + assert.equal(weightsEdit({ tensor: "w", baseSha256: A, editedSha256: B, ...weightsDelta([[0, 1]], [[0.5, 1]]) }), true); + assert.deepEqual(one(byKind(await posted(), "notification")).payload, { kind: "weights.edit", tensor: "w", shape: [1, 2], changed: 1, l2: 0.5, maxAbs: 0.5, baseSha256: A, editedSha256: B }); +}); + +// ---------------------------------------------------------------- parity with the Python SDK + +type Run = (edit: () => void, sessionId: string | null) => void; + +// One entry per scenario in the fixture; test_weights.py has the same list. The functions return +// nothing, so no span records an output the Python `with` block would not have. +const SCENARIOS: Record = { + alone: (edit) => edit(), + withSession: (edit, sid) => withSession(sid!, edit), + trace: (edit) => trace("run", edit), + traceWithSessionId: (edit, sid) => trace("run", edit, { sessionId: sid! }), + toolSpan: (edit) => trace("run", () => span("tool", "patch", edit)), + stepSpan: (edit) => trace("run", () => span("step", "patch", edit)), +}; + +test("every scenario in the fixture runs here", () => { + assert.deepEqual(Object.keys(SCENARIOS).sort(), Object.keys(fx.scenarios).sort()); +}); + +for (const [name, sc] of Object.entries(fx.scenarios)) { + test(`the same events as the Python SDK: ${name}`, async () => { + let queued = false; + SCENARIOS[name]!(() => void (queued = weightsEdit(fx.edit)), sc.sessionId); + assert.equal(queued, true); + const events = await posted(); + const got = events.map((e) => ({ + kind: e.kind, + keys: Object.keys(e).filter((k) => !fx.ignoredEventKeys.includes(k)).sort(), + payload: Object.fromEntries(Object.entries(e.payload).map(([k, v]) => [k, fx.volatilePayloadKeys.includes(k) ? "" : v])), + })); + assert.deepEqual(got, sc.events.map((e) => (e.kind === "notification" ? { ...e, payload: fx.payload } : e))); + + // One trace, one session, and the edit hangs off the event the fixture names. + const first = events[0]!; + const note = one(byKind(events, "notification")); + assert.deepEqual([...new Set(events.map((e) => e.traceId))], [first.traceId]); + assert.deepEqual([...new Set(events.map((e) => e.sessionId))], [sc.sessionId ?? first.sessionId]); + if (sc.parent === null) assert.equal(note.spanId, first.spanId); + else { + const parent = one(byKind(events, sc.parent)); + assert.equal(note.parentId, parent.id); + assert.equal(note.parentSpanId, parent.spanId); + assert.notEqual(note.spanId, parent.spanId); + } + if (note.turnId !== undefined) assert.equal(note.turnId, note.traceId); + }); +} + +test("the same delta as the Python SDK", () => { + assert.deepEqual(weightsDelta(fx.delta.base, fx.delta.edited), fx.delta.result); +}); + +// ---------------------------------------------------------------- a real server + +const [major = 0, minor = 0] = process.versions.node.split(".").map(Number); +// The same condition as WeightsEditServerTest in test_weights.py. +const serveOk = existsSync(CLI) && (major > 22 || (major === 22 && minor >= 15)); + +test("round trip through a real coding.sh serve: the event comes back with the secret redacted", { skip: serveOk ? false : "needs node >= 22.15 and the core package" }, async () => { + const root = mkdtempSync(join(tmpdir(), "csh-sdk-weights-")); + const ingest = "csh_ik_" + "w".repeat(43); + const read = "csh_rk_" + "r".repeat(43); + // Nothing the server reads or writes is the user's: every home it knows is under root. + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: join(root, "home"), + USERPROFILE: join(root, "home"), + CODING_SH_HOME: join(root, "coding.sh"), + CLAUDE_CONFIG_DIR: join(root, "claude"), + CODEX_HOME: join(root, "codex"), + XDG_CONFIG_HOME: join(root, "xdg-config"), + XDG_DATA_HOME: join(root, "xdg-data"), + CURSOR_CONFIG_DIR: join(root, "cursor"), + OPENCODE_CONFIG_DIR: join(root, "opencode"), + OPENCODE_DB: join(root, "opencode.db"), + CODING_SH_NO_AUTO_INGEST: "1", + CODING_SH_INGEST_KEYS: `demo:${ingest}`, + CODING_SH_READ_KEYS: `demo:${read}`, + NODE_NO_WARNINGS: "1", + }; + delete env.CODING_SH_DEV_REAL_STORE; + const proc = spawn(process.execPath, ["--no-warnings", CLI, "serve", "--port", "0"], { env, cwd: root, stdio: ["ignore", "pipe", "pipe"] }); + const exited = once(proc, "exit"); + let stderr = ""; + proc.stderr.on("data", (c) => (stderr += String(c))); + try { + const lines = createInterface({ input: proc.stdout }); + const url = await Promise.race([ + once(lines, "line").then(([l]) => String(l).trim()), + exited.then(() => ""), + new Promise((r) => setTimeout(() => r(""), 30_000).unref()), + ]); + assert.match(url, /^http:\/\//, `serve did not start: ${stderr || url}`); + _reset(); + init({ url, key: ingest, service: "weights-test", flushIntervalMs: 5 }); + assert.equal(weightsEdit({ tensor: `blocks.0.${FAKE_TOKEN}`, shape: [2, 3], changed: 2, l2: 0.5, maxAbs: 0.4, baseSha256: A, editedSha256: B, note: `deploy with ${FAKE_TOKEN}` }), true); + assert.equal(await flush(5000), true); + assert.equal(stats().failed, 0); + + let items: { kind: string; payload: Record }[] = []; + const until = Date.now() + 10_000; + for (;;) { + const res = await fetch(`${url}/api/events?limit=50`, { headers: { authorization: `Bearer ${read}` } }); + assert.equal(res.status, 200); + const body = (await res.json()) as { items?: typeof items } | typeof items; + items = Array.isArray(body) ? body : (body.items ?? []); + if (items.some((i) => i.kind === "notification") || Date.now() > until) break; + await new Promise((r) => setTimeout(r, 100)); + } + const e = one(items.filter((i) => i.kind === "notification")); + const p = e.payload; + assert.equal(p.kind, "weights.edit"); + assert.deepEqual([p.shape, p.changed, p.l2, p.maxAbs], [[2, 3], 2, 0.5, 0.4]); + assert.deepEqual([p.baseSha256, p.editedSha256], [A, B]); + assert.ok(!JSON.stringify(e).includes(FAKE_TOKEN)); + assert.equal(p.tensor, "blocks.0.[redacted:github-token]"); + assert.equal(p.note, "deploy with [redacted:github-token]"); + } finally { + await shutdown(1000); + proc.kill(); + // The store's files are open until the server exits; remove them only after it has. + await Promise.race([exited, new Promise((r) => setTimeout(r, 10_000).unref())]); + if (proc.exitCode === null && proc.signalCode === null) { + proc.kill("SIGKILL"); + await exited; + } + rmSync(root, { recursive: true, force: true }); + } +}); From 0b65eb97ec9bf1c919c7ea1daf868c942f19ff4a Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:12:44 -0600 Subject: [PATCH 04/17] feat(otel): a weight edit reads the same whether it arrives over OTLP or leaves through the otlp sink A span or a log record named weights.edit with coding_sh.weights.* attributes is now read as the weights.edit notification the SDK sends, so the session gets the same stat, tag and line. Before, a span became a generic span and a root span lost its attributes. The otlp sink and serve's forward send one weights.edit span per edit with the same attributes, and the invoke_agent span carries coding_sh.stats.weight_edits and coding_sh.stats.weights_changed. Metadata mode keeps the counts and the norms: the forward allowlist drops the tensor name, shape, hashes and note. --- CHANGELOG.md | 1 + docs/PRODUCTION.md | 4 +- docs/reference/api.md | 2 + docs/reference/stores.md | 2 + packages/coding.sh/src/core/weights.ts | 24 +++++ packages/coding.sh/src/otel/map.ts | 31 +++++++ packages/coding.sh/src/server/forward.ts | 6 +- packages/coding.sh/src/sinks/otlp.ts | 16 ++++ packages/coding.sh/test/otel/weights.test.ts | 90 +++++++++++++++++++ .../coding.sh/test/server/forward.test.ts | 8 ++ packages/coding.sh/test/sinks/otlp.test.ts | 71 ++++++++++++++- 11 files changed, 250 insertions(+), 5 deletions(-) create mode 100644 packages/coding.sh/test/otel/weights.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f451bf07..252a5895 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ grouped by area, with links to the docs for the details. - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) - **Sessions.** A session that recorded weight edits gets `stats.weightEdits` (edits, distinct tensors, weights changed) and the `weights:edited` tag, and is listed even when it holds nothing else. `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit instead of a blank notification (MCP dropped them), and `coding.sh sessions --where 'weights.changed>1000'` filters by size. Stored sessions are folded again. [Docs](https://coding.sh/docs/api) +- **OpenTelemetry.** A span or a log record named `weights.edit` with `coding_sh.weights.*` attributes is read as a weight edit, and the `otlp` sink and `[serve.forward]` send one `weights.edit` span per edit, so an edit reads the same whether it arrived from an SDK or over OTLP. The `invoke_agent` span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. Metadata mode keeps the counts and the norms only. [Docs](https://coding.sh/docs/api) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/PRODUCTION.md b/docs/PRODUCTION.md index 28e441e7..5a8c1016 100644 --- a/docs/PRODUCTION.md +++ b/docs/PRODUCTION.md @@ -265,11 +265,11 @@ max_buffer_spans = 20000 ``` - OTLP posted to `/v1/traces` keeps its traceId, spanId and parent ids. Native `/v1/ingest` batches - are sent as `invoke_agent`, `execute_tool` and `chat` spans, the same mapping as the `otlp` sink. + are sent as `invoke_agent`, `execute_tool`, `chat` and `weights.edit` spans, the same mapping as the `otlp` sink. Every resource gets `coding_sh.project` set to the ingest key's project. - `metadata` mode keeps an allowlist: gen_ai operation, provider, model, response id and finish reasons, tool name, call id and type, agent and conversation ids, token usage, `service.*`, - `telemetry.*`, `coding_sh.*` (except the session title and repo root), and numbers and booleans. + `telemetry.*`, `coding_sh.*` (except the session title, the repo root, and a weight edit's tensor name, shape, hashes and note), and numbers and booleans. Messages, prompts, tool arguments and results, `input.value`, `output.value`, span event attributes other than `exception.type`, and status messages are dropped. `full` mode sends every attribute after `capture.redact`. - Spans are sent in batches of up to 1,000, about once a second. A network error, 408, 429 or 5xx diff --git a/docs/reference/api.md b/docs/reference/api.md index f1261277..185f10c2 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -731,6 +731,8 @@ One more `notification` kind records weight edits. The Python SDK's `coding_sh.w - A session with a weight edit gets the `weights:edited` tag. `coding.sh sessions --flag weights:edited` lists them, and `--where 'weights.changed>1000'` filters by size. - `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. - Metadata mode: the SDK leaves out `note`. A server with `capture.mode = "metadata"` also drops `tensor`, `shape` and both hashes, and keeps `changed`, `l2` and `maxAbs`. +- OpenTelemetry in: a span or a log record named `weights.edit` sent to `/v1/traces` or `/v1/logs` is read as the same event. Its fields are the attributes `coding_sh.weights.tensor`, `.shape` (an array of integers), `.changed`, `.l2`, `.max_abs`, `.base_sha256`, `.edited_sha256` and `.note`. A malformed attribute is left out and the edit still counts. +- OpenTelemetry out: the `otlp` sink and `[serve.forward]` send one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. - The caller computes the hashes and norms. The SDK checks the values, drops an invalid edit with one warning, and never raises into the app. Numpy and torch scalars count as numbers. - `weights_edit()` returns `True` only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. diff --git a/docs/reference/stores.md b/docs/reference/stores.md index de81ac1b..f233a832 100644 --- a/docs/reference/stores.md +++ b/docs/reference/stores.md @@ -182,6 +182,7 @@ What lands in `otel_traces`, with `ServiceName = 'coding.sh'`: | `invoke_agent ` | session (an open one can have several rows) | `gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.request.model`, `gen_ai.conversation.id` (the session id), `gen_ai.usage.*`, `coding_sh.cost_usd`, `coding_sh.stats.*` (events, turns, tool_calls, tool_errors, files_changed, tests_passed, tests_failed), `coding_sh.session.flags`, `vcs.ref.head.revision` | | `execute_tool ` | tool call | `gen_ai.operation.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.agent.name`; `StatusCode` is `Error` when the call failed | | `chat ` | model call | `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.provider.name`, `gen_ai.usage.*`, `coding_sh.cost_usd` | +| `weights.edit` | weight edit | under `coding_sh.weights.`: `changed`, `l2` and `max_abs`; `mode = "full"` adds `tensor`, `shape`, `base_sha256`, `edited_sha256` and `note` | `gen_ai.usage.*` is `input_tokens`, `output_tokens`, `cache_read.input_tokens` and `cache_write.input_tokens`. `input_tokens` is the whole prompt, cache included; the uncached part @@ -192,6 +193,7 @@ trace, and `Duration` is in nanoseconds. - Map columns store every attribute as a string: cast with `toFloat64OrZero` and `toUInt64OrZero`. `coding_sh.session.flags` is a JSON array string. - Cost is on the `invoke_agent` span (the session) and on each `chat` span. Sum one, never both. +- A session that edited weights also has `weight_edits` and `weights_changed` under `coding_sh.stats.`, and one `weights.edit` span per edit. - `mode = "full"` adds prompts, code and tool output as span attributes. - The `invoke_agent` span is sent when the session ends. `coding.sh serve` and `coding.sh collector` also send it while a session is open: after 10 quiet seconds, and every 60 seconds while busy. diff --git a/packages/coding.sh/src/core/weights.ts b/packages/coding.sh/src/core/weights.ts index 6b1fa34f..615ffee5 100644 --- a/packages/coding.sh/src/core/weights.ts +++ b/packages/coding.sh/src/core/weights.ts @@ -54,6 +54,30 @@ export function weightsEditOf(p: Record): WeightsEdit { return out; } +/** + * OTLP: a span or a log record named `weights.edit` carries the edit in these attributes. + * otel/map.ts reads them and sinks/otlp.ts writes them, so an edit keeps its fields both ways. + */ +export const WEIGHTS_EDIT_ATTRS: Record = { + tensor: "coding_sh.weights.tensor", + shape: "coding_sh.weights.shape", + changed: "coding_sh.weights.changed", + l2: "coding_sh.weights.l2", + maxAbs: "coding_sh.weights.max_abs", + baseSha256: "coding_sh.weights.base_sha256", + editedSha256: "coding_sh.weights.edited_sha256", + note: "coding_sh.weights.note", +}; +/** The attributes that name the tensor or the checkpoints: metadata capture leaves them out, as it does the payload fields. */ +export const WEIGHTS_EDIT_CONTENT_ATTRS: string[] = (["tensor", "shape", "baseSha256", "editedSha256", "note"] as const).map((k) => WEIGHTS_EDIT_ATTRS[k]); + +/** A weights.edit payload as OTLP attributes: the fields that are present and well-formed. */ +export function weightsEditAttrs(p: Record): Record { + const out: Record = {}; + for (const [k, v] of Object.entries(weightsEditOf(p))) out[WEIGHTS_EDIT_ATTRS[k as keyof WeightsEdit]] = v; + return out; +} + export function foldWeightEdits(s: WeightEditState, e: AgentEvent): void { if (!isWeightsEdit(e)) return; const w = weightsEditOf(e.payload); diff --git a/packages/coding.sh/src/otel/map.ts b/packages/coding.sh/src/otel/map.ts index 1c5991f4..7c5ab3ab 100644 --- a/packages/coding.sh/src/otel/map.ts +++ b/packages/coding.sh/src/otel/map.ts @@ -35,6 +35,7 @@ import { SCHEMA_VERSION } from "../schema/index.ts"; import { hashParts, sha256Hex } from "../core/ids.ts"; import { noteBlobHits, type RedactionTally, type Redactor, redactBlobValue, redactionTally } from "../core/redact.ts"; import { geoValue, tierValue } from "../core/pricing.ts"; +import { WEIGHTS_EDIT_ATTRS, WEIGHTS_EDIT_KIND, weightsEditOf } from "../core/weights.ts"; import { anthropicPriceHints } from "../adapters/_shared/normalize.ts"; import { type DurableTag, durableOf, durableSessionId } from "./durable.ts"; import { @@ -795,6 +796,29 @@ function errorMessage(span: OtelSpan): string | undefined { return ex ? str(ex.attributes, "exception.message", "exception.type") : undefined; } +/** + * The payload of a span or log record named `weights.edit`: `kind` plus the well-formed + * coding_sh.weights.* attributes (WEIGHTS_EDIT_ATTRS), the event the SDKs' weights_edit helpers send. + * A number or a shape sent as a string is read as the number or the list it spells. + */ +function weightsEditDraft(d: Draft, a: Attrs): Draft { + const K = WEIGHTS_EDIT_ATTRS; + const w = weightsEditOf({ + tensor: str(a, K.tensor), + shape: maybeJson(a[K.shape]), + changed: num(a, K.changed), + l2: num(a, K.l2), + maxAbs: num(a, K.maxAbs), + baseSha256: str(a, K.baseSha256), + editedSha256: str(a, K.editedSha256), + note: str(a, K.note), + }); + d.payload.kind = WEIGHTS_EDIT_KIND; + // Assigned, not set(): an empty shape is a scalar's shape and a zero is a number. + Object.assign(d.payload, w); + return d; +} + async function mapSpan(c: SpanCtx, ctx: MapContext, store: Blobs): Promise { const { span, traceId, spanId } = c; const a = span.attributes; @@ -840,6 +864,8 @@ async function mapSpan(c: SpanCtx, ctx: MapContext, store: Blobs): Promise new Draft(store, max, raw); + // A span named weights.edit is a weight edit, whatever else its attributes say. + const weightsEdit = span.name === WEIGHTS_EDIT_KIND; const agentName = str(a, "gen_ai.agent.name", "agent.name") ?? (/^invoke_agent\s+(.+)$/.exec(span.name)?.[1] || undefined); // ---- model call @@ -916,6 +942,7 @@ async function mapSpan(c: SpanCtx, ctx: MapContext, store: Blobs): Promise 0 ? { durationMs } : {}); } else if (klass === "llm" || klass === "embedding") { await llm(0); } else if (klass === "tool") { @@ -1332,6 +1361,8 @@ async function mapLog(rec: OtelLogRecord, resource: Attrs, scope: OtelScope, ctx .set("plugin", str(a, "plugin.name", "skill.plugin_id")) .set("event", name); emit(0, "notification", ts, d); + } else if (suffix === WEIGHTS_EDIT_KIND) { + emit(0, "notification", ts, weightsEditDraft(draft(), a)); } else if (suffix === "agent_response" && content(str(a, "response"))) { // Codex: a final answer of the main agent or of a spawned subagent (otel.log_agent_responses). const d = draft().set("responseLength", num(a, "response_length")).set("truncated", bool(a, "response_truncated") || undefined); diff --git a/packages/coding.sh/src/server/forward.ts b/packages/coding.sh/src/server/forward.ts index ce13e3ed..3f8bf73b 100644 --- a/packages/coding.sh/src/server/forward.ts +++ b/packages/coding.sh/src/server/forward.ts @@ -5,7 +5,7 @@ // // - Spans received over OTLP keep their own trace and span ids (they line up // with the app's other spans in the backend). Native /v1/ingest batches are -// converted with toOtlpRequest (invoke_agent / execute_tool / chat spans). +// converted with toOtlpRequest (invoke_agent / execute_tool / chat / weights.edit spans). // - Every resource gets `coding_sh.project`. // - Metadata mode is an allowlist (see keepInMetadata); full mode keeps every // attribute and runs serve's redactor over every string. @@ -17,6 +17,7 @@ import type { Redactor } from "../core/redact.ts"; import type { OtelValue, ResourceSpans } from "../otel/model.ts"; import type { OpenSession } from "../core/types.ts"; import { otlpTracesUrl, toOtlpOpenSessionRequest, toOtlpRequest } from "../sinks/otlp.ts"; +import { WEIGHTS_EDIT_CONTENT_ATTRS } from "../core/weights.ts"; import { DEFAULT_TIMEOUT_MS, parseRetryAfter, redactUrl } from "../sinks/post.ts"; import type { ForwardConfig } from "./config.ts"; import type { Logger } from "./log.ts"; @@ -102,7 +103,8 @@ const KEEP_EXACT = new Set([ "error.type", ]); const KEEP_PREFIX = ["gen_ai.usage.", "service.", "telemetry.", "coding_sh."]; -const DROP_EXACT = new Set(["coding_sh.session.title", "coding_sh.repo.root"]); +// Content under coding_sh.*: the session title, the repo root, and what names a weight edit's tensor and checkpoints. +const DROP_EXACT = new Set(["coding_sh.session.title", "coding_sh.repo.root", ...WEIGHTS_EDIT_CONTENT_ATTRS]); const isScalarNonString = (v: AnyValue): boolean => "boolValue" in v || "intValue" in v || "doubleValue" in v; diff --git a/packages/coding.sh/src/sinks/otlp.ts b/packages/coding.sh/src/sinks/otlp.ts index e7613415..933b40dc 100644 --- a/packages/coding.sh/src/sinks/otlp.ts +++ b/packages/coding.sh/src/sinks/otlp.ts @@ -7,6 +7,7 @@ // see writeOpenSessions) // tool.start/tool.end → span "execute_tool {tool}" (child, emitted with tool.end) // assistant.message → span "chat {model}" (child, carries token usage) +// weights.edit → span "weights.edit" (child, one per edit: coding_sh.weights.*) // Token usage follows the GenAI gen_ai.usage.* attributes: input_tokens counts the // whole prompt (uncached + cache reads + cache writes), and the cache parts go to // cache_read.input_tokens and cache_write.input_tokens (renamed from @@ -19,6 +20,7 @@ import type { AgentEvent, Session, Usage } from "../schema/index.ts"; import type { BlobStore, OpenSession, Sink, SinkAck, SinkConfig } from "../core/types.ts"; import { hashParts } from "../core/ids.ts"; import { sessionToMetadata, toMetadata } from "../core/redact.ts"; +import { isWeightsEdit, WEIGHTS_EDIT_KIND, weightsEditAttrs } from "../core/weights.ts"; import { postJson } from "./post.ts"; export const SEMCONV_SCHEMA_URL = "https://opentelemetry.io/schemas/1.37.0"; @@ -214,6 +216,18 @@ export function toOtlpSpans(batch: AgentEvent[], sessions: Session[], mode: "ful : {}), }), }); + } else if (isWeightsEdit(e)) { + // Metadata mode has already dropped the tensor's name, its shape, both hashes and the note (toMetadata above). + spans.push({ + traceId, + spanId: hashParts("otel-span", "weights", e.id).slice(0, 16), + parentSpanId, + name: WEIGHTS_EDIT_KIND, + kind: SPAN_KIND_INTERNAL, + startTimeUnixNano: nanos(e.ts), + endTimeUnixNano: nanos(e.ts + (e.durationMs ?? 0)), + attributes: attrs({ "gen_ai.agent.name": e.agent, "coding_sh.event.id": e.id, ...weightsEditAttrs(e.payload) }), + }); } else if (e.kind === "session.end") { const s = sessionsById.get(e.sessionId); if (!s) continue; @@ -250,6 +264,8 @@ function sessionSpan(s: Session, endMs: number, full: boolean, traceId: string): "coding_sh.parent.session.id": s.parent?.sessionId, "coding_sh.stats.tests_passed": s.stats.testsPassed, "coding_sh.stats.tests_failed": s.stats.testsFailed, + "coding_sh.stats.weight_edits": s.stats.weightEdits?.events, + "coding_sh.stats.weights_changed": s.stats.weightEdits?.weightsChanged, "vcs.ref.head.revision": s.repo?.head || undefined, ...(full ? { "coding_sh.session.title": s.title, "coding_sh.repo.root": s.repo?.root } : {}), }), diff --git a/packages/coding.sh/test/otel/weights.test.ts b/packages/coding.sh/test/otel/weights.test.ts new file mode 100644 index 00000000..c3eb1d10 --- /dev/null +++ b/packages/coding.sh/test/otel/weights.test.ts @@ -0,0 +1,90 @@ +// A weight edit over OTLP: a span or a log record named `weights.edit` with coding_sh.weights.* +// attributes becomes the notification the SDKs' weights_edit helpers send, so the session gets +// the same stat, tag and line whichever way the edit arrived. +import assert from "node:assert/strict"; +import { describe, it } from "node:test"; +import type { AgentEvent } from "../../src/schema/index.ts"; +import { buildSession } from "../../src/core/sessions.ts"; +import { isWeightsEdit, weightsEditText } from "../../src/core/weights.ts"; +import { decodeLogsJson, decodeTraceJson } from "../../src/otel/json.ts"; +import { type MapContext, mapLogs, mapTraces, TraceSessionMap } from "../../src/otel/map.ts"; +import { iAttr, otlpLogs, otlpSpan, otlpTraces, sAttr } from "./_encode.ts"; +import { MemBlobs } from "./_memstore.ts"; + +type J = Record; +const A = "a".repeat(64); +const B = "b".repeat(64); +const TRACE = "5b8efff798038103d269b633813fc60c"; +const dAttr = (key: string, v: number): J => ({ key, value: { doubleValue: v } }); +const shapeAttr = (dims: number[]): J => ({ key: "coding_sh.weights.shape", value: { arrayValue: { values: dims.map((d) => ({ intValue: String(d) })) } } }); +const FULL: J[] = [ + sAttr("coding_sh.weights.tensor", "layers.3.mlp.down_proj.weight"), + shapeAttr([4096, 11008]), + iAttr("coding_sh.weights.changed", 1204), + dAttr("coding_sh.weights.l2", 0.0183), + dAttr("coding_sh.weights.max_abs", 0.004), + sAttr("coding_sh.weights.base_sha256", A), + sAttr("coding_sh.weights.edited_sha256", B), + sAttr("coding_sh.weights.note", "rank-1 edit"), +]; +const PAYLOAD = { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4096, 11008], changed: 1204, l2: 0.0183, maxAbs: 0.004, baseSha256: A, editedSha256: B, note: "rank-1 edit" }; +const LINE = "layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"; + +const ctx = (): MapContext => { + const blobs = new MemBlobs(); + return { maxInlineBytes: 4096, blob: (d) => blobs.put(d), traceSessions: new TraceSessionMap() }; +}; +const spans = async (...list: J[]): Promise => (await mapTraces(decodeTraceJson(otlpTraces("trainer", list)), ctx())).events; +const edits = (events: AgentEvent[]) => events.filter(isWeightsEdit); + +describe("a weights.edit span", () => { + it("under a parent span becomes the weight edit notification, and nothing else", async () => { + const events = await spans( + otlpSpan(TRACE, "1111111111111111", undefined, "edit run", [], 0, 5000), + otlpSpan(TRACE, "2222222222222222", "1111111111111111", "weights.edit", FULL, 1000, 1200), + ); + const own = events.filter((e) => e.spanId === "2222222222222222"); + assert.deepEqual(own.map((e) => e.kind), ["notification"]); + assert.deepEqual(own[0]!.payload, PAYLOAD); + assert.equal(own[0]!.durationMs, 200); + assert.equal(weightsEditText(own[0]!.payload), LINE); + const s = buildSession(events); + assert.deepEqual(s.stats.weightEdits, { events: 1, tensors: 1, weightsChanged: 1204 }); + assert.ok(s.flags.includes("weights:edited")); + }); + + it("as the root span of its trace still records the edit", async () => { + const events = await spans(otlpSpan(TRACE, "1111111111111111", undefined, "weights.edit", FULL, 0, 200)); + assert.deepEqual(edits(events).map((e) => e.payload), [PAYLOAD]); + assert.ok(events.some((e) => e.kind === "session.start"), "the root span still opens the session"); + assert.deepEqual(buildSession(events).stats.weightEdits, { events: 1, tensors: 1, weightsChanged: 1204 }); + }); + + it("keeps a scalar's empty shape and a zero, reads numbers sent as strings, and drops what is malformed", async () => { + const one = async (attrs: J[]) => edits(await spans(otlpSpan(TRACE, "1111111111111111", undefined, "run", [], 0, 9), otlpSpan(TRACE, "3333333333333333", "1111111111111111", "weights.edit", attrs, 1, 2)))[0]!.payload; + assert.deepEqual(await one([sAttr("coding_sh.weights.tensor", "bias"), shapeAttr([]), iAttr("coding_sh.weights.changed", 0), iAttr("coding_sh.weights.l2", 0)]), { kind: "weights.edit", tensor: "bias", shape: [], changed: 0, l2: 0 }); + assert.deepEqual(await one([sAttr("coding_sh.weights.shape", "[2, 3]"), sAttr("coding_sh.weights.changed", "6")]), { kind: "weights.edit", shape: [2, 3], changed: 6 }); + assert.deepEqual(await one([sAttr("coding_sh.weights.shape", "wide"), iAttr("coding_sh.weights.changed", -4), dAttr("coding_sh.weights.l2", -1)]), { kind: "weights.edit" }); + assert.deepEqual(await one([]), { kind: "weights.edit" }, "an edit with no attributes still counts as an edit"); + }); + + it("a span with another name is not a weight edit, whatever its attributes", async () => { + const events = await spans(otlpSpan(TRACE, "1111111111111111", undefined, "run", [], 0, 9), otlpSpan(TRACE, "4444444444444444", "1111111111111111", "weights.load", FULL, 1, 2)); + assert.deepEqual(edits(events), []); + assert.equal(Object.hasOwn(buildSession(events).stats, "weightEdits"), false); + }); +}); + +describe("a weights.edit log record", () => { + it("becomes the same notification", async () => { + const events = (await mapLogs(decodeLogsJson(otlpLogs("trainer", "weights.edit", [sAttr("session.id", "run-7"), ...FULL])), ctx())).events; + assert.deepEqual(events.map((e) => [e.kind, e.sessionId]), [["notification", "run-7"]]); + assert.deepEqual(events[0]!.payload, PAYLOAD); + }); + + it("named by its event.name attribute too", async () => { + const req = otlpLogs("trainer", "edited one tensor", [sAttr("event.name", "weights.edit"), iAttr("coding_sh.weights.changed", 12)]); + const events = (await mapLogs(decodeLogsJson(req), ctx())).events; + assert.deepEqual(events.map((e) => e.payload), [{ kind: "weights.edit", changed: 12 }]); + }); +}); diff --git a/packages/coding.sh/test/server/forward.test.ts b/packages/coding.sh/test/server/forward.test.ts index fa2f9aa4..9c9b898e 100644 --- a/packages/coding.sh/test/server/forward.test.ts +++ b/packages/coding.sh/test/server/forward.test.ts @@ -231,6 +231,14 @@ describe("serve forward: metadata allowlist and full-mode redaction", () => { ["coding_sh.run_id", str, true], ["coding_sh.session.title", str, false], ["coding_sh.repo.root", str, false], + ["coding_sh.weights.changed", { intValue: "1204" }, true], + ["coding_sh.weights.l2", { doubleValue: 0.0183 }, true], + ["coding_sh.weights.max_abs", { doubleValue: 0.004 }, true], + ["coding_sh.weights.tensor", str, false], + ["coding_sh.weights.shape", { arrayValue: { values: [{ intValue: "4096" }, { intValue: "11008" }] } }, false], + ["coding_sh.weights.base_sha256", str, false], + ["coding_sh.weights.edited_sha256", str, false], + ["coding_sh.weights.note", str, false], ["gen_ai.input.messages", str, false], ["gen_ai.output.messages", str, false], ["gen_ai.system_instructions", str, false], diff --git a/packages/coding.sh/test/sinks/otlp.test.ts b/packages/coding.sh/test/sinks/otlp.test.ts index d82dec65..a3b9b0dc 100644 --- a/packages/coding.sh/test/sinks/otlp.test.ts +++ b/packages/coding.sh/test/sinks/otlp.test.ts @@ -7,7 +7,7 @@ import { describe, it } from "node:test"; import type { AgentEvent, Session, Usage } from "../../src/schema/index.ts"; import { type KeyValue, OtlpSink, toOtlpRequest, toOtlpSpans } from "../../src/sinks/otlp.ts"; import { decodeTraceJson } from "../../src/otel/json.ts"; -import { mapTraces, TraceSessionMap } from "../../src/otel/map.ts"; +import { type MapContext, mapTraces, TraceSessionMap } from "../../src/otel/map.ts"; import { startReceiver } from "../../src/otel/receiver.ts"; import { MemBlobs, MemIndex } from "../otel/_memstore.ts"; @@ -153,3 +153,72 @@ describe("otlp sink → OTLP receiver round-trip", () => { assert.equal(new Set(spans.map((s) => s.spanId)).size, 2); }); }); + +describe("otlp sink: weight edits", () => { + const A = "a".repeat(64); + const B = "b".repeat(64); + const payload = { kind: "weights.edit", tensor: "layers.3.mlp.down_proj.weight", shape: [4096, 11008], changed: 1204, l2: 0.0183, maxAbs: 0.004, baseSha256: A, editedSha256: B, note: "rank-1 edit" }; + const edit = (id: string, p: Record = payload): AgentEvent => ({ id, v: 0, sessionId: "otlp-sink-s1", ts: T0 + 2_000, agent: "sdk", kind: "notification", payload: p }); + const weights = (a: Record) => Object.fromEntries(Object.entries(a).filter(([k]) => k.startsWith("coding_sh.weights."))); + const ctx = (): MapContext => ({ maxInlineBytes: 4096, blob: (d) => new MemBlobs().put(d), traceSessions: new TraceSessionMap() }); + + it("sends one weights.edit span per edit, under the session span", () => { + const spans = toOtlpSpans([edit("w1"), edit("w2", { ...payload, tensor: "layers.4.mlp.down_proj.weight" })], [], "full"); + assert.deepEqual(spans.map((s) => s.name), ["weights.edit", "weights.edit"]); + assert.notEqual(spans[0]!.spanId, spans[1]!.spanId); + assert.equal(spans[0]!.parentSpanId, toOtlpSpans([{ ...edit("end"), kind: "session.end", payload: {} }], [session()], "full")[0]!.spanId); + assert.deepEqual(weights(attrsOf(spans[0]!.attributes)), { + "coding_sh.weights.tensor": "layers.3.mlp.down_proj.weight", + "coding_sh.weights.shape": { arrayValue: { values: [{ intValue: "4096" }, { intValue: "11008" }] } }, + "coding_sh.weights.changed": 1204, + "coding_sh.weights.l2": 0.0183, + "coding_sh.weights.max_abs": 0.004, + "coding_sh.weights.base_sha256": A, + "coding_sh.weights.edited_sha256": B, + "coding_sh.weights.note": "rank-1 edit", + }); + }); + + it("metadata mode keeps the counts and the norms, and no name, shape, hash or note", () => { + const spans = toOtlpSpans([edit("w1")], [], "metadata"); + assert.equal(spans.length, 1); + const a = attrsOf(spans[0]!.attributes); + assert.deepEqual(weights(a), { "coding_sh.weights.changed": 1204, "coding_sh.weights.l2": 0.0183, "coding_sh.weights.max_abs": 0.004 }); + assert.equal(spans[0]!.attributes.some((kv) => kv.key === "coding_sh.weights.shape"), false); + }); + + it("another notification is not exported", () => { + assert.deepEqual(toOtlpSpans([edit("n1", { kind: "hook.context", text: "x" }), edit("n2", { message: "weights.edit" })], [], "full"), []); + }); + + it("the session span carries the edit and weight counts only when there are edits", () => { + const end: AgentEvent = { ...edit("end"), kind: "session.end", payload: {} }; + const s = session(); + s.stats.weightEdits = { events: 2, tensors: 2, weightsChanged: 66_560 }; + s.flags = ["weights:edited"]; + for (const mode of ["full", "metadata"] as const) { + const a = attrsOf(toOtlpSpans([end], [s], mode)[0]!.attributes); + assert.equal(a["coding_sh.stats.weight_edits"], 2, mode); + assert.equal(a["coding_sh.stats.weights_changed"], 66_560, mode); + } + const none = attrsOf(toOtlpSpans([end], [session()], "full")[0]!.attributes); + assert.equal(Object.hasOwn(none, "coding_sh.stats.weight_edits") || Object.hasOwn(none, "coding_sh.stats.weights_changed"), false); + }); + + it("the receiver reads an exported edit back as the same payload", async () => { + const cases: Record[] = [ + payload, + { kind: "weights.edit", tensor: "bias", shape: [], changed: 0, l2: 0, maxAbs: 0 }, + { kind: "weights.edit", tensor: "w", shape: [2, 3], changed: 6, l2: 2, maxAbs: 1 }, + { kind: "weights.edit", changed: 1204, l2: 0.0183, maxAbs: 0.004 }, + { kind: "weights.edit" }, + ]; + for (const [i, p] of cases.entries()) { + const req = JSON.parse(JSON.stringify(toOtlpRequest([edit(`rt${i}`, p)], [], "full"))); + const { events } = await mapTraces(decodeTraceJson(req), ctx()); + const back = events.filter((e) => e.kind === "notification"); + assert.equal(back.length, 1, JSON.stringify(p)); + assert.deepEqual(back[0]!.payload, p); + } + }); +}); From e8e75c4d9ead50a0536743b3e58af1f13fa00803 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:24:22 -0600 Subject: [PATCH 05/17] fix(sessions): a weight edit with no readable field keeps its line, and an impossible shape or count is not printed as a total MCP's session_get dropped an edit whose payload had no well-formed field. The line no longer says "3 of 0 weights" when the count does not fit in the shape, reads no shape of more than 16 dimensions (the SDK's limit), and never rounds a norm to Infinity. The docs now say which edits serve's forward sends. --- CHANGELOG.md | 2 +- docs/reference/api.md | 4 ++-- packages/coding.sh/src/core/weights.ts | 16 ++++++++++++---- packages/coding.sh/src/mcp/data.ts | 3 ++- packages/coding.sh/src/viewer/model.ts | 2 +- packages/coding.sh/src/viewer/ui/index.html | 9 +++++---- packages/coding.sh/test/core/weights.test.ts | 7 +++++++ packages/coding.sh/test/mcp/weights.test.ts | 10 ++++++---- .../coding.sh/test/viewer/weights-row.test.ts | 7 +++++++ 9 files changed, 43 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 252a5895..ae8c7170 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ grouped by area, with links to the docs for the details. - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) - **Sessions.** A session that recorded weight edits gets `stats.weightEdits` (edits, distinct tensors, weights changed) and the `weights:edited` tag, and is listed even when it holds nothing else. `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit instead of a blank notification (MCP dropped them), and `coding.sh sessions --where 'weights.changed>1000'` filters by size. Stored sessions are folded again. [Docs](https://coding.sh/docs/api) -- **OpenTelemetry.** A span or a log record named `weights.edit` with `coding_sh.weights.*` attributes is read as a weight edit, and the `otlp` sink and `[serve.forward]` send one `weights.edit` span per edit, so an edit reads the same whether it arrived from an SDK or over OTLP. The `invoke_agent` span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. Metadata mode keeps the counts and the norms only. [Docs](https://coding.sh/docs/api) +- **OpenTelemetry.** A span or a log record named `weights.edit` with `coding_sh.weights.*` attributes is read as a weight edit, and the `otlp` sink sends one `weights.edit` span per edit (`[serve.forward]` does too for edits posted to `/v1/ingest`), so an edit reads the same whether it arrived from an SDK or over OTLP. The `invoke_agent` span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. Metadata mode keeps the counts and the norms only. [Docs](https://coding.sh/docs/api) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/reference/api.md b/docs/reference/api.md index 185f10c2..1a0770da 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -729,10 +729,10 @@ One more `notification` kind records weight edits. The Python SDK's `coding_sh.w | `stats.weightEdits` (`events`, `tensors`, `weightsChanged`) | Weight edits the session recorded: how many, how many distinct tensor names, and their `changed` counts added up (a weight two edits changed counts twice). Absent when the session recorded none. `tensors` is absent when no edit kept its name | - A session with a weight edit gets the `weights:edited` tag. `coding.sh sessions --flag weights:edited` lists them, and `--where 'weights.changed>1000'` filters by size. -- `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. +- `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. A shape of more than 16 dimensions is not shown, and the total is left out when the count does not fit in the shape. An edit with no readable field still gets a line. - Metadata mode: the SDK leaves out `note`. A server with `capture.mode = "metadata"` also drops `tensor`, `shape` and both hashes, and keeps `changed`, `l2` and `maxAbs`. - OpenTelemetry in: a span or a log record named `weights.edit` sent to `/v1/traces` or `/v1/logs` is read as the same event. Its fields are the attributes `coding_sh.weights.tensor`, `.shape` (an array of integers), `.changed`, `.l2`, `.max_abs`, `.base_sha256`, `.edited_sha256` and `.note`. A malformed attribute is left out and the edit still counts. -- OpenTelemetry out: the `otlp` sink and `[serve.forward]` send one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. +- OpenTelemetry out: the `otlp` sink sends one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. `[serve.forward]` sends the same span for an edit posted to `/v1/ingest` and passes on a `weights.edit` span posted to `/v1/traces` as it came; an edit that arrived as a log record is not forwarded. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. - The caller computes the hashes and norms. The SDK checks the values, drops an invalid edit with one warning, and never raises into the app. Numpy and torch scalars count as numbers. - `weights_edit()` returns `True` only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. diff --git a/packages/coding.sh/src/core/weights.ts b/packages/coding.sh/src/core/weights.ts index 615ffee5..3cfe122f 100644 --- a/packages/coding.sh/src/core/weights.ts +++ b/packages/coding.sh/src/core/weights.ts @@ -9,6 +9,8 @@ export const WEIGHTS_EDIT_KIND = "weights.edit"; export const WEIGHTS_EDITED = "weights:edited"; /** Distinct tensor names kept per session; `tensors` stops counting there. */ const TENSORS_MAX = 10_000; +/** Dimensions a shape may have (what the SDKs accept); a longer one is not read as a shape. */ +const SHAPE_MAX = 16; /** What the fold keeps. Serializable (node:v8). */ export interface WeightEditState { @@ -44,7 +46,7 @@ export function weightsEditOf(p: Record): WeightsEdit { if (v !== undefined) out[k] = v; }; set("tensor", text(p.tensor)); - if (Array.isArray(p.shape) && p.shape.every((d) => count(d) !== undefined)) out.shape = p.shape as number[]; + if (Array.isArray(p.shape) && p.shape.length <= SHAPE_MAX && p.shape.every((d) => count(d) !== undefined)) out.shape = p.shape as number[]; set("changed", count(p.changed)); set("l2", norm(p.l2)); set("maxAbs", norm(p.maxAbs)); @@ -97,7 +99,11 @@ export function weightEditsOf(s: WeightEditState | undefined): WeightEditStats | export const weightEditFlags = (s: WeightEditState | undefined): string[] => (s?.events ? [WEIGHTS_EDITED] : []); const int = (n: number): string => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ","); -const real = (n: number): string => String(Number(n.toPrecision(4))); +/** Four significant digits; a number that rounds past the largest double is written as it is. */ +const real = (n: number): string => { + const r = Number(n.toPrecision(4)); + return String(Number.isFinite(r) ? r : n); +}; const short = (sha: string): string => sha.slice(0, 8); /** @@ -112,8 +118,10 @@ export function weightsEditText(p: Record): string { const dims = w.shape?.length ? `[${w.shape.join("×")}]` : ""; if (w.tensor || dims) parts.push([w.tensor, dims].filter(Boolean).join(" ")); if (w.changed !== undefined) { - const total = w.shape?.length ? w.shape.reduce((a, b) => a * b, 1) : undefined; - parts.push(`${int(w.changed)}${total !== undefined && Number.isSafeInteger(total) ? ` of ${int(total)}` : ""} ${w.changed === 1 && total === undefined ? "weight" : "weights"}`); + // The tensor's size, when the shape gives one the count fits in. + const size = w.shape?.length ? w.shape.reduce((a, b) => a * b, 1) : undefined; + const total = size !== undefined && Number.isSafeInteger(size) && w.changed <= size ? size : undefined; + parts.push(`${int(w.changed)}${total !== undefined ? ` of ${int(total)}` : ""} ${w.changed === 1 && total === undefined ? "weight" : "weights"}`); } if (w.l2 !== undefined) parts.push(`L2 ${real(w.l2)}`); if (w.maxAbs !== undefined) parts.push(`max ${real(w.maxAbs)}`); diff --git a/packages/coding.sh/src/mcp/data.ts b/packages/coding.sh/src/mcp/data.ts index 1dac8ec5..743d5a58 100644 --- a/packages/coding.sh/src/mcp/data.ts +++ b/packages/coding.sh/src/mcp/data.ts @@ -332,7 +332,8 @@ export function timeline(events: AgentEvent[], startedAt: number): TimelineEntry push("compact", `context compacted (${str(p.trigger) ?? "auto"})`); break; case "notification": - if (p.kind === WEIGHTS_EDIT_KIND) push("weights", clip(weightsEditText(p), 240)); + // An edit with no readable field still gets its line: push() leaves out an empty one. + if (p.kind === WEIGHTS_EDIT_KIND) push("weights", clip(weightsEditText(p), 240) || "no details recorded"); else push("notification", clip(str(p.message) ?? str(p.text), 160)); break; default: diff --git a/packages/coding.sh/src/viewer/model.ts b/packages/coding.sh/src/viewer/model.ts index 5bd80fdd..c92563cb 100644 --- a/packages/coding.sh/src/viewer/model.ts +++ b/packages/coding.sh/src/viewer/model.ts @@ -312,7 +312,7 @@ export function rowLabel(e: AgentEvent): string { case "span": return text(p.name); case "notification": - return isWeightsEdit(e) ? cut(weightsEditText(p), 200) : e.kind; + return isWeightsEdit(e) ? cut(weightsEditText(p), 200) || "weights.edit" : e.kind; default: return e.kind; } diff --git a/packages/coding.sh/src/viewer/ui/index.html b/packages/coding.sh/src/viewer/ui/index.html index a8173d9d..58366526 100644 --- a/packages/coding.sh/src/viewer/ui/index.html +++ b/packages/coding.sh/src/viewer/ui/index.html @@ -3787,14 +3787,15 @@ const norm = (v) => typeof v === "number" && Number.isFinite(v) && v >= 0; const text = (v) => (typeof v === "string" && v ? v : ""); const int = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ","); - const real = (n) => String(Number(n.toPrecision(4))); - const shape = Array.isArray(p.shape) && p.shape.every(count) ? p.shape : undefined; + const real = (n) => { const r = Number(n.toPrecision(4)); return String(Number.isFinite(r) ? r : n); }; + const shape = Array.isArray(p.shape) && p.shape.length <= 16 && p.shape.every(count) ? p.shape : undefined; const parts = []; const dims = shape && shape.length ? "[" + shape.join("×") + "]" : ""; if (text(p.tensor) || dims) parts.push([text(p.tensor), dims].filter(Boolean).join(" ")); if (count(p.changed)) { - const total = shape && shape.length ? shape.reduce((a, b) => a * b, 1) : undefined; - parts.push(int(p.changed) + (total !== undefined && Number.isSafeInteger(total) ? " of " + int(total) : "") + (p.changed === 1 && total === undefined ? " weight" : " weights")); + const size = shape && shape.length ? shape.reduce((a, b) => a * b, 1) : undefined; + const total = size !== undefined && Number.isSafeInteger(size) && p.changed <= size ? size : undefined; + parts.push(int(p.changed) + (total !== undefined ? " of " + int(total) : "") + (p.changed === 1 && total === undefined ? " weight" : " weights")); } if (norm(p.l2)) parts.push("L2 " + real(p.l2)); if (norm(p.maxAbs)) parts.push("max " + real(p.maxAbs)); diff --git a/packages/coding.sh/test/core/weights.test.ts b/packages/coding.sh/test/core/weights.test.ts index f35f56b4..b41c6be7 100644 --- a/packages/coding.sh/test/core/weights.test.ts +++ b/packages/coding.sh/test/core/weights.test.ts @@ -67,6 +67,13 @@ describe("weightsEditText", () => { assert.equal(weightsEditText({ tensor: "w", shape: [2, "x"], changed: 2 }), "w · 2 weights", "a shape that is not all counts is not shown"); }); + it("says `of` the tensor's size only when the count fits in it, and reads no shape of more than 16 dimensions", () => { + assert.equal(weightsEditText({ tensor: "w", shape: [0, 5], changed: 3 }), "w [0×5] · 3 weights"); + assert.equal(weightsEditText({ tensor: "w", shape: [2, 3], changed: 7 }), "w [2×3] · 7 weights"); + assert.equal(weightsEditText({ tensor: "w", shape: Array.from({ length: 17 }, () => 2), changed: 4 }), "w · 4 weights"); + assert.equal(weightsEditText({ l2: Number.MAX_VALUE }), `L2 ${Number.MAX_VALUE}`, "never Infinity"); + }); + it("keeps four significant digits of a norm", () => { assert.equal(weightsEditText({ l2: 0.018300000000000004, maxAbs: 1234.5678 }), "L2 0.0183 · max 1235"); assert.equal(weightsEditText({ l2: 1e-9 }), "L2 1e-9"); diff --git a/packages/coding.sh/test/mcp/weights.test.ts b/packages/coding.sh/test/mcp/weights.test.ts index 16a1ce65..cbfc4cc6 100644 --- a/packages/coding.sh/test/mcp/weights.test.ts +++ b/packages/coding.sh/test/mcp/weights.test.ts @@ -26,6 +26,7 @@ before(async () => { { ...base, id: "w-0", ts: t0, kind: "session.start", payload: { name: "edit layer 15" } }, { ...base, id: "w-1", ts: t0 + 100, kind: "notification", payload: edit("model.layers.15.mlp.down_proj.weight", 65536, A, B) }, { ...base, id: "w-2", ts: t0 + 200, kind: "notification", payload: edit("model.layers.16.mlp.down_proj.weight", 1024, B, C) }, + { ...base, id: "w-2b", ts: t0 + 250, kind: "notification", payload: { kind: "weights.edit" } }, { ...base, id: "w-3", ts: t0 + 300, kind: "session.end", payload: {} }, ]; await ctx.index.putEvents(evs); @@ -45,7 +46,7 @@ describe("weight edits over MCP", () => { const r = await c.call("sessions_list", { flag: "weights:edited" }); const sessions = r.structuredContent!.sessions as { id: string; flags?: string[]; weightEdits?: unknown }[]; assert.deepEqual(sessions.map((s) => s.id), [SID]); - assert.deepEqual(sessions[0]!.weightEdits, { events: 2, tensors: 2, weightsChanged: 66560 }); + assert.deepEqual(sessions[0]!.weightEdits, { events: 3, tensors: 2, weightsChanged: 66560 }); assert.ok(sessions[0]!.flags!.includes("weights:edited")); const where = await c.call("sessions_list", { where: "weights.changed>1000" }); assert.deepEqual((where.structuredContent!.sessions as { id: string }[]).map((s) => s.id), [SID]); @@ -57,21 +58,22 @@ describe("weight edits over MCP", () => { it("session_get has one timeline line per edit", async () => { const r = await c.call("session_get", { sessionId: SID }); const d = r.structuredContent as { session: { weightEdits?: unknown }; timeline: { kind: string; text: string }[]; timelineTotal: number }; - assert.deepEqual(d.session.weightEdits, { events: 2, tensors: 2, weightsChanged: 66560 }); + assert.deepEqual(d.session.weightEdits, { events: 3, tensors: 2, weightsChanged: 66560 }); assert.deepEqual( d.timeline.map((t) => [t.kind, t.text]), [ ["weights", "model.layers.15.mlp.down_proj.weight [4096×11008] · 65,536 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"], ["weights", "model.layers.16.mlp.down_proj.weight [4096×11008] · 1,024 of 45,088,768 weights · L2 0.0183 · max 0.004 · bbbbbbbb → cccccccc"], + ["weights", "no details recorded"], ], ); - assert.equal(d.timelineTotal, 2); + assert.equal(d.timelineTotal, 3, "an edit with no readable field keeps its line"); }); it("the session resource has the weight edits row and both lines", async () => { const r = await c.request("resources/read", { uri: `codingsh://session/${SID}` }); const text = (r.result!.contents as { text: string }[])[0]!.text; - assert.match(text, /Weight edits.*2 edits, 2 tensors, 66,560 weights changed/); + assert.match(text, /Weight edits.*3 edits, 2 tensors, 66,560 weights changed/); assert.match(text, /model\.layers\.15\.mlp\.down_proj\.weight \[4096×11008\]/); assert.match(text, /model\.layers\.16\.mlp\.down_proj\.weight \[4096×11008\]/); }); diff --git a/packages/coding.sh/test/viewer/weights-row.test.ts b/packages/coding.sh/test/viewer/weights-row.test.ts index 362c12f2..efc2a37c 100644 --- a/packages/coding.sh/test/viewer/weights-row.test.ts +++ b/packages/coding.sh/test/viewer/weights-row.test.ts @@ -34,6 +34,12 @@ test("the page and the CLI write the same line for a weight edit", () => { { tensor: "w", shape: [1e9, 1e9], changed: 5 }, { l2: 0.018300000000000004, maxAbs: 1234.5678 }, { l2: 1e-9, changed: 1234567 }, + { l2: Number.MAX_VALUE, maxAbs: Number.MAX_VALUE }, + { tensor: "w", shape: [0, 5], changed: 3 }, + { tensor: "w", shape: [2, 3], changed: 7 }, + { tensor: "w", shape: [1], changed: 1 }, + { tensor: "w", shape: Array.from({ length: 16 }, () => 1), changed: 1 }, + { tensor: "w", shape: Array.from({ length: 17 }, () => 1), changed: 1 }, ]; for (const p of payloads) assert.equal(page(p), weightsEditText(p), JSON.stringify(p)); assert.equal(page(FULL), "layers.3.mlp.down_proj.weight [4096×11008] · 1,204 of 45,088,768 weights · L2 0.0183 · max 0.004 · aaaaaaaa → bbbbbbbb"); @@ -44,6 +50,7 @@ const ev = (i: number, kind: AgentEvent["kind"], payload: Record { assert.equal(rowLabel(ev(0, "notification", FULL)), weightsEditText(FULL)); + assert.equal(rowLabel(ev(2, "notification", { kind: "weights.edit" })), "weights.edit", "an edit with no readable field still has a label"); assert.equal(rowLabel(ev(1, "notification", { kind: "hook.context", text: "x" })), "notification"); }); From 0636b9c14247c857308cf2fa56b1bcb942b9042f Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:34:06 -0600 Subject: [PATCH 06/17] fix(mcp): session_get's description names weight edits without growing the tool surface The words added for weight edits are paid for by a shorter opening and a shorter note on the turn argument, so the description budget still holds when other tools add words. --- packages/coding.sh/src/mcp/tools.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/coding.sh/src/mcp/tools.ts b/packages/coding.sh/src/mcp/tools.ts index 6fba3609..fb16a05a 100644 --- a/packages/coding.sh/src/mcp/tools.ts +++ b/packages/coding.sh/src/mcp/tools.ts @@ -407,7 +407,7 @@ function toolDefs(): ToolDef[] { name: "session_get", title: "Get a session", description: - "One session as a digest: summary (with waste: repeated reads and prompt-cache breaks), skills used, per-turn index, and a timeline of prompts, replies, tool calls (with failures and durations), model calls, file changes, tests and weight edits. Pass turn to zoom into one turn; page with cursor.", + "A session digest: summary (with waste: repeated reads and prompt-cache breaks), skills used, per-turn index, and a timeline of prompts, replies, tool calls (with failures and durations), model calls, file changes, tests and weight edits. Pass turn to zoom in; page with cursor.", inputSchema: input( { sessionId: SESSION_ID, From f5da5512991ceb9c1c5d1a05b9a77088a44ff1c2 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:13:12 -0600 Subject: [PATCH 07/17] feat(evals): exp run --weights records which checkpoint a run used, and exp compare names both exp run --weights stores metadata.weights = { sha256 } (64 hex, lowercased; coding.sh hashes nothing). exp compare shows each side's stamp in the normal and the --given header, warns when both sides ran the same weights and neither regrades the other, and says so when only one side is stamped. exp show, the exp run summary, the viewer and the MCP tools experiments_list and experiment_compare carry the same facts. --arm takes one stamp per arm (--weights base= --weights edited=). --watch and matrix specs refuse the flag. --resume keeps the stored stamp and refuses a different one; the stamp is part of the job config. A regrade inherits the stamp. --- CHANGELOG.md | 1 + docs/guides/evals.md | 44 ++- docs/reference/api.md | 1 + docs/reference/cli.md | 2 + packages/coding.sh/src/cli/commands/exp.ts | 99 ++++- packages/coding.sh/src/cli/registry.ts | 7 + packages/coding.sh/src/experiments/ab.ts | 6 +- packages/coding.sh/src/experiments/compare.ts | 20 +- packages/coding.sh/src/experiments/render.ts | 15 +- packages/coding.sh/src/experiments/run.ts | 21 +- packages/coding.sh/src/experiments/weights.ts | 53 +++ packages/coding.sh/src/mcp/tools.ts | 23 +- packages/coding.sh/src/viewer/ui/index.html | 20 +- .../test/docs/guide-commands.test.ts | 15 +- .../test/experiments/weights-stamp.test.ts | 364 ++++++++++++++++++ packages/coding.sh/test/llmcache/cli.test.ts | 15 + packages/coding.sh/test/mcp/server.test.ts | 57 +++ packages/coding.sh/test/mcp/toolevals.test.ts | 4 +- packages/coding.sh/test/viewer/server.test.ts | 18 + .../coding.sh/test/viewer/ui-browser.test.ts | 49 ++- 20 files changed, 790 insertions(+), 44 deletions(-) create mode 100644 packages/coding.sh/src/experiments/weights.ts create mode 100644 packages/coding.sh/test/experiments/weights-stamp.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d0c7ba07..8e50ee9a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ grouped by area, with links to the docs for the details. - **Lint.** New rules: vendor-literal, site-voice, universal-transition, reduced-motion, global-install and personal-data. `slop.mjs --strict-allowlist` fails on allowlist entries that no longer match anything. [Contributing](https://github.com/jackulau/coding.sh/blob/main/CONTRIBUTING.md) - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) +- **Evals.** `coding.sh exp run --weights ` records which checkpoint a run's task ran (the SHA-256 of the weights file, which you compute). `exp compare`, `exp show`, the viewer and the MCP tools show each side's stamp, and a comparison warns when both sides ran the same weights or only one side recorded them. [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/guides/evals.md b/docs/guides/evals.md index 6b06a312..2427c28d 100644 --- a/docs/guides/evals.md +++ b/docs/guides/evals.md @@ -527,10 +527,18 @@ measurement"). To compare two configs graded by different grader versions, regra with the newer graders and compare that. `--json` returns the new `experiment` and the `diff` (`flips`, `toFail`, `toPass`, `toUngraded`, `fromUngraded`, `graders`). +A regrade keeps the original's [weights stamp](#compare-an-edited-model-with-its-base): the +outputs came from that checkpoint. `exp compare` of an experiment with its regrade shows the same +stamp on both sides and does not warn about it. + `coding.sh exp run --arm base=a.json --arm cand=b.json` runs two task configs as one A/B. Cases go in a random order, and each case runs in both arms back to back, in random arm order, so drift hits both arms the same. Both arms are stored (as `-base` and `-cand`) and share a pair -id. `--seed` fixes the order, `--max-cases N` runs a random N cases. +id. `--seed` fixes the order, `--max-cases N` runs a random N cases. With `--arm`, `--weights` +names the arm each stamp belongs to (`--weights base= --weights cand=`); one arm +may go without. A bare `--weights ` is a usage error there, because two arms rarely run +one checkpoint. `--replay` with its control arm is an A/B too and refuses `--weights`; with +`--no-control` it runs one task and takes the stamp. Add `--stop-early` to stop once the answer is clear. It is a group sequential test with O'Brien-Fleming alpha spending at `--looks` (default 4) evenly spaced points of the case budget, @@ -645,7 +653,8 @@ compare ` gives the per-case diff of one cell, and treats interleaved. `--cell NAME` (repeatable) runs some cells; the baseline always runs. `--seed` fixes the order, `--resume` picks up an interrupted matrix, and `--json` returns `matrix`, `plan`, `experiments` (cell to id) and `comparisons` (each with `pHolm` and `holmVerdict`). Up to 256 -cells. +cells. A matrix spec does not take `--weights`: one stamp cannot name the checkpoint of every +cell. Run each checkpoint as its own `exp run --weights `. Several matrices against one baseline pay for the baseline each time. `--reuse baseline` copies the baseline cell's finished case-trials from earlier experiments instead of running them again @@ -779,15 +788,28 @@ count such as tokens used has to become a pass or fail before it is a grader. {"kind": "json-path", "name": "correct", "path": "$.answer"} ``` -Run each model, then compare: +Run each model, then compare. `--weights` records which checkpoint each task ran: the SHA-256 of +its weights file, 64 hex characters. You compute it, for example with +`shasum -a 256 model.safetensors`; coding.sh stores the value and hashes nothing. ```sh -coding.sh exp run -d math --task base.json -g terminated.json -g correct.json --name base -coding.sh exp run -d math --task edited.json -g terminated.json -g correct.json --name edited +coding.sh exp run -d math --task base.json -g terminated.json -g correct.json --name base \ + --weights 0496e5111097627dd9de19e48925a9143c6e2566b4f29c3f39ecc908f3b424ee +coding.sh exp run -d math --task edited.json -g terminated.json -g correct.json --name edited \ + --weights 58899f90572cba67c56b85687d69714292f845c7830f9971cc1b332ee047b2a7 coding.sh exp compare base edited coding.sh exp compare base edited --given terminated ``` +Both compares name each side's checkpoint by the first 12 characters of its stamp +(`weights 0496e5111097` on A, `weights 58899f90572c` on B), and `--json` carries the whole value +as `comparison.a.weights.sha256` and `comparison.b.weights.sha256`. When both sides carry the same +stamp, the report warns that `B − A` is not a difference between two models. When only one side +carries a stamp, it says so, and the other side reads `weights not recorded`. `exp show`, the +`exp run` summary, the viewer and the MCP tools `experiments_list` and `experiment_compare` show +the same stamp. `exp run` without `--weights` stores no stamp, and a comparison of two such runs +prints nothing about weights. + The first compare prints a paired delta for `terminated` and one for `correct`, each with its bootstrap interval. If the edited model only stopped looping, `terminated` is up and `correct` is up with it. `--given terminated` keeps the cases where `terminated` passes in both runs, and @@ -876,6 +898,12 @@ rounds as a run that was never interrupted. script a command task runs) and timeouts, not concurrency. If any of them changed, `--resume` refuses and names the parts: `Can't resume exp_…: its config changed since it started (dataset, task)`. Mixing trials of two configs would give a result of neither. +- A [weights stamp](#compare-an-edited-model-with-its-base) is part of the config. `--resume ` + without `--weights` keeps the stamp the run started with, and with a different `--weights` it + refuses: the finished trials would be stamped with weights they did not run. `--resume` with no + id matches a run by its config, so pass the same `--weights` again; without it the resume + refuses and names `weights` as the part that changed. An `--arm` run keeps and checks one stamp + per arm in the same way, and its stamps are in the `arms` part of the config. - The first Ctrl-C starts no new trials and lets the ones in flight finish and be saved. The second stops them. The exit code is 130, and the command to continue is printed (and is the `resume` field of `--json` output). @@ -904,7 +932,8 @@ rounds as a run that was never interrupted. kill then leaves a partial record: `exp list` marks it `(partial)` and `exp show` says how many case runs it holds. `exp run … --resume ` continues it when no job has that id: the finished case-trials are kept and only the rest run, under the same id. It refuses, and names - what changed, when the dataset version, task, graders, trials or case selection differ. + what changed, when the dataset version, task, graders, trials, case selection or `--weights` + differ. Without `--weights` it keeps the stamp the experiment has. `--retry-errors` with `--resume ` also runs its task-errored case-trials again, on a finished experiment too. @@ -957,7 +986,8 @@ prints one line and its delta against the run before it: - When only the graders changed (a rubric edit, a grader file), the stored outputs of the last run are graded again with no task calls (`regrade`). The new experiment records `regradeOf`. - A spec that fails to parse is reported and watching goes on. Enter reruns, `q` quits, Ctrl-C - exits 0. `--watch` does not combine with `--json`, `--resume` or `--arm`. + exits 0. `--watch` does not combine with `--json`, `--resume`, `--arm` or `--weights` (a watch + session reruns as the task changes, so one stamp would not stay true). ## Metric A/B (ms, bytes) diff --git a/docs/reference/api.md b/docs/reference/api.md index 220898e7..924df2ca 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -1477,6 +1477,7 @@ These fields are present only when they apply: | `samples`, `summary.caseVerdicts` | `exp run --samples adaptive` | `"samples": "adaptive"`, and verdicts `passed`, `failed`, `tieBreaks`. The `passRate` interval is then biased, so use the verdicts | | `regression` | `--stop-on-regression` stopped the run (status `cancelled`) | `{caseId, baseline}` | | `metadata.checkpoint` | A partial record written while the run was going (status `cancelled`) | `done`, `total`, `at` | +| `metadata.weights` | `exp run --weights ` | `sha256`: the SHA-256 of the checkpoint the task ran, 64 hex characters as given to the flag, stored in lowercase. `/api/compare` carries it as `a.weights` and `b.weights`, with `weightsWarning` when both are the same or only one side has one | | `metadata.selection` | A run of part of the dataset | `from` (`{kind: "failing"\|"errors", experiment}`), `cases`, `sample`, `seed`, `firstN`, `selected`, `of` | | `metadata.seq` | `exp run --seq K` | `k`, `feedback`, `history` | | `summary.seq` | `exp run --seq K` | `k`, `feedback`, `history`, `curve` (`[{k, rate, lo, hi}]`, Seq@1..K), `egs`, `lgs`, `unchangedRate`, `costToSolve`; with 2+ trials also `passAtK` (`[{k, rate}]`), `front` (`[{protocol, k, cost, success, overall}]`) and `costUnit` (`usd` or `attempts`) | diff --git a/docs/reference/cli.md b/docs/reference/cli.md index b847b3d5..5dd25a9a 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1515,6 +1515,7 @@ coding.sh exp run [spec] [options] | `--replay ` | | Serve this recorded experiment's model calls again, up to each case-trial's first changed call; later calls run live. Adds the recorded config as a control arm (A) unless --no-control. | | `--replay-until ` | | With --replay: serve only calls 1..N from the recording; call N+1 onward runs live. | | `--series ` | | Tag the experiment as the next round of an iteration loop and print whether the loop should stop (see exp plateau) | +| `--weights ` | | Record the checkpoint the task ran: the SHA-256 of the weights file, 64 hex characters, which you compute (coding.sh hashes nothing). exp compare shows each side's and warns when both ran the same. With --arm: one per arm, as ARM=<SHA256>. Not with --watch or a matrix spec. Repeatable. | | `--watch` | | Rerun on every change to the spec, the task's script, grader files or --watch-path; a grader-only change regrades the stored outputs. Enter reruns, q quits. | | `--watch-path ` | | With --watch: more files whose change reruns the task (repeatable; also the spec's "watch" field) | | `--session` | `true` | In an experiment session (exp session new), add the run to its timeline and print the delta against the previous run; --no-session skips it. | @@ -1537,6 +1538,7 @@ coding.sh exp run matrix.json --cell model=opus,prompt=short coding.sh exp run matrix.json --reuse baseline --plan coding.sh exp run spec.json --record coding.sh exp run cand.json --replay exp_01J… --replay-until 3 +coding.sh exp run edited.json --name edited --weights 5f2c8a1e9b3d4c7f6a0e1d2b3c4f5a6e7d8c9b0a1f2e3d4c5b6a7f8e9d0c1b2a ``` ### `coding.sh exp session` diff --git a/packages/coding.sh/src/cli/commands/exp.ts b/packages/coding.sh/src/cli/commands/exp.ts index d4a4caa5..97e9d83f 100644 --- a/packages/coding.sh/src/cli/commands/exp.ts +++ b/packages/coding.sh/src/cli/commands/exp.ts @@ -44,6 +44,7 @@ import { sortableId } from "../../core/ids.ts"; import { jobConfig, taskFingerprint } from "../../jobs/config.ts"; import type { Job, JobKind } from "../../jobs/store.ts"; import { positiveInt, UsageError } from "../args.ts"; +import { parseWeights, shortWeights, weightsOf, withWeights } from "../../experiments/weights.ts"; import { finishJob, infraStopNote, openJob, resumeField, resumeNote, retryFlag, statusCommand, trialTimeout } from "../jobs.ts"; import { gateFlags, jsonArg, list, num, shellQuote, str, withContext } from "../common.ts"; import { ago, CliError, plural } from "../output.ts"; @@ -359,6 +360,8 @@ async function runSpec(c: CommandContext, store: FileEvalStore, specFile: string if (resumeFrom.status !== "cancelled" && !retryErrors) throw new CliError(`${resumeFrom.id} already finished; nothing to resume (--retry-errors runs its errored case-trials again)`, { reason: "already_complete" }); jobCtx = { ...c, flags: { ...c.flags, resume: false } }; } + // The weights stamp: the flag's, else the one the run being continued already carries. + const weights = resumeFrom ? (weightsFlag(c) ?? weightsOf(resumeFrom)) : (await resumedWeights(c, "exp run", [weightsFlag(c)]))[0]; const { spec, dataset, graders } = loaded; if (gating.adaptive && (spec.trials ?? 1) > 1) throw new UsageError("--samples adaptive sets the trials (2 per case, a 3rd when they disagree); drop trials from the spec or --trials", { reason: "invalid_argument" }); if (gating.baseline && gating.baseline.dataset.id !== dataset.id) { @@ -379,8 +382,11 @@ async function runSpec(c: CommandContext, store: FileEvalStore, specFile: string ...(gating.llm ? { llm: { mode: gating.llm.mode, source: gating.llm.source, until: gating.llm.until } } : {}), ...(selection ? { selection: selection.config } : {}), ...(seq ? { seq: { k: seq.k, feedback: str(c.flags.feedback) ?? "reasons", history: seq.history ?? "full" } } : {}), + // Undefined parts are not hashed, so a run without a stamp keeps the config hash it had. + weights, }); const job = await openJob(jobCtx, { kind: "exp run", name: resumeFrom?.name ?? loaded.name, config, newId: () => sortableId("exp_"), concurrency: spec.concurrency ?? 4 }); + await rememberWeights(job, [weights]); // A resumed job: the partial result it stored also holds case-trials finished before a crash cut the job short. const priorId = !resumeFrom && job.scope().state("experimentId"); if (priorId) { @@ -401,7 +407,7 @@ async function runSpec(c: CommandContext, store: FileEvalStore, specFile: string ...(gating.baseline ? { stopOnRegression: gating.baseline } : {}), ...(spec.concurrency ? { concurrency: spec.concurrency } : {}), ...(spec.split ? { split: spec.split } : {}), - ...(continuation || seriesFlag(c) ? { metadata: { ...(continuation ? { control: "skipped" } : {}), ...(seriesFlag(c) ? { series: seriesFlag(c) } : {}) } } : {}), + ...(continuation || seriesFlag(c) || weights ? { metadata: withWeights({ ...(continuation ? { control: "skipped" } : {}), ...(seriesFlag(c) ? { series: seriesFlag(c) } : {}) }, weights) } : {}), ...(selection ? { selection: selection.opts } : {}), ...(resumeFrom ? { resumeFrom } : {}), ...(retryErrors ? { retryErrors } : {}), @@ -415,7 +421,7 @@ async function runSpec(c: CommandContext, store: FileEvalStore, specFile: string }), ) .catch((e: unknown) => { - if (e instanceof ResumeMismatchError) throw new CliError(e.message, { reason: "config_changed", next: ["Run it with the original dataset, task, graders, trials and case selection, or start a new run without --resume"] }); + if (e instanceof ResumeMismatchError) throw new CliError(e.message, { reason: "config_changed", next: ["Run it with the original dataset, task, graders, trials, case selection and --weights (or none, to keep its stamp), or start a new run without --resume"] }); if (e instanceof RangeError) throw new UsageError(e.message, { reason: "invalid_argument" }); throw e; }) @@ -470,6 +476,76 @@ function seriesFlag(c: CommandContext): string | undefined { return v.trim(); } +const WEIGHTS_HELP = "hash the file yourself, e.g. shasum -a 256 model.safetensors"; + +function weightsValue(v: string): string { + const sha = parseWeights(v); + if (sha === undefined) { + throw new UsageError(`--weights takes the SHA-256 of the checkpoint file: 64 hex characters (got ${JSON.stringify(v.length > 80 ? `${v.slice(0, 80)}…` : v)}); ${WEIGHTS_HELP}`, { reason: "invalid_argument" }); + } + return sha; +} + +/** + * `--weights `: the checkpoint a run of one task used, as its user + * hashed it. Stored lowercase in `metadata.weights`; undefined without the flag. + */ +function weightsFlag(c: CommandContext): string | undefined { + const values = list(c.flags.weights); + if (c.flags.weights !== undefined && !values.length) throw new UsageError(`--weights takes the SHA-256 of the checkpoint file: 64 hex characters; ${WEIGHTS_HELP}`, { reason: "invalid_argument" }); + if (!values.length) return undefined; + if (values.length > 1) throw new UsageError(`--weights takes one SHA-256 for a run of one task (got ${values.length}); with two --arm options, give one per arm as --weights ARM=`, { reason: "invalid_argument" }); + const v = values[0]!; + const eq = v.indexOf("="); + if (eq > 0 && NAME.test(v.slice(0, eq))) throw new UsageError("--weights ARM= needs two --arm options; a run of one task takes --weights ", { reason: "invalid_argument" }); + return weightsValue(v); +} + +/** `--weights ARM=` with two `--arm` options: each arm with the stamp given for it, if any. */ +function armWeights(c: CommandContext, arms: readonly [AbArm, AbArm]): [AbArm, AbArm] { + const values = list(c.flags.weights); + const of = new Map(); + for (const v of values) { + const eq = v.indexOf("="); + const arm = eq > 0 ? v.slice(0, eq) : ""; + if (!NAME.test(arm)) { + throw new UsageError(`with --arm, name the arm each stamp belongs to: --weights ${arms[0].name}= --weights ${arms[1].name}=`, { reason: "invalid_argument" }); + } + if (!arms.some((a) => a.name === arm)) throw new UsageError(`--weights ${arm}=…: no arm named ${JSON.stringify(arm)} (arms: ${arms.map((a) => a.name).join(", ")})`, { reason: "invalid_argument" }); + if (of.has(arm)) throw new UsageError(`--weights names arm ${JSON.stringify(arm)} twice`, { reason: "invalid_argument" }); + of.set(arm, weightsValue(v.slice(eq + 1))); + } + return arms.map((a) => (of.has(a.name) ? { ...a, weights: of.get(a.name)! } : a)) as [AbArm, AbArm]; +} + +/** + * The stamps of the run `--resume ` names, in place of `given` when no + * `--weights` was passed: a resumed run keeps the weights its finished + * case-trials ran under. A `--weights` that differs from them is refused. + * Runs before the job's config is hashed, since the stamps are part of it. + * `labels` names the arms of an A/B; a run of one task has one unnamed stamp. + */ +async function resumedWeights(c: CommandContext, kind: JobKind, given: (string | undefined)[], labels: readonly string[] = []): Promise<(string | undefined)[]> { + const ref = str(c.flags.resume); + const v = ref ? await JobStore.forEnv(c.env).find(ref) : undefined; + if (!v || v.header.kind !== kind) return given; + const stored = Array.isArray(v.header.state.weights) ? (v.header.state.weights as unknown[]) : []; + const was = given.map((_, i) => parseWeights(stored[i])); + if (given.every((x) => x === undefined)) return was; + const i = given.findIndex((x, j) => x !== was[j]); + if (i < 0) return given; + const text = (x: string | undefined) => (x ? shortWeights(x) : "none"); + throw new CliError(`Can't resume ${v.header.id}: ${labels[i] ? `arm ${labels[i]}` : "it"} ran weights ${text(was[i])}, and --weights gives ${text(given[i])}. Its finished trials would be stamped with weights they did not run`, { + reason: "config_changed", + next: ["Resume it without --weights to keep its stamp, or start a new run without --resume"], + }); +} + +/** Remember a job's stamps (see resumedWeights). Nothing is written when there are none. */ +async function rememberWeights(job: Job, weights: (string | undefined)[]): Promise { + if (weights.some((x) => x !== undefined) && job.scope().state("weights") === undefined) await job.scope().setState("weights", weights.map((x) => x ?? null)); +} + /** A stored series: completed experiments tagged `metadata.series`, oldest first. */ async function storedSeries(store: FileEvalStore, name: string | undefined, datasetId?: string): Promise { const all = await store.listExperiments(datasetId ? { datasetId } : {}); @@ -665,7 +741,7 @@ export async function runArms( ): Promise<{ result: AbResult; job: Job }> { const { spec, dataset, graders } = loaded; const config = specConfig(c, loaded, { - arms: arms.map((a) => ({ name: a.name, task: taskFingerprint(a.task, c.cwd) })), + arms: arms.map((a) => ({ name: a.name, task: taskFingerprint(a.task, c.cwd), weights: a.weights })), seed: extra.seed, maxCases: extra.maxCases, stopEarly: extra.stopEarly, @@ -673,6 +749,7 @@ export async function runArms( const job = await openJob(c, { kind, name: loaded.name, config, newId: () => sortableId("pair_"), concurrency: spec.concurrency ?? 4 }); // The job id doubles as the pair id, so --resume takes either. if (job.header.state.pairId === undefined) await job.scope().setState("pairId", job.id); + await rememberWeights(job, arms.map((a) => a.weights)); const name = job.header.name; const label = `${dataset.name} (${arms[0].name} vs ${arms[1].name}, interleaved)`; const spin = c.out.spin(`Running ${label}…`); @@ -881,6 +958,14 @@ export async function run(c: CommandContext): Promise { if (matrix || armFlags.length || c.flags.watch === true) throw new UsageError(`${flag} runs one task as a gate; it does not combine with ${matrix ? "a matrix spec" : armFlags.length ? "--arm" : "--watch"}`, { reason: "invalid_argument" }); if (adaptive && c.flags.trials !== undefined) throw new UsageError("--samples adaptive sets the trials (2 per case, a 3rd when they disagree); drop --trials", { reason: "invalid_argument" }); } + // --weights names one checkpoint for one run: a watch reruns when files change and a matrix runs several configs, so a fixed hash would mislabel them. + if (c.flags.weights !== undefined) { + const matrix = !!c.positionals[0] && (await isMatrixFile(resolve(c.cwd, c.positionals[0]))); + if (matrix || c.flags.watch === true) { + throw new UsageError(`--weights stamps one run with one checkpoint; it does not combine with ${matrix ? "a matrix spec" : "--watch"}. Run each checkpoint as its own exp run --weights `, { reason: "invalid_argument" }); + } + if (!armFlags.length) weightsFlag(c); + } // A spec with "matrix" or "variants" runs every cell against its baseline cell. if (c.positionals[0] && (await isMatrixFile(resolve(c.cwd, c.positionals[0])))) { if (c.flags.seq !== undefined) throw new UsageError("--seq runs one task; it does not combine with a matrix spec", { reason: "invalid_argument" }); @@ -907,7 +992,9 @@ export async function run(c: CommandContext): Promise { if (armFlags.length) { if (str(c.flags.task)) throw new UsageError("Use either --task or two --arm options, not both", { reason: "invalid_argument" }); if (c.flags.watch === true) throw new UsageError("--watch reruns one task; for an A/B of harness edits on save use coding.sh exp ab --watch", { reason: "invalid_argument" }); - const arms = await parseArms(armFlags); + const parsed = armWeights(c, await parseArms(armFlags)); + const armStamps = await resumedWeights(c, "exp ab", parsed.map((a) => a.weights), parsed.map((a) => a.name)); + const arms = parsed.map(({ weights: _w, ...a }, i) => (armStamps[i] ? { ...a, weights: armStamps[i] } : a)) as [AbArm, AbArm]; const loaded = await loadSpec(c, store, c.positionals[0], false); // X-11: both arms record, or both replay one source at one boundary (A is the control). const llmExp = await import("./exp-llm.ts"); @@ -955,8 +1042,8 @@ export async function run(c: CommandContext): Promise { // X-11: --replay runs the recorded config as a control arm next to this task, unless --no-control. const llm = await llmExp.llmPlan(c, store, c.positionals[0]); if (llm?.source && llm.control) { - if (adaptive || baseline || seriesFlag(c) !== undefined || picked.length) { - throw new UsageError(`--replay with a control arm runs an A/B; pass --no-control to combine it with ${picked.length ? `--${picked[0]}` : "--samples, --stop-on-regression or --series"}`, { reason: "invalid_argument" }); + if (adaptive || baseline || seriesFlag(c) !== undefined || picked.length || c.flags.weights !== undefined) { + throw new UsageError(`--replay with a control arm runs an A/B; pass --no-control to combine it with ${picked.length ? `--${picked[0]}` : c.flags.weights !== undefined ? "--weights" : "--samples, --stop-on-regression or --series"}`, { reason: "invalid_argument" }); } return llmExp.replayWithControl(c, store, llm, c.positionals[0], seed); } diff --git a/packages/coding.sh/src/cli/registry.ts b/packages/coding.sh/src/cli/registry.ts index 81dabba5..7684832c 100644 --- a/packages/coding.sh/src/cli/registry.ts +++ b/packages/coding.sh/src/cli/registry.ts @@ -1118,6 +1118,12 @@ export const COMMANDS: CommandSpec[] = [ { name: "replay", type: "string", value: "", desc: "Serve this recorded experiment's model calls again, up to each case-trial's first changed call; later calls run live. Adds the recorded config as a control arm (A) unless --no-control" }, { name: "replay-until", type: "number", value: "", desc: "With --replay: serve only calls 1..N from the recording; call N+1 onward runs live" }, { name: "series", type: "string", value: "", desc: "Tag the experiment as the next round of an iteration loop and print whether the loop should stop (see exp plateau)" }, + { + name: "weights", + type: "list", + value: "", + desc: "Record the checkpoint the task ran: the SHA-256 of the weights file, 64 hex characters, which you compute (coding.sh hashes nothing). exp compare shows each side's and warns when both ran the same. With --arm: one per arm, as ARM=. Not with --watch or a matrix spec", + }, { name: "watch", type: "boolean", desc: "Rerun on every change to the spec, the task's script, grader files or --watch-path; a grader-only change regrades the stored outputs. Enter reruns, q quits" }, { name: "watch-path", type: "list", value: "", desc: "With --watch: more files whose change reruns the task (repeatable; also the spec's \"watch\" field)" }, { name: "session", type: "boolean", desc: "In an experiment session (exp session new), add the run to its timeline and print the delta against the previous run; --no-session skips it", default: "true" }, @@ -1138,6 +1144,7 @@ export const COMMANDS: CommandSpec[] = [ "coding.sh exp run matrix.json --reuse baseline --plan", "coding.sh exp run spec.json --record", "coding.sh exp run cand.json --replay exp_01J… --replay-until 3", + "coding.sh exp run edited.json --name edited --weights 5f2c8a1e9b3d4c7f6a0e1d2b3c4f5a6e7d8c9b0a1f2e3d4c5b6a7f8e9d0c1b2a", ], }, { diff --git a/packages/coding.sh/src/experiments/ab.ts b/packages/coding.sh/src/experiments/ab.ts index 546c8e61..e47f0e7f 100644 --- a/packages/coding.sh/src/experiments/ab.ts +++ b/packages/coding.sh/src/experiments/ab.ts @@ -22,11 +22,14 @@ import { planSequential, sequentialLook, sequentialOutcome, type LookResult, typ import { llmMetadata, openLlmCache, type LlmRunOpts } from "../llmcache/run.ts"; import { caseRates, experimentResult, prepareGraders, priorRun, rememberIdentity, runTrial, selectCases, storeResult, type TrialEnv } from "./run.ts"; import { resolveTask, type TaskSpec } from "./task.ts"; +import { withWeights } from "./weights.ts"; export interface AbArm { /** Arm name; the experiment is stored as `-`. */ name: string; task: TaskSpec; + /** SHA-256 of the checkpoint this arm runs (`exp run --weights ARM=`); stored as the arm's `metadata.weights`. */ + weights?: string; } export interface StopEarlyOpts { @@ -205,6 +208,7 @@ export async function runAb(opts: RunAbOpts): Promise { const cancelled = settled.some((run, j) => run === undefined && jobs[j]!.ci < keep) || !!infraStop?.report; const sequential = plan ? sequentialOutcome(plan, history) : undefined; const results = ([0, 1] as const).map((i) => { + const metadata = withWeights(opts.llm ? { llm: llmMetadata(opts.llm, opts.arms[i].task) } : undefined, opts.arms[i].weights); const r = experimentResult({ id: ids[i], name: names[i], @@ -219,7 +223,7 @@ export async function runAb(opts: RunAbOpts): Promise { runs: settled.filter((run, j): run is CaseRun => run !== undefined && jobs[j]!.arm === i && jobs[j]!.ci < keep), boot: { iters: opts.bootstrapIters ?? 2000, seed: 0x5eed }, infraStop: infraStop?.report, - ...(opts.llm ? { metadata: { llm: llmMetadata(opts.llm, opts.arms[i].task) } } : {}), + ...(metadata ? { metadata } : {}), }); r.pair = { id: pairId, arm: opts.arms[i].name, experiments: ids, seed, ...(sequential ? { sequential } : {}) }; return r; diff --git a/packages/coding.sh/src/experiments/compare.ts b/packages/coding.sh/src/experiments/compare.ts index 7eb8f902..78dead98 100644 --- a/packages/coding.sh/src/experiments/compare.ts +++ b/packages/coding.sh/src/experiments/compare.ts @@ -22,6 +22,7 @@ import type { HumanLabel } from "../judge/calibrate.ts"; import { graderPpi, labelMajority, type GraderPpi } from "./ppi.ts"; import type { ReplicationConclusion } from "./replicate.ts"; import { selectionWarning } from "./select.ts"; +import { stampedWith, weightsOf, weightsWarning, type WeightsStamp } from "./weights.ts"; export interface CaseDelta { caseId: string; @@ -230,8 +231,15 @@ export function clusterMap(cases: readonly Case[], key: string): Map (t[0] === t[1] ? `trial ${t[0]}` : `trials ${t[0]}-${t[1]}`); diff --git a/packages/coding.sh/src/experiments/render.ts b/packages/coding.sh/src/experiments/render.ts index c290cfbe..af66d090 100644 --- a/packages/coding.sh/src/experiments/render.ts +++ b/packages/coding.sh/src/experiments/render.ts @@ -22,6 +22,7 @@ import type { ExperimentBlock, ExpSession, SessionBlock } from "./session.ts"; import { stabilityLine } from "./stability.ts"; import type { SweepResult } from "./sweep.ts"; import type { Turnover } from "./turnover.ts"; +import { shortWeights, weightsOf } from "./weights.ts"; import { fixed, ms, paint, pct, pValue, signedPct, table, truncate, type RenderOpts } from "./text.ts"; import { formatBestSet } from "../stats/mcb.ts"; @@ -112,6 +113,9 @@ export function renderExperiment(r: ExperimentResult, opts: RenderOpts & { verif const out: string[] = []; out.push(`${paint(r.name, "bold", color)} ${paint(r.id, "dim", color)}`); out.push(`dataset ${r.dataset.name}@${r.dataset.version}${r.dataset.split ? ` (${r.dataset.split})` : ""} task ${r.task.kind}: ${r.task.ref} trials ${r.trials}`); + // The checkpoint the task ran, as given to exp run --weights: whole, so it can be checked against the file's hash. + const weights = weightsOf(r); + if (weights) out.push(`weights ${weights}`); if (r.status === "cancelled" && s.n === 0) { out.push(paint("cancelled before any case run finished: nothing was graded", "yellow", color)); return out.join("\n"); @@ -250,14 +254,19 @@ export function renderComparison(c: Comparison, opts: RenderOpts & { a?: Experim const out: string[] = []; // A forced compare of a cancelled experiment keeps saying it is partial. const partial = (e?: ExperimentResult) => (e?.status === "cancelled" ? ` ${paint(`cancelled: ${e.runs.length} case run${e.runs.length === 1 ? "" : "s"} only`, "yellow", color)}` : ""); + // Which checkpoint each side ran (exp run --weights), once either side says: the first 12 hex characters, or that it was not recorded. + const stamped = c.a.weights !== undefined || c.b.weights !== undefined; + const weights = (x: Comparison["a"]) => (stamped ? ` weights ${x.weights ? shortWeights(x.weights.sha256) : "not recorded"}` : ""); if (c.given && !c.given.kept) { const g = c.given; - out.push(`${paint("A", "bold", color)} ${c.a.name} ${paint(c.a.id, "dim", color)}`, `${paint("B", "bold", color)} ${c.b.name} ${paint(c.b.id, "dim", color)}`); + out.push(`${paint("A", "bold", color)} ${c.a.name} ${paint(c.a.id, "dim", color)}${weights(c.a)}`, `${paint("B", "bold", color)} ${c.b.name} ${paint(c.b.id, "dim", color)}${weights(c.b)}`); + if (c.weightsWarning) out.push(paint(`! ${c.weightsWarning}`, "yellow", color)); out.push(paint(`! --given ${g.grader}: no case passes ${g.grader} in both A and B (${g.dropped} dropped), so there is nothing to compare`, "yellow", color)); return out.join("\n"); } - out.push(`${paint("A", "bold", color)} ${c.a.name} ${paint(c.a.id, "dim", color)} pass ${pct(c.a.passRate)}${partial(opts.a)}`); - out.push(`${paint("B", "bold", color)} ${c.b.name} ${paint(c.b.id, "dim", color)} pass ${pct(c.b.passRate)}${partial(opts.b)}`); + out.push(`${paint("A", "bold", color)} ${c.a.name} ${paint(c.a.id, "dim", color)} pass ${pct(c.a.passRate)}${partial(opts.a)}${weights(c.a)}`); + out.push(`${paint("B", "bold", color)} ${c.b.name} ${paint(c.b.id, "dim", color)} pass ${pct(c.b.passRate)}${partial(opts.b)}${weights(c.b)}`); + if (c.weightsWarning) out.push(paint(`! ${c.weightsWarning}`, "yellow", color)); if (c.given) out.push(paint(`! ${c.given.note}`, "yellow", color)); if (c.datasetWarning) out.push(paint(`! ${c.datasetWarning}`, "yellow", color)); if (c.gradeWarning) out.push(paint(`! ${c.gradeWarning}`, "yellow", color)); diff --git a/packages/coding.sh/src/experiments/run.ts b/packages/coding.sh/src/experiments/run.ts index fb2485ee..4b3648f0 100644 --- a/packages/coding.sh/src/experiments/run.ts +++ b/packages/coding.sh/src/experiments/run.ts @@ -28,6 +28,7 @@ import { ADAPTIVE_MAX, ADAPTIVE_TRIALS, allPassKeys, CaseTracker, caseVerdicts, import { selectRunCases, type CaseSelection, type SelectOpts } from "./select.ts"; import { checkSeq, markUnchanged, seqCritic, seqFeedback, seqFeedbackName, seqTaskError, summarizeSeq, type SeqOpts } from "./seq.ts"; import { resolveTask, type ResolvedTask, type TaskContext, type TaskSpec } from "./task.ts"; +import { shortWeights, weightsOf, withWeights } from "./weights.ts"; import type { LlmCache } from "../llmcache/proxy.ts"; import { llmMetadata, openLlmCache, withLlmJob, type LlmRunOpts, type TaskLlm } from "../llmcache/run.ts"; @@ -101,6 +102,7 @@ export interface RunExperimentOpts { * finished case-trials are kept and only the rest run. It must have run the * same dataset version, task, graders, trials and selection * (ResumeMismatchError names what changed). Infra-failed trials always run again. + * Its weights stamp (`metadata.weights`) is kept when `metadata` gives none; a different one is a mismatch. */ resumeFrom?: ExperimentResult; /** With resumeFrom: also run again the case-trials whose task errored. */ @@ -126,9 +128,9 @@ export interface RunExperimentOpts { llm?: LlmRunOpts; } -/** Thrown when resumeFrom ran with a different dataset, task, graders, trials or selection. */ +/** Thrown when resumeFrom ran with a different dataset, task, graders, trials, selection or weights stamp. */ export class ResumeMismatchError extends Error { - /** What changed: "dataset", "task", "graders", "trials", "selection". */ + /** What changed: "dataset", "task", "graders", "trials", "selection", "seq", "weights". */ readonly fields: string[]; constructor(id: string, fields: string[], detail: string[]) { super(`can't resume ${id}: ${detail.join("; ")}. Its finished case-trials would mix with ones measured differently`); @@ -140,7 +142,7 @@ export class ResumeMismatchError extends Error { /** Throw ResumeMismatchError unless `prev` measured what this run measures. */ export function assertResumable( prev: ExperimentResult, - now: { dataset: Pick; task: ResolvedTask["info"]; graders: readonly Pick[]; trials: number; selection?: CaseSelection; seq?: unknown }, + now: { dataset: Pick; task: ResolvedTask["info"]; graders: readonly Pick[]; trials: number; selection?: CaseSelection; seq?: unknown; weights?: string }, ): void { const fields: string[] = []; const detail: string[] = []; @@ -172,6 +174,12 @@ export function assertResumable( fields.push("seq"); detail.push("the --seq settings changed"); } + // A stamp given now must be the stored one; none given keeps the stored one (runExperiment carries it over). + const was = weightsOf(prev); + if (now.weights !== undefined && now.weights !== was) { + fields.push("weights"); + detail.push(`weights changed (${was ? shortWeights(was) : "none"} → ${shortWeights(now.weights)})`); + } if (fields.length) throw new ResumeMismatchError(prev.id, fields, detail); } @@ -567,11 +575,14 @@ export async function runExperiment(opts: RunExperimentOpts): Promise(); diff --git a/packages/coding.sh/src/experiments/weights.ts b/packages/coding.sh/src/experiments/weights.ts new file mode 100644 index 00000000..f6d34fee --- /dev/null +++ b/packages/coding.sh/src/experiments/weights.ts @@ -0,0 +1,53 @@ +// The weights stamp of an experiment: the SHA-256 of the checkpoint file the +// task ran, as the user gave it to `exp run --weights`. coding.sh hashes +// nothing itself; it stores the value in `metadata.weights` and shows it where +// two experiments are compared, so "base vs edited" names both checkpoints and +// a comparison of one checkpoint with itself is called out. +import { isSha256Hex } from "../core/ids.ts"; +import type { ExperimentResult } from "../datasets/types.ts"; + +/** `ExperimentResult.metadata.weights`. */ +export interface WeightsStamp { + /** SHA-256 of the checkpoint file: 64 lowercase hex characters. */ + sha256: string; +} + +/** A `--weights` value as stored: trimmed and lowercased; undefined when it is not 64 hex characters. */ +export function parseWeights(v: unknown): string | undefined { + const s = typeof v === "string" ? v.trim().toLowerCase() : undefined; + return isSha256Hex(s) ? s : undefined; +} + +/** The stamp an experiment carries, or undefined (never stamped, or a value that is not a SHA-256). */ +export function weightsOf(e: Pick | undefined): string | undefined { + const w = e?.metadata?.weights; + return w && typeof w === "object" ? parseWeights((w as { sha256?: unknown }).sha256) : undefined; +} + +/** The first 12 hex characters, for a line of text. */ +export const shortWeights = (sha256: string): string => sha256.slice(0, 12); + +/** Metadata with the stamp set; `metadata` itself when there is no stamp. */ +export function withWeights(metadata: Record | undefined, sha256: string | undefined): Record | undefined { + return sha256 === undefined ? metadata : { ...(metadata ?? {}), weights: { sha256 } satisfies WeightsStamp }; +} + +/** `o` with `weights` set when there is a stamp; `o` itself otherwise, so an unstamped object gains no key. */ +export function stampedWith(o: T, sha256: string | undefined): T & { weights?: WeightsStamp } { + return sha256 === undefined ? o : { ...o, weights: { sha256 } }; +} + +/** + * What a comparison says about the two stamps beyond showing them. Undefined + * when neither side is stamped, when they differ, or when one side regrades + * the other's outputs (the same weights by construction). + */ +export function weightsWarning(a: string | undefined, b: string | undefined, regrade: boolean): string | undefined { + if (a === undefined && b === undefined) return undefined; + if (a === undefined || b === undefined) { + const [has, lacks, sha] = a !== undefined ? ["A", "B", a] : ["B", "A", b!]; + return `only ${has} records which weights it ran (${shortWeights(sha)}); ${lacks} was run without --weights, so nothing here says whether the two ran different weights`; + } + if (a !== b || regrade) return undefined; + return `A and B ran the same weights (${shortWeights(a)}): B − A is not a difference between two models`; +} diff --git a/packages/coding.sh/src/mcp/tools.ts b/packages/coding.sh/src/mcp/tools.ts index 64f2b145..f73b85ee 100644 --- a/packages/coding.sh/src/mcp/tools.ts +++ b/packages/coding.sh/src/mcp/tools.ts @@ -85,6 +85,8 @@ const N = { type: "number" }; const I = { type: "integer" }; const B = { type: "boolean" }; const arr = (items: Obj) => ({ type: "array", items }); +/** An experiment's weights stamp (`exp run --weights`): the SHA-256 of the checkpoint its task ran. */ +const WEIGHTS_OUT = out({ sha256: S }); // Output schemas name the fields worth knowing about; results may carry more // (objects are open), which keeps every tool definition cheap to send each turn. @@ -975,11 +977,11 @@ function toolDefs(): ToolDef[] { { name: "experiments_list", title: "List experiments", - description: "List dataset experiments, newest first, with dataset, trials, and pass rate with its confidence interval. Compare two with experiment_compare.", + description: "List dataset experiments, newest first, with dataset, trials, pass rate with its interval, and weights (a run's checkpoint SHA-256). Compare two with experiment_compare.", inputSchema: input({ dataset: { type: "string", description: "Only experiments on this dataset id." }, name: { type: "string", description: "Only experiments with this exact name." }, limit: LIMIT(20, 100), cursor: CURSOR }), outputSchema: out( { - experiments: arr(out({ id: S, name: S, dataset: S, passRate: N }, ["id", "name"])), + experiments: arr(out({ id: S, name: S, dataset: S, passRate: N, weights: WEIGHTS_OUT }, ["id", "name"])), nextCursor: S, }, ["experiments"], @@ -987,7 +989,7 @@ function toolDefs(): ToolDef[] { annotations: READ, async run(a, c) { const ctx = await c.store(); - const { FileEvalStore } = await import("../datasets/store.ts"); + const [{ FileEvalStore }, { weightsOf, stampedWith }] = await Promise.all([import("../datasets/store.ts"), import("../experiments/weights.ts")]); const store = new FileEvalStore({ env: c.env, index: ctx.index }); const offset = cursorArg(a); const limit = numArg(a, "limit", 20); @@ -996,7 +998,8 @@ function toolDefs(): ToolDef[] { const all = await store.listExperiments({ limit: offset + limit + 1, ...(dataset ? { datasetId: dataset } : {}), ...(name ? { name } : {}) }); const page = all.slice(offset, offset + limit); return { - experiments: page.map((e) => ({ + // Each row carries the checkpoint its run recorded (exp run --weights); no key when it recorded none. + experiments: page.map((e) => stampedWith({ id: e.id, name: e.name, dataset: e.dataset.id, @@ -1011,7 +1014,7 @@ function toolDefs(): ToolDef[] { passRateLo: round(e.summary.passRate.lo), passRateHi: round(e.summary.passRate.hi), createdAt: iso(e.createdAt), - })), + }, weightsOf(e))), ...(all.length > offset + limit ? { nextCursor: String(offset + limit) } : {}), }; }, @@ -1031,8 +1034,8 @@ function toolDefs(): ToolDef[] { ), outputSchema: out( { - baseline: out({ id: S, passRate: N }), - candidate: out({ id: S, passRate: N }), + baseline: out({ id: S, passRate: N, weights: WEIGHTS_OUT }), + candidate: out({ id: S, passRate: N, weights: WEIGHTS_OUT }), verdict: { type: "string", enum: ["better", "worse", "inconclusive"] }, diff: out({ mean: N, lo: N, hi: N }), paired: I, @@ -1078,11 +1081,13 @@ function toolDefs(): ToolDef[] { ...cmp.power.warnings.map((w) => w.message), ...(cmp.turnover.warning ? [cmp.turnover.warning] : []), ]; + // First: with one checkpoint on both sides, nothing below compares two models. + if (cmp.weightsWarning) warnings.unshift(cmp.weightsWarning); const gate = cmp.gate; const t = cmp.turnover; return { - baseline: { id: cmp.a.id, name: cmp.a.name, passRate: round(cmp.a.passRate) }, - candidate: { id: cmp.b.id, name: cmp.b.name, passRate: round(cmp.b.passRate) }, + baseline: { ...cmp.a, passRate: round(cmp.a.passRate) }, + candidate: { ...cmp.b, passRate: round(cmp.b.passRate) }, verdict: cmp.verdict, diff: { mean: round(cmp.diff.mean), lo: round(cmp.diff.lo), hi: round(cmp.diff.hi) }, paired: cmp.paired, diff --git a/packages/coding.sh/src/viewer/ui/index.html b/packages/coding.sh/src/viewer/ui/index.html index 882306b6..5ec114ad 100644 --- a/packages/coding.sh/src/viewer/ui/index.html +++ b/packages/coding.sh/src/viewer/ui/index.html @@ -6156,7 +6156,7 @@ h("div", { class: "head-row" }, h("h1", { title: d.id }, d.name || d.id), h("div", { class: "badges" }, d.status === "cancelled" ? h("span", { class: "badge warn" }, "cancelled · partial") : null, copyBtn(d.id, "Copy experiment id", "experiment id", "Copy id"))), // Facts wrap between items, never inside one; the task command gets its own line. - h("p", { class: "meta fact-line" }, ...[datasetText(d), (d.trials || 1) > 1 ? plural(d.trials, "trial") + " per case" : "", fmtDate(d.createdAt || 0)].filter(Boolean).map((t) => h("span", null, t))), + h("p", { class: "meta fact-line" }, ...[datasetText(d), (d.trials || 1) > 1 ? plural(d.trials, "trial") + " per case" : "", fmtDate(d.createdAt || 0)].filter(Boolean).map((t) => h("span", null, t)), weightsFact(d.metadata && d.metadata.weights, "")), d.task ? h("p", { class: "meta mono truncate task-line", title: d.task.kind + " " + d.task.ref }, // Each argument wraps as a whole, so a phone never splits "agents-md-v2" at its hyphen. ...(d.task.kind + " " + d.task.ref).split(" ").flatMap((t, i) => [i ? " " : null, h("span", { class: "tok" }, t)])) : null), @@ -6273,10 +6273,18 @@ /** A symmetric axis that ends on a round number: ±10, 20, 25, 50 or 100 pts. */ function niceSpan(x) { return [0.1, 0.2, 0.25, 0.5, 1].find((v) => v >= x * 1.1) || 1; } /** The side of the verdict card: role, name (when it says more than the role) and pass rate. */ -function side(cls, role, e) { +/** `stamped`: one side at least records its weights (`exp run --weights`), so both say what they recorded. */ +function side(cls, role, e, stamped) { const name = e.name || e.id; return h("div", { class: "side " + cls }, h("span", { class: "meta" }, name.toLowerCase() === role ? role : role + " · " + name), - h("a", { class: "pct", href: expHref(e.id), title: e.id }, pct1(e.passRate))); + h("a", { class: "pct", href: expHref(e.id), title: e.id }, pct1(e.passRate)), + stamped ? weightsFact(e.weights, "weights not recorded") : null); +} +/** A weights stamp in 12 hex characters, the whole SHA-256 in its title. `none`: the text for a run that recorded no stamp ("" shows nothing). */ +function weightsFact(w, none) { + const sha = w && typeof w.sha256 === "string" ? w.sha256 : ""; + if (!sha && !none) return null; + return h("span", { class: "meta weights", title: sha ? "SHA-256 of the checkpoint this run recorded with --weights: " + sha : "This run was started without --weights" }, sha ? "weights " + sha.slice(0, 12) : none); } /** One decimal, so both sides add up to the difference shown in tenths of a point. */ function pct1(x) { return ((x || 0) * 100).toFixed(1) + "%"; } @@ -6329,7 +6337,7 @@ : "Inconclusive"; // Axis: symmetric around zero, wide enough for the interval, at least ±10 pts. const span = niceSpan(Math.max(Math.abs(c.diff.lo), Math.abs(c.diff.hi))); - const warnings = [c.datasetWarning, c.gradeWarning, ...(c.partial || []).map((id) => (id === c.a.id ? "Baseline" : "Candidate") + " was cancelled; only its finished cases are compared.")].filter(Boolean).map((w) => ({ text: w })); + const warnings = [c.weightsWarning, c.datasetWarning, c.gradeWarning, ...(c.partial || []).map((id) => (id === c.a.id ? "Baseline" : "Candidate") + " was cancelled; only its finished cases are compared.")].filter(Boolean).map((w) => ({ text: w })); // small-n already states the detection limit; below-mde would repeat it. const pw = (c.power && c.power.warnings) || []; const smallN = pw.some((w) => w.code === "small-n"); @@ -6343,9 +6351,9 @@ h("div", { class: "verdict-line" }, h("h1", null, headline), h("span", { class: "meta" }, plural(c.paired, "paired case") + " · McNemar p = " + (c.mcnemar.p < 0.001 ? "<0.001" : (Math.round(c.mcnemar.p * 1000) / 1000)) + (mde && !smallN ? " · " + mde : ""))), h("div", { class: "vs" }, - side("a", "baseline", c.a), + side("a", "baseline", c.a, !!(c.a.weights || c.b.weights)), h("span", { class: "arrow", "aria-hidden": "true" }, "→"), - side("b", "candidate", c.b), + side("b", "candidate", c.b, !!(c.a.weights || c.b.weights)), h("div", { class: "side diff-side" }, h("span", { class: "meta" }, "difference"), h("span", { class: "pct delta-big " + (tone === "better" ? "up" : tone === "worse" ? "down" : "") }, signedPts(c.diff.mean)))), h("div", { class: "delta-ci" }, h("div", { class: "section-label" }, h("span", { class: "meta" }, "95% CI " + signedPts(c.diff.lo) + " to " + signedPts(c.diff.hi))), diff --git a/packages/coding.sh/test/docs/guide-commands.test.ts b/packages/coding.sh/test/docs/guide-commands.test.ts index 744f5ba9..2a34f2a5 100644 --- a/packages/coding.sh/test/docs/guide-commands.test.ts +++ b/packages/coding.sh/test/docs/guide-commands.test.ts @@ -273,11 +273,16 @@ process.stdout.write(JSON.stringify({answer:right?String(n):"?",finish_reason:do for (const argv of cmds) assert.doesNotThrow(() => checkArgs(argv), `coding.sh ${argv.join(" ")} parses`); assert.deepEqual(cmds.map((c) => c.slice(0, 2).join(" ")), ["dataset import", "exp run", "exp run", "exp compare", "exp compare"]); assert.ok(cmds[4]!.includes("--given") && cmds[4]!.includes("terminated")); + // Each run names its checkpoint: two different SHA-256s, as literal values. + const stamps = [cmds[1]!, cmds[2]!].map((c) => c[c.indexOf("--weights") + 1]!); + for (const s of stamps) assert.match(s, /^[0-9a-f]{64}$/, "exp run carries --weights "); + assert.notEqual(stamps[0], stamps[1], "base and edited are two checkpoints"); const out: string[] = []; for (const argv of cmds) { out.push(ok([...argv.map((a) => (/^(?:math\.jsonl|\w+\.json)$/.test(a) ? join(proj, a) : a)), "--json"]).stdout); } - const graders = (stdout: string) => (JSON.parse(stdout) as { comparison: { graders: GraderDelta[]; given?: { kept: number; dropped: number } } }).comparison; + type Side = { weights?: { sha256: string } }; + const graders = (stdout: string) => (JSON.parse(stdout) as { comparison: { a: Side; b: Side; weightsWarning?: string; graders: GraderDelta[]; given?: { kept: number; dropped: number } } }).comparison; const plain = graders(out[3]!); assert.deepEqual(plain.graders.map((g) => g.id), ["terminated", "correct"]); for (const g of plain.graders) assert.ok(g.diff.mean > 0, `${g.id} is up: ${JSON.stringify(plain)}`); @@ -286,6 +291,14 @@ process.stdout.write(JSON.stringify({answer:right?String(n):"?",finish_reason:do const given = graders(out[4]!); assert.deepEqual([given.given?.kept, given.given?.dropped], [2, 2]); assert.equal(given.graders.find((g) => g.id === "correct")!.diff.mean, 0); + // Both compares carry each side's stamp, in --json whole and in text by its first 12 characters. + for (const c of [plain, given]) assert.deepEqual([c.a.weights?.sha256, c.b.weights?.sha256, c.weightsWarning], [...stamps, undefined]); + for (const argv of [cmds[3]!, cmds[4]!]) { + const text = ok(argv).stdout; + assert.match(text, new RegExp(`^A base .*weights ${stamps[0]!.slice(0, 12)}$`, "m"), text); + assert.match(text, new RegExp(`^B edited .*weights ${stamps[1]!.slice(0, 12)}$`, "m"), text); + for (const s of stamps) assert.ok(recipe.includes("`weights " + s.slice(0, 12) + "`"), "the prose quotes the short stamp compare prints"); + } }); it("fails when the section is gone", () => { diff --git a/packages/coding.sh/test/experiments/weights-stamp.test.ts b/packages/coding.sh/test/experiments/weights-stamp.test.ts new file mode 100644 index 00000000..af58fe71 --- /dev/null +++ b/packages/coding.sh/test/experiments/weights-stamp.test.ts @@ -0,0 +1,364 @@ +// exp run --weights : the checkpoint an experiment ran, stored as metadata.weights and +// shown by exp compare, so "base vs edited" names both checkpoints and a comparison of one +// checkpoint with itself is called out. The hashes below are of labels, not of files: the CLI +// never hashes anything itself. +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { after, before, describe, it } from "node:test"; +import { createDataset } from "../../src/datasets/dataset.ts"; +import type { ExperimentResult, Grader } from "../../src/datasets/types.ts"; +import { runAb } from "../../src/experiments/ab.ts"; +import { compareExperiments } from "../../src/experiments/compare.ts"; +import { renderComparison, renderExperiment } from "../../src/experiments/render.ts"; +import { regradeExperiment, ResumeMismatchError, runExperiment } from "../../src/experiments/run.ts"; +import { parseWeights, weightsOf } from "../../src/experiments/weights.ts"; +import { tmpEnv, type TmpEnv } from "../adapters/_helpers.ts"; +import { CLI, cli, ok } from "../query/_seed.ts"; + +const sha = (label: string) => createHash("sha256").update(label).digest("hex"); +const BASE = sha("base checkpoint"); +const EDITED = sha("edited checkpoint"); +const short = (s: string) => s.slice(0, 12); +const stamp = (r: ExperimentResult) => (r.metadata?.weights as { sha256?: string } | undefined)?.sha256; +const stamped = (r: ExperimentResult) => Object.hasOwn(r.metadata ?? {}, "weights"); + +const ds = createDataset({ name: "wt", createdAt: 1, cases: [1, 2, 3, 4].map((n) => ({ id: `m${n}`, input: n, expected: n })) }); +const exact: Grader = { id: "exact", version: "1", kind: "custom", grade: async (c) => (c.output === c.expected ? { score: 1, pass: true } : { score: 0, pass: false }) }; +const run = (name: string, weights?: string, task: (n: unknown) => unknown = (n) => n) => + runExperiment({ name, dataset: ds, task, graders: [exact], metadata: weights ? { weights: { sha256: weights } } : undefined }); + +describe("the weights stamp in compareExperiments", () => { + it("different stamps: each side carries its own, with no warning; no stamps: nothing about weights", async () => { + const c = compareExperiments(await run("base", BASE), await run("edited", EDITED)); + assert.deepEqual([c.a.weights, c.b.weights], [{ sha256: BASE }, { sha256: EDITED }]); + assert.equal(Object.hasOwn(c, "weightsWarning"), false); + + const a = await run("a"); + assert.equal(stamped(a), false); + const none = compareExperiments(a, await run("b")); + assert.equal(Object.hasOwn(none.a, "weights"), false); + assert.equal(Object.hasOwn(none.b, "weights"), false); + assert.equal(Object.hasOwn(none, "weightsWarning"), false); + assert.doesNotMatch(renderComparison(none, { color: false }), /weights/); + }); + + it("the same stamp on both sides warns; a regrade of the other side does not", async () => { + const a = await run("base", BASE); + const twin = await run("twin", BASE); + const c = compareExperiments(a, twin); + assert.match(c.weightsWarning!, new RegExp(`^A and B ran the same weights \\(${short(BASE)}\\)`)); + assert.match(renderComparison(c, { color: false }), new RegExp(`^! A and B ran the same weights \\(${short(BASE)}\\)`, "m")); + + const regraded = await regradeExperiment({ result: a, dataset: ds, graders: [exact] }); + assert.equal(stamp(regraded), BASE, "a regrade inherits the stamp of the outputs it grades"); + for (const [x, y] of [[a, regraded], [regraded, a]] as const) { + const r = compareExperiments(x, y); + assert.deepEqual([r.a.weights, r.b.weights], [{ sha256: BASE }, { sha256: BASE }]); + assert.equal(Object.hasOwn(r, "weightsWarning"), false); + } + }); + + it("a stamp on one side only is said plainly", async () => { + const c = compareExperiments(await run("plain"), await run("edited", EDITED)); + assert.equal(Object.hasOwn(c.a, "weights"), false); + assert.match(c.weightsWarning!, new RegExp(`^only B records which weights it ran \\(${short(EDITED)}\\); A was run without --weights`)); + const text = renderComparison(c, { color: false }); + assert.match(text, /^A plain \S+ {2}pass 100\.0% {2}weights not recorded$/m); + assert.match(text, new RegExp(`^B edited \\S+ {2}pass 100\\.0% {2}weights ${short(EDITED)}$`, "m")); + assert.match(text, /^! only B records which weights it ran/m); + }); + + it("both headers show the stamps: the normal one and the one --given prints when no case is kept", async () => { + const gate: Grader = { id: "even", version: "1", kind: "custom", grade: async (c) => ((c.output as number) % 2 === 0 ? { score: 1, pass: true } : { score: 0, pass: false }) }; + const of = (name: string, weights: string, task: (n: unknown) => unknown) => runExperiment({ name, dataset: ds, task, graders: [gate, exact], metadata: { weights: { sha256: weights } } }); + const a = await of("base", BASE, () => 1); + const b = await of("edited", EDITED, (n) => n); + const none = compareExperiments(a, b, { given: "even" }); + assert.equal(none.given?.kept, 0); + for (const c of [compareExperiments(a, b), none]) { + const text = renderComparison(c, { color: false }); + assert.match(text, new RegExp(`^A base \\S+.* {2}weights ${short(BASE)}$`, "m")); + assert.match(text, new RegExp(`^B edited \\S+.* {2}weights ${short(EDITED)}$`, "m")); + } + assert.match(renderComparison(none, { color: false }), /nothing to compare/); + assert.match(renderExperiment(a, { color: false }), new RegExp(`^weights ${BASE}$`, "m")); + assert.doesNotMatch(renderExperiment(await run("plain"), { color: false }), /weights/); + }); + + it("a stored value that is not a SHA-256 is not a stamp; any case reads as lowercase", () => { + assert.equal(weightsOf({ metadata: { weights: { sha256: "abc" } } }), undefined); + assert.equal(weightsOf({ metadata: { weights: BASE } }), undefined); + assert.equal(weightsOf({ metadata: { checkpoint: { done: 1, total: 4, at: 1 } } }), undefined); + assert.equal(weightsOf({ metadata: { weights: { sha256: BASE.toUpperCase() } } }), BASE); + assert.equal(parseWeights(` ${BASE.toUpperCase()}\n`), BASE); + assert.equal(parseWeights(BASE.slice(1)), undefined); + assert.equal(parseWeights(`${BASE.slice(2)}zz`), undefined); + }); +}); + +describe("the weights stamp across resume and A/B runs", () => { + const flaky = (broken: () => boolean) => (n: unknown) => { + if (n === 2 && broken()) throw new Error("boom"); + return n; + }; + + it("a resumed run keeps the stored stamp when none is given, and refuses a different one", async () => { + let broken = true; + const task = flaky(() => broken); + const first = await runExperiment({ name: "r", dataset: ds, task, graders: [exact], metadata: { weights: { sha256: BASE } } }); + assert.equal(first.summary.errors, 1); + broken = false; + const again = await runExperiment({ name: "r", dataset: ds, task, graders: [exact], resumeFrom: first, retryErrors: true }); + assert.equal(again.summary.errors, 0); + assert.equal(stamp(again), BASE); + const same = await runExperiment({ name: "r", dataset: ds, task, graders: [exact], resumeFrom: first, retryErrors: true, metadata: { weights: { sha256: BASE } } }); + assert.equal(stamp(same), BASE); + await assert.rejects( + runExperiment({ name: "r", dataset: ds, task, graders: [exact], resumeFrom: first, retryErrors: true, metadata: { weights: { sha256: EDITED } } }), + (e: unknown) => e instanceof ResumeMismatchError && e.fields.includes("weights") && new RegExp(`weights changed \\(${short(BASE)} → ${short(EDITED)}\\)`).test(e.message), + ); + }); + + it("a run stored without a stamp does not gain one on resume", async () => { + let broken = true; + const task = flaky(() => broken); + const first = await runExperiment({ name: "r", dataset: ds, task, graders: [exact] }); + broken = false; + await assert.rejects( + runExperiment({ name: "r", dataset: ds, task, graders: [exact], resumeFrom: first, retryErrors: true, metadata: { weights: { sha256: EDITED } } }), + (e: unknown) => e instanceof ResumeMismatchError && /weights changed \(none → /.test(e.message), + ); + const again = await runExperiment({ name: "r", dataset: ds, task, graders: [exact], resumeFrom: first, retryErrors: true }); + assert.equal(stamped(again), false); + }); + + it("runAb stamps each arm with its own weights, and an arm without any stays unstamped", async () => { + const both = await runAb({ name: "ab", dataset: ds, graders: [exact], seed: 1, arms: [{ name: "base", task: (n: unknown) => n, weights: BASE }, { name: "edited", task: (n: unknown) => n, weights: EDITED }] }); + assert.deepEqual([stamp(both.a), stamp(both.b)], [BASE, EDITED]); + const one = await runAb({ name: "ab", dataset: ds, graders: [exact], seed: 1, arms: [{ name: "base", task: (n: unknown) => n }, { name: "edited", task: (n: unknown) => n, weights: EDITED }] }); + assert.equal(stamped(one.a), false); + assert.equal(stamp(one.b), EDITED); + }); +}); + +describe("exp run --weights from the CLI", () => { + let env: TmpEnv; + let proj = ""; + let edited = ""; + const G = ['{"kind":"json-path","name":"terminated","path":"$.finish_reason","equals":"stop"}', '{"kind":"json-path","name":"correct","path":"$.answer"}']; + const task = (model: string) => JSON.stringify({ kind: "command", command: process.execPath, args: [join(proj, "gen.mjs")], env: { MODEL: model } }); + const runArgs = (model: string, name: string, ...more: string[]) => ["exp", "run", "-d", "wt", "--task", task(model), "-g", G[0]!, "-g", G[1]!, "--name", name, ...more]; + const exp = (r: { stdout: string }) => (JSON.parse(r.stdout) as { experiment: ExperimentResult }).experiment; + const names = () => (JSON.parse(ok(["exp", "list", "--json"]).stdout) as { experiments: { name: string }[] }).experiments.map((e) => e.name); + + before(() => { + env = tmpEnv(); + proj = join(env.root, "proj"); + mkdirSync(proj, { recursive: true }); + // MODEL=base finishes m1 and m2; edited finishes every case and misses m3; stuck finishes none. + // m2 exits 1 while ./broken exists, and m3 hangs while ./hang exists. + writeFileSync( + join(proj, "gen.mjs"), + `import {existsSync} from "node:fs";let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{const n=Number(JSON.parse(s).slice(1));const m=process.env.MODEL; +if(n===2&&existsSync(${JSON.stringify(join(proj, "broken"))}))process.exit(1); +if(n===3&&existsSync(${JSON.stringify(join(proj, "hang"))})){setInterval(()=>{},1000);return} +const done=m==="edited"||(m==="base"&&n<=2);const right=m==="edited"?n!==3:m==="base"&&n<=2; +process.stdout.write(JSON.stringify({answer:right?"ok":"wrong",finish_reason:done?"stop":"length"}))});\n`, + ); + const cases = [1, 2, 3, 4].map((n) => ["--case", JSON.stringify({ id: `m${n}`, input: `m${n}`, expected: { answer: "ok" } })]).flat(); + ok(["dataset", "add", "wt", ...cases]); + ok(runArgs("base", "base", "--weights", BASE)); + // Uppercase in, as some hashing tools print it. + edited = ok(runArgs("edited", "edited", "--weights", EDITED.toUpperCase())).stdout; + ok(runArgs("base", "twin", "--weights", BASE)); + ok(runArgs("stuck", "stuck", "--weights", sha("stuck checkpoint"))); + ok(runArgs("base", "plain")); + }); + after(() => env.restore()); + + it("stores metadata.weights lowercase, and shows it in the run summary and exp show", () => { + assert.match(edited, new RegExp(`^weights ${EDITED}$`, "m")); + const shown = ok(["exp", "show", "edited"]).stdout; + assert.match(shown, new RegExp(`^weights ${EDITED}$`, "m")); + const e = exp(ok(["exp", "show", "edited", "--json"])); + assert.deepEqual(e.metadata?.weights, { sha256: EDITED }); + }); + + it("without the flag, metadata.weights is absent", () => { + const e = exp(ok(["exp", "show", "plain", "--json"])); + assert.equal(stamped(e), false); + assert.doesNotMatch(ok(["exp", "show", "plain"]).stdout, /weights/); + }); + + it("exp compare shows both stamps in the header: short in text, whole in --json", () => { + const text = ok(["exp", "compare", "base", "edited"]).stdout; + assert.match(text, new RegExp(`^A base exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(BASE)}$`, "m")); + assert.match(text, new RegExp(`^B edited exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(EDITED)}$`, "m")); + assert.doesNotMatch(text, /same weights|only [AB] records/); + const c = JSON.parse(ok(["exp", "compare", "base", "edited", "--json"]).stdout).comparison; + assert.deepEqual([c.a.weights, c.b.weights], [{ sha256: BASE }, { sha256: EDITED }]); + assert.equal("weightsWarning" in c, false); + }); + + it("the --given header shows them too, with cases kept and with none", () => { + const kept = ok(["exp", "compare", "base", "edited", "--given", "terminated"]).stdout; + assert.match(kept, new RegExp(`^A base exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(BASE)}$`, "m")); + assert.match(kept, new RegExp(`^B edited exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(EDITED)}$`, "m")); + const none = ok(["exp", "compare", "stuck", "edited", "--given", "terminated"]).stdout; + assert.match(none, /nothing to compare/); + assert.match(none, new RegExp(`^A stuck exp_\\w+ {2}weights ${short(sha("stuck checkpoint"))}$`, "m")); + assert.match(none, new RegExp(`^B edited exp_\\w+ {2}weights ${short(EDITED)}$`, "m")); + }); + + it("equal stamps warn in text and --json", () => { + const text = ok(["exp", "compare", "base", "twin"]).stdout; + assert.match(text, new RegExp(`^! A and B ran the same weights \\(${short(BASE)}\\): B − A is not a difference between two models$`, "m")); + const c = JSON.parse(ok(["exp", "compare", "base", "twin", "--json"]).stdout).comparison; + assert.match(c.weightsWarning, /^A and B ran the same weights/); + }); + + it("a regrade inherits the stamp and its pair does not warn", () => { + const r = exp(ok(["exp", "regrade", "base", "-g", G[1]!, "--name", "regraded", "--json"])); + assert.deepEqual(r.metadata?.weights, { sha256: BASE }); + assert.equal(typeof r.metadata?.regradeOf, "string"); + const text = ok(["exp", "compare", "base", "regraded"]).stdout; + assert.match(text, new RegExp(`^B regraded exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(BASE)}$`, "m")); + assert.doesNotMatch(text, /same weights/); + }); + + it("a stamp on one side only says so; no stamp on either says nothing", () => { + const text = ok(["exp", "compare", "plain", "edited"]).stdout; + assert.match(text, /^A plain exp_\w+ {2}pass [\d.]+% {2}weights not recorded$/m); + assert.match(text, /^! only B records which weights it ran \(\w{12}\); A was run without --weights/m); + ok(runArgs("edited", "plain2")); + const bare = ok(["exp", "compare", "plain", "plain2"]); + assert.doesNotMatch(bare.stdout, /weights/); + assert.equal("weights" in JSON.parse(ok(["exp", "compare", "plain", "plain2", "--json"]).stdout).comparison.a, false); + }); + + it("a value that is not 64 hex characters is a usage error and nothing runs", () => { + for (const bad of [BASE.slice(0, 40), `${BASE.slice(2)}zz`, "model.safetensors", `sha256:${BASE}`]) { + const r = cli(runArgs("base", "bad", "--weights", bad)); + assert.equal(r.status, 2, `${bad}: ${r.stderr}`); + assert.match(r.stderr, /--weights takes the SHA-256 of the checkpoint file: 64 hex characters/); + } + const two = cli(runArgs("base", "bad", "--weights", BASE, "--weights", EDITED)); + assert.equal(two.status, 2); + assert.match(two.stderr, /--weights takes one SHA-256 for a run of one task/); + const named = cli(runArgs("base", "bad", "--weights", `base=${BASE}`)); + assert.equal(named.status, 2); + assert.match(named.stderr, /--weights ARM= needs two --arm options/); + assert.equal(names().includes("bad"), false); + }); + + it("--arm takes one stamp per arm, by name", () => { + const arms = ["--arm", `base=${task("base")}`, "--arm", `edited=${task("edited")}`]; + const args = ["exp", "run", "-d", "wt", "-g", G[0]!, "-g", G[1]!, ...arms, "--seed", "1"]; + const r = JSON.parse(ok([...args, "--name", "ab", "--weights", `edited=${EDITED}`, "--weights", `base=${BASE.toUpperCase()}`, "--json"]).stdout) as { experiments: ExperimentResult[]; comparison: { a: { weights: unknown }; b: { weights: unknown } } }; + assert.deepEqual(r.experiments.map((e) => [e.name, e.metadata?.weights]), [["ab-base", { sha256: BASE }], ["ab-edited", { sha256: EDITED }]]); + assert.deepEqual([r.comparison.a.weights, r.comparison.b.weights], [{ sha256: BASE }, { sha256: EDITED }]); + + const text = ok([...args, "--name", "ab1", "--weights", `edited=${EDITED}`]).stdout; + assert.match(text, /^A ab1-base exp_\w+ {2}pass [\d.]+% {2}weights not recorded$/m); + assert.match(text, new RegExp(`^B ab1-edited exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(EDITED)}$`, "m")); + assert.equal(stamped(exp(ok(["exp", "show", "ab1-base", "--json"]))), false); + + for (const [bad, why] of [ + [["--weights", BASE], /with --arm, name the arm each stamp belongs to: --weights base= --weights edited=/], + [["--weights", `nope=${BASE}`], /--weights nope=…: no arm named "nope" \(arms: base, edited\)/], + [["--weights", `base=${BASE}`, "--weights", `base=${EDITED}`], /--weights names arm "base" twice/], + [["--weights", "base=abc"], /--weights takes the SHA-256 of the checkpoint file: 64 hex characters/], + ] as const) { + const x = cli([...args, "--name", "abbad", ...bad]); + assert.equal(x.status, 2, `${bad.join(" ")}: ${x.stderr}`); + assert.match(x.stderr, why); + } + assert.equal(names().some((n) => n.startsWith("abbad")), false); + }); + + it("--watch and a matrix spec refuse the flag before anything runs", () => { + const watch = cli(runArgs("base", "w", "--weights", BASE, "--watch")); + assert.equal(watch.status, 2, watch.stderr); + assert.match(watch.stderr, /--weights stamps one run with one checkpoint; it does not combine with --watch/); + const spec = join(proj, "matrix.json"); + writeFileSync(spec, JSON.stringify({ dataset: "wt", graders: [JSON.parse(G[1]!)], variants: { base: { task: JSON.parse(task("base")) }, edited: { task: JSON.parse(task("edited")) } }, baseline: "base" })); + const matrix = cli(["exp", "run", spec, "--weights", BASE]); + assert.equal(matrix.status, 2, matrix.stderr); + assert.match(matrix.stderr, /--weights stamps one run with one checkpoint; it does not combine with a matrix spec/); + assert.deepEqual(names().filter((n) => n === "w" || n.includes("matrix")), []); + }); + + it("--resume --retry-errors keeps the stamp, and refuses a different one", () => { + writeFileSync(join(proj, "broken"), ""); + const first = exp(ok(runArgs("edited", "flaky", "--weights", BASE, "--json"))); + assert.equal(first.summary.errors, 1); + rmSync(join(proj, "broken")); + const other = cli(runArgs("edited", "flaky", "--resume", first.id, "--retry-errors", "--weights", EDITED)); + assert.equal(other.status, 1, other.stderr); + assert.match(other.stderr, new RegExp(`can't resume ${first.id}: weights changed \\(${short(BASE)} → ${short(EDITED)}\\)`)); + const again = exp(ok(runArgs("edited", "flaky", "--resume", first.id, "--retry-errors", "--json"))); + assert.equal(again.id, first.id); + assert.equal(again.summary.errors, 0); + assert.deepEqual(again.metadata?.weights, { sha256: BASE }); + }); + + /** Start a run that hangs on m3, wait until `ready` returns its id, then kill it. */ + async function killMidRun(args: string[], ready: () => string | undefined): Promise { + writeFileSync(join(proj, "hang"), ""); + const childEnv = { ...process.env, CODING_SH_NO_UPDATE_CHECK: "1", CODING_SH_NO_AUTO_INGEST: "1", NO_COLOR: "1" }; + const child = spawn(process.execPath, ["--no-warnings", CLI, ...args, "-c", "1"], { env: childEnv, stdio: "ignore" }); + let id: string | undefined; + for (let i = 0; i < 80 && !id; i++) { + await new Promise((r) => setTimeout(r, 250)); + id = ready(); + } + const exited = child.exitCode !== null || child.signalCode !== null ? Promise.resolve() : new Promise((r) => child.once("exit", r)); + child.kill("SIGKILL"); + await exited; + rmSync(join(proj, "hang")); + assert.ok(id, `the run never got far enough to kill (home ${existsSync(env.dataHome) ? "exists" : "missing"})`); + return id; + } + + it("a killed run resumes with its stamp, and neither a different stamp nor a bare --resume takes it over", async () => { + const id = await killMidRun(runArgs("edited", "killed", "--weights", BASE), () => { + const l = cli(["exp", "list", "--json"]); + const e = l.status === 0 ? (JSON.parse(l.stdout) as { experiments: { id: string; name: string; partial?: boolean; runs: number }[] }).experiments.find((x) => x.name === "killed") : undefined; + return e?.partial && e.runs >= 2 ? e.id : undefined; + }); + // The partial record written mid-run already carries the stamp. + assert.deepEqual(exp(ok(["exp", "show", id, "--json"])).metadata?.weights, { sha256: BASE }); + + const other = cli(runArgs("edited", "killed", "--resume", id, "--weights", EDITED)); + assert.equal(other.status, 1, other.stderr); + assert.match(other.stderr, new RegExp(`Can't resume ${id}: it ran weights ${short(BASE)}, and --weights gives ${short(EDITED)}`)); + // Without an id, --resume finds a run by its config, and the stamp is part of it. + const bare = cli(runArgs("edited", "killed", "--resume")); + assert.equal(bare.status, 1, bare.stderr); + assert.match(bare.stderr, /its config changed since it started \(weights\)/); + + const done = exp(ok(runArgs("edited", "killed", "--resume", id, "--json"))); + assert.equal(done.id, id); + assert.equal(done.status, "completed"); + assert.equal(done.runs.length, 4); + assert.deepEqual(done.metadata?.weights, { sha256: BASE }); + }); + + it("a killed --arm run resumes with each arm's stamp, and refuses a different one", async () => { + const args = ["exp", "run", "-d", "wt", "-g", G[0]!, "-g", G[1]!, "--arm", `base=${task("base")}`, "--arm", `edited=${task("edited")}`, "--seed", "1", "--name", "abk"]; + const id = await killMidRun([...args, "--weights", `base=${BASE}`, "--weights", `edited=${EDITED}`], () => { + const l = cli(["exp", "status", "--json"]); + const runs = l.status === 0 ? (JSON.parse(l.stdout) as { runs: { id: string; name: string; done: number }[] }).runs : []; + return runs.find((x) => x.name === "abk" && x.done >= 2)?.id; + }); + const other = cli([...args, "--resume", id, "--weights", `base=${BASE}`, "--weights", `edited=${BASE}`]); + assert.equal(other.status, 1, other.stderr); + assert.match(other.stderr, new RegExp(`Can't resume ${id}: arm edited ran weights ${short(EDITED)}, and --weights gives ${short(BASE)}`)); + const done = JSON.parse(ok([...args, "--resume", id, "--json"]).stdout) as { outcome: string; experiments: ExperimentResult[] }; + assert.equal(done.outcome, "completed"); + assert.deepEqual(done.experiments.map((e) => [e.name, e.runs.length, e.metadata?.weights]), [["abk-base", 4, { sha256: BASE }], ["abk-edited", 4, { sha256: EDITED }]]); + }); +}); diff --git a/packages/coding.sh/test/llmcache/cli.test.ts b/packages/coding.sh/test/llmcache/cli.test.ts index 72d9d8e9..821053cb 100644 --- a/packages/coding.sh/test/llmcache/cli.test.ts +++ b/packages/coding.sh/test/llmcache/cli.test.ts @@ -132,6 +132,21 @@ describe("exp run --record / --replay from the CLI", () => { assert.match(t.stdout, /LLM calls: 8 \(4 cached, 4 live\)/); }); + it("--weights with a control arm is a usage error; with --no-control the replayed run carries the stamp", async () => { + const sha = "ab12".repeat(16); + const before = stub.calls(); + // The control arm replays the recorded checkpoint, so one stamp cannot name both arms. + const refused = await cli(["exp", "run", spec("wt", "sys A"), "--replay", recId, "--weights", sha]); + assert.equal(refused.status, 2, refused.stderr); + assert.match(refused.stderr, /--replay with a control arm runs an A\/B; pass --no-control to combine it with --weights/); + assert.equal(stub.calls() - before, 0, "nothing ran"); + const alone = await cli(["exp", "run", spec("wt", "sys A"), "--replay", recId, "--no-control", "--weights", sha, "--json"]); + assert.equal(alone.status, 0, alone.stderr); + const meta = JSON.parse(alone.stdout).experiment.metadata; + assert.deepEqual(meta.weights, { sha256: sha }); + assert.equal(meta.llm.control, "skipped"); + }); + it("the spec's llm field records too, and flags fail fast with a reason", async () => { const r = await cli(["exp", "run", spec("specrec", "sys A", { record: true }), "--json"]); assert.equal(r.status, 0, r.stderr); diff --git a/packages/coding.sh/test/mcp/server.test.ts b/packages/coding.sh/test/mcp/server.test.ts index 35c569cd..fa89d60b 100644 --- a/packages/coding.sh/test/mcp/server.test.ts +++ b/packages/coding.sh/test/mcp/server.test.ts @@ -2,10 +2,14 @@ // every tool against a seeded store, resources, and JSON-RPC error paths. import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { after, before, describe, it } from "node:test"; +import { openContext } from "../../src/core/context.ts"; +import { FileEvalStore } from "../../src/datasets/store.ts"; +import { CHARACTER_LIMIT } from "../../src/mcp/tools.ts"; import { schemaViolations } from "../../src/toolevals/jsonschema.ts"; import { connectMcp, resetEraCache } from "../../src/toolevals/mcp/client.ts"; import { MAIN, RawClient, SEED, cleanup, makeWorld, seed, type World } from "./helpers.ts"; @@ -322,6 +326,59 @@ describe("tools", () => { assert.match(await callError("experiment_compare", { baseline: SEED.expA, candidate: "exp-missing" }), /No experiment with id "exp-missing"/); }); + it("experiments_list and experiment_compare carry each run's weights stamp, and warn when both ran the same", async () => { + const sha = (label: string) => createHash("sha256").update(label).digest("hex"); + const [base, edited] = [sha("base checkpoint"), sha("edited checkpoint")]; + // A world of its own, so the lists other tests read keep their two experiments. + const ww = makeWorld(); + await seed(ww); + const ctx = await openContext({ env: ww.env, cwd: ww.root }); + try { + const store = new FileEvalStore({ env: ww.env, index: ctx.index }); + const [a, b] = await Promise.all([store.getExperiment(SEED.expA), store.getExperiment(SEED.expB)]); + await store.putExperiment({ ...a!, id: "exp-w-base", name: "w-base", metadata: { weights: { sha256: base } } }); + await store.putExperiment({ ...b!, id: "exp-w-edited", name: "w-edited", metadata: { weights: { sha256: edited } } }); + await store.putExperiment({ ...b!, id: "exp-w-twin", name: "w-twin", metadata: { weights: { sha256: base } } }); + // A full page of stamped rows, for the answer-size check below. + for (let i = 0; i < 100; i++) await store.putExperiment({ ...a!, id: `exp-many-${String(i).padStart(3, "0")}`, name: "many", metadata: { weights: { sha256: sha(`many ${i}`) } } }); + } finally { + await ctx.close(); + } + const wc = new RawClient(ww.env, ww.repo); + try { + await wc.initialize(); + // `call` checks every answer against the tool's outputSchema. + const row = async (name: string) => ((await call("experiments_list", { name }, wc)).experiments as { id: string; weights?: { sha256: string } }[])[0]!; + assert.deepEqual([(await row("w-base")).id, (await row("w-base")).weights], ["exp-w-base", { sha256: base }]); + assert.deepEqual((await row("w-edited")).weights, { sha256: edited }); + const plain = await row("baseline"); + assert.equal(plain.id, SEED.expA); + assert.equal(Object.hasOwn(plain, "weights"), false, "a run without a stamp has no weights key"); + + const many = await wc.call("experiments_list", { name: "many", limit: 100 }); + assert.ok(many.content[0]!.text.length <= CHARACTER_LIMIT, `100 stamped rows are ${many.content[0]!.text.length} chars`); + const page = many.structuredContent as { experiments: { weights?: { sha256: string } }[]; limitUsed?: number; nextCursor?: string }; + assert.ok(page.experiments.length > 0 && page.experiments.every((r) => /^[0-9a-f]{64}$/.test(r.weights?.sha256 ?? "")), "every row keeps its full stamp"); + assert.ok(page.experiments.length === 100 || page.nextCursor, "rows that did not fit are behind the cursor"); + + const cmp = await call("experiment_compare", { baseline: "exp-w-base", candidate: "exp-w-edited" }, wc); + assert.deepEqual([(cmp.baseline as { weights?: unknown }).weights, (cmp.candidate as { weights?: unknown }).weights], [{ sha256: base }, { sha256: edited }]); + assert.equal((cmp.warnings as string[]).some((x) => /weights/.test(x)), false, "different stamps: nothing to warn about"); + + const twin = await call("experiment_compare", { baseline: "exp-w-base", candidate: "exp-w-twin" }, wc); + assert.equal((twin.warnings as string[])[0], `A and B ran the same weights (${base.slice(0, 12)}): B − A is not a difference between two models`); + const oneSided = await call("experiment_compare", { baseline: SEED.expA, candidate: "exp-w-edited" }, wc); + assert.equal(Object.hasOwn(oneSided.baseline as object, "weights"), false); + assert.deepEqual((oneSided.candidate as { weights?: unknown }).weights, { sha256: edited }); + assert.match((oneSided.warnings as string[])[0]!, /^only B records which weights it ran \([0-9a-f]{12}\); A was run without --weights/); + const neither = await call("experiment_compare", { baseline: SEED.expA, candidate: SEED.expB }, wc); + assert.equal(Object.hasOwn(neither.baseline as object, "weights"), false); + assert.equal((neither.warnings as string[]).some((x) => /weights/.test(x)), false); + } finally { + await wc.close(); + } + }); + it("exp_session_note and exp_session_summary keep the experiment session the CLI reads", async () => { const cli = (...args: string[]) => { const r = spawnSync(process.execPath, ["--no-warnings", CLI_MAIN, ...args], { env: w.env, cwd: w.repo, encoding: "utf8" }); diff --git a/packages/coding.sh/test/mcp/toolevals.test.ts b/packages/coding.sh/test/mcp/toolevals.test.ts index 3afb152d..5e4ca1ec 100644 --- a/packages/coding.sh/test/mcp/toolevals.test.ts +++ b/packages/coding.sh/test/mcp/toolevals.test.ts @@ -73,7 +73,9 @@ describe("tool evals against the coding.sh MCP server", () => { // tree was one 679 KB reply. // limitUsed in the output schema of the 21 tools that can return it, and session_replay's paging // fields (treeTotal, changesTotal, filesTotal, unchanged, truncated, note), add about 0.95 KB, 0.1 KB of it descriptions. - assert.ok(s.surfaceBytes < 43_900, `surface is ${s.surfaceBytes} bytes`); + // weights (the checkpoint a run recorded with exp run --weights) on experiments_list rows and on + // both sides of experiment_compare adds about 0.2 KB of output schema and 23 bytes of description. + assert.ok(s.surfaceBytes < 44_100, `surface is ${s.surfaceBytes} bytes`); assert.ok(s.descriptionBytes < 11_850, `descriptions are ${s.descriptionBytes} bytes`); for (const it of s.items) assert.ok(it.bytes < 4_000, `${it.name} is ${it.bytes} bytes`); }); diff --git a/packages/coding.sh/test/viewer/server.test.ts b/packages/coding.sh/test/viewer/server.test.ts index a8a21976..05bbb49d 100644 --- a/packages/coding.sh/test/viewer/server.test.ts +++ b/packages/coding.sh/test/viewer/server.test.ts @@ -622,6 +622,24 @@ describe("viewer server: experiment compare", () => { assert.ok(c.diff.lo <= c.diff.mean && c.diff.mean <= c.diff.hi); }); + test("each side carries its weights stamp, and the same stamp twice is a warning", async () => { + const [x, y] = ["ab12".repeat(16), "cd34".repeat(16)]; + await index.putExperiment(experimentResult("w-base", { metadata: { weights: { sha256: x } } })); + await index.putExperiment(experimentResult("w-edit", { metadata: { weights: { sha256: y } } })); + await index.putExperiment(experimentResult("w-twin", { metadata: { weights: { sha256: x } } })); + const cmp = async (a: string, b: string) => (await req(v, `/api/compare?a=${a}&b=${b}`)).json(); + const differ = await cmp("w-base", "w-edit"); + assert.deepEqual([differ.a.weights, differ.b.weights, differ.weightsWarning], [{ sha256: x }, { sha256: y }, undefined]); + assert.match((await cmp("w-base", "w-twin")).weightsWarning, /^A and B ran the same weights \(ab12ab12ab12\)/); + const oneSided = await cmp("base", "w-edit"); + assert.equal("weights" in oneSided.a, false); + assert.match(oneSided.weightsWarning, /^only B records which weights it ran \(cd34cd34cd34\)/); + const neither = await cmp("base", "cand"); + assert.deepEqual(["weights" in neither.a, "weights" in neither.b, "weightsWarning" in neither], [false, false, false]); + // The experiment itself is served with its metadata, which the experiment page reads the stamp from. + assert.deepEqual((await req(v, "/api/experiments/w-base")).json().metadata, { weights: { sha256: x } }); + }); + test("rejects bad ids and missing experiments", async () => { assert.equal((await req(v, "/api/compare?a=base")).status, 400); assert.equal((await req(v, "/api/compare?a=..%2Fx&b=base")).status, 400); diff --git a/packages/coding.sh/test/viewer/ui-browser.test.ts b/packages/coding.sh/test/viewer/ui-browser.test.ts index 4f0e55c2..cd43d2fa 100644 --- a/packages/coding.sh/test/viewer/ui-browser.test.ts +++ b/packages/coding.sh/test/viewer/ui-browser.test.ts @@ -628,6 +628,9 @@ describe("viewer UI: timeline, inspector and compare", { skip: chrome ? false : const WORK = "w-1"; const API = "api-1"; const CHEAP = "cheap-1"; + // Two checkpoints' SHA-256s: any 64 hex characters. + const SHA_BASE = "ab12".repeat(16); + const SHA_EDIT = "cd34".repeat(16); const STEPS = "document.querySelectorAll('#steps .st')"; const selected = "document.querySelector('#steps .st[aria-selected=true]')"; const stepWhere = (re: string) => `[...${STEPS}].find((t) => new RegExp(${JSON.stringify(re)}).test(t.querySelector('.lbl').textContent))`; @@ -676,9 +679,15 @@ describe("viewer UI: timeline, inspector and compare", { skip: chrome ? false : ], }); const exps = [cand, base]; + // Stamped with `exp run --weights`. Reached by id only, so the list stays at two rows. + const stamped = [ + experimentResult("exp-w-base", { name: "base", metadata: { weights: { sha256: SHA_BASE } } }), + experimentResult("exp-w-edit", { name: "edited", metadata: { weights: { sha256: SHA_EDIT } } }), + experimentResult("exp-w-twin", { name: "twin", metadata: { weights: { sha256: SHA_BASE } } }), + ]; v = await startViewer({ index, blobs: new FakeBlobs(), port: 0, updateStatus: async () => ({ disabled: "test" }) as any, - experiments: { listExperiments: async () => exps, getExperiment: async (id) => exps.find((e) => e.id === id) }, + experiments: { listExperiments: async () => exps, getExperiment: async (id) => [...exps, ...stamped].find((e) => e.id === id) }, }); browser = await launchBrowser(chrome!); }); @@ -852,6 +861,44 @@ describe("viewer UI: timeline, inspector and compare", { skip: chrome ? false : await page.close(); } }); + + test("weights stamp: the experiment page and both compare sides show it; the same stamp twice is a warning", async () => { + const stamps = "[...document.querySelectorAll('.vs .side .weights')].map((e) => [e.textContent, e.title])"; + const notes = "[...document.querySelectorAll('.verdict .alert.note span')].map((e) => e.textContent)"; + const compare = async (a: string, b: string): Promise<{ stamps: [string, string][]; notes: string[] }> => { + await page.eval(`location.hash = "#/experiments/${a}/vs/${b}"`); + await page.waitFor(`document.querySelector('.vs .side.a a.pct')?.getAttribute('href') === "#/experiments/${a}" && document.querySelector('.vs .side.b a.pct')?.getAttribute('href') === "#/experiments/${b}"`, `compare ${a} with ${b}`); + return page.eval(`({ stamps: ${stamps}, notes: ${notes} })`); + }; + await open("#/experiments/exp-w-base"); + try { + const fact = await page.waitFor("(() => { const e = document.querySelector('.exp-top .fact-line .weights'); return e && [e.textContent, e.title]; })()", "the stamp on the experiment page"); + assert.equal(fact[0], "weights " + SHA_BASE.slice(0, 12)); + assert.ok(fact[1].endsWith(SHA_BASE), "the title carries the whole SHA-256"); + + const differ = await compare("exp-w-base", "exp-w-edit"); + assert.deepEqual(differ.stamps.map((x) => x[0]), ["weights " + SHA_BASE.slice(0, 12), "weights " + SHA_EDIT.slice(0, 12)]); + assert.ok(differ.stamps[1]![1].endsWith(SHA_EDIT)); + assert.equal(differ.notes.some((t) => /weights/.test(t)), false, "two different stamps need no warning"); + + const same = await compare("exp-w-base", "exp-w-twin"); + assert.equal(same.notes[0], `A and B ran the same weights (${SHA_BASE.slice(0, 12)}): B − A is not a difference between two models`); + + const oneSided = await compare("exp-base", "exp-w-edit"); + assert.deepEqual(oneSided.stamps.map((x) => x[0]), ["weights not recorded", "weights " + SHA_EDIT.slice(0, 12)]); + assert.match(oneSided.notes[0]!, /^only B records which weights it ran/); + + const neither = await compare("exp-base", "exp-cand"); + assert.deepEqual(neither.stamps, [], "no stamp on either side: nothing about weights"); + assert.equal(neither.notes.some((t) => /weights/.test(t)), false); + + await page.eval(`location.hash = "#/experiments/exp-base"`); + await page.waitFor("document.querySelector('.exp-top h1')?.getAttribute('title') === 'exp-base'", "an unstamped experiment page"); + assert.equal(await page.eval("document.querySelector('.exp-top .fact-line .weights')"), null); + } finally { + await page.close(); + } + }); }); describe("viewer UI: benchmarks", { skip: chrome ? false : "no Chrome/Chromium found (set CODING_SH_TEST_CHROME)" }, () => { From 05688cc98bc4dbf65e9dab760a3a6473f11b6443 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:14:12 -0600 Subject: [PATCH 08/17] fix(evals): exp metrics prints a small non-zero value with its digits, not 0.000 A count or score below 0.0005 (a gain per edited weight of 0.0001) printed as 0.000 and read as zero. It now prints two significant digits; values that three decimals already show are unchanged. --- CHANGELOG.md | 1 + packages/coding.sh/src/experiments/metrics.ts | 6 +++++- .../coding.sh/test/experiments/metrics.test.ts | 16 ++++++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e50ee9a..320f447a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ grouped by area, with links to the docs for the details. ### Fixed +- **Evals.** `exp metrics` prints a count or score below 0.0005 with two significant digits (`0.0001`) instead of `0.000`, so a small value such as a gain per edited weight no longer reads as zero. [Metric A/B](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#metric-ab-ms-bytes) - **MCP.** A request cancelled with `notifications/cancelled` gets no reply (a cancel for `initialize` is ignored), a cursor on `tools/list`, `prompts/list` or `resources/templates/list` is an error, a request with a null id gets an error instead of no answer (a null-id `notifications/` message is still a notification), and a 2025-03-26 session accepts batches. [MCP server](https://coding.sh/docs/mcp#protocol) - **MCP.** Tool answers stay under 25,000 characters: a longer page has fewer rows and `limitUsed`, and a tool with a `cursor` also returns `nextCursor`. `session_replay` pages its files and cuts text and lists only when an answer would not fit, saying so with `truncated` (a range was 5 million characters). [MCP server](https://coding.sh/docs/mcp#tools) - **MCP.** `eval_init` and `score_add` are marked destructive, since both can replace stored data, and `score_list`'s error names its `since` argument instead of the CLI's `--since`. [MCP server](https://coding.sh/docs/mcp#tools) diff --git a/packages/coding.sh/src/experiments/metrics.ts b/packages/coding.sh/src/experiments/metrics.ts index 5d7c516e..8b140c27 100644 --- a/packages/coding.sh/src/experiments/metrics.ts +++ b/packages/coding.sh/src/experiments/metrics.ts @@ -348,7 +348,11 @@ function fmt(v: number, unit: MetricUnit, signed = false): string { return `${sign}${Math.round(a)}B`; } if (Number.isInteger(a)) return `${sign}${a}`; - return `${sign}${Math.abs(a) < 1 ? a.toFixed(3) : a.toFixed(1)}`; + if (Math.abs(a) >= 1) return `${sign}${a.toFixed(1)}`; + // Below 1: three decimals, or two significant digits where three decimals would read as zero + // (a gain per edited weight of 0.0001 is not 0.000). + const fixed = a.toFixed(3); + return `${sign}${Number(fixed) === 0 ? String(Number(a.toPrecision(2))) : fixed}`; } export function renderMetrics(r: MetricExpResult, opts: RenderOpts): string { diff --git a/packages/coding.sh/test/experiments/metrics.test.ts b/packages/coding.sh/test/experiments/metrics.test.ts index 62a4fdfa..2c69e9f5 100644 --- a/packages/coding.sh/test/experiments/metrics.test.ts +++ b/packages/coding.sh/test/experiments/metrics.test.ts @@ -117,6 +117,22 @@ describe("runMetrics", () => { assert.match(text, /js_bytes .*fixed.* worse/); }); + it("a small non-zero value keeps its digits: 0.0001 does not print as 0.000", async () => { + // A gain per edited weight: 29.45 points over 1023 weights, against 0.0001. + const r = await runMetrics({ + arms: [fnArm("small", () => ({ gain_per_weight: 29.45 / 1023, tiny: -0.00004, zero: 0, half: 0.5 })), fnArm("wide", () => ({ gain_per_weight: 0.0001, tiny: 0.00000012, zero: 0, half: 0.25 }))], + pairs: 3, + seed: 1, + higher: ["gain_per_weight"], + }); + const row = (name: string) => renderMetrics(r, { color: false }).split("\n").find((l) => l.startsWith(name))!.split(/\s{2,}/); + // metric, A, B, delta: three decimals where they say something, two significant digits below that. + assert.deepEqual(row("gain_per_weight").slice(0, 4), ["gain_per_weight ↑", "0.029", "0.0001", "-0.029 (-99.7%)"]); + assert.deepEqual(row("tiny").slice(1, 3), ["-0.00004", "1.2e-7"]); + assert.deepEqual(row("zero").slice(1, 3), ["0", "0"]); + assert.deepEqual(row("half").slice(1, 4), ["0.500", "0.250", "-0.250 (-50.0%)"]); + }); + it("--higher flips the direction", async () => { const r = await runMetrics({ arms: [fnArm("a", () => ({ score: 70 })), fnArm("b", () => ({ score: 80 }))], From 6a2c4b8b4e9d1f6f1b603da47cf8366674e5fc8f Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:15:15 -0600 Subject: [PATCH 09/17] docs(evals): how to report an edit's size and gain to exp metrics The Metric A/B guide names three keys for an arm that reports a model edit (edited_weights, gain_pp, gain_per_weight), says the two gain keys need --higher, and shows the table. A CLI test runs a stub arm that prints them and checks the three rows and their direction. --- docs/guides/evals.md | 36 ++++++++++++++++ .../test/experiments/metrics.test.ts | 43 +++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/docs/guides/evals.md b/docs/guides/evals.md index 2427c28d..dc8e1fc2 100644 --- a/docs/guides/evals.md +++ b/docs/guides/evals.md @@ -1013,6 +1013,42 @@ the same, like the byte count of a fixed build, has no noise: a non-zero delta i `--json` prints the whole result, runs included. `--fail-on-worse` exits 1 when any metric got worse, for CI gates. scripts/perf/ uses it to gate the coding.sh site on Lighthouse metrics. +### An edit's size and gain + +To weigh a model edit by what it cost, make each arm one edit, and have its command print how +many weights the edit changed and what it gained: + +```json +{"edited_weights": 1023, "gain_pp": 29.45, "gain_per_weight": 0.0288} +``` + +- `edited_weights`: the number of weights the edit changed. Lower is better, which is the default. +- `gain_pp`: the gain over the unedited base, in percentage points. +- `gain_per_weight`: `gain_pp` divided by `edited_weights`. + +Your command computes the three numbers. coding.sh treats them like any other metric: it pairs +the runs and compares them. Name the two gain keys in `--higher`. Without it every metric is +lower-is-better, and the edit with the larger gain reads as `worse`: + +```sh +coding.sh exp metrics --arm small='python3 edit_report.py small' \ + --arm wide='python3 edit_report.py wide' --higher 'gain_*' +``` + +``` +metric small wide Δ 95% CI p (adj) verdict +----------------- ----- ------- ------------------- ------ ------- ------- +edited_weights 1023 294912 +293889 (+28728.2%) fixed — worse +gain_per_weight ↑ 0.029 0.00011 -0.029 (-99.6%) fixed — worse +gain_pp ↑ 29.4 31.2 +1.8 (+5.9%) fixed — better +``` + +`↑` marks a metric where higher is better. Here the wide edit changes more weights, gains 1.8 +points more, and gains far less per weight. None of the three names ends in `_ms` or `_bytes`, so +they print as plain numbers, and a value below 0.0005 keeps two significant digits (`0.00011`) +and does not print as `0.000`. A command that prints the same numbers on every run has no noise: +the CI column reads `fixed`, and a non-zero delta is a change. + ## Scores A score is one typed value on a session, an event, a dataset case or an eval run: a number, true diff --git a/packages/coding.sh/test/experiments/metrics.test.ts b/packages/coding.sh/test/experiments/metrics.test.ts index 2c69e9f5..0f974975 100644 --- a/packages/coding.sh/test/experiments/metrics.test.ts +++ b/packages/coding.sh/test/experiments/metrics.test.ts @@ -269,6 +269,49 @@ describe("coding.sh exp metrics", () => { assert.match(aa.stdout, /0 better · 0 worse · 2 same · 0 inconclusive/); }); + it("an arm that reports an edit's size and gain: three rows, and the gain keys read higher-is-better only with --higher", () => { + // Prints the keys the guide names for a model edit: how many weights it changed, the points it + // gained, and the gain per weight. argv: . + writeFileSync( + join(root, "edit.mjs"), + `process.stdin.resume().on("end", () => { + const [n, gain] = process.argv.slice(2).map(Number); + console.log(JSON.stringify({ edited_weights: n, gain_pp: gain, gain_per_weight: gain / n })); + });`, + ); + const args = ["exp", "metrics", "--arm", "small=node edit.mjs 1023 29.45", "--arm", "wide=node edit.mjs 294912 31.2", "--pairs", "3", "--seed", "1"]; + type Row = { name: string; better: string; verdict: string }; + const rows = (extra: string[]) => { + const r = cli([...args, ...extra, "--json"]); + assert.equal(r.status, 0, r.stderr); + return (JSON.parse(r.stdout) as { experiment: { metrics: Row[] } }).experiment.metrics.map((m) => [m.name, m.better, m.verdict]); + }; + // The wide edit changes more weights (worse: lower is better), gains more points, and gains less per weight. + const named = rows(["--higher", "gain_pp,gain_per_weight"]); + assert.deepEqual(named, [ + ["edited_weights", "lower", "worse"], + ["gain_per_weight", "higher", "worse"], + ["gain_pp", "higher", "better"], + ]); + assert.deepEqual(rows(["--higher", "gain_*"]), named, "the * wildcard names both gain keys"); + // Without --higher every key is lower-is-better, so both gains read backwards. + assert.deepEqual(rows([]), [ + ["edited_weights", "lower", "worse"], + ["gain_per_weight", "lower", "better"], + ["gain_pp", "lower", "worse"], + ]); + const text = cli([...args, "--higher", "gain_*"]); + assert.equal(text.status, 0, text.stderr); + // metric, A, B, delta, then CI, p and verdict. + const row = (name: string) => text.stdout.split("\n").find((l) => l.startsWith(name))!.split(/\s{2,}/); + assert.deepEqual(row("edited_weights").slice(0, 4), ["edited_weights", "1023", "294912", "+293889 (+28728.2%)"]); + assert.deepEqual(row("gain_pp").slice(0, 4), ["gain_pp ↑", "29.4", "31.2", "+1.8 (+5.9%)"]); + // 31.2 / 294912 is 0.000106: it prints with its digits, not as 0.000. + assert.deepEqual(row("gain_per_weight").slice(0, 4), ["gain_per_weight ↑", "0.029", "0.00011", "-0.029 (-99.6%)"]); + for (const name of ["edited_weights", "gain_pp", "gain_per_weight"]) assert.equal(row(name)[4], "fixed", "the same number every run has no noise"); + assert.deepEqual(["edited_weights", "gain_pp", "gain_per_weight"].map((n) => row(n).at(-1)), ["worse", "better", "worse"]); + }); + it("usage errors exit 2 and a broken arm exits 1 with the next step", () => { const one = cli(["exp", "metrics", "--arm", "a=node bench.mjs"]); assert.equal(one.status, 2); From 0153b03dfb6882b147fd33765809bfb080fe69fb Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:41:37 -0600 Subject: [PATCH 10/17] fix(stores): the Postgres and ClickHouse stores filter by weights.edits and weights.changed The two --where keys had SQL only in the SQLite store, so a query that used them was refused on the other two. --- packages/store-clickhouse/src/where.ts | 2 ++ packages/store-postgres/src/where.ts | 2 ++ 2 files changed, 4 insertions(+) diff --git a/packages/store-clickhouse/src/where.ts b/packages/store-clickhouse/src/where.ts index 775faf26..b38d7ddf 100644 --- a/packages/store-clickhouse/src/where.ts +++ b/packages/store-clickhouse/src/where.ts @@ -72,6 +72,8 @@ export const KEY_SQL: Record = { skill: ["f"], hook: ["f"], "hook.tokens": [stat("hookContext.tokens")], + "weights.edits": [stat("weightEdits.events")], + "weights.changed": [stat("weightEdits.weightsChanged")], since: ["started_at"], until: ["started_at"], duration: ["(ended_at - started_at)"], diff --git a/packages/store-postgres/src/where.ts b/packages/store-postgres/src/where.ts index 8e7439e9..e5baab2e 100644 --- a/packages/store-postgres/src/where.ts +++ b/packages/store-postgres/src/where.ts @@ -62,6 +62,8 @@ export const KEY_SQL: Record = { skill: ["f.v"], hook: ["f.v"], "hook.tokens": [stat("hookContext,tokens")], + "weights.edits": [stat("weightEdits,events")], + "weights.changed": [stat("weightEdits,weightsChanged")], since: ["s.started_at"], until: ["s.started_at"], duration: ["(s.ended_at - s.started_at)"], From 1474141bc0c232c5b2a35a86d5b0b51b7a5b7167 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:41:37 -0600 Subject: [PATCH 11/17] docs: what a weight edit is, in concepts, and how one arrives over OTLP, in adapters --- docs/ADAPTERS.md | 3 +++ docs/concepts.md | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/docs/ADAPTERS.md b/docs/ADAPTERS.md index 1d98d242..e6ed7ce9 100644 --- a/docs/ADAPTERS.md +++ b/docs/ADAPTERS.md @@ -262,6 +262,9 @@ Apps in production don't use an adapter. They send data over the network to `cod | Vercel AI Gateway trace drain | `POST /v1/traces` | `src/otel/map.ts` plus the gateway enricher | `otel…` | | Claude Code, Codex and Gemini CLI running remotely with OTel logs on | `POST /v1/logs` | `src/otel/map.ts` log events | `otel…` | +- A span or a log record named `weights.edit` with `coding_sh.weights.*` attributes is stored as + the weight edit event the SDKs send (`notification`, `payload.kind: "weights.edit"`); the + attributes are listed in [reference/api.md](reference/api.md). - Hooks, transcript backfill and `coding.sh run` stay local. Serve does not take spool lines. - The ingest key picks the project, and session ids are stored as `/`. - SDK sessions that never send `session.end` have no `endedAt` yet; see the open item in diff --git a/docs/concepts.md b/docs/concepts.md index ead8591c..025b1cdb 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -62,6 +62,18 @@ test ran. A flag with a colon, such as `skill:pdf`, is a tag, not a failure. Every flag and what sets it: [Flags and derails](https://coding.sh/docs/query#flags-and-derails). +### Weight edit + +A change a script made to a model's weights, recorded as one event: the tensor's name and shape, +how many weights changed, the size of the change, and the hash of the tensor before and after. A +session that recorded one has the `weights:edited` tag and counts its edits in `stats.weightEdits`. + +- In the viewer: each edit is a row on the session page, and the header counts them. +- In the CLI: `coding.sh sessions --flag weights:edited` lists the sessions, and + `coding.sh show ` prints one line per edit. +- Recorded by the Python SDK's `coding_sh.weights_edit()`, or by any OpenTelemetry SDK as a span + named `weights.edit`. The fields are in the [API reference](https://coding.sh/docs/api). + ### Cost What a session's model calls cost, from their token counts and each model's price. A `~` marks an From e87ba688817c47991320f6ef2b7b1683c8b7623b Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:43:09 -0600 Subject: [PATCH 12/17] fix(sdk): weights_edit() takes a 0-d tensor as the number it holds l2=delta.norm() in torch is a 0-d tensor, not a Python number, so the edit was refused and dropped with a warning while the docs said torch scalars count. A value with ndim 0 and item() is now read through item(); a tensor of more than one value is still refused. --- CHANGELOG.md | 1 + packages/sdk-python/src/coding_sh/_weights.py | 15 ++++++++ packages/sdk-python/tests/test_weights.py | 35 +++++++++++++++++++ 3 files changed, 51 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d0c7ba07..ce474f12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,7 @@ grouped by area, with links to the docs for the details. ### Fixed +- **SDK.** `coding_sh.weights_edit()` reads a 0-d tensor or array as the number it holds, so `l2=delta.norm()` records the edit. Before, a torch scalar was refused and the edit was dropped with a warning, though the docs said torch scalars count as numbers. [Docs](https://coding.sh/docs/api) - **MCP.** A request cancelled with `notifications/cancelled` gets no reply (a cancel for `initialize` is ignored), a cursor on `tools/list`, `prompts/list` or `resources/templates/list` is an error, a request with a null id gets an error instead of no answer (a null-id `notifications/` message is still a notification), and a 2025-03-26 session accepts batches. [MCP server](https://coding.sh/docs/mcp#protocol) - **MCP.** Tool answers stay under 25,000 characters: a longer page has fewer rows and `limitUsed`, and a tool with a `cursor` also returns `nextCursor`. `session_replay` pages its files and cuts text and lists only when an answer would not fit, saying so with `truncated` (a range was 5 million characters). [MCP server](https://coding.sh/docs/mcp#tools) - **MCP.** `eval_init` and `score_add` are marked destructive, since both can replace stored data, and `score_list`'s error names its `since` argument instead of the CLI's `--since`. [MCP server](https://coding.sh/docs/mcp#tools) diff --git a/packages/sdk-python/src/coding_sh/_weights.py b/packages/sdk-python/src/coding_sh/_weights.py index 2ba6b364..abe17a71 100644 --- a/packages/sdk-python/src/coding_sh/_weights.py +++ b/packages/sdk-python/src/coding_sh/_weights.py @@ -33,13 +33,28 @@ MAX_DIMS = 16 +def _scalar(v: Any) -> Any: + """A 0-d tensor or array (what torch's `.norm()` and numpy's `np.array(x)` give) as the number + it holds. Anything else is returned unchanged, for the checks below to accept or refuse.""" + if isinstance(v, numbers.Number): + return v + try: + if getattr(v, "ndim", None) == 0 and callable(getattr(v, "item", None)): + return v.item() + except Exception: + pass + return v + + def _count(v: Any, what: str) -> int: + v = _scalar(v) if isinstance(v, bool) or not isinstance(v, numbers.Integral) or operator.index(v) < 0: raise ValueError("%s must be a non-negative integer, got %r" % (what, v)) return operator.index(v) def _norm(v: Any, what: str) -> float: + v = _scalar(v) if isinstance(v, bool) or not isinstance(v, numbers.Real): raise ValueError("%s must be a finite number >= 0, got %r" % (what, v)) f = float(v) diff --git a/packages/sdk-python/tests/test_weights.py b/packages/sdk-python/tests/test_weights.py index de3ea499..12af820b 100644 --- a/packages/sdk-python/tests/test_weights.py +++ b/packages/sdk-python/tests/test_weights.py @@ -86,6 +86,22 @@ def __repr__(self) -> str: numbers.Real.register(FakeFloat) +class FakeTensor: + """A tensor the way torch's are: no numbers ABC, a number only through ndim == 0 and item().""" + + def __init__(self, v: Any, ndim: int = 0) -> None: + self.v = v + self.ndim = ndim + + def item(self) -> Any: + if self.ndim != 0: + raise ValueError("only one element tensors can be converted to Python scalars") + return self.v + + def __repr__(self) -> str: + return "tensor(%r)" % (self.v,) + + def _edit(**kw: Any) -> bool: args: Dict[str, Any] = dict( tensor="layers.3.mlp.down_proj.weight", @@ -206,6 +222,25 @@ def test_accepts_numpy_like_numbers(self) -> None: self.assertIs(_edit(changed=FakeInt(-1)), False) self.assertIs(_edit(l2=FakeFloat(float("nan"))), False) + def test_accepts_zero_dim_tensors(self) -> None: + # What delta.norm(), delta.abs().max() and (delta != 0).sum() return in torch. + start_sdk(self.server) + self.assertTrue(_edit(changed=FakeTensor(12), shape=[FakeTensor(4), 8], l2=FakeTensor(0.25), max_abs=FakeTensor(0.1))) + (e,) = by_kind(collect(self.server), "notification") + p = e["payload"] + self.assertEqual((p["changed"], p["shape"], p["l2"], p["maxAbs"]), (12, [4, 8], 0.25, 0.1)) + for v in (p["changed"], p["l2"], p["maxAbs"]): + self.assertIn(type(v), (int, float)) + + def test_refuses_tensors_that_are_not_one_number(self) -> None: + start_sdk(self.server) + self.assertIs(_edit(l2=FakeTensor([0.25, 0.5], ndim=1)), False) + self.assertIs(_edit(changed=FakeTensor(1.5)), False) + self.assertIs(_edit(l2=FakeTensor(float("inf"))), False) + self.assertIs(_edit(max_abs=FakeTensor("0.1")), False) + self.assertIs(_edit(changed=FakeTensor(True)), False) + self.assertEqual(by_kind(collect(self.server), "notification"), []) + def test_warns_once_per_process(self) -> None: start_sdk(self.server) with self.assertLogs("coding_sh", level="WARNING") as cm: From 23e6bddc2ca226112197a182d8bf433fbeeec2e7 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:50:10 -0600 Subject: [PATCH 13/17] docs: the weight edit changelog lines fit in 60 words --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae8c7170..e4633124 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,8 +16,8 @@ grouped by area, with links to the docs for the details. - **Lint.** New rules: vendor-literal, site-voice, universal-transition, reduced-motion, global-install and personal-data. `slop.mjs --strict-allowlist` fails on allowlist entries that no longer match anything. [Contributing](https://github.com/jackulau/coding.sh/blob/main/CONTRIBUTING.md) - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) -- **Sessions.** A session that recorded weight edits gets `stats.weightEdits` (edits, distinct tensors, weights changed) and the `weights:edited` tag, and is listed even when it holds nothing else. `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit instead of a blank notification (MCP dropped them), and `coding.sh sessions --where 'weights.changed>1000'` filters by size. Stored sessions are folded again. [Docs](https://coding.sh/docs/api) -- **OpenTelemetry.** A span or a log record named `weights.edit` with `coding_sh.weights.*` attributes is read as a weight edit, and the `otlp` sink sends one `weights.edit` span per edit (`[serve.forward]` does too for edits posted to `/v1/ingest`), so an edit reads the same whether it arrived from an SDK or over OTLP. The `invoke_agent` span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. Metadata mode keeps the counts and the norms only. [Docs](https://coding.sh/docs/api) +- **Sessions.** A session that recorded weight edits gets `stats.weightEdits` (edits, distinct tensors, weights changed) and the `weights:edited` tag, and is listed even when it holds nothing else. `coding.sh show`, the viewer and MCP `session_get` print one line per edit, and `--where 'weights.changed>1000'` filters by size. Stored sessions are folded again. [Docs](https://coding.sh/docs/api) +- **OpenTelemetry.** A span or log record named `weights.edit` with `coding_sh.weights.*` attributes is read as a weight edit, and the `otlp` sink sends one `weights.edit` span per edit (so does `[serve.forward]`, for edits posted to `/v1/ingest`). The `invoke_agent` span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. Metadata mode keeps only the counts and norms. [Docs](https://coding.sh/docs/api) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) From a20f34c3dbf4575256beec923c639afa5122a7c1 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:42:06 -0600 Subject: [PATCH 14/17] docs: the weight edit notes name both SDKs --- docs/concepts.md | 5 +++-- docs/reference/api.md | 8 ++++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 025b1cdb..0130a040 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -71,8 +71,9 @@ session that recorded one has the `weights:edited` tag and counts its edits in ` - In the viewer: each edit is a row on the session page, and the header counts them. - In the CLI: `coding.sh sessions --flag weights:edited` lists the sessions, and `coding.sh show ` prints one line per edit. -- Recorded by the Python SDK's `coding_sh.weights_edit()`, or by any OpenTelemetry SDK as a span - named `weights.edit`. The fields are in the [API reference](https://coding.sh/docs/api). +- Recorded by the Python SDK's `coding_sh.weights_edit()`, the TypeScript SDK's `weightsEdit()`, + or any OpenTelemetry SDK as a span named `weights.edit`. The fields are in the + [API reference](https://coding.sh/docs/api). ### Cost diff --git a/docs/reference/api.md b/docs/reference/api.md index c3ce7618..3516064a 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -734,11 +734,11 @@ One more `notification` kind records weight edits. The Python SDK's `coding_sh.w - OpenTelemetry in: a span or a log record named `weights.edit` sent to `/v1/traces` or `/v1/logs` is read as the same event. Its fields are the attributes `coding_sh.weights.tensor`, `.shape` (an array of integers), `.changed`, `.l2`, `.max_abs`, `.base_sha256`, `.edited_sha256` and `.note`. A malformed attribute is left out and the edit still counts. - OpenTelemetry out: the `otlp` sink sends one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. `[serve.forward]` sends the same span for an edit posted to `/v1/ingest` and passes on a `weights.edit` span posted to `/v1/traces` as it came; an edit that arrived as a log record is not forwarded. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. -- The caller computes the hashes and norms. The SDK checks the values, drops an invalid edit with one warning, and never raises into the app. Numpy and torch scalars count as numbers. -- `weights_edit()` returns `True` only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. -- `coding_sh.weights_delta()` computes `shape`, `changed`, `l2` and `max_abs` for two small array-likes in pure Python. It is a helper you call on purpose, so it raises `ValueError` on bad input. +- The caller computes the hashes and norms. Each SDK checks the values, drops an invalid edit with one warning, and never raises into the app. In Python, numpy and torch scalars count as numbers. +- `weights_edit()` (Python) and `weightsEdit()` (TypeScript) return true only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. +- `coding_sh.weights_delta()` computes `shape`, `changed`, `l2` and `max_abs` for two small array-likes in pure Python. It is a helper you call on purpose, so it raises `ValueError` on bad input. The TypeScript SDK's `weightsDelta()` does the same for nested arrays and typed arrays, and throws a `TypeError` or a `RangeError`. - The server stores the payload without validating these fields, after redaction. -- A call outside `trace()` and `with_session()` makes a session of its own, so put a run of edits inside one of them. +- A call outside `trace()` and `with_session()` (`withSession()` in TypeScript) makes a session of its own, so put a run of edits inside one of them. A missing session, or one outside a scoped key's project, gets 404: From c552d6d2dc97f014d6ea4967f1c2f27161248d02 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:43:14 -0600 Subject: [PATCH 15/17] docs: say which tensors the Python weights helper reads as numbers --- docs/reference/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/api.md b/docs/reference/api.md index 3516064a..7c1489d1 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -734,7 +734,7 @@ One more `notification` kind records weight edits. The Python SDK's `coding_sh.w - OpenTelemetry in: a span or a log record named `weights.edit` sent to `/v1/traces` or `/v1/logs` is read as the same event. Its fields are the attributes `coding_sh.weights.tensor`, `.shape` (an array of integers), `.changed`, `.l2`, `.max_abs`, `.base_sha256`, `.edited_sha256` and `.note`. A malformed attribute is left out and the edit still counts. - OpenTelemetry out: the `otlp` sink sends one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. `[serve.forward]` sends the same span for an edit posted to `/v1/ingest` and passes on a `weights.edit` span posted to `/v1/traces` as it came; an edit that arrived as a log record is not forwarded. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. -- The caller computes the hashes and norms. Each SDK checks the values, drops an invalid edit with one warning, and never raises into the app. In Python, numpy and torch scalars count as numbers. +- The caller computes the hashes and norms. Each SDK checks the values, drops an invalid edit with one warning, and never raises into the app. In Python, numpy scalars and 0-d tensors (what torch's `.norm()` returns) count as numbers. - `weights_edit()` (Python) and `weightsEdit()` (TypeScript) return true only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. - `coding_sh.weights_delta()` computes `shape`, `changed`, `l2` and `max_abs` for two small array-likes in pure Python. It is a helper you call on purpose, so it raises `ValueError` on bad input. The TypeScript SDK's `weightsDelta()` does the same for nested arrays and typed arrays, and throws a `TypeError` or a `RangeError`. - The server stores the payload without validating these fields, after redaction. From 31e0826918bd1fae22309ec022060828b6ef03c9 Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 15:00:31 -0600 Subject: [PATCH 16/17] feat(evals): exp list shows which checkpoint each run used --- CHANGELOG.md | 2 +- docs/guides/evals.md | 3 ++- packages/coding.sh/src/cli/commands/exp.ts | 18 ++++++++++++------ .../test/experiments/weights-stamp.test.ts | 12 ++++++++++++ 4 files changed, 27 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 320f447a..7b6b3fa2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ grouped by area, with links to the docs for the details. - **Lint.** New rules: vendor-literal, site-voice, universal-transition, reduced-motion, global-install and personal-data. `slop.mjs --strict-allowlist` fails on allowlist entries that no longer match anything. [Contributing](https://github.com/jackulau/coding.sh/blob/main/CONTRIBUTING.md) - **Doctor.** `coding.sh doctor` counts recent model calls past a physical limit (speed, time to first token, duration) and names the session with the most. [Troubleshooting](https://github.com/jackulau/coding.sh/blob/main/docs/troubleshooting.md) - **SDK.** `coding_sh.weights_edit()` in the Python SDK records each weight edit a script makes as one `notification` event with `payload.kind: "weights.edit"`: tensor name, shape, weights changed, L2 and max-abs of the change, and the base and edited sha256. It adds no event kind, and `coding_sh.weights_delta()` computes the counts and norms for small arrays without numpy. [Docs](https://coding.sh/docs/api) -- **Evals.** `coding.sh exp run --weights ` records which checkpoint a run's task ran (the SHA-256 of the weights file, which you compute). `exp compare`, `exp show`, the viewer and the MCP tools show each side's stamp, and a comparison warns when both sides ran the same weights or only one side recorded them. [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) +- **Evals.** `coding.sh exp run --weights ` records which checkpoint a run's task ran (the SHA-256 of the weights file, which you compute). `exp compare`, `exp show`, `exp list`, the viewer and the MCP tools show the stamp, and a comparison warns when both sides ran the same weights or only one recorded them. [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **Evals.** `coding.sh exp compare A B --given ` compares only the cases where that grader passes in both runs, and prints how many it kept and dropped. With a `terminated` grader this separates "finished more answers" from "got more of the finished answers right". [Compare an edited model with its base](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#compare-an-edited-model-with-its-base) - **MCP.** In 2026-07-28 requests, list, read and discover results carry `ttlMs` and `cacheScope`: the tool, prompt and template lists are cached for an hour, `server/discover` for five minutes, and sessions are not cached. [MCP server](https://coding.sh/docs/mcp#protocol) diff --git a/docs/guides/evals.md b/docs/guides/evals.md index dc8e1fc2..be5aa727 100644 --- a/docs/guides/evals.md +++ b/docs/guides/evals.md @@ -807,7 +807,8 @@ as `comparison.a.weights.sha256` and `comparison.b.weights.sha256`. When both si stamp, the report warns that `B − A` is not a difference between two models. When only one side carries a stamp, it says so, and the other side reads `weights not recorded`. `exp show`, the `exp run` summary, the viewer and the MCP tools `experiments_list` and `experiment_compare` show -the same stamp. `exp run` without `--weights` stores no stamp, and a comparison of two such runs +the same stamp. `exp list` adds a `WEIGHTS` column when a listed run has one, and its `--json` +rows carry `weights.sha256`. `exp run` without `--weights` stores no stamp, and a comparison of two such runs prints nothing about weights. The first compare prints a paired delta for `terminated` and one for `correct`, each with its diff --git a/packages/coding.sh/src/cli/commands/exp.ts b/packages/coding.sh/src/cli/commands/exp.ts index 97e9d83f..22dea01b 100644 --- a/packages/coding.sh/src/cli/commands/exp.ts +++ b/packages/coding.sh/src/cli/commands/exp.ts @@ -44,7 +44,7 @@ import { sortableId } from "../../core/ids.ts"; import { jobConfig, taskFingerprint } from "../../jobs/config.ts"; import type { Job, JobKind } from "../../jobs/store.ts"; import { positiveInt, UsageError } from "../args.ts"; -import { parseWeights, shortWeights, weightsOf, withWeights } from "../../experiments/weights.ts"; +import { parseWeights, shortWeights, stampedWith, weightsOf, withWeights } from "../../experiments/weights.ts"; import { finishJob, infraStopNote, openJob, resumeField, resumeNote, retryFlag, statusCommand, trialTimeout } from "../jobs.ts"; import { gateFlags, jsonArg, list, num, shellQuote, str, withContext } from "../common.ts"; import { ago, CliError, plural } from "../output.ts"; @@ -1316,17 +1316,23 @@ export async function run(c: CommandContext): Promise { const limit = positiveInt(c.flags.limit, "limit") ?? 20; const all = await store.listExperiments({ limit }); if (out.json) { - out.data({ experiments: all.map((e) => ({ id: e.id, name: e.name, status: e.status ?? "completed", ...(isCheckpoint(e) ? { partial: true } : {}), dataset: e.dataset, trials: e.trials, runs: e.runs.length, createdAt: new Date(e.createdAt).toISOString(), summary: e.summary })) }); + // Each row carries the checkpoint its run recorded (exp run --weights); no key when it recorded none. + out.data({ experiments: all.map((e) => stampedWith({ id: e.id, name: e.name, status: e.status ?? "completed", ...(isCheckpoint(e) ? { partial: true } : {}), dataset: e.dataset, trials: e.trials, runs: e.runs.length, createdAt: new Date(e.createdAt).toISOString(), summary: e.summary }, weightsOf(e))) }); return 0; } if (!all.length) { out.note("No experiments"); return 0; } - out.table( - ["ID", "NAME", "DATASET", "RUNS", "CREATED"], - all.map((e) => [e.id, e.status === "cancelled" ? `${e.name} (${isCheckpoint(e) ? "partial" : "cancelled"})` : e.name, `${e.dataset.name}@${e.dataset.version}`, plural(e.runs.length, "run"), ago(e.createdAt)]), - ); + // The WEIGHTS column is there only when a listed run recorded its checkpoint. + const stamps = all.map((e) => weightsOf(e)); + const head = ["ID", "NAME", "DATASET", "RUNS", "CREATED"]; + const rows = all.map((e) => [e.id, e.status === "cancelled" ? `${e.name} (${isCheckpoint(e) ? "partial" : "cancelled"})` : e.name, `${e.dataset.name}@${e.dataset.version}`, plural(e.runs.length, "run"), ago(e.createdAt)]); + if (stamps.some((w) => w !== undefined)) { + head.push("WEIGHTS"); + rows.forEach((r, i) => r.push(stamps[i] ? shortWeights(stamps[i]) : "")); + } + out.table(head, rows); return 0; } } diff --git a/packages/coding.sh/test/experiments/weights-stamp.test.ts b/packages/coding.sh/test/experiments/weights-stamp.test.ts index af58fe71..dc991de6 100644 --- a/packages/coding.sh/test/experiments/weights-stamp.test.ts +++ b/packages/coding.sh/test/experiments/weights-stamp.test.ts @@ -193,6 +193,18 @@ process.stdout.write(JSON.stringify({answer:right?"ok":"wrong",finish_reason:don assert.doesNotMatch(ok(["exp", "show", "plain"]).stdout, /weights/); }); + it("exp list shows each run's stamp: short in a WEIGHTS column, whole in --json, and nothing for a run without one", () => { + const rows = (JSON.parse(ok(["exp", "list", "--json"]).stdout) as { experiments: { name: string; weights?: { sha256: string } }[] }).experiments; + const row = (name: string) => rows.find((r) => r.name === name)!; + assert.deepEqual(row("edited").weights, { sha256: EDITED }); + assert.deepEqual(row("base").weights, { sha256: BASE }); + assert.equal(Object.hasOwn(row("plain"), "weights"), false); + const text = ok(["exp", "list"]).stdout; + assert.match(text, /^ID\s+NAME\s+DATASET\s+RUNS\s+CREATED\s+WEIGHTS$/m); + assert.match(text, new RegExp(`^exp_\\w+\\s+edited\\s.*\\s${short(EDITED)}$`, "m")); + assert.match(text, /^exp_\w+\s+plain\s.*\sago\s*$/m, "a run without a stamp has an empty cell"); + }); + it("exp compare shows both stamps in the header: short in text, whole in --json", () => { const text = ok(["exp", "compare", "base", "edited"]).stdout; assert.match(text, new RegExp(`^A base exp_\\w+ {2}pass [\\d.]+% {2}weights ${short(BASE)}$`, "m")); From d3bad9e3f60836f828ffc74d4443b8e8102dcdef Mon Sep 17 00:00:00 2001 From: Jack <72348727+Jack-GitHub12@users.noreply.github.com> Date: Thu, 12 Nov 2026 16:08:23 -0600 Subject: [PATCH 17/17] docs(api): the weight edit notes fit the docs page's bullet budget --- docs/reference/api.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/reference/api.md b/docs/reference/api.md index 7c1489d1..a4cf1b16 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -729,14 +729,29 @@ One more `notification` kind records weight edits. The Python SDK's `coding_sh.w | `stats.weightEdits` (`events`, `tensors`, `weightsChanged`) | Weight edits the session recorded: how many, how many distinct tensor names, and their `changed` counts added up (a weight two edits changed counts twice). Absent when the session recorded none. `tensors` is absent when no edit kept its name | - A session with a weight edit gets the `weights:edited` tag. `coding.sh sessions --flag weights:edited` lists them, and `--where 'weights.changed>1000'` filters by size. -- `coding.sh show`, the viewer and the MCP server's `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. A shape of more than 16 dimensions is not shown, and the total is left out when the count does not fit in the shape. An edit with no readable field still gets a line. +- `coding.sh show`, the viewer and MCP `session_get` print one line per edit: the tensor and its shape, the weights changed out of the total, both norms and the first 8 characters of each hash. +- A shape of more than 16 dimensions is not shown, and the total is left out when the count does not fit in the shape. An edit with no readable field still gets a line. - Metadata mode: the SDK leaves out `note`. A server with `capture.mode = "metadata"` also drops `tensor`, `shape` and both hashes, and keeps `changed`, `l2` and `maxAbs`. -- OpenTelemetry in: a span or a log record named `weights.edit` sent to `/v1/traces` or `/v1/logs` is read as the same event. Its fields are the attributes `coding_sh.weights.tensor`, `.shape` (an array of integers), `.changed`, `.l2`, `.max_abs`, `.base_sha256`, `.edited_sha256` and `.note`. A malformed attribute is left out and the edit still counts. -- OpenTelemetry out: the `otlp` sink sends one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span, which carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. `[serve.forward]` sends the same span for an edit posted to `/v1/ingest` and passes on a `weights.edit` span posted to `/v1/traces` as it came; an edit that arrived as a log record is not forwarded. In `metadata` mode the span keeps `.changed`, `.l2` and `.max_abs` only. +- OpenTelemetry in: a span or log record named `weights.edit`, sent to `/v1/traces` or `/v1/logs`, is read as the same event. Its fields are the attributes below. A malformed one is left out and the edit still counts. +- OpenTelemetry out: the `otlp` sink sends one `weights.edit` span per edit with the same attributes, as a child of the session's `invoke_agent` span. That span carries `coding_sh.stats.weight_edits` and `coding_sh.stats.weights_changed`. +- `[serve.forward]` sends the same span for an edit posted to `/v1/ingest`, and passes on a `weights.edit` span posted to `/v1/traces` as it came. An edit that arrived as a log record is not forwarded. +- In `metadata` mode the span keeps `coding_sh.weights.changed`, `.l2` and `.max_abs` only. + +| Attribute | Payload field | +|---|---| +| `coding_sh.weights.tensor` | `tensor` | +| `coding_sh.weights.shape` | `shape`, an array of integers | +| `coding_sh.weights.changed` | `changed` | +| `coding_sh.weights.l2` | `l2` | +| `coding_sh.weights.max_abs` | `maxAbs` | +| `coding_sh.weights.base_sha256` | `baseSha256` | +| `coding_sh.weights.edited_sha256` | `editedSha256` | +| `coding_sh.weights.note` | `note` | - The caller computes the hashes and norms. Each SDK checks the values, drops an invalid edit with one warning, and never raises into the app. In Python, numpy scalars and 0-d tensors (what torch's `.norm()` returns) count as numbers. - `weights_edit()` (Python) and `weightsEdit()` (TypeScript) return true only when the event was queued. A `note` that is not a string is dropped with a warning, and the edit is still recorded. -- `coding_sh.weights_delta()` computes `shape`, `changed`, `l2` and `max_abs` for two small array-likes in pure Python. It is a helper you call on purpose, so it raises `ValueError` on bad input. The TypeScript SDK's `weightsDelta()` does the same for nested arrays and typed arrays, and throws a `TypeError` or a `RangeError`. +- `coding_sh.weights_delta()` computes `shape`, `changed`, `l2` and `max_abs` for two small array-likes in pure Python. It is a helper you call on purpose, so it raises `ValueError` on bad input. +- The TypeScript SDK's `weightsDelta()` does the same for nested arrays and typed arrays, and throws a `TypeError` or a `RangeError`. - The server stores the payload without validating these fields, after redaction. - A call outside `trace()` and `with_session()` (`withSession()` in TypeScript) makes a session of its own, so put a run of edits inside one of them.