Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .claude/skills/verify-open-pstack/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: verify-open-pstack
description: Verify an Open Pstack PR's exact candidate head in native Claude Code, Codex and T3 Code sessions, judge them with machine-checked assertions, and publish the exact-SHA live gate.
description: Verify an Open Pstack PR's exact candidate head in native Claude Code, Codex, T3 Code and Grok sessions, judge them with machine-checked assertions, and publish the exact-SHA live gate.
---

# Verify Open Pstack
Expand Down Expand Up @@ -53,11 +53,13 @@ Claude runs with normal operator config/login and session-only `claude --plugin-

`t3:` features run through the T3 transport in `features/t3.md`: a run-owned disposable T3 server whose Claude provider gets the same session-only plugin flags, one `t3 thread start --wait --json` per case, and the parent's Claude transcript parsed like the Claude stream. Classification never adds `t3` on its own and self-test does not reach it; the doctor checks the T3 service binary only when a `t3:` feature is named.

`grok:` features run through the Grok transport in `features/grok.md`: one `grok agent --no-leader --plugin-dir <exact-head checkout>/plugins/pstack --model grok-4.7 --reasoning-effort low stdio` process per case, driven over ACP, with permission requests answered from the recipe tools and the rest refused. Before the prompt, `_x.ai/plugins/list` must show the candidate as the only loaded `pstack`, by path and tree hash. Classification never adds `grok` on its own and self-test does not reach it; the doctor checks the Grok binary, its agent flags and the `grok models` login line only when a `grok:` feature is named.

Candidate commands and provider descendants use the recorded native-login arrangement, never GitHub publisher credentials. No extra login, implicit timeout, or weaker-model fallback is permitted. If Claude's plugin flags fail to exclude installed pstack or Codex replaces the auth symlink instead of writing through it, stop and reopen the design.

For every named feature, exercise its native surface and retain real native tool calls and concrete fixture effects. Asset consumers come from the pinned plugin manifests; the current `plugins/pstack/assets/logo.png` is consumed by the Codex manifest only and therefore requires the Codex installed surface, not an invented Claude asset exercise. The session recorder streams each session's stdout and stderr to private files under `sessions/<harness>/<feature>/` in the run root and keeps them on any exit. For Codex it also copies the session's rollout from the run-owned home, because only the rollout records which `SKILL.md` each skill mention resolved to. A nonzero session exit fails the case. The trusted parent then parses the native stream and evaluates the case's assertions on the native skill load, the commands that ran, and files the session wrote in its fixture, and copies those files next to the stream. A model's claim that something passed never counts, and neither do direct CLI tests.

For setup only, snapshot real `${CLAUDE_CONFIG_DIR:-$HOME/.claude}/pstack-models.md` and `CLAUDE.md`, including original absence, before exercise. Restore exact bytes or absence afterwards on success or failure and verify restoration before any publication. Restoration failure stops the run and publishes nothing.
For setup only, snapshot real `${CLAUDE_CONFIG_DIR:-$HOME/.claude}/pstack-models.md` and `CLAUDE.md`, and `${GROK_HOME:-$HOME/.grok}/rules/pstack-models.md`, including original absence, before exercise. Restore exact bytes or absence afterwards on success or failure and verify restoration before any publication. Restoration failure stops the run and publishes nothing.

For `project-skill`, invoke `/verify-open-pstack` in Claude and `$verify-open-pstack` in Codex from the candidate checkout. Ask the discovered pinned skill to run `verify.sh doctor --candidate --output <fresh-run-owned-directory>`. Assert the skill load, the doctor command, and the canonical-path doctor output; forbid recursive verification or publication. This does not replace required plugin exercises.

Expand Down Expand Up @@ -92,6 +94,8 @@ Read receipts as data, never shell input. Do not put credentials in prompts or p
- scripts/core.ts
- scripts/doctor.ts
- scripts/github.ts
- scripts/grok.test.ts
- scripts/grok.ts
- scripts/harness.test.ts
- scripts/harness.ts
- scripts/io.test.ts
Expand Down
3 changes: 2 additions & 1 deletion .claude/skills/verify-open-pstack/features/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The ownership table below says which features a changed path reaches. `SKILL.md`

A change to `features/recipes.ts` can name any feature that has a recipe.

A change to `plugins/pstack/.claude-plugin/plugin.json`, `.claude-plugin/marketplace.json`, or `plugins/pstack/.codex-plugin/plugin.json` is version-only when the merge base and the head hold the same regular file with the same mode, both sides parse as JSON objects, at least one version string changed (top-level `version`, or `plugins[].version` in the marketplace), and nothing else differs. A version-only change reaches only `hook:startup`: the two Claude files in Claude, the Codex manifest in Codex. `.agents/plugins/marketplace.json` has no version and always takes the shared expansion.
A change to `plugins/pstack/.claude-plugin/plugin.json`, `.claude-plugin/marketplace.json`, or `plugins/pstack/.codex-plugin/plugin.json` is version-only when the merge base and the head hold the same regular file with the same mode, both sides parse as JSON objects, at least one version string changed (top-level `version`, or `plugins[].version` in the marketplace), and nothing else differs. A version-only change reaches only `hook:startup`: the two Claude files in Claude, the Codex manifest in Codex. `.agents/plugins/marketplace.json` has no version and always takes the shared expansion. `plugins/pstack/.grok-plugin/plugin.json` always takes the shared expansion too, because pstack's startup hook does not fire in Grok.

Asset ownership is consumer-driven, not blanket shared ownership. Inspect the exact candidate's plugin manifests. At present, `plugins/pstack/assets/logo.png` is referenced only by `plugins/pstack/.codex-plugin/plugin.json`, so a logo-only change selects the installed Codex asset surface and does not invent a Claude exercise. New shipped assets must be added to `assets` and must have a real manifest consumer; missing, ambiguous, or unknown consumption fails closed.

Expand All @@ -36,6 +36,7 @@ The feature documents below define sub-features, user entry points, driving inst
- `shipped-tools.md`: watch, plan checks, audit, statechart, and evidence logging.
- `assets.md`: manifest-declared installed assets on actual consuming harnesses.
- `t3.md`: a T3 Code parent dispatching a role through `delegate_task` (`t3:dispatch`), on a run-owned disposable T3 server.
- `grok.md`: a Grok parent's native and runner lanes and agent visibility (`grok:dispatch`), and setup writing the Grok sheet (`grok:setup`), over ACP.
- `hooks.md`: the SessionStart hook's context reaching a new session (`hook:startup`), and the Windows runner beside it.
- `recipes.ts`: the headless cases and machine-checked assertions for every selectable feature. A named feature without one fails the run as `missing-recipe` before any session starts. A recipe proves a lookup or a read with a fixture value the model cannot derive from the prompt, such as a per-run `crypto.randomUUID()` marker, compared in the structured result. A command's text can show which tool the parent called, but it never proves what that tool read or produced, so do not add command-text or tool-output pattern checks for that.
- `agent-preload:poteto-agent` (Claude only): the parent dispatches `pstack:poteto-agent` through the Agent tool and asks it for the first Core principle of the preloaded poteto-mode skill without invoking `Skill` or reading files. The check needs that Agent call, `Laziness Protocol` in the reply the parent writes to `result.md`, and no `Skill` call or `Read` of poteto-mode's `SKILL.md` among the child's own tool calls.
Expand Down
21 changes: 21 additions & 0 deletions .claude/skills/verify-open-pstack/features/grok.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Grok parent

## Sub-features

`grok:dispatch`: a Grok parent running the installed `poteto-mode` routes an `inherit-parent` lane natively through `spawn_subagent` with `pstack:poteto-agent`, an explicit `grok:` lane through `pstack-runner --parent grok --provider grok`, and a Codex lane through `pstack-runner --parent grok`, and Grok offers only the model-free pstack agents. `grok:setup`: `/pstack:setup-pstack` on a Grok parent writes `${GROK_HOME:-$HOME/.grok}/rules/pstack-models.md`. Select them for a change to the Grok manifest, `provider-dispatch.md`, `codex-tools.md`, `setup-pstack/SKILL.md`, `recipes.ts` or the Grok transport. A path that reaches them never adds them to every run; name them with `--feature grok:dispatch` or `--feature grok:setup`.

## How to get to it (user POV)

Start a Grok session with the candidate plugin loaded and invoke `/pstack:poteto-mode` or `/pstack:setup-pstack` by name. pstack's startup hook does not fire in Grok.

## Driving it with verify-open-pstack

Each case starts one `grok agent --no-leader --plugin-dir <exact-head checkout>/plugins/pstack --model grok-4.7 --reasoning-effort low stdio` process with the operator's real HOME and Grok home, and drives it over ACP: `initialize`, `session/new` with the fixture cwd, `_x.ai/plugins/list`, and one `session/prompt`. The transport answers `session/request_permission` with allow-once for the tools in `GROK_TOOLS` and reject-once for every other tool, so no prompt waits and no always-approve mode is set. It closes stdin after the turn, and the process exits. Every message in both directions goes to `sessions/grok/<feature>/<case>.jsonl`. No operator Grok config is written.

## Gotchas

Grok also discovers the operator's installed pstack through its Claude-compat scan of `~/.claude/plugins`, and the `--plugin-dir` copy shadows it. Before the prompt, the transport requires `_x.ai/plugins/list` to report exactly one enabled `pstack`, whose root is the candidate path and whose tree hash equals the pinned candidate's, and otherwise stops. The response is retained as `<case>.plugins.json`. Grok has no Skill tool. A slash command expands in Grok's own prompt, which records the skill in a `<skills_referenced>` block of the parent's `chat_history.jsonl`; a skill counts as loaded when that block names the candidate's `SKILL.md`, or when a parent `read_file` of it completed. A subagent's updates arrive under its own session id, and its `SubagentCompleted` output names that id. A turn that ends with a tool call still running, or with a child session that has no `turn_completed`, fails the case. The transport copies the session's `summary.json`, which must show `grok-4.7` at `low`, `tool_definitions.json`, which holds the `spawn_subagent` `subagent_type` enum, and `chat_history.jsonl`, from `${GROK_HOME:-$HOME/.grok}/sessions/<cwd>/<session>/` into the run root. On macOS the Grok runner lane needs the `/var/run/docker.sock` prerequisite in the model catalog.

## Headless

The `dispatch` case writes a fixture sheet with three read-only roles: `why investigators, synthesizer` on `inherit-parent`, `how explorer` on `grok:grok-4.7@low` and `how explainer` on `codex:gpt-6-luna@low`, and a `marker-N.txt` per lane holding a per-run `crypto.randomUUID()` the prompt never contains. It asserts the candidate `poteto-mode` load; an agent enum holding `pstack:poteto-agent` and `pstack:comment-sicko` and no `pstack:pstack-*` agent; exactly one `spawn_subagent` call, with `pstack:poteto-agent`, whose `SubagentCompleted` output holds lane 1's marker and whose child ran tools; no runner receipt for lane 1; and for lanes 2 and 3 a `pstack-runner --parent grok` command for that provider and a receipt that is `complete`, on that route, `read-only`, proved by the provider's Model proof (`provider-report` for Grok, `pinned-argv` for Codex), from this session, with the lane's marker in its output. The `configure` setup case answers setup with every model-family role on Grok 4.7 at low effort and every writer sandbox opt-in declined, and asserts that the Grok sheet was written this session, holds only `grok:grok-4.7@low`, and has no `full-access` setting line. The parent snapshots the Grok sheet with the other setup files, and restores and verifies it on success or failure before any publication.
82 changes: 81 additions & 1 deletion .claude/skills/verify-open-pstack/features/recipes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ export async function readCatalog(location: string): Promise<Catalog> {
const kept = new Map(rows(section('Kept predecessors and normalization').split('\nRolling aliases:')[0]!).slice(1)
.map(([provider, model, , current]): [string, string] => [`${provider}:${model}`, `${provider}:${current}`]));
const [header = [], ...routes] = rows(section('Routes'));
const column = { claude: header.indexOf('Claude Code parent'), codex: header.indexOf('Codex parent'), t3: -1 };
const column = { claude: header.indexOf('Claude Code parent'), codex: header.indexOf('Codex parent'), t3: -1, grok: -1 };
if (!defaults.size || column.claude < 0 || column.codex < 0) throw new Error('catalog-unreadable');
return { defaults, kept, route(harness, descriptor) {
if (!descriptor.includes(':')) return 'native';
Expand Down Expand Up @@ -801,11 +801,91 @@ const t3Dispatch: Case = {
},
};

const GROK_LANES = [
{ role: 'why investigators, synthesizer', descriptor: 'inherit-parent' },
{ role: 'how explorer', descriptor: 'grok:grok-4.7@low', provider: 'grok', model: 'grok-4.7', effort: 'low', proof: 'provider-report' },
{ role: 'how explainer', descriptor: 'codex:gpt-6-luna@low', provider: 'codex', model: 'gpt-6-luna', effort: 'low', proof: 'pinned-argv' },
] as const;
const GROK_AGENTS = ['pstack:poteto-agent', 'pstack:comment-sicko'];

export function grokAgents(agents: string[] | undefined): string {
if (!agents || GROK_AGENTS.some(agent => !agents.includes(agent))) throw new Error('grok-model-free-agents-missing');
const lane = agents.find(agent => agent.startsWith('pstack:pstack-'));
if (lane) throw new Error(`grok-claude-lane-agent-offered:${lane}`);
return 'agents:model-free-only';
}

export function grokReceipt(receipt: Record<string, any>, lane: { provider: string; model: string; effort: string; proof: 'provider-report' | 'pinned-argv' }, output: string, started: number): void {
if (receipt.status !== 'complete') throw new Error(`receipt-status:${receipt.status}`);
if (receipt.parent !== 'grok' || receipt.provider !== lane.provider || receipt.model !== lane.model || receipt.effort !== lane.effort || receipt.mode !== 'read-only') throw new Error(`receipt-route-mismatch:${lane.provider}`);
const proved = lane.proof === 'provider-report'
? receipt.modelVerified === true && receipt.modelEvidence === 'provider-report'
: receipt.modelVerified === false && receipt.modelEvidence === 'pinned-argv' && receipt.reportedModel === null;
if (!proved) throw new Error(`receipt-model-unverified:${lane.provider}`);
if (receipt.outputPath !== output || !(Date.parse(receipt.startedAt) >= started)) throw new Error(`receipt-not-this-session:${lane.provider}`);
}

async function grokSkill(record: SessionRecord, f: Fixture, skill: string): Promise<string> {
if (!record.skills.includes(await realpath(join(f.location, 'skills', skill)))) throw new Error(`skill-not-loaded-from-candidate:pstack:${skill}`);
return 'skill-loaded-from-candidate';
}

const grokDispatch: Case = {
id: 'dispatch',
async prepare(f) {
await files(f.dir, { 'pstack-models.md': sheetText(GROK_LANES.map(lane => [lane.role, [lane.descriptor]])),
...Object.fromEntries(GROK_LANES.map((_, index) => [`marker-${index + 1}.txt`, `${crypto.randomUUID()}\n`])) });
},
prompt: f => `${invoke(f.harness, 'poteto-mode', `Do no engineering work. For this task the pstack model sheet is ${join(f.dir, 'pstack-models.md')}; use it in place of any configured sheet. Dispatch exactly one read-only lane for each of these roles, in this order, and nothing else: ${GROK_LANES.map(lane => JSON.stringify(lane.role)).join(', ')}. Lane N's whole task: "Read ${join(f.dir, 'marker-N.txt')} and reply with its exact contents. Do not edit files." A lane that needs an output or receipt path uses ${join(f.dir, 'lane-N.output.md')} and ${join(f.dir, 'lane-N.receipt.json')}. Wait for every lane to finish, then reply with each lane's result.`)} ${rules(f.dir)}`,
async assert(record, f) {
const assertions = [await grokSkill(record, f, 'poteto-mode'), grokAgents(record.agents)];
const markers = await Promise.all(GROK_LANES.map(async (_, index) => (await readFile(join(f.dir, `marker-${index + 1}.txt`), 'utf8')).trim()));
const spawned = record.tools.filter(t => t.name === 'spawn_subagent' && !t.failed);
if (spawned.length !== 1 || spawned[0]!.input.subagent_type !== 'pstack:poteto-agent') throw new Error(`native-lanes:${spawned.map(t => t.input.subagent_type).join(',')}`);
const native = spawned[0]!, completed = parseJson(native.result) as Record<string, any> | undefined;
if (completed?.type !== 'SubagentCompleted' || completed.subagent_type !== 'pstack:poteto-agent' || !String(completed.output).includes(markers[0]!)) throw new Error('native-lane-result-mismatch');
if (!record.tools.some(t => t.parent === native.id)) throw new Error('native-child-did-not-run');
if (await lstat(join(f.dir, 'lane-1.receipt.json')).catch(() => undefined)) throw new Error('native-lane-used-runner');
assertions.push('native:pstack:poteto-agent');
const paths = [join(f.dir, 'pstack-models.md')];
for (const [index, lane] of GROK_LANES.entries()) {
if (!('provider' in lane)) continue;
const at = (name: string) => join(f.dir, `lane-${index + 1}.${name}`);
if (!record.commands.some(c => c.includes('pstack-runner') && c.includes('--parent grok') && c.includes(`--provider ${lane.provider}`))) throw new Error(`runner-not-run:${lane.provider}`);
grokReceipt(JSON.parse(await readFile(at('receipt.json'), 'utf8')), lane, at('output.md'), record.started);
if (!(await readFile(at('output.md'), 'utf8')).includes(markers[index]!)) throw new Error(`marker-missing:${lane.provider}`);
assertions.push(`runner:${lane.descriptor}`);
paths.push(at('receipt.json'), at('output.md'));
}
const exchange = join(f.dir, 'native-lane.json');
await writeFile(exchange, JSON.stringify({ request: native.input, response: completed }, null, 2));
return { assertions: [...assertions, 'markers-returned'], files: [exchange, ...paths] };
},
};

const GROK_SETUP_DESCRIPTOR = 'grok:grok-4.7@low';
const grokSetup: Case = {
id: 'configure',
prompt: f => `${invoke(f.harness, 'setup-pstack', `Use these answers and ask nothing else. Role assignments: change every role assigned to a model-matrix family to the Grok family, and keep inherit-parent and auto roles unchanged. Grok requested effort: low. Decline every writer sandbox opt-in. I confirm every write. Write the setup report to ${join(f.dir, 'result.md')}.`)} Do not push, post, or contact GitHub.`,
async assert(record, f) {
const sheet = join(f.configHome, 'rules/pstack-models.md'), assertions = [await grokSkill(record, f, 'setup-pstack'), await written(record, sheet)];
const text = await readFile(sheet, 'utf8'), descriptors = text.match(DESCRIPTOR) ?? [];
if (!descriptors.length || descriptors.some(d => d !== GROK_SETUP_DESCRIPTOR)) throw new Error('sheet-descriptors-mismatch');
if (settingLines(text).some(line => line.endsWith(': full-access'))) throw new Error('setting-full-access');
return { assertions: [...assertions, `sheet:${GROK_SETUP_DESCRIPTOR}`, 'no-full-access'], files: [sheet] };
},
};

export function recipe(feature: string, harness: Harness, routes: string[] = []): Case[] {
if (harness === 't3') {
if (feature === 'dispatch') return [t3Dispatch];
throw new Error(`missing-recipe:${harness}/${feature}`);
}
if (harness === 'grok') {
if (feature === 'dispatch') return [grokDispatch];
if (feature === 'setup') return [grokSetup];
throw new Error(`missing-recipe:${harness}/${feature}`);
}
if (feature === 'assets:codex') throw new Error('unsupported-native-consumer:assets:codex');
if (feature === 'project-skill') return [projectSkill];
if (feature === 'hook:startup') return [startupHook];
Expand Down
Loading
Loading