Measure the CI/CD after benchmark for v2.3 #569

Part of #554. Rulings: #554 rulings 1, 2, 3, 4, 5, 7. Work on main. This issue changes no repo file. Its output is a "Measured" comment here and the same text on #554.

Summary

• What. Measure the pipeline after the v2.3 CI/CD work with one fixed method. Compare it with the frozen baseline below.
• Why. The tracker computes about 620 s per (build): deploy after Build the deploy images while the quality gates run #565 lands, and a stretch of about 540 s after set E, from measured steps. This issue is the end-to-end check.
• Today. Time to live (M1) has a median of 1090 s and a p90 of 1386 s, over 14 deploys from 2026-09-22 to 2026-10-20. Run 37431581531 took 1510 s; run 39398067267 took 1132 s.
• Owns. The metric definitions, sample rules, statistics rules, baseline, "after" protocol and targets.

Depends on / Blocks

• Depends on Build the deploy images while the quality gates run #565. The "after" sample starts at the first deploy that contains it.
• Depends on Write a timing table to every CI/CD run summary #556. Each run's 📊 Run Report summary is the quick read, and it shows free_gb and load1.
• Reads, does not wait for: Start the extension and webapp unit tests without waiting for backend validation #558 (set B, so every after row contains it); Run only the CI suites a pull request or push can affect #563 (the docs-only push target needs it); Prune Docker build cache toward a free-disk target and keep rollback images #564 (it lands before Build the deploy images while the quality gates run #565, so every after row contains it); Make the webapp Docker installs fast and cacheable #559, Skip the duplicate type check inside the webapp Docker build #560, Keep the Actions cache under its limit and pin Bun in CI #561 and Stop extension README media from triggering extension test suites #562 (each changes a measured stage).
• Blocks: nothing.

Current behaviour

Links are pinned to d0a53756f8676556358b6adc6c2397d2ed7c64c8. They are the "before" state; #565 moves some on purpose.

• Triage reads only the tip commit subject (prod.docs.plus.yml:108-112; AGENTS.md:416). So the script reads head_commit.message. displayTitle can show the workflow name, as in run 36437965916.
• Every number comes from production runs, read with read-only gh and git calls. Docker timings do not transfer from a laptop (AGENTS.md:293), and a cache fix cannot be measured under --no-cache (AGENTS.md:301). No run writes its own timing table today.

What we measure

• createdAt is 2–6 s after the push. Record queue_s, the first job's startedAt minus createdAt, so a GitHub queue delay stays visible.
• Today M1 minus (M2 + M3 + M4 + M5) is 19–39 s: checkout, 🔧 Ensure Infrastructure, migrate and verify.
• After Build the deploy images while the quality gates run #565, M1 equals the later of M2g and M2i, plus W, plus the deploy job time.

Extra fields per deploy separate a real gain from a lucky cache.

"Previous deployed SHA" is the last earlier push or workflow_dispatch run whose deploy succeeded on any attempt. A re-run that deployed counts as a base. Example: run 37976785981 is warm only when re-run 37906965365 is its base. Without that base, it reads yes, yes, 81.9 h.

Sample rules

• Count a push run when its 🚀 Deploy Production job ended success on attempt 1.
• Read the subject from head_commit.message (gh api repos/docs-plus/docs.plus/actions/runs/<id> -q .head_commit.message).
• Leave out failed runs, cancelled runs and re-runs. A re-run carries a human gap: run 37906965365, attempt 2, reads 2569 s.
• Split by "Ext suite ran" and by cache class. Warm means gap_h is 24 or less, with no bun.lock change and no extension change. Every other deploy is cold.
• Report a (build): back run on its own row. It runs no front gates, so it stays out of the pooled median.
• Statistics. The median of an even n is the mean of the two middle values, rounded half up. The p90 is nearest-rank: sort ascending, then take item ceil(0.9 × n). Show the p90 only when n is 10 or more.
• A slice with n below 5 is a hint, not a result.
• The baseline sample ends with run 39398067267 (2026-10-20 09:04 UTC).

Baseline: (build): deploys

Seconds after run start, attempt 1. Read on 2026-10-20 with the Step 1 script; free_gb is the "Disk after prune" log value. queue_s was 2–3 s on every row except 35723691814 (11 s).

• The first row has no diff base, so its diff fields are unknown.
• Both webapp installs were CACHED on the three fastest image builds (M3 316–472 s). Neither was CACHED on the other 11 deploys.
• Left out: 37906965365 (re-run), 39274130383 and 38689468498 (the (build):front subject typo), 37904552781 (flaky test), 35723391885 (cancelled).

Baseline Summary

W was 2–3 s on every row, because the image build ran inside the deploy job.

Inside M3, from the production BuildKit logs

Parsed from deploy jobs 118549257854, 118217497479, 118142985559, 115281929767, 114780192678 and 114262648443.

Baseline: PR and non-deploy push runs (M6)

Successful runs from 2026-09-20 to 2026-10-20:

• Every PR ran 5 or 6 Ext suites and the backend unit suite. Backend E2E does not run on PRs.
• Run 39283678757 changed a README and a GIF, yet ran the mermaid suite (194 s) and Build Verification (161 s).

Target behaviour

One "Measured, " comment here and on #554 holds the "after" result. It uses the metrics and rules above. It shows the pooled median, min–max and every slice. It gives a verdict per target and per slice, with run ids. It names the stage behind each miss.

The "after" protocol

1. Measure normal deploys only. Measure whether the backend image build still needs --no-cache #566 is the one exception, with its own procedure.
2. Each change set gets a quick read from the 📊 Run Report of its next one or two deploys. The sibling issue owns it. It is not the result.
3. The sample window starts at the first (build): push run that contains the Build the deploy images while the quality gates run #565 merge commit (Step 3.2). It ends at the 8th qualifying deploy, or 30 days after the start, whichever comes first. Post the median verdict at 8. Judge the p90 in a second "Measured" comment at n of 10 or more.
4. A change set landing inside the window does not restart the count. Each row records the later change sets it contains.
5. A slice is a clear win when its "after" median is below its "before" minimum: All 777, Ext suite ran 1052, No Ext suite 777, Warm 777, Cold 971.

Targets

The targets are the author's call, from the tracker estimate.

Implementation steps

Work in a scratch folder outside the repo. Never commit the scripts. "Comment here" means a comment on this issue.

Step 0. Read and check the base

On current main, check names and behaviour, not pinned line numbers.

1. Update the clone: git -C <repo> fetch origin main. The script needs every deployed SHA locally.
2. Confirm the two dependencies are on main.

   git -C <repo> log origin/main --format='%H %cs %s' -S '📦 Verify images present' -- .github/workflows/prod.docs.plus.yml | tail -1
   git -C <repo> log origin/main --format='%H %cs %s' -S '📊 Run Report' -- .github/workflows/prod.docs.plus.yml | tail -1

   Each command must print one commit. The first SHA is IMAGES_EARLY_SHA. If either prints nothing, stop and comment here.
3. Confirm each step name exists exactly once.

   for n in '🩺 Verify Deployment' '🏗️ Build Docker Images' '💾 Pre-deploy disk guard' '🚀 Deploy Services (rolling replace)' '🚀 Deploy Production' '🏗️ Build Verification' '📊 Run Report'; do
     printf '%s: ' "$n"; git -C <repo> show origin/main:.github/workflows/prod.docs.plus.yml | grep -cF "name: $n"
   done

   Expect 1 on every line. If a name changed, update its regex in bench.ts and note it in the result comment.
4. Confirm the job names fit the script's regexes. On the first after run, run gh run view <runId> -R docs-plus/docs.plus --json jobs -q '.jobs[].name'.
   • The images job (🧱 Build Images in Build the deploy images while the quality gates run #565) must not match /Deploy Production/ or /Build Verification/.
   • The report job must match /report/i, or M6 counts it.
   • Only 🚀 Deploy Production may match /Deploy Production/.
     If any rule fails, fix the regex in bench.ts and note it.
5. Confirm the parsed log text. With the images job id from Step 0.4, run gh run view -R docs-plus/docs.plus --job <imagesJobId> --log | grep -E 'Disk after prune: [0-9]+ GB free|\[webapp (build-extensions|builder) [^]]*\] RUN bun install'.
   • Expect one Disk after prune line. If Prune Docker build cache toward a free-disk target and keep rollback images #564 rewrote it, change the free_gb regex, or read free_gb from the 📊 Run Report table.
   • Expect both [webapp build-extensions …] and [webapp builder …] install lines. If Make the webapp Docker installs fast and cacheable #559 renamed a stage, change the install regex.
   • If Try running the backend and webapp image builds at the same time #567 is on main, install lines start with [front] , which bench.ts matches. Expect each end line to carry the same prefix.
6. Confirm log retention: gh api repos/docs-plus/docs.plus/actions/permissions/artifact-and-log-retention. On 2026-10-20 it printed {"days":90,"maximum_allowed_days":90}, so the 2026-09-22 logs expire about 2026-12-21. Then cached_installs and free_gb read null for old rows; the frozen values stay the baseline.
7. Confirm the window has closed (Step 3.3) before Step 4.

Step 1. Write the two scripts

Write both files in ~/ci-bench/ or another scratch folder. They call only gh and git diff. bench.ts prints one JSON row per run.

// bench.ts: bun bench.ts <repoDir> <runId> ...  → one JSON row per run, oldest first
// Read-only. Diff fields compare each run with the last earlier run in the list that deployed on any
// attempt (a re-run that deployed is a valid base). Pass every push and dispatch run, oldest first.
const REPO = "docs-plus/docs.plus";
const [repoDir, ...ids] = Bun.argv.slice(2);
const sh = (cmd: string[]) => {
  const p = Bun.spawnSync(cmd);
  if (p.exitCode !== 0) throw new Error(`${cmd.join(" ")}: ${p.stderr.toString()}`);
  return p.stdout.toString();
};
// GitHub reports a step that never ran with a zero date (0001-01-01). Treat it as missing.
const ms = (s?: string | null) => (s && !s.startsWith("0001") ? Date.parse(s) : null);
const rows: any[] = [];
let prev: { sha: string; t0: number } | null = null;
for (const id of ids) {
  // Run createdAt comes from the run, not from the attempt: `--attempt 1` reports the attempt's own time.
  const run = JSON.parse(sh(["gh", "api", `repos/${REPO}/actions/runs/${id}`]));
  const d = JSON.parse(sh(["gh", "run", "view", id, "-R", REPO, "--attempt", "1", "--json", "jobs"]));
  const subject = run.head_commit.message.split("\n")[0];
  const t0 = Date.parse(run.created_at);
  const off = (s?: string | null) => { const v = ms(s); return v === null ? null : Math.round((v - t0) / 1000); };
  const dur = (s: any) => { const a = ms(s?.startedAt), b = ms(s?.completedAt); return a === null || b === null ? null : Math.round((b - a) / 1000); };
  // Search the steps of every job: after the images-early change, the image steps live in a new job.
  const all = d.jobs.flatMap((j: any) => (j.steps ?? []).map((s: any) => ({ ...s, job: j })));
  const step = (re: RegExp) => all.find((s: any) => re.test(s.name) && s.conclusion === "success");
  const job = (re: RegExp) => d.jobs.find((j: any) => re.test(j.name));
  const deployJob = job(/Deploy Production/);
  const build = step(/Build Docker Images/);
  const m2g = off(job(/Build Verification/)?.completedAt);
  const m2i = off(build?.completedAt);
  const wStart = off(deployJob?.startedAt);
  const done = d.jobs.filter((j: any) => j.conclusion === "success" && !/report/i.test(j.name));
  const row: any = {
    id, sha: run.head_sha, subject, event: run.event, conclusion: run.conclusion, attempts: run.run_attempt,
    deploy: deployJob?.conclusion ?? null,
    ext: d.jobs.filter((j: any) => /Ext \(/.test(j.name) && j.conclusion === "success").length,
    queue_s: Math.min(...d.jobs.map((j: any) => off(j.startedAt) ?? Infinity)),
    m1: off(step(/Verify Deployment/)?.completedAt),
    m2g, m2i,
    m3: dur(build), m4: dur(step(/disk guard/)), m5: dur(step(/Deploy Services/)),
    m2: wStart,
    // Before images-early the image build runs inside the deploy job, so W is the gap after M2g only.
    w: wStart === null || m2g === null ? null
      : build?.job.databaseId === deployJob?.databaseId ? wStart - m2g : wStart - Math.max(m2g, m2i ?? 0),
    m6: done.length ? Math.max(...done.map((j: any) => off(j.completedAt) ?? 0)) : null,
    typecheck: d.jobs.some((j: any) => /Type Check/.test(j.name) && j.conclusion === "success"),
    obs: d.jobs.some((j: any) => /Deploy Observability/.test(j.name) && j.conclusion === "success"),
    lock_changed: null, ext_src_changed: null, gap_h: null, cached_installs: null, free_gb: null,
  };
  if (build) {
    const log = sh(["gh", "run", "view", "-R", REPO, "--job", String(build.job.databaseId), "--log"]).split("\n");
    let hits = 0;
    for (const [i, line] of log.entries()) {
      const m = line.match(/(\[(?:front|backend)\] )?(#\d+) \[webapp (build-extensions|builder) [^\]]*\] RUN bun install/);
      if (!m) continue;
      const [, pre = "", num, stage] = m;
      if (log.slice(0, i).some((l) => l.includes(`${pre}${num} [webapp ${stage} `))) continue;
      const end = log.slice(i + 1).find((l) => l.includes(`${pre}${num} CACHED`) || l.includes(`${pre}${num} DONE`));
      if (end?.includes("CACHED")) hits++;
    }
    row.cached_installs = hits;
    const free = log.map((l) => l.match(/Disk after prune: (\d+) GB free/)).find(Boolean);
    row.free_gb = free ? Number(free[1]) : null;
  }
  if (prev) {
    const same = (path: string) => {
      const code = Bun.spawnSync(["git", "-C", repoDir, "diff", "--quiet", prev!.sha, run.head_sha, "--", path]).exitCode;
      if (code !== 0 && code !== 1) throw new Error(`git diff failed for ${id}; run git fetch origin main`);
      return code === 0;
    };
    row.lock_changed = !same("bun.lock");
    row.ext_src_changed = !same("extensions/");
    row.gap_h = Math.round(((t0 - prev.t0) / 3600000) * 10) / 10;
  }
  rows.push(row);
  const last = run.run_attempt > 1
    ? JSON.parse(sh(["gh", "run", "view", id, "-R", REPO, "--json", "jobs"])).jobs.find((j: any) => /Deploy Production/.test(j.name))
    : deployJob;
  if (last?.conclusion === "success") prev = { sha: run.head_sha, t0 };
}
console.log(JSON.stringify(rows, null, 1));

stats.ts applies the sample and statistics rules. --from and --to fail on an unknown run id. --before adds the before minimum and clear-win verdict per slice.

// stats.ts: bun stats.ts rows.json [--from <runId>] [--to <runId>] [--before before.json]  → Markdown
// Applies the sample rules and the statistics rules to bench.ts output. --from and --to bound the window.
const args = Bun.argv.slice(2);
const opt = (k: string) => { const i = args.indexOf(k); return i < 0 ? null : args[i + 1]; };
const load = async (p: string) => JSON.parse(await Bun.file(p).text()) as any[];
const at = (rows: any[], id: string) => {
  const i = rows.findIndex((r) => r.id === id);
  if (i < 0) throw new Error(`run ${id} is not in the input`);
  return i;
};
const cut = (rows: any[], from: string | null, to: string | null) =>
  rows.slice(from ? at(rows, from) : 0, to ? at(rows, to) + 1 : rows.length);
const backOnly = (r: any) => /^\(build\):/.test(r.subject) && !/front/.test(r.subject);
const warm = (r: any) => r.gap_h !== null && r.gap_h <= 24 && r.lock_changed === false && r.ext_src_changed === false;
// Sample rule: a push whose deploy job ended success on attempt 1. bench.ts reads attempt 1 only.
const pooled = (rows: any[]) => rows.filter((r) => r.event === "push" && r.deploy === "success" && r.m1 !== null && !backOnly(r));
const slices = (rs: any[]): [string, any[]][] => [
  ["All", rs],
  ["Ext suite ran", rs.filter((r) => r.ext > 0)],
  ["No Ext suite", rs.filter((r) => r.ext === 0)],
  ["Warm", rs.filter(warm)],
  ["Cold", rs.filter((r) => !warm(r))],
];
const median = (v: number[]) => {
  const s = [...v].sort((a, b) => a - b), n = s.length, h = Math.floor(n / 2);
  return n === 0 ? null : n % 2 ? s[h] : Math.floor((s[h - 1] + s[h]) / 2 + 0.5); // even n: round half up
};
const p90 = (v: number[]) => { const s = [...v].sort((a, b) => a - b); return s[Math.ceil(0.9 * s.length) - 1]; };
const col = (rs: any[], k: string) => rs.map((r) => r[k]).filter((x) => typeof x === "number");

const rows = cut(await load(args[0]), opt("--from"), opt("--to"));
const before = opt("--before") ? new Map(slices(pooled(await load(opt("--before")!)))) : null;
console.log("| Slice | n | M1 median | M1 p90 | M1 min–max | M2 median | M2g median | M2i median | W median | M3 median |" + (before ? " Before min | Clear win |" : ""));
console.log("|---|---|---|---|---|---|---|---|---|---|" + (before ? "---|---|" : ""));
for (const [name, rs] of slices(pooled(rows))) {
  const m1 = col(rs, "m1"), med = median(m1);
  const p = m1.length >= 10 ? String(p90(m1)) : `— (max ${m1.length ? Math.max(...m1) : "—"})`;
  const range = m1.length ? `${Math.min(...m1)}–${Math.max(...m1)}` : "—";
  let tail = "";
  if (before) {
    const bmin = Math.min(...col(before.get(name)!, "m1"));
    const verdict = med === null ? "—" : med < bmin ? (rs.length < 5 ? "yes (hint, n<5)" : "yes") : "no";
    tail = ` ${bmin} | ${verdict} |`;
  }
  console.log(`| ${name} | ${rs.length} | ${med ?? "—"} | ${p} | ${range} | ${median(col(rs, "m2")) ?? "—"} | ${median(col(rs, "m2g")) ?? "—"} | ${median(col(rs, "m2i")) ?? "—"} | ${median(col(rs, "w")) ?? "—"} | ${median(col(rs, "m3")) ?? "—"} |${tail}`);
}
for (const r of rows.filter((r) => r.event === "push" && r.deploy === "success" && backOnly(r)))
  console.log(`\n(build): back row: run ${r.id}, M1 ${r.m1} s`);

Check: bun --version prints 1.4.0 or later.

Step 2. Prove the scripts on the baseline

Run the scripts on every baseline push run, oldest first. Re-run 37906965365 must be in the list, as a diff base.

cd ~/ci-bench
gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --event push --limit 400 \
  --json databaseId,createdAt > pushes.json
bun -e 'const r = JSON.parse(await Bun.file("pushes.json").text());
console.log(r.filter((x) => x.createdAt >= "2026-09-22T11:50:00Z" && x.createdAt <= "2026-10-20T09:05:00Z")
  .reverse().map((x) => x.databaseId).join(" "))' | xargs bun bench.ts <repo> > before.json
bun stats.ts before.json

• Pass: every slice row equals the Baseline Summary to the second, and every before.json row matches the 14-row table. This passed on 2026-10-20.
• If the 2026-09 logs have expired (Step 0.6), compare timing fields only.
• If any timing value differs, stop and comment here with the row.

Step 3. List the after runs

1. List every push and workflow_dispatch run from 39398067267 to now, oldest first, so the first after deploy has its real diff base.

   gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --limit 400 \
     --json databaseId,createdAt,event -q '[.[] | select(.event == "push" or .event == "workflow_dispatch")]' > pushes.json
   bun -e 'const r = JSON.parse(await Bun.file("pushes.json").text());
   console.log(r.filter((x) => x.createdAt >= "2026-10-20T09:04:00Z").reverse().map((x) => x.databaseId).join(" "))' \
     | xargs bun bench.ts <repo> > after-all.json

   Dispatch rows are diff bases only; stats.ts keeps push rows. List each under "Left out" as "A/B dispatch (Measure whether the backend image build still needs --no-cache #566)".
2. FIRST_AFTER is the first push row whose subject starts with (build): and for which git -C <repo> merge-base --is-ancestor <IMAGES_EARLY_SHA> <row sha> && echo contains prints contains.
3. LAST_AFTER. A qualifying row has event push, deploy success, a numeric m1, and a subject that starts with (build): and contains front. A (build): back row does not count; stats.ts prints it on its own line. Count from FIRST_AFTER, oldest first.
   • LAST_AFTER is the 8th qualifying row, or, if day 30 comes first, the last qualifying row created before FIRST_AFTER createdAt plus 30 days. Read createdAt from pushes.json.
   • END_TIME is the createdAt of LAST_AFTER. The second comment uses the same rule with 10 in place of 8. If day 30 comes first, it reports "p90 not measured, n = ".
   • Check: bun stats.ts after-all.json --from <FIRST_AFTER> --to <LAST_AFTER> prints an All row with n of 8, or the day-30 count.
4. Tag each qualifying row with the later change sets it contains. Find each merge SHA, then run git -C <repo> merge-base --is-ancestor <merge SHA> <row sha> && echo contains.

   git -C <repo> log origin/main --format='%H %cs %s' -S 'BACK_PID' -- .github/workflows/prod.docs.plus.yml | tail -1         # parallel-compose
   git -C <repo> log origin/main --format='%H %cs %s' -S 'outputFileTracingExcludes' -- apps/webapp/next.config.js | tail -1   # build-traces trim
   git -C <repo> log origin/main --format='%H %cs %s' -S 'backend_no_cache' -- .github/workflows/prod.docs.plus.yml | head -1   # backend-nocache-ab Step 6, input removed

   An empty line means the change set is not on main; tag it "no". If a sibling used another string, read its merge commit there.
   • For Measure whether the backend image build still needs --no-cache #566: if git -C <repo> show origin/main:.github/workflows/prod.docs.plus.yml | grep -c backend_no_cache prints more than 0, the A/B test is still open. Tag every row "A/B only".
   • Otherwise the head -1 commit is its Step 6 change. Tag a row 'drop' only when it contains that commit and the maintainer ruled 'drop' there. A 'keep' ruling changes no deploy, so tag it 'no change'.
5. If the window has not ended (Step 3.3), stop and come back later.

Step 4. Add free_gb and load1

• For each qualifying row, read free_gb and load1 from the 📊 Run Report table at https://github.com/docs-plus/docs.plus/actions/runs/<id>.
• The bench.ts free_gb is a cross-check. If they differ, use the report value and note it. If the report shows n/a, use the bench.ts value and say so.
• If load1 reads n/a, record n/a. Do not read the host.

Step 5. Compute the result

1. Deploys: bun stats.ts after-all.json --from <FIRST_AFTER> --to <LAST_AFTER> --before before.json. The p90 prints only at n of 10 or more; judge it in the second comment.
2. M6. Run bench.ts on the PR runs in the same window.

   gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --event pull_request --status success --limit 200 \
     --json databaseId,createdAt -q '[.[] | select(.createdAt >= "<FIRST_AFTER createdAt>" and .createdAt <= "<END_TIME>")] | reverse | .[].databaseId' \
     | xargs bun bench.ts <repo> > after-pr.json

   • PR median: the median of m6 over after-pr.json. Also report how many PRs ran 0 Ext suites.
   • Root docs-only push: an after-all.json row between FIRST_AFTER and LAST_AFTER with event push, conclusion success, deploy not success, typecheck false and ext 0. A push can hold more than one commit, so confirm with git -C <repo> diff --name-only <previous push row sha> <row sha> that every changed file is docs-only. Report its m6, or "no sample".
3. Name the stage that missed, for every missed target. The later of the M2g and M2i medians is the long path. A W median above 10 s (an author threshold) means the prod runner queued the deploy. The rest is the deploy job: M1 minus the later of M2g and M2i, minus W.

Step 6. Draft the result comment

Post it as this issue's SP3 (#554) record. This layout extends the SP3 template with the window, the row table and the target verdicts. Draft it in the scratch folder.

## Measured, <YYYY-MM-DD>

Window: <FIRST_AFTER> (<date>) to <last run id> (<date>). n = <n> qualifying deploys.
Left out: <run id (reason)>, …

| Run | Date | Ext suites | M2g | M2i | W | M4 | M3 | M5 | **M1** | queue_s | lock_changed | ext_src_changed | gap_h | cached_installs | obs | free_gb | load1 | Class | Later sets |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

<stats.ts output with --before>

| Target | Before | After | Verdict |
|---|---|---|---|
| M1 median ≤ 620 s (stretch 540 s, rows with set E) | 1090 | | |
| M1 p90 ≤ 720 s (n ≥ 10) | 1386 | | |
| M6 PR median lower | 519 | | |
| M6 root docs-only push ≤ 120 s | 217 | | |

Missed stage: <M2g / M2i / W / deploy job, with medians>, or "none".
Stop condition: <not met / met, with the value>.

Review checkpoint

• git -C <repo> status --short must show no change from this work.
• Show the maintainer the draft comment, before.json and after-all.json. Then stop.
• After the maintainer reads it, post it here and on [Tracking] Faster, change-aware CI/CD pipeline #554, with a link back here.
• No git add, git commit, git push, git stash or branch.

Acceptance criteria

• Step 0 found IMAGES_EARLY_SHA and the 📊 Run Report commit on main, and every step name once.
• Step 2 reproduced the Baseline Summary to the second, or the comment states which fields the expired logs blocked.
• FIRST_AFTER is the first (build): push run that contains IMAGES_EARLY_SHA, checked with git merge-base --is-ancestor.
• LAST_AFTER follows Step 3.3, and stats.ts ran with --from <FIRST_AFTER> --to <LAST_AFTER>.
• Every left-out run in the window is listed with its reason.
• Every qualifying row shows every metric and extra field, or n/a.
• All, Ext suite ran, No Ext suite, Warm and Cold each show n, median, min–max and the clear-win verdict. Each slice with n below 5 is marked as a hint.
• Each of the four targets has a verdict, and each miss names its stage.
• The p90 verdict is in a second comment at n of 10 or more, or that comment reads "p90 not measured, n = ".
• The same comment exists on [Tracking] Faster, change-aware CI/CD pipeline #554, with a link back here.
• git status --short shows no change from this work.

Verification

• No workflow or triage code changes here, so actionlint with .github/actionlint.yaml, bash .github/scripts/parse-build-trigger.test.sh and a dry_run_subject dispatch do not apply.
• Script check: Step 2.
• The measurement: read with bench.ts, stats.ts and the 📊 Run Report table. M1 and M6 rows are in Targets; clear-win minimums are in the protocol.

Accountability

• Who. The builder reads the runs and drafts the comment. The maintainer reads the draft before posting and rules on the verdict.
• When. At the 8th qualifying deploy or day 30, whichever comes first. A second comment follows at n of 10 for the p90.
• Stop condition, which means "revert". Any one of these:
  • the after All median of M1 is above 1090 s;
  • a slice with n of 5 or more has an after median above its before median: Ext suite ran 1354, No Ext suite 994, Warm 946, Cold 1132;
  • any qualifying after deploy has M1 above 1510 s, the before maximum.
• When it is met: name the missed stage, from M2g, M2i and W. Comment on Build the deploy images while the quality gates run #565 and recommend the revert of the Build the deploy images while the quality gates run #565 commit only. The maintainer decides. Change no workflow from this issue.
• Disk check. If any after row has free_gb below 20, comment on Prune Docker build cache toward a free-disk target and keep rollback images #564 with the run id and the value.

Rollback

• This issue changes no repo file and no host.
• Fix a wrong result with a new comment headed "Correction, " that names the wrong and right value. Do not delete the first comment.
• On the stop condition, Build the deploy images while the quality gates run #565 reverts per SP4 ([Tracking] Faster, change-aware CI/CD pipeline #554): revert only IMAGES_EARLY_SHA, and keep Prune Docker build cache toward a free-disk target and keep rollback images #564. The next run reads the old workflow from the new tip.

Docs to update

None. #557 owns the stale CACHED and deploy-gap facts in AGENTS.md §Docker.

Out of scope

• Extra deploys to make numbers (ruling 4), a dashboard, a metrics stack or a new CI vendor (ruling 5).
• Committing the scripts, or any workflow edit. Write a timing table to every CI/CD run summary #556 owns the in-run timing table.
• Fixing a missed target here. Comment on the owning sibling issue.
• Laptop timings, and judging a cache fix under --no-cache.
• Reading or touching the prod host. Do not ssh.
• Reopening the GHCR question. The tracker reopens it only when free disk stays under the Prune Docker build cache toward a free-disk target and keep rollback images #564 target across a week of deploys.
• Skipping the backend images on (build): front (ruling 7), Turbopack, a BuildKit cache mount with a 168h prune, and a deploy gate on chatroom E2E. The tracker rejected or deferred each one.

Read first

• [Tracking] Faster, change-aware CI/CD pipeline #554, Build the deploy images while the quality gates run #565 and Write a timing table to every CI/CD run summary #556, in full.
• AGENTS.md §Docker (lines 286–307 at the pinned commit) and §Learned Workspace Facts (line 416).
• .github/workflows/prod.docs.plus.yml on current main: the images, deploy and report jobs.
• .github/WORKFLOWS.md, for the current job order.