Build the deploy images while the quality gates run #565

Part of #554, change set D. Work on main. Rulings: #554 rulings 1, 6, 7, 11, 12, 13, 14, 15, 16.

Summary

• Today. The gate path ends, and 🚀 Deploy Production starts at a median of 417 s. Then 🏗️ Build Docker Images runs inside that job on the prod host, for a median of 546 s.
• The image build reads no gate output. The workflow's own comment says every step before 🔧 Ensure Infrastructure leaves production untouched (L646-L648).
• After. A new images job builds the four images while the gates run. deploy waits for both, checks the images, then rolls as today.
• Saving. About 340 s median time to live (range 257–586 s) on the 14 baseline runs. About 476 s (range 320–731 s) with Make the webapp Docker installs fast and cacheable #559 and Skip the duplicate type check inside the webapp Docker build #560. Both are computed with the formulas in Target behaviour, not measured.
  • Each run uses its own 🔒 Security Audit end (45–80 s), M2g, M4 and M3. W is taken as 0. The second figure takes 145 s off M3.
• Example. In run 39398067267, Build Verification ended at +538 s. Job 118549257854 then built images from +544 s to +1057 s (513 s). Time to live was 1132 s.
• Also here: a cleanup job for unshipped images, an images failure notice, the cancel-safe if, and the docs.
• Metrics. M1–M5, M2g, M2i and W are defined in Measure the CI/CD after benchmark for v2.3 #569. The step names are the metric keys. This issue keeps them byte-identical.
• Baseline. 14 (build): deploys from 2026-09-22 to 2026-10-20, each success on attempt 1. Runs: 35723691814, 36324777594, 36437965916, 36473904247, 36550492709, 37431581531, 37440315600, 37976785981, 38068638114, 38239392350, 38392614215, 39275929831, 39297468969, 39398067267. The values are in Accountability.

Depends on / Blocks

• Lands after sets A to C (ruling 15).
• Depends on Roll back only after the deploy has started rolling services #555. Today 🚨 Rollback on Failure is a bare if: failure() (L855-L857). Without that fix, a failed 📦 Verify images present would run the rollback on all five live services, at the failed SHA (Roll back only after the deploy has started rolling services #555, Current behaviour item 8).
• Depends on Prune Docker build cache toward a free-disk target and keep rollback images #564. This issue moves the disk guard into images. That guard must already prune toward a free-disk target and keep the rollback images. It is the backstop for images that the cleanup job does not remove.
• Coordinates with Write a timing table to every CI/CD run summary #556. Step 7 points its report job at the moved steps.
• Blocks Measure the CI/CD after benchmark for v2.3 #569. The final "after" benchmark starts only after this change set lands.
• Does not wait for Restrict the production self-hosted runner to main deploys #552 (ruling 12).

Current behaviour

Links are pinned to d0a53756f8676556358b6adc6c2397d2ed7c64c8. WF below means .github/workflows/prod.docs.plus.yml.

1. One job does the build and the roll (WF L420-L441). It has needs: [triage, build] (L427) and the -deploy group with cancel-in-progress: false (L428-L430). Its if (WF L431-L435) is:

   if: |
     always() &&
     needs.build.result == 'success' &&
     ((github.event_name == 'push' && needs.triage.outputs.app_deploy == 'true') ||
      (github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main' && inputs.force_deploy))

2. The steps run in this order:

   • 🔐 Prepare Environment (WF L448-L571). It writes deploy-service.sh to RUNNER_TEMP (L454-L534). It copies the host env file to .env.production and appends DEPLOY_TAG. It runs the DSN and email guards (L536-L559). It stashes PREVIOUS_TAG in GITHUB_ENV (L561-L570).
   • 💾 Pre-deploy disk guard (WF L573-L594).
   • 📂 Verify build context (monorepo root) (WF L596-L610).
   • 🏗️ Build Docker Images (WF L612-L642).
   • 🔧 Ensure Infrastructure. Its first action is echo "DEPLOY_STARTED=true" >> "$GITHUB_ENV" (WF L644-L649). It writes no step output and has no id.
   • 🗄️ Migrate database (one-shot gate) (WF L690-L699), then the roll, verify, stash, cleanup, summary and annotation steps.

3. The drift warning. "a second copy of this function is a drift waiting to drain the fleet" (WF L450-L453). So the env guards must not be pasted into a second job.

4. The failure notice reads DEPLOY_STARTED. Unset means "aborted before touching prod (env guard or build); production unchanged" (WF L951-L965). Its Telegram credentials come from the host file named at L957. Step 3 copies that OBS_ENV= line once into the images notice. Add the path nowhere else.

5. migrate has no build: section. It is only image: docsplus-hocuspocus:${DEPLOY_TAG:-latest} (docker-compose.prod.yml L160-L168). If the image is missing, migrate fails after DEPLOY_STARTED is set. That is past the point where the rollback runs.

6. The four image names are docsplus-rest-api, docsplus-hocuspocus, docsplus-webapp and docsplus-admin (docker-compose.prod.yml L171, L289, L491, L590). hocuspocus-worker and migrate reuse docsplus-hocuspocus.

7. The concurrency comment is wrong (WF L415-L418). It says "a later push still cancels an in-flight roll". A running always() job survives a cancel. See [Tracking] Faster, change-aware CI/CD pipeline #554 SP1, runs 34223310158 → 34224526419. An images job under always() would keep building for about 9 min on the live host after its run was superseded.

8. observability-deploy needs only [triage], on purpose (WF L970-L985). Lines 973-978 say that a reusable-workflow call does not run reliably under always() when a needed job is skipped. The one prod runner already serializes it with the app jobs, at a cost of about 31 s.

9. security already runs early. It needs only [triage] (WF L214-L222). In run 39398067267 it ended at +62 s.

10. The skip_quality_gates escape in build is (needs.security.result == 'success' || (github.event_name == 'workflow_dispatch' && inputs.skip_quality_gates)) (WF L373).

11. The webapp build uploads sourcemaps to GlitchTip when SENTRY_AUTH_TOKEN is set (apps/webapp/next.config.js L437-L450; build args at docker-compose.prod.yml L513-L516).

12. Two comments describe the old order: the header (WF L5-L9) and the Build Verification note (WF L346-L349).

13. .github/WORKFLOWS.md describes the stages and the three-way failure notice (L10-L15).

Target behaviour

• A (build): push (or a force_deploy dispatch on main) runs images on prod.docs.plus as soon as triage and security pass. Lint, typecheck, back-validation, the Ext suites, webapp unit and Build Verification run at the same time.
• images runs the env guards, the disk guard and the image build (Step 3). It touches no running service.
• deploy starts only when build and images both succeeded. It checks that the images exist (Step 4). Every step from 🔧 Ensure Infrastructure onward is unchanged.
• A later push cancels a running or pending images job. If images has already passed, that run may still deploy.
  • If images was cancelled and the new tip is not a (build): commit, nothing deploys. Then push a new (build): tip, or run a force_deploy dispatch on main.
• A failed gate leaves deploy skipped. Production is unchanged. The cleanup job removes that run's four image tags, unless another run of the same commit is live.
• "Re-run failed jobs" after a failed gate stops at 📦 Verify images present. No rollback runs.
• A failure in images sends its own Telegram notice.
• PRs, non-deploy pushes, (build): back no-deploy, (build): observability and dry-run dispatches skip images, deploy and the cleanup job.
• The sourcemap upload now runs before the gates pass. This is accepted. The repo is public, so the sourcemaps expose no private source. A failed run can leave sourcemaps for a release that never shipped.
• Expected numbers (computed, not measured):
  • The images job starts within about 30 s of security ending. Add about 31 s when 🔭 Deploy Observability holds the runner.
  • M2i ≈ 🔒 Security Audit end + 3 + M4 + M3 + 5 s. The 3 s is the job start. In run 39398067267 the security end was +62 s, so M2i ≈ 65 + M4 + M3 + 5 s.
  • M1 ≈ max(M2g, M2i) + W + about 85 s. The 85 s is job 118549257854 without its build (597 s − 513 s = 84 s), plus the new verify step.
  • The M1 targets are in Accountability.

Implementation steps

Read every file in Read first before Step 0. "Comment here" means a comment on this GitHub issue, never a code comment. Do not commit inside any step.

Step 0. Read and check the base

#555 and #564 change the deploy job first, so pinned line numbers will have moved. Find each target by step name and content. Stop only when the meaning differs.

1. Show what changed since the pin, and read each diff. Expect commits from Roll back only after the deploy has started rolling services #555, Prune Docker build cache toward a free-disk target and keep rollback images #564 and other tracker issues:

   git log --oneline d0a53756f8676556358b6adc6c2397d2ed7c64c8..HEAD -- .github/workflows/prod.docs.plus.yml docker-compose.prod.yml .github/WORKFLOWS.md scripts/ci

2. Rollback guard on main. The if of 🚨 Rollback on Failure must require the ROLL_STARTED marker from Roll back only after the deploy has started rolling services #555:

   grep -n -A3 'name: 🚨 Rollback on Failure' .github/workflows/prod.docs.plus.yml

   If the if is still a bare failure(), stop and comment here.

3. Disk guard on main. 💾 Pre-deploy disk guard must meet four conditions:

   • it has no docker builder prune -af --filter "until=24h" line;
   • it prunes toward a free-space target;
   • it keeps the LAST_GOOD_TAG_FILE tag and the current DEPLOY_TAG;
   • it has a 20 GB hard guard.

   Note its step id (Prune Docker build cache toward a free-disk target and keep rollback images #564 names it disk_guard) and its free-disk outputs (Prune Docker build cache toward a free-disk target and keep rollback images #564 names them free_gb_before and free_gb). Note whether the guard reads PREVIOUS_TAG. Note the outputs: block that Prune Docker build cache toward a free-disk target and keep rollback images #564 added to deploy. If any condition fails, stop and comment here.

4. report job. Run grep -n -E '^ report:|free_gb|Build Docker Images|disk guard' .github/workflows/prod.docs.plus.yml. Note how it finds M3, M4 and free disk.

5. migrate still has no build: key, and the image names are unchanged:

   awk '/^  migrate:/,/^  rest-api:/' docker-compose.prod.yml
   grep -n 'image: docsplus-' docker-compose.prod.yml

   Expect a migrate block with an image: line and no build: line, and the four names from Current behaviour item 6. If migrate gained a build: section, or a name changed, stop and comment here.

6. The deploy header still matches Current behaviour item 1: same needs, same -deploy group and the same if text. If the if changed in meaning, stop and comment here.

7. observability-deploy still has needs: [triage].

8. Lint the base: actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml. Expect exit 0. If it already fails, comment here with the output before you edit.

Step 1. Move the env copy and guards into one shared script

Create scripts/ci/prepare-deploy-env.sh. Both prod jobs call it, so the guards exist once, and images aborts before a 9-minute build when a key is missing.

#!/usr/bin/env bash
# Builds the compose env file for one prod job and refuses a host env that lacks
# a key the deploy needs. Both prod jobs call it, so the guards exist once.
set -eo pipefail
: "${ENV_SOURCE:?}" "${ENV_FILE:?}" "${DEPLOY_TAG:?}"

Below that header, paste one block. It starts at the comment # Compose `--env-file` is the single source of truth. It ends after the second for k in …; done loop. At the pin this block is WF L536-L559.

• Move it without any change except removing the 10-space YAML indent.
• Keep the cp, the echo "DEPLOY_TAG=${DEPLOY_TAG}" >> "${ENV_FILE}" line, both for k in loops and both exit 1 lines byte-identical.
• Do not write the host env file path into the script. The script reads it from ENV_SOURCE, which the workflow env: block sets.

Check it: bash -n scripts/ci/prepare-deploy-env.sh exits 0.

Step 2. deploy calls the shared script

In 🔐 Prepare Environment of the deploy job:

1. Keep the deploy-service.sh heredoc first and unchanged. Its comment says it is written first so a later failure can still source it.

2. Replace the block you moved in Step 1 with one line:

   bash scripts/ci/prepare-deploy-env.sh

3. Keep the PREVIOUS_TAG stash and echo "✅ Environment ready" unchanged.

Check it: grep -c 'EMAIL_UNSUBSCRIBE_SECRET' .github/workflows/prod.docs.plus.yml prints 0. grep -c 'EMAIL_UNSUBSCRIBE_SECRET' scripts/ci/prepare-deploy-env.sh prints at least 1.

Step 3. Add the images job

Insert this job after the build job and before the STAGE 3 comment block.

  # ===========================================================================
  # STAGE 2b — DEPLOY IMAGES (prod host, beside the gates)
  # ===========================================================================
  images:
    name: 🧱 Build Images
    runs-on: prod.docs.plus
    timeout-minutes: 30
    needs: [triage, security]
    # Own group: in the `-deploy` group a new pending `images` job would
    # cancel a pending `deploy`. Never share that group.
    concurrency:
      group: ${{ github.workflow }}-images
      cancel-in-progress: false
    # `!cancelled()`, not `always()`: a running `always()` job survives a newer
    # push (run 34223310158), so a stale build would hold the live host for ~9 min.
    if: |
      !cancelled() &&
      needs.triage.result == 'success' &&
      (needs.security.result == 'success' || (github.event_name == 'workflow_dispatch' && inputs.skip_quality_gates)) &&
      ((github.event_name == 'push' && needs.triage.outputs.app_deploy == 'true') ||
       (github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main' && inputs.force_deploy))
    permissions:
      contents: read
    outputs:
      # Use the step id and output names you noted in Step 0.3.
      free_gb_before: ${{ steps.disk_guard.outputs.free_gb_before }}
      free_gb: ${{ steps.disk_guard.outputs.free_gb }}

    steps:
      - name: 📦 Checkout (images)
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 1

      - name: 🔐 Prepare Build Environment
        run: bash scripts/ci/prepare-deploy-env.sh

      # The moved disk guard keeps the rollback images only through PREVIOUS_TAG.
      - name: 📌 Read last-good tag
        run: echo "PREVIOUS_TAG=$(cat "${LAST_GOOD_TAG_FILE}" 2>/dev/null || true)" >> "$GITHUB_ENV"

      # MOVE here, unchanged, from `deploy`: `💾 Pre-deploy disk guard` (with its id),
      # `📂 Verify build context (monorepo root)` and `🏗️ Build Docker Images`.

      - name: 📣 Notify image build failure
        if: failure()
        run: |
          # Copy the OBS_ENV= line of `📣 Notify deploy failure` here verbatim.
          TEXT="⚠️ docs.plus deploy of ${DEPLOY_TAG} aborted before touching prod (env guard, disk guard or image build); production unchanged"
          bash scripts/ci/notify-telegram.sh "${OBS_ENV}" "${TEXT}"

Rules for this job:

• The if. The event and ref clause is the deploy clause, copied exactly. Do not add always(). Do not add needs.build.
• The step names. 💾 Pre-deploy disk guard and 🏗️ Build Docker Images (the M4 and M3 keys) move byte-identical. No other step may match /Build Docker Images/ or /disk guard/. The job name must not match /Deploy Production/ or /Build Verification/.
• Move the build step as it is on main. Keep any flags that other tracker issues added, for example the NEXT_SKIP_TYPECHECK build arg from Skip the duplicate type check inside the webapp Docker build #560.
• No environment: key. The job changes no live service, and GitHub then records one deployment per run, on deploy.
• No deploy-service.sh here. This job rolls nothing. It exports PREVIOUS_TAG for the moved disk guard. Without it, the guard prints holds no commit SHA, sets SWEEP=0 and never sweeps a tag. The disk then fills until the 20 GB guard stops deploys.
• If Prune Docker build cache toward a free-disk target and keep rollback images #564 shipped other output names, use the names you noted in Step 0.3. If it shipped no free-disk output, delete the outputs: block.

Check it: grep -c 'name: 🏗️ Build Docker Images' .github/workflows/prod.docs.plus.yml prints 1, and grep -c 'name: 💾 Pre-deploy disk guard' .github/workflows/prod.docs.plus.yml prints 1.

Step 4. Change the deploy job

1. Header. Change needs and the if. Replace the outputs: block that Prune Docker build cache toward a free-disk target and keep rollback images #564 added with started: ${{ steps.infra.outputs.started }} only. The free-disk outputs now live on images (Step 3). Keep exactly one outputs: key. Keep the -deploy group, timeout-minutes: 45, environment and permissions unchanged.

       needs: [triage, build, images]
       # (concurrency block unchanged)
       if: |
         always() &&
         needs.build.result == 'success' &&
         needs.images.result == 'success' &&
         ((github.event_name == 'push' && needs.triage.outputs.app_deploy == 'true') ||
          (github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main' && inputs.force_deploy))
       # (environment and permissions unchanged)
       outputs:
         started: ${{ steps.infra.outputs.started }}

   deploy keeps always(). That is what protects a roll in flight.

2. Remove the three steps that Step 3 moved.

3. Add 📦 Verify images present as the first step after 🔐 Prepare Environment:

         - name: 📦 Verify images present
           # "Re-run failed jobs" skips a green `images` job, and `migrate` has no
           # build section, so a missing image must stop here, before DEPLOY_STARTED.
           run: |
             MISSING=()
             for img in docsplus-webapp docsplus-rest-api docsplus-hocuspocus docsplus-admin; do
               docker image inspect "${img}:${DEPLOY_TAG}" >/dev/null 2>&1 || MISSING+=("${img}:${DEPLOY_TAG}")
             done
             if [ "${#MISSING[@]}" -gt 0 ]; then
               echo "::error::images missing; use Re-run all jobs; production unchanged (${MISSING[*]})"
               exit 1
             fi
             echo "✅ All four images present at ${DEPLOY_TAG}"

4. 🔧 Ensure Infrastructure exposes started. Add id: infra. Add one line right after the DEPLOY_STARTED line:

         - name: 🔧 Ensure Infrastructure
           id: infra
           run: |
             # (three comment lines; change only the list, see below)
             echo "DEPLOY_STARTED=true" >> "$GITHUB_ENV"
             echo "started=true" >> "$GITHUB_OUTPUT"
             # (rest unchanged)

   In the comment, change (env guard, / disk guard, image build) to (env guard, image check). The disk guard and the build now run in images. Keep the rest of the three lines as Roll back only after the deploy has started rolling services #555 left them.

5. 📣 Notify deploy failure. In the "aborted before touching prod" text, replace (env guard, disk guard or build) with (env guard or missing images). If main still has (env guard or build), replace that instead. Keep the rest of the step as Roll back only after the deploy has started rolling services #555 left it.

Check it:

awk '/^  deploy:/,/^  images-cleanup:/' .github/workflows/prod.docs.plus.yml | grep -n -E '^      - name: |needs:|started' | head -8

Expect these eight lines, in this order:

• needs: [triage, build, images]
• started: ${{ steps.infra.outputs.started }}
• - name: 📦 Checkout Code
• - name: 🔐 Prepare Environment
• - name: 📦 Verify images present
• - name: 🔧 Ensure Infrastructure
• echo "started=true" >> "$GITHUB_OUTPUT"
• - name: 🗄️ Migrate database (one-shot gate)

Step 5. Fix the comments

1. The concurrency comment above deploy (WF L415-L418 at the pin). Replace those four lines with:

     # The job-level `…-deploy` group only serializes a SECOND run's deploy behind
     # this one. The workflow-level `…-quality` group does not cancel a running
     # `always()` job; the new run waits for it (run 34223310158 → 34224526419).
     # A job whose if uses always() (build, deploy, images-cleanup) may still run.

2. The header (WF L9 at the pin). Replace # 4. Deploy: docker compose build on the prod host → blue-green rollout with:

   #   4. Images: docker compose build on the prod host, beside the gates
   #   5. Deploy: verify images → blue-green rollout, after every gate passed

   Change # 5. Report: … from Write a timing table to every CI/CD run summary #556 to # 6. Report: ….

3. The Build Verification note (WF L346-L347 at the pin). Change "The deploy job rebuilds them on the prod host anyway (decided 2026-05)" to "The images job builds them on the prod host (decided 2026-05)". Keep the rest of the sentence.

Check it: grep -n 'still cancels an in-flight roll' .github/workflows/prod.docs.plus.yml prints nothing.

Step 6. Add the cleanup job

Insert it after deploy and before the STAGE 4 comment. It is the cleanup step of ruling 11.

  images-cleanup:
    name: 🧹 Remove Unshipped Images
    runs-on: prod.docs.plus
    timeout-minutes: 5
    needs: [images, deploy]
    if: |
      always() &&
      needs.images.result != 'skipped' &&
      (needs.deploy.result == 'skipped' || needs.deploy.outputs.started != 'true')
    permissions:
      contents: read
      actions: read
    steps:
      - name: 🧹 Remove this run's images
        env:
          GH_TOKEN: ${{ github.token }}
          RUNS_API: ${{ github.api_url }}/repos/${{ github.repository }}/actions/workflows/prod.docs.plus.yml/runs?head_sha=${{ github.sha }}&per_page=1
        run: |
          set -uo pipefail
          # A dispatch or a re-run of the same commit reuses DEPLOY_TAG.
          # Keep the images while another run of this sha is live, or when the
          # query fails. The next disk guard removes them later.
          count() {
            curl -sf --max-time 10 -H "Authorization: Bearer ${GH_TOKEN}" \
              -H 'Accept: application/vnd.github+json' "${RUNS_API}&status=$1" \
              | grep -o '"total_count": *[0-9]*' | grep -o '[0-9]*$'
          }
          for s in queued waiting pending requested in_progress; do
            n=$(count "${s}") || { echo "::warning::Run query failed; keeping ${DEPLOY_TAG} images"; exit 0; }
            [ "${s}" = in_progress ] && n=$((n - 1)) # this run is in progress
            if [ "${n}" -gt 0 ]; then echo "ℹ️ Another run of ${DEPLOY_TAG} is ${s}; keeping its images"; exit 0; fi
          done

          LAST_GOOD=$(cat "${LAST_GOOD_TAG_FILE}" 2>/dev/null || true)
          if [ "${DEPLOY_TAG}" = "${LAST_GOOD}" ]; then
            echo "ℹ️ ${DEPLOY_TAG} is the last-good tag; keeping it"; exit 0
          fi

          # No -f: Docker refuses to remove an image that a container uses.
          for img in docsplus-webapp docsplus-rest-api docsplus-hocuspocus docsplus-admin; do
            docker rmi "${img}:${DEPLOY_TAG}" || echo "ℹ️ kept or absent: ${img}:${DEPLOY_TAG}"
          done

Notes:

• The job checks for a live run with curl on the host. 🩺 Verify Deployment already calls curl on this runner (WF L776-L778). Do not assume gh or jq on the host.
• The job subtracts its own run from the in_progress count, or it would never remove anything.
• The job never fails the run. A missing image, an image in use, or a failed query each prints a line and continues.
• A cancelled run. It is not proven whether this job runs after a cancel. Both cases are safe: it removes the tags, or the next disk guard (Prune Docker build cache toward a free-disk target and keep rollback images #564) does.
• A deploy that started (started == 'true') skips this job, even when it failed. Its tags may be in use or needed for a rollback. The next disk guard handles them.

Check it: actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml exits 0.

Step 7. Point the report job at the moved steps

1. Add images and images-cleanup to its needs.
2. If it finds M3 or M4 only inside the 🚀 Deploy Production job, make it search the steps of every job. Step names are unique in the workflow, so a search across all jobs stays correct.
3. Change FREE_GB (and LOAD1, if present) from needs.deploy.outputs.… to needs.images.outputs.…. If no FREE_GB line exists, add FREE_GB: ${{ needs.images.outputs.free_gb }}. If the disk guard exports load1, add load1 to the images outputs too.
4. Make sure it computes W as 🚀 Deploy Production startedAt minus the later of M2g and M2i.

Check it: grep -n -E 'needs\.deploy\.outputs\.(free|load1)' .github/workflows/prod.docs.plus.yml prints nothing.

Step 8. Update .github/WORKFLOWS.md

In the workflows/prod.docs.plus.yml entry (L10-L15 at the pin):

1. Stages line. Replace it with: "Stages: triage (parse-build-trigger.sh) → quality gates and the prod-host images build → build verification → deploy chain (app → observability → uptime-kuma, fail-fast) → run report".
2. Deploy guard line. Change "or the deploy aborts before building" to "or the images job aborts before building".
3. Failure line.
   • Change aborted before touching prod (production unchanged) to aborted before touching prod (env guard or missing images; production unchanged). If main still has (env guard/build — production unchanged), change that to (env guard or missing images; production unchanged) instead.
   • Add two sentences: "The images job sends its own notice when the image build fails. 🧹 Remove Unshipped Images removes the images of a run whose deploy never started."
4. Add one line under the deploy guard line: "A later push to main while the images job runs cancels that run's deploy. To ship it, push a new (build): tip or run a force_deploy dispatch."

Check it: grep -n -E 'images|later push' .github/WORKFLOWS.md shows the four edits.

Step 9. Local checks

Run every command in Verification, part A. Fix any failure before the checkpoint.

Review checkpoint

Show the change set and stop. Do not commit, stash or push.

git status --short
git diff --stat
git diff -- .github/workflows/prod.docs.plus.yml .github/WORKFLOWS.md
git diff --no-index /dev/null scripts/ci/prepare-deploy-env.sh

Expect changes only in .github/workflows/prod.docs.plus.yml, .github/WORKFLOWS.md and the new scripts/ci/prepare-deploy-env.sh. The maintainer reviews the diff, then commits and pushes per SP2 (#554). This issue is its own commit, separate from #564, so a revert keeps the disk fix.

Acceptance criteria

• actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml exits 0.
• bash .github/scripts/parse-build-trigger.test.sh passes, unchanged.
• scripts/ci/prepare-deploy-env.sh exits 0 with all six keys set, and exits 1 with an ::error:: line when GLITCHTIP_DSN is missing (Verification A3).
• images has needs: [triage, security], an if that starts with !cancelled(), no always(), no needs.build, and timeout-minutes: 30.
• The images if carries the skip_quality_gates escape on the security clause, in the same form as build at WF L373 (ruling 13).
• images uses the group ${{ github.workflow }}-images with cancel-in-progress: false, and it has no environment: key.
• 💾 Pre-deploy disk guard and 🏗️ Build Docker Images each appear exactly once in the workflow, inside images.
• images has a failure notice that contains "aborted before touching prod".
• deploy keeps the name 🚀 Deploy Production, the -deploy group and always(). It has needs: [triage, build, images] and needs.images.result == 'success' in its if.
• 📦 Verify images present is the first step after 🔐 Prepare Environment in deploy, and it comes before 🔧 Ensure Infrastructure.
• 🔧 Ensure Infrastructure has id: infra and writes started=true to $GITHUB_OUTPUT, and deploy exposes started.
• observability-deploy still has needs: [triage], and no job needs images except deploy, images-cleanup and report.
• images-cleanup has the exact if from Step 6 and permissions: actions: read.
• The concurrency comment no longer says that a later push cancels an in-flight roll.
• A dry_run_subject dispatch on main shows images, deploy and images-cleanup as skipped (Verification B1).
• The first PR run after the merge shows images, deploy and images-cleanup as skipped.
• The first real (build): deploy ends success. images starts within 30 s of security ending (60 s when observability ran), and W is 30 s or less.
• On that deploy, 📦 Verify images present and 💾 Stash this tag as last-good both print the run's head SHA.
• On the first real deploy, the images disk guard log shows 🔒 Keeping tags: with the last-good SHA. It shows no holds no commit SHA warning.
• deploy has exactly one outputs: key, and it holds only started. images exposes free_gb_before and free_gb from steps.disk_guard.
• On that deploy, M1 is at most max(M2g, M2i) + 120 s.
• A Measured, <date> comment per SP3 ([Tracking] Faster, change-aware CI/CD pipeline #554) exists here for that run (see Accountability).
• First time it happens: a run with a failed gate shows images-cleanup removing its four tags, or logging why it kept them.
• First time it happens: a run superseded while its images job was running shows images ended cancelled, not success.
• First time a gate fails on its own: "Re-run failed jobs" ends at 📦 Verify images present with no rollback step run. Do not force a gate failure for this check.

Verification

A. Local, before the review checkpoint

1. Lint the workflow:

   actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml

2. Run the trigger tests as a regression check. Triage does not change:

   bash .github/scripts/parse-build-trigger.test.sh

3. Test the shared script with fake values. Run from the repo root:

   bash -n scripts/ci/prepare-deploy-env.sh
   T=$(mktemp -d)
   printf '%s=x\n' GLITCHTIP_DSN NEXT_PUBLIC_GLITCHTIP_DSN EMAIL_PROVIDER EMAIL_FROM EMAIL_UNSUBSCRIBE_SECRET PUBLIC_RESTAPI_URL > "$T/env"
   ENV_SOURCE="$T/env" ENV_FILE="$T/out" DEPLOY_TAG=abc bash scripts/ci/prepare-deploy-env.sh; echo "rc=$?"
   tail -1 "$T/out"
   grep -v '^GLITCHTIP_DSN=' "$T/env" > "$T/env2"
   ENV_SOURCE="$T/env2" ENV_FILE="$T/out2" DEPLOY_TAG=abc bash scripts/ci/prepare-deploy-env.sh; echo "rc=$?"
   rm -rf "$T"

   Expect rc=0 and DEPLOY_TAG=abc first. Then expect an ::error::GLITCHTIP_DSN is missing or empty line and rc=1.

4. Confirm the step names once more:

   grep -n -E 'name: (🏗️ Build Docker Images|💾 Pre-deploy disk guard|📦 Verify images present|🚀 Deploy Production|🧱 Build Images)' .github/workflows/prod.docs.plus.yml

B. After the maintainer pushes

1. Dry-run dispatch. It proves the skips only, because force_deploy is false. Run SP1 ([Tracking] Faster, change-aware CI/CD pipeline #554).

   gh workflow run prod.docs.plus.yml --repo docs-plus/docs.plus --ref main -f dry_run_subject='(build): front'
   gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --event workflow_dispatch --limit 1
   gh run view <RUN> --repo docs-plus/docs.plus --json jobs --jq '.jobs[] | [.name, .conclusion] | @tsv'

   Expect 🧱 Build Images, 🚀 Deploy Production and 🧹 Remove Unshipped Images as skipped. The gates run.

2. There is no rehearsal of the image path. The first real (build): push is the first test. The maintainer pushes it at a quiet hour and watches it.

3. Read the metrics from that run, and from any later run, with this command:

   gh run view <RUN> --repo docs-plus/docs.plus --json createdAt,jobs > run.json
   jq -r '
     (.createdAt | fromdate) as $t0
     | [.jobs[].steps[]?] as $steps
     | def off($t): if $t then (($t | fromdate) - $t0) else null end;
       def step($re): [$steps[] | select(.name | test($re))][0];
       def job($re): [.jobs[] | select(.name | test($re))][0];
       def dur($x): if $x then (($x.completedAt | fromdate) - ($x.startedAt | fromdate)) else null end;
     {
       M1: off(step("Verify Deployment").completedAt),
       M2g: off(job("Build Verification").completedAt),
       M2i: off(step("Build Docker Images").completedAt),
       M3: dur(step("Build Docker Images")),
       M4: dur(step("disk guard")),
       M5: dur(step("Deploy Services")),
       images_start: off(job("Build Images").startedAt),
       security_end: off(job("Security Audit").completedAt),
       deploy_start: off(job("Deploy Production").startedAt),
       deploy_job_s: dur(job("Deploy Production"))
     }
     | .W = (if .deploy_start and .M2g and .M2i then .deploy_start - ([.M2g, .M2i] | max) else null end)' run.json

   On run 39398067267 this command prints M1 1132, M2g 538, M2i 1057, M3 513, M4 0 and M5 57. Before this change, this command prints a negative W, because it skips the same-job rule of Write a timing table to every CI/CD run summary #556 and Measure the CI/CD after benchmark for v2.3 #569. Read a pre-Build the deploy images while the quality gates run #565 W from those issues.

Accountability

Record per SP3 (#554) within one day of the first real (build): deploy after the merge. Add the images and deploy job ids, deploy_job_s, images_start and security_end from Verification B3.

• The maintainer checks that comment against the Acceptance criteria. After three successful deploys, the builder posts a second Measured, <date> comment with all three runs.
• Checks that wait for an event: a failed gate, a superseded run and a re-run. Each one gets its own Measured, <date> comment, with the run id, when it first happens.
• A missed speed target is not a revert. If M1 does not fall below the 777 s baseline minimum after three deploys, comment here with the metrics. Name the stage that missed, from M2g, M2i and W.

Rollback

Revert per SP4 (#554) at once if any of these happens:

• deploy rolls a tag that is not its run's head SHA;
• a rollback runs on a run that failed before 🔧 Ensure Infrastructure;
• deploy starts while any gate failed and skip_quality_gates was not set;
• the cleanup job or the disk guard removes the last-good tag, or an image a container uses;
• an images job keeps building more than 2 min after its run was cancelled.

No host step and no GitHub setting is needed. Leftover images from early builds stay on the host until the next disk guard removes them.

Docs to update

• .github/workflows/prod.docs.plus.yml: the header, the Build Verification note and the concurrency comment (Step 5).
• .github/WORKFLOWS.md: the workflows/prod.docs.plus.yml entry, the stages, deploy guard and failure lines (Step 8).
• Checked, no change needed. Each still describes deploy correctly:
  • apps/hocuspocus.server/CLAUDE.md:273 (the Traefik copy);
  • docker-compose.prod.yml:10;
  • Makefile:388.

Out of scope

• Do not build on hosted runners or push to a registry (GHCR). The images stay on the prod host (decided 2026-05). See [Tracking] Faster, change-aware CI/CD pipeline #554 §Not doing.
• Do not change the build commands. --no-cache is Measure whether the backend image build still needs --no-cache #566, parallel compose calls are Try running the backend and webapp image builds at the same time #567, and the trace step is Measure and trim the Next.js build-trace step in the webapp build #568.
• Do not change the disk guard's prune or thresholds (Prune Docker build cache toward a free-disk target and keep rollback images #564), or the rollback and its markers (Roll back only after the deploy has started rolling services #555).
• Do not run Build Verification in parallel with the gates. After this issue it is off the critical path.
• Do not add a needs edge from observability-deploy to images (WF L973-L978).
• Do not edit the uptime-kuma-deploy concurrency comment (WF L993-L997). It makes the same stale "cancelling the parent run" claim. Make CI failures reach Telegram, and fix stale CI facts #557 owns stale CI facts, so comment there if it still stands.
• Do not change runner settings (Restrict the production self-hosted runner to main deploys #552).
• Follow the settled Docker rules in [Tracking] Faster, change-aware CI/CD pipeline #554 §Shared rules (AGENTS.md §Docker). No laptop timings.
• Do not touch dependencies. The metascraper pin stays at 5.50.6, and Bun stays at 1.4.0 or newer.
• No new tests. actionlint, the dry-run dispatch and the first real deploy check the workflow wiring.

Read first

• .github/workflows/prod.docs.plus.yml, all of it, on main.
• docker-compose.prod.yml, the migrate block and the four image: lines.
• scripts/ci/notify-telegram.sh.
• .github/WORKFLOWS.md, the workflows/prod.docs.plus.yml entry.
• .github/actionlint.yaml.
• AGENTS.md: §Docker (L286-L308), §Git And Commits, and the tip-commit fact (L416).
• [Tracking] Faster, change-aware CI/CD pipeline #554, Roll back only after the deploy has started rolling services #555, Prune Docker build cache toward a free-disk target and keep rollback images #564, Write a timing table to every CI/CD run summary #556 and Restrict the production self-hosted runner to main deploys #552.