Skip to content

Try running the backend and webapp image builds at the same time #567

Open

Description

@HMarzban

Part of #554. Change set E. Work on main. Rulings: #554 rulings 1, 2, 4, 5, 7, 11, 14, 15, 16.

Summary

  • Today. 🏗️ Build Docker Images runs two docker compose … build calls in series. The backend call must finish before the front call starts.
  • Measured cost. On six deploys, the backend call took 69–80 s and the front call took 414–487 s. The gap between them was 0 s. M3 equals their sum.
  • After. Both calls start at once with &. The step checks both exit codes with wait, so a failed call still fails the step.
  • Expected saving. 45–70 s off M3. The ceiling is the backend call time. CPU contention on the 4-vCPU host lowers the saving. Nobody has measured that contention.
  • This is a trial over two normal deploys. The maintainer keeps or reverts it from M3 and the host numbers the step records.
Run Build job M3 (s) Backend call (s) Front call (s)
39398067267 118549257854 513 76 436
39297468969 118217497479 483 69 414
39275929831 118142985559 550 74 475
38392614215 115281929767 542 70 471
38239392350 114780192678 559 71 487
38068638114 114262648443 557 80 477

Each call runs from its first Image docsplus-<name>:<sha> Building log line to its Image docsplus-<name>:<sha> Built line. The backend uses docsplus-hocuspocus; the front uses docsplus-webapp. The Step 0.5 script reads the same lines.

The admin image finished 96–112 s after the front call started. So the backend call overlaps the most CPU-heavy part of the front call: the admin build and both webapp installs.

These six deploys are history only. Sets B and D change M3 before this issue lands. Step 0 reads a fresh baseline.

Depends on / Blocks

Current behaviour

Links are pinned to d0a53756f8676556358b6adc6c2397d2ed7c64c8. After #565, these lines sit in the images job at new line numbers. Step 0 finds them by name.

  1. The step runs the two calls in series (prod.docs.plus.yml:612-642). The backend call is at :633-634:
    docker compose -f "${COMPOSE_FILE}" --env-file "${ENV_FILE}" \
      build --no-cache rest-api hocuspocus-server
    The front call is at :639-640:
    docker compose -f "${COMPOSE_FILE}" --env-file "${ENV_FILE}" \
      build webapp admin-dashboard
  2. Inside one call, the named services already build at once (:636-638). The series is between the two calls.
  3. The calls stay separate because of --no-cache. Only the backend call passes it (:628-632). One call cannot apply the flag to some services only.
  4. The compose file has no no_cache key, and other callers read it.
  5. Host ([Tracking] Faster, change-aware CI/CD pipeline #554 §Host facts): 4 vCPUs. next build runs several workers, so two builds at once compete for CPU. The calls write the same layers as today, so disk use does not change. Prune Docker build cache toward a free-disk target and keep rollback images #564 owns disk.
  6. Settled Docker rules: [Tracking] Faster, change-aware CI/CD pipeline #554 §Shared rules. --parallel is not a docker compose build flag in Compose v2 (AGENTS.md:300). Docker build timings do not transfer from a laptop (AGENTS.md:293). No build-once-then-docker tag (AGENTS.md:294). Never measure a cache fix under --no-cache (AGENTS.md:301). Prod deploys read only the pushed tip commit subject (AGENTS.md:416).

Target behaviour

  • The step keeps its name, 🏗️ Build Docker Images, and its job.
  • The backend call and the front call start at the same time, in that one step. Each keeps its exact arguments from main.
  • Each call streams live into the job log. Every line starts with [backend] or [front] , so a hung call still shows its log.
  • The step waits for both calls. When one call fails, the other runs to its end. A failed backend call then reports only when the front call ends, up to about 487 s later.
  • The step fails when either call exits non-zero, and the error names both exit codes.
  • The step summary gets an "Image build calls" block:
    • the wall time of the step;
    • each call's exit code and seconds;
    • load1_max, mem_avail_min_mb, swap_used_start_mb, swap_used_max_mb and the sample count, sampled every 5 s;
    • restarts_during_build, unhealthy_before and unhealthy_after for the live containers.
  • The load sampler and the restart check stay after the trial.
  • Nothing else changes: the compose file, the Dockerfiles, the other steps, stage and the Makefile.

Implementation steps

Read every file in Read first before Step 0. "Comment here" means a comment on this GitHub issue, never a code comment.

Step 0. Read and check the base

  1. Confirm Build the deploy images while the quality gates run #565 is merged on main.
    git show main:.github/workflows/prod.docs.plus.yml \
      | awk '/^  [a-z][a-z0-9-]*:$/ {job=$1} /name: 🏗️ Build Docker Images/ {print job, NR}'
    At the pinned commit it prints deploy: 612. It must now print images: with some line number. If it prints deploy:, stop and comment here.
  2. Confirm the Measure whether the backend image build still needs --no-cache #566 A/B is finished. Its backend_no_cache input must be gone from workflow_dispatch. After a drop, Measure whether the backend image build still needs --no-cache #566 must also have its second "Measured, " comment for the next two (build): deploys. If either check fails, stop and comment here.
  3. Print the step and copy both calls.
    git show main:.github/workflows/prod.docs.plus.yml | grep -n -A 60 "name: 🏗️ Build Docker Images"
  4. Confirm the compose file has no no_cache key. Run git grep -n no_cache main -- docker-compose.prod.yml. Expect no output.
  5. Read the baseline, before you edit anything.
    • List recent deploy runs:
      gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --event push --branch main \
        --status success --limit 100 --json databaseId,createdAt,attempt,headSha
    • Keep a run only when its attempt is 1 and its subject starts with (build):. Read the subject with gh api repos/docs-plus/docs.plus/actions/runs/<id> -q .head_commit.message.
    • Take the last 3 such runs on main whose build step ran in the images job. The script below prints that job as build_job_name. It must not be 🚀 Deploy Production.
    • Run git fetch origin main first, so every SHA exists locally. The last baseline SHA is the headSha of the latest of the three runs. Each baseline run must build the same way as main today:
      • git show <headSha>:.github/workflows/prod.docs.plus.yml | grep -n -A 60 'name: 🏗️ Build Docker Images' must show the same two calls, with the same arguments, as Step 0.3.
      • git diff --quiet <headSha> <last baseline SHA> -- apps/webapp/docker/Dockerfile.bun apps/admin-dashboard/docker/Dockerfile.bun apps/hocuspocus.server/docker/Dockerfile.bun docker-compose.prod.yml apps/webapp/next.config.js apps/admin-dashboard/next.config.js must exit 0.
      • If a run fails either check, skip it and walk further back. The images job proves only that Build the deploy images while the quality gates run #565 landed, not set B or the Measure whether the backend image build still needs --no-cache #566 result.
    • If fewer than 3 such runs exist, stop and wait for more normal deploys. Do not fall back to runs that fail the checks.
    • Note each run's cache class:
      • Warm: the previous build started 24 h or less before it, and git diff --quiet <previous SHA> <SHA> -- bun.lock extensions/ exits 0.
      • The previous build is the latest earlier prod.docs.plus.yml run, of any event or conclusion, where bun x2-read.ts <id> prints a non-null m3. Its headSha is the previous SHA. Compare the two runs' createdAt for the 24 h rule.
      • List the candidates with gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml --branch main --limit 100 --json databaseId,createdAt,event,conclusion,headSha. Walk back from the baseline run, one run at a time.
      • Cold: every other run.
    • Save this read-only script outside the repo as x2-read.ts. Run bun x2-read.ts <runId> <runId> <runId>.
      // bun x2-read.ts <runId> ... → one JSON row per run (read-only gh calls)
      const R = "docs-plus/docs.plus";
      const gh = (...a: string[]) => Bun.spawnSync(["gh", ...a]).stdout.toString();
      for (const id of Bun.argv.slice(2)) {
        const d = JSON.parse(gh("run", "view", id, "--repo", R, "--json", "jobs,createdAt,attempt,headSha"));
        const t0 = Date.parse(d.createdAt);
        const off = (s?: string) => (s ? Math.round((Date.parse(s) - t0) / 1000) : null);
        const dur = (s: any) => (s ? Math.round((Date.parse(s.completedAt) - Date.parse(s.startedAt)) / 1000) : null);
        const steps = d.jobs.flatMap((j: any) => (j.steps ?? []).map((s: any) => ({ ...s, job: j })));
        const step = (re: RegExp) => steps.find((s: any) => re.test(s.name));
        const build = steps.find((s: any) => /Build Docker Images/.test(s.name) && s.conclusion !== "skipped");
        const log = build ? gh("run", "view", "--repo", R, "--job", String(build.job.databaseId), "--log") : "";
        const at = (re: RegExp) => {
          const m = log.split("\n").find((l) => re.test(l))?.match(/\d{4}-\d\d-\d\dT[\d:.]+Z/);
          return m ? Date.parse(m[0]) : NaN;
        };
        const sec = (a: number, b: number) => (Number.isNaN(a) || Number.isNaN(b) ? null : Math.round((b - a) / 1000));
        const tag = d.headSha;
        const bs = at(new RegExp(`Image docsplus-hocuspocus:${tag} Building`));
        const be = at(new RegExp(`Image docsplus-hocuspocus:${tag} Built`));
        const fs = at(new RegExp(`Image docsplus-webapp:${tag} Building`));
        const fe = at(new RegExp(`Image docsplus-webapp:${tag} Built`));
        console.log(JSON.stringify({
          id, attempt: d.attempt, sha: tag.slice(0, 9),
          build_job: build?.job.databaseId ?? null, build_job_name: build?.job.name ?? null,
          m1: off(step(/Verify Deployment/)?.completedAt),
          m2g: off(d.jobs.find((j: any) => /Build Verification/.test(j.name))?.completedAt),
          m2i: off(build?.completedAt), m3: dur(build),
          back_s: sec(bs, be), front_s: sec(fs, fe), overlap_s: Math.max(0, sec(Math.max(bs, fs), Math.min(be, fe)) ?? 0),
        }));
      }
    • Check the script on run 39398067267 first. It prints build_job_name 🚀 Deploy Production, m3 513, back_s 76, front_s 436 and overlap_s 0. A run whose build step was skipped or never started prints m3 null.
    • Post the three rows and their cache classes as a comment here, headed "Baseline, ". Name the full last baseline SHA.
  6. Check that the image path is the long one. After Build the deploy images while the quality gates run #565, M1 follows the later of M2g and M2i. If m2i is below m2g on all three baseline runs, this issue cannot lower M1. Stop, and comment here with the numbers and a proposal to close this issue.

Step 1. Run the two calls at once

File: .github/workflows/prod.docs.plus.yml, step 🏗️ Build Docker Images, in the images job.

  1. Keep the step's name and env lines. Keep the comment block above the backend call, which starts "rest-api, hocuspocus-server and hocuspocus-worker share one build definition".
  2. Replace the rest of the run: block with the block below. Put the two calls from Step 0.3 in the two build_call lines. Write only the arguments that follow build, not the docker compose prefix.
  3. Delete the old 3-line comment that starts "No --parallel: it is not a docker compose build flag". The new comment replaces it.
      - name: 🏗️ Build Docker Images
        env:
          DOCKER_BUILDKIT: '1'
          COMPOSE_DOCKER_CLI_BUILD: '1'
          # Keep every other env line from main here, such as SKIP_QUALITY_GATES.
        run: |
          echo "🔨 Building images with tag: ${DEPLOY_TAG}"
          # (keep the existing comment block about the shared backend build and --no-cache here)

          BUILD_DIR="${RUNNER_TEMP}/image-build"
          mkdir -p "${BUILD_DIR}"
          : > "${BUILD_DIR}/times.txt"

          # Sum of restart counts over running containers; a rise means a live container died during the build.
          restarts() { { docker ps -q | xargs -r docker inspect -f '{{.RestartCount}}' 2>/dev/null || true; } | awk '{s += $1} END {print s + 0}'; }
          unhealthy() { { docker ps -q --filter health=unhealthy || true; } | awk 'END {print NR}'; }
          RESTARTS_BEFORE=$(restarts)
          UNHEALTHY_BEFORE=$(unhealthy)

          # Host load, available memory and used swap every 5 s, for the keep-or-revert call (#567).
          ( while sleep 5; do
              printf '%s %s %s\n' "$(date +%s)" "$(cut -d' ' -f1 /proc/loadavg)" \
                "$(awk '/^MemAvailable:/ {a=$2} /^SwapTotal:/ {t=$2} /^SwapFree:/ {f=$2} END {print int(a/1024), int((t-f)/1024)}' /proc/meminfo)"
            done ) > "${BUILD_DIR}/host.txt" 2>/dev/null &
          SAMPLER_PID=$!
          # Stop the sampler and both build calls on every exit, also on a cancel.
          stop_all() {
            kill "${SAMPLER_PID}" 2>/dev/null || true
            for p in "${BACK_PID:-}" "${FRONT_PID:-}"; do
              [ -z "$p" ] && continue
              pkill -TERM -P "$p" 2>/dev/null || true
              kill "$p" 2>/dev/null || true
            done
          }
          trap stop_all EXIT
          trap 'exit 143' INT TERM

          # Streams one compose build call with a name prefix, so a hung call still shows its log.
          build_call() {
            local name=$1 start=$SECONDS rc=0
            shift
            docker compose -f "${COMPOSE_FILE}" --env-file "${ENV_FILE}" build "$@" 2>&1 \
              | sed -u "s/^/[${name}] /" || rc=$?
            echo "${name} exit=${rc} seconds=$((SECONDS - start))" >> "${BUILD_DIR}/times.txt"
            return "${rc}"
          }

          # The calls stay separate because only the backend passes --no-cache. They run
          # at once, and Compose v2 has no --parallel flag (AGENTS.md §Docker).
          STEP_START=$SECONDS
          build_call backend --no-cache rest-api hocuspocus-server &
          BACK_PID=$!
          build_call front webapp admin-dashboard &
          FRONT_PID=$!

          BACK_RC=0
          wait "${BACK_PID}" || BACK_RC=$?
          FRONT_RC=0
          wait "${FRONT_PID}" || FRONT_RC=$?
          kill "${SAMPLER_PID}" 2>/dev/null || true

          RESTARTS_AFTER=$(restarts)
          UNHEALTHY_AFTER=$(unhealthy)
          HOST_LINE=$(awk 'NR == 1 {l = $2; m = $3; s0 = $4; s = $4} {if ($2 > l) l = $2; if ($3 < m) m = $3; if ($4 > s) s = $4} END {printf "load1_max %s, mem_avail_min_mb %s, swap_used_start_mb %s, swap_used_max_mb %s, samples %d", l, m, s0, s, NR}' "${BUILD_DIR}/host.txt")
          {
            echo "### 🏗️ Image build calls"
            echo "- wall: $((SECONDS - STEP_START)) s"
            sed 's/^/- /' "${BUILD_DIR}/times.txt"
            echo "- host: ${HOST_LINE}"
            echo "- live containers: restarts_during_build $((RESTARTS_AFTER - RESTARTS_BEFORE)), unhealthy_before ${UNHEALTHY_BEFORE}, unhealthy_after ${UNHEALTHY_AFTER}"
          } | tee -a "${GITHUB_STEP_SUMMARY}"

          if [ "${BACK_RC}" -ne 0 ] || [ "${FRONT_RC}" -ne 0 ]; then
            echo "::error::Image build failed: backend exit ${BACK_RC}, front exit ${FRONT_RC}. Read the [backend] and [front] lines above."
            exit 1
          fi
          echo "✅ Images built"

Notes for the builder:

  • || BACK_RC=$?. prod.docs.plus.yml:64-67 sets shell: bash, so the step runs bash --noprofile --norc -eo pipefail. A bare wait on a failed call exits before the second exit code is read.
  • || rc=$? in build_call. Under pipefail, the pipe status is the docker compose exit code. Without the guard, set -e ends the call before it writes times.txt.
  • stop_all. A & job in non-interactive bash ignores SIGINT. So the step stops both calls and their docker compose children itself.
  • || true inside stop_all, restarts and unhealthy. set -e also applies inside a trap, and a container can stop between docker ps and docker inspect. Either would change the step's exit code.
  • The front call with Skip the duplicate type check inside the webapp Docker build #560. Keep its two lines that set NEXT_SKIP_TYPECHECK above STEP_START. The call then reads build_call front --build-arg "NEXT_SKIP_TYPECHECK=${NEXT_SKIP_TYPECHECK}" webapp admin-dashboard &.
  • The backend call after Measure whether the backend image build still needs --no-cache #566. If --no-cache is gone, the call reads build_call backend rest-api hocuspocus-server &. Then replace "because only the backend passes --no-cache" in the new comment with: "The calls stay separate so each keeps its own flags."
  • {{.RestartCount}} has no $, so GitHub does not read it as an expression. Line 740 of the pinned file uses {{.Names}} the same way.
  • The [backend] and [front] lines keep GitHub's timestamps, so x2-read.ts still reads each call's seconds.

Check:

git diff --stat
git diff .github/workflows/prod.docs.plus.yml
git diff -U0 .github/workflows/prod.docs.plus.yml | grep -E '^[-+] +[A-Z_]+: '

Expect one file changed, only inside 🏗️ Build Docker Images. The third command must print nothing, so the env: lines are unchanged.

Step 2. Lint the workflow

Install shellcheck first, so actionlint also lints the run: block.

actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml

Expect exit code 0 and no output.

Step 3. Prove the exit-code logic with a fake docker

A one-off check outside the repo, not a committed test.

  1. Copy the run: block from Step 1 into step.sh in a scratch folder. Remove the 10-space YAML indent.
  2. Create bin/docker and run chmod +x bin/docker:
    #!/bin/bash
    case "$1" in ps|inspect) exit 0 ;; esac
    echo "fake docker $*"
    case "$*" in
      *rest-api*) sleep 2; [ "$FAIL_SVC" = back ] && exit 3 ;;
      *webapp*) sleep 3; [ "$FAIL_SVC" = front ] && exit 4 ;;
    esac
    exit 0
  3. Run the three cases from that folder:
    for f in none back front; do
      mkdir -p "t-$f"; : > "t-$f/sum.md"
      PATH="$PWD/bin:$PATH" FAIL_SVC=$f RUNNER_TEMP="$PWD/t-$f" GITHUB_STEP_SUMMARY="$PWD/t-$f/sum.md" \
        DEPLOY_TAG=abc COMPOSE_FILE=c.yml ENV_FILE=e.env bash --noprofile --norc -eo pipefail step.sh
      echo "FAIL_SVC=$f step exit=$?"
    done
  4. Pass rules:
    • none ends with ✅ Images built and step exit=0. The wall time is about 3 s, not 5 s.
    • back prints backend exit 3, front exit 0 and step exit=1.
    • front prints backend exit 0, front exit 4 and step exit=1.
    • Every fake line starts with [backend] or [front] .
    • grep -c -e '- backend exit=' -e '- front exit=' t-<case>/sum.md prints 2 in every case.
    • On macOS, /proc does not exist, so the host line shows empty values. On Linux it shows numbers.
  5. Prove the final check by sabotage. Restore each line after its leg.
    • Change if [ "${BACK_RC}" -ne 0 ] || [ "${FRONT_RC}" -ne 0 ]; then to if [ "${BACK_RC}" -ne 0 ]; then. The front case must print ✅ Images built and step exit=0.
    • Change wait "${BACK_PID}" || BACK_RC=$? to wait "${BACK_PID}". The back case must print step exit=3 with no summary block and no ::error:: line.
    • In build_call, delete || rc=$? from the sed -u line. The back case prints step exit=1, but t-back/sum.md holds no - backend exit= line.
    • Run all three cases again. Expect the pass rules in item 4.
  6. Wait 6 s, then run ps -ax -o pid,command | grep step.sh | grep -v grep. Expect no output.
  7. Cancel check.
    • In bin/docker, change sleep 3 to sleep 30.
    • Run PATH="$PWD/bin:$PATH" FAIL_SVC=none RUNNER_TEMP="$PWD/t-none" GITHUB_STEP_SUMMARY="$PWD/t-none/sum.md" DEPLOY_TAG=abc COMPOSE_FILE=c.yml ENV_FILE=e.env bash --noprofile --norc -eo pipefail step.sh & S=$!; sleep 1; kill -TERM "$S"; sleep 3; ps -ax -o pid,command | grep '[b]in/docker'.
    • Expect no output. Use TERM, not INT, because a test shell starts & jobs with SIGINT ignored.
    • Restore sleep 3.

Review checkpoint

Show git diff, the actionlint output and the Step 3 output. Confirm the log-prefix handling in #569 and #568. Then stop. Do not run git add, git commit, git push or git stash. The maintainer reviews the diff, then commits and pushes per SP2 (#554), as its own commit.

Acceptance criteria

  • Step 0 checks passed, and the "Baseline, " comment holds three runs with their cache classes.
  • The step is still named 🏗️ Build Docker Images and still sits in the images job.
  • Both calls start with & in that one step. Each keeps its exact arguments from main.
  • The step's env: lines are unchanged. git diff -U0 .github/workflows/prod.docs.plus.yml | grep -E '^[-+] +[A-Z_]+: ' prints nothing.
  • Before merge, Measure the CI/CD after benchmark for v2.3 #569 Step 1 and Measure and trim the Next.js build-trace step in the webapp build #568 Step 5, item 4, still handle the prefix, or a comment there names it.
  • docker-compose.prod.yml, stage.docs.plus.yml, Makefile and every Dockerfile are unchanged.
  • No --parallel was added anywhere.
  • actionlint exits 0 with the repo config.
  • The Step 3 fake-docker cases pass, and the sabotage case failed as described.
  • The Step 3 cancel check leaves no fake docker process.
  • On the first real deploy, the step summary shows the "Image build calls" block with numbers in every field.
  • The job log shows [backend] and [front] lines that overlap in time.
  • Two "Measured, " comments exist here, one per deploy.
  • A maintainer ruling comment says keep or revert, and the AGENTS.md §Docker bullet records the result.

Verification

Before merge:

actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml
git diff --exit-code docker-compose.prod.yml .github/workflows/stage.docs.plus.yml Makefile
grep -n -- "--parallel" .github/workflows/prod.docs.plus.yml | grep -v "#"
  • The second command must exit 0. The third must print nothing; it drops comment lines, because the new comment names --parallel.
  • bash .github/scripts/parse-build-trigger.test.sh and a dry_run_subject dispatch do not apply. This issue changes no triage, job if, needs or gating.

After merge, read each of the first two deploys with bun x2-read.ts <runId> (Step 0.5), and copy the "Image build calls" block from the step summary. Expected values are in Accountability. overlap_s must be close to back_s, which shows the calls ran at once. front_s minus the baseline front_s of the same class is the contention cost.

Accountability

  • Who pushes. The maintainer pushes the two measured deploys, at an hour the maintainer judges quiet. They are the first two (build): push runs whose tree contains this change, and the push that carries it can be the first. They are normal deploys with real changes; no force_deploy dispatch counts.
  • A measured deploy counts only when its two calls carry the Step 0.3 arguments and git diff --quiet <last baseline SHA> <measured SHA> -- apps/webapp/docker/Dockerfile.bun apps/admin-dashboard/docker/Dockerfile.bun apps/hocuspocus.server/docker/Dockerfile.bun docker-compose.prod.yml apps/webapp/next.config.js apps/admin-dashboard/next.config.js exits 0. Otherwise comment here, take a new Step 0.5 baseline, and measure the next deploys.
  • Who reads. The builder of this issue, or the agent the maintainer names, reads each run after 🩺 Verify Deployment ends.
  • Record per SP3 ([Tracking] Faster, change-aware CI/CD pipeline #554), one comment per deploy. Metrics as defined in Measure the CI/CD after benchmark for v2.3 #569.
    • Metric row: M3, read from each of the two measured runs, against the class baseline, expected 45–70 s lower.
    • M2i falls by the same amount. M1 falls by the M3 saving only when M2i is later than M2g. M2g, M4, M5, W and M6 stay unchanged.
    • Also add the build job id, the cache class and its baseline runs, back_s, front_s, overlap_s, and the "Image build calls" block.
  • The class baseline is the median M3 of the Step 0.5 runs in the deploy's cache class. With two runs, it is their mean. With one run, it is that run's M3. With none, it is the median M3 of all three baseline runs, and the comment says so.
  • Who rules. The maintainer, in one comment after the second deploy.
  • Keep when both deploys pass every stop condition, and M3 is at least 30 s below the class baseline on both.
  • Stop condition: revert when any of these holds on either deploy:
    • either call fails with exit code 137, signal: killed or cannot allocate memory;
    • mem_avail_min_mb is below 2048;
    • swap_used_max_mb minus swap_used_start_mb is above 512;
    • restarts_during_build is above 0, or unhealthy_after is above unhealthy_before;
    • 🩺 Verify Deployment fails on that run;
    • M3 is not below the class baseline.
  • A stop condition on the first measured deploy. Do not wait for a second deploy. Comment here the same day with the numbers. The maintainer reverts before the next (build): push.
  • A mixed result (both pass every stop condition, but one saves less than 30 s) needs a third deploy. Comment here with the numbers. The maintainer rules after the third one.

Rollback

Revert per SP4 (#554) on a stop condition in Accountability. The commit touches only .github/workflows/prod.docs.plus.yml. No host step is needed: the step writes only under ${RUNNER_TEMP}, which the runner clears after each job. A failed build leaves production unchanged, because the build ends before 🔧 Ensure Infrastructure.

Docs to update

  • In this change: only the new comment inside the step (Step 1). It replaces the old --parallel comment.
  • After the maintainer's ruling, in the revert commit, or in its own commit for a keep: add one bullet to AGENTS.md §Docker after the --parallel bullet (line 300 at the pinned commit). Name the two run ids, M3 before and after, the cache class, mem_avail_min_mb, and "kept" or "reverted".
  • .github/WORKFLOWS.md: no change. It does not describe the image build calls.

Out of scope

Read first

Activity

  1. added this to the v2.3.0 milestone on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions