Skip to content

Prune Docker build cache toward a free-disk target and keep rollback images #564

Open

Description

@HMarzban

Part of #554. Rulings: #554 rulings 1, 5, 11, 12, 14, 15, 16.

Summary

  • Today. The 💾 Pre-deploy disk guard in 🚀 Deploy Production prunes by age (until=24h). On 2026-10-20 it reclaimed 0 B on all three deploys:

    Run Job Free before prune Image prune Cache prune Free after prune M4
    39398067267 118549257854 47 GB Total reclaimed space: 0B Total: 0B 46 GB 0 s
    39412528288 118596011402 33 GB Total reclaimed space: 0B Total: 0B 32 GB 1 s
    39414945889 118603664970 29 GB Total reclaimed space: 0B Total: 0B 28 GB 0 s
  • The risk. One deploy wrote 14 GB. A disk with 15 GB free passes the 10 GB guard. The build then fills the root disk of the live containers (ENOSPC).

  • After. See Target behaviour. This is the disk-guard half of the ruling 11 cleanup. It is a safety fix with no time saving, and must not move M1 (Measure the CI/CD after benchmark for v2.3 #569).

Depends on / Blocks

Current behaviour

Links are pinned to d0a53756f8676556358b6adc6c2397d2ed7c64c8. Host facts are on #554. Extra host facts from the same read: disk /dev/vda1; BuildKit v0.26.2; images 12.03 GB (11.32 GB reclaimable); docker builder prune lists --max-used-space, --min-free-space, --reserved-space, no --keep-storage. Step 0 reads them again.

  1. The disk guard prunes by age (prod.docs.plus.yml:573-594):

    docker image prune -af --filter "until=24h" 2>/dev/null || true
    docker builder prune -af --filter "until=24h" 2>/dev/null || true

    With several deploys a day, nothing is old enough, so both free 0 B. After a gap over 24 h, the whole cache goes. 2>/dev/null || true hides every error.

  2. docker image prune -af cannot exclude one tag. After a manual docker compose down, no container holds the last-good image. Once it is 24 h old, the prune removes it, and the next rollback fails.

  3. The hard guard aborts only below 10 GB (:584-594):

    if [ "${AVAIL_GB}" -lt 10 ]; then
      echo "::error::Less than 10 GB free on /. Aborting deploy. SSH to host and run 'docker system prune -af --volumes'."

    The hint is unsafe. -a can remove a last-good image and all build cache. --volumes removes unused anonymous volumes (docker/cli system/prune.go:51); docker volume prune -a could remove redis-data and traefik-certs. The new text names no volume command.

  4. The rollback tag. DEPLOY_TAG is the commit SHA (:77). LAST_GOOD_TAG_FILE is at :80. 🔐 Prepare Environment reads it into PREVIOUS_TAG first (:561-570). Rollback reads only PREVIOUS_TAG (:876-880). No step reads last-good-tag.previous (:805-807).

  5. Rollback needs all four images at PREVIOUS_TAG (:870-889). If one is missing, it exits 1. Its comment cites the age filter:

    # tag must still exist locally. The cleanup step honors --filter
    # until=24h, so within a 24h window this works reliably; outside
  6. The image names are docsplus-hocuspocus (migrate, hocuspocus-server, hocuspocus-worker), docsplus-rest-api, docsplus-webapp and docsplus-admin. All use :${DEPLOY_TAG:-latest} (docker-compose.prod.yml:161, :171, :289, :398, :491, :590).

  7. The deploy job has no outputs: (:420-441).

  8. 🧹 Cleanup prunes dangling images only (:812-818). It stays.

  9. Only buildx has the size flags. At v29.1.3 the CLI's docker builder prune has only --keep-storage (docker/cli prune.go:57-61). buildx v0.30.1 has the three size flags (docker/buildx prune.go:178-187). On the host, docker builder prune routes to buildx. buildx errors on BuildKit older than v0.17.0 (prune.go:114-123).

  10. How BuildKit v0.26.2 treats the size flags (manager.go:1090-1120):

    • Below --min-free-space, it deletes least-recently-used records to cover the shortfall, never below --reserved-space. At or above the target, the strict < at line 1118 deletes one record.
    • "Free" is Bfree (disk_unix.go:17-21), which counts the ext4 root reserve. The guard's df --output=avail reads Bavail, so it reads about R GiB less. df showed 158G size, 105G used, 47G free (run 39398067267), so R is about 6 GiB.
    • 15GB and the guard's / 1024 / 1024 are binary units. Read every "GB" here as GiB.

Target behaviour

  • The guard removes each old docsplus-{webapp,rest-api,hocuspocus,admin} tag with docker rmi, without -f or an age filter. It keeps DEPLOY_TAG, PREVIOUS_TAG and every image a container uses, and no last-good-tag.previous tag.
  • A LAST_GOOD_TAG_FILE with no 40-character SHA, or a failed in-use list, skips the sweep with a warning. A missing PREVIOUS_TAG image also warns.
  • docker buildx prune -af --reserved-space 15GB --min-free-space 45GB, called by name, prunes the build cache. A failure warns. By df the target is about 45 − R, about 39 GiB. That leaves about 25 GiB after a 14 GB build.
  • The guard aborts below 20 GB free, with a safe hint only.
  • The deploy job exposes free_gb_before, free_gb and load1; the step summary shows both free values (ruling 5).
  • The rollback comment no longer cites until=24h.

Implementation steps

Read the Read first files before Step 0. "Comment here" means a comment on this issue.

Step 0. Read and check the base

  1. On current main, check that the cited lines still match:

    sed -n '573,594p' .github/workflows/prod.docs.plus.yml   # the disk guard, exactly as quoted above
    sed -n '426,430p' .github/workflows/prod.docs.plus.yml   # timeout-minutes, needs, concurrency
    sed -n '870,874p' .github/workflows/prod.docs.plus.yml   # the rollback comment
    grep -c 'Pre-deploy disk guard' .github/workflows/prod.docs.plus.yml   # expect 1
    grep -n -A2 'name: 🚨 Rollback on Failure' .github/workflows/prod.docs.plus.yml   # the if: line must not be a bare if: failure()
    • Lines moved, same text: use the new numbers. Text changed: stop, re-derive, comment here.
    • Stop and comment here if Build the deploy images while the quality gates run #565 already moved the guard, or the rollback if: is still a bare if: failure().
    • If the deploy job already has outputs:, add the keys to it.
  2. Host check (read-only; the maintainer runs it if you have no host shell). Post the output here as "Host check, ". Never post IP addresses, ports or env values.

    docker version --format '{{.Server.Version}}'      # expect 29.1.3 or later
    docker buildx version                              # expect v0.30.1 or later
    docker buildx ls                                   # the default builder uses driver "docker"
    docker buildx inspect default | grep -i 'buildkit version'   # expect v0.17.0 or later (v0.26.2 on 2026-10-20)
    docker buildx prune --help                         # must list --reserved-space, --min-free-space, --max-used-space
    docker builder prune --help                        # record whether it shows the same flags
    df --output=target "$(docker info --format '{{.DockerRootDir}}')"   # must print "/"
    stat -f -c 'bsize=%S free=%f avail=%a' /           # R GiB = (free - avail) * bsize / 2^30
    docker system df
    cat /opt/projects/prod.docs.plus/.deploy/last-good-tag   # expect one 40-character SHA
    docker image ls 'docsplus-*' --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.CreatedSince}}'
    • Stop and comment here if docker buildx prune --help lacks --min-free-space, or the Docker root is not on /. Also stop if BuildKit is older than v0.17.0. buildx checks that only when the prune runs.
    • ID check. Run docker inspect -f '{{.Image}}' <a running webapp container id>. The ID must match the running tag's ID in docker image ls docsplus-webapp --no-trunc --format '{{.Tag}} {{.ID}}'. If not, stop and comment here.
    • Record R.

Step 1. Expose free disk from the deploy job

  1. In .github/workflows/prod.docs.plus.yml, add an outputs: block after needs: [triage, build] (line 427):

        needs: [triage, build]
        outputs:
          free_gb_before: ${{ steps.disk_guard.outputs.free_gb_before }}
          free_gb: ${{ steps.disk_guard.outputs.free_gb }}
          load1: ${{ steps.disk_guard.outputs.load1 }}
        concurrency:
  2. Wire into the report. Run grep -n '^ report:' .github/workflows/prod.docs.plus.yml.

Step 2. Replace the disk guard step

Replace lines 573-594 with this step. Keep the step name: M4 finds the step by the text disk guard.

      - name: 💾 Pre-deploy disk guard
        id: disk_guard
        run: |
          free_gib() { echo $(( $(df --output=avail / | tail -1) / 1024 / 1024 )); }
          FREE_BEFORE=$(free_gib)
          echo "📊 Disk before prune: ${FREE_BEFORE} GB free"
          df -h / | tail -1

          # Rollback needs every image at PREVIOUS_TAG (read from
          # LAST_GOOD_TAG_FILE above). `docker image prune -a` cannot exclude
          # one tag, so remove old docsplus-* tags one by one instead. Keep this
          # run's tag, PREVIOUS_TAG and every image a container uses.
          SWEEP=1
          KEEP_TAGS=" ${DEPLOY_TAG} "
          if [ -f "${LAST_GOOD_TAG_FILE}" ]; then
            if [[ "${PREVIOUS_TAG:-}" =~ ^[0-9a-f]{40}$ ]]; then
              KEEP_TAGS+="${PREVIOUS_TAG} "
            else
              echo "::warning::${LAST_GOOD_TAG_FILE} holds no commit SHA. Skipping the image sweep."
              SWEEP=0
            fi
          fi
          # -a: a stopped container still pins its image.
          if ! USED_IDS=$(docker ps -aq | xargs -r docker inspect -f '{{.Image}}' | sort -u); then
            echo "::warning::Could not list in-use images. Skipping the image sweep."
            SWEEP=0
          fi
          if [ "${SWEEP}" -eq 1 ]; then
            echo "🔒 Keeping tags:${KEEP_TAGS}and every image a container uses"
            for repo in docsplus-webapp docsplus-rest-api docsplus-hocuspocus docsplus-admin; do
              while read -r tag id; do
                case "${KEEP_TAGS}" in *" ${tag} "*) continue ;; esac
                if grep -qxF "${id}" <<< "${USED_IDS}"; then continue; fi
                # No -f: Docker refuses an image that a container still uses.
                if docker rmi "${repo}:${tag}" >/dev/null; then
                  echo "🗑️  Removed ${repo}:${tag}"
                else
                  echo "::warning::docker rmi refused ${repo}:${tag}. Kept it."
                fi
              done < <(docker image ls "${repo}" --no-trunc --format '{{.Tag}} {{.ID}}' | grep -v '^<none> ' || true)
            done
          fi
          docker image prune -f || echo "::warning::Dangling image prune failed."
          if [ -n "${PREVIOUS_TAG:-}" ]; then
            for img in docsplus-webapp docsplus-rest-api docsplus-hocuspocus docsplus-admin; do
              docker image inspect "${img}:${PREVIOUS_TAG}" >/dev/null 2>&1 \
                || echo "::warning::Rollback image ${img}:${PREVIOUS_TAG} is missing. A failed deploy cannot roll back."
            done
          fi

          # Prune the build cache toward a free-space target, oldest first, and
          # never below the reserved size. No age filter: on 2026-10-20 the
          # 24 h filter freed 0 B on three deploys. Called as buildx, because
          # only buildx has --min-free-space; `docker builder` routes to it.
          if ! docker buildx prune -af --reserved-space 15GB --min-free-space 45GB; then
            echo "::warning::docker buildx prune failed. The hard guard below still runs."
          fi

          FREE_AFTER=$(free_gib)
          echo "📊 Disk after prune: ${FREE_AFTER} GB free"
          docker system df
          {
            echo "free_gb_before=${FREE_BEFORE}"
            echo "free_gb=${FREE_AFTER}"
            echo "load1=$(cut -d' ' -f1 /proc/loadavg)"
          } >> "$GITHUB_OUTPUT"
          echo "💾 Free disk on /: ${FREE_BEFORE} GB before the prune, ${FREE_AFTER} GB after" >> "$GITHUB_STEP_SUMMARY"

          # Hard guard. One build wrote 14 GB on 2026-10-20 (47 GB free, then 33 GB).
          if [ "${FREE_AFTER}" -lt 20 ]; then
            echo "::error::Less than 20 GB free on / after the prune. Aborting before the build; production is unchanged. Free space by hand with 'docker buildx prune -af' (build cache only). Do not prune images or volumes: rollback needs the last-good images, and redis-data and traefik-certs are volumes."
            df -h /
            exit 1
          fi

Why: each command that may fail sits in an if, an || or a process substitution, so bash -eo pipefail does not abort. The in-use check compares image IDs and includes stopped containers, such as migrate. The sweep runs first, so the cache prune deletes less. The abort leaves production alone: 🔧 Ensure Infrastructure sets DEPLOY_STARTED later (:644-649), given the #555 gate.

Check: install shellcheck, then actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml exits 0.

Step 3. Fix the rollback comment

Replace lines 871-874 inside 🚨 Rollback on Failure with:

          # tag must still exist locally. The disk guard never removes the
          # PREVIOUS_TAG images. If a person removed one by hand, we fail
          # loudly rather than partially-rollback into a mixed-version cluster.

Keep line 870 (# Multi-image precondition (A2): ...). Change no code in that step.

Check: grep -n 'until=24h' .github/workflows/prod.docs.plus.yml prints one line only, docker image prune -f --filter "until=24h" 2>/dev/null || true in 🧹 Cleanup.

Step 4. Rehearse the sweep locally

A logic check, so a laptop is fine (AGENTS.md §Docker bans laptop timings only). It uses FROM scratch images under ztest-sweep. Run it from the repo root:

bash <<'EOF'
set -eo pipefail
T=$(mktemp -d)
R=ztest-sweep
A=$(printf 'a%.0s' {1..40}); B=$(printf 'b%.0s' {1..40}); C=$(printf 'c%.0s' {1..40}); E=$(printf 'e%.0s' {1..40})
for t in "$A" "$B" "$C" "$E"; do
  printf 'FROM scratch\nLABEL sweep=%s\n' "$t" | docker build -q -t "$R:$t" - >/dev/null
done
CID=$(docker create "$R:$C" /none)   # a stopped container pins tag c
sed -n '/^          SWEEP=1$/,/^          docker image prune -f/p' .github/workflows/prod.docs.plus.yml \
  | sed -e 's/^          //' -e "s/^\(  for repo in \).*; do\$/\1$R; do/" -e '$d' > "$T/sweep.sh"
echo "$A" > "$T/last-good-tag"
DEPLOY_TAG=$E PREVIOUS_TAG=$A LAST_GOOD_TAG_FILE=$T/last-good-tag bash -eo pipefail "$T/sweep.sh"
echo "left: $(docker image ls "$R" --format '{{.Tag}}' | cut -c1 | sort | tr -d '\n')"   # expect: ace
DEPLOY_TAG=$E PREVIOUS_TAG=garbage LAST_GOOD_TAG_FILE=$T/last-good-tag bash -eo pipefail "$T/sweep.sh"   # expect: the warning, no removal
docker rm "$CID" >/dev/null
for t in "$A" "$C" "$E"; do docker rmi "$R:$t" >/dev/null; done
rm -rf "$T"
EOF

Pass: the first run removes only ztest-sweep:bbbb… and prints left: ace. The second prints holds no commit SHA. Skipping the image sweep. Afterwards, docker image ls ztest-sweep lists nothing.

Step 5. Docs

Edit every file in Docs to update.

Review checkpoint

Show git diff --stat, git diff, the actionlint and Step 4 output and the host-check comment. Expect changes in exactly .github/workflows/prod.docs.plus.yml, .github/WORKFLOWS.md, AGENTS.md, CHANGELOG.md and apps/webapp/docker/Dockerfile.bun. Then stop. Do not run git add, git commit, git push or git stash, and do not create a branch. The maintainer commits and pushes per SP2 (#554), as its own commit.

Acceptance criteria

  • 🚨 Rollback on Failure has no bare if: failure(). The gate from Roll back only after the deploy has started rolling services #555 is in place.
  • The host-check comment exists. docker buildx prune --help lists --min-free-space, BuildKit is v0.17.0 or later, the Docker root is on /, and the ID check matched.
  • The 💾 Pre-deploy disk guard step has id: disk_guard. It contains no docker image prune -a, no docker builder prune and no until= filter.
  • The cache prune is docker buildx prune -af --reserved-space 15GB --min-free-space 45GB, with no 2>/dev/null || true.
  • The hard guard aborts below 20, and its text names neither --volumes nor an image prune.
  • The deploy job has outputs.free_gb_before, outputs.free_gb and outputs.load1.
  • If the report job exists, its 📊 Write timing table step has FREE_GB: ${{ needs.deploy.outputs.free_gb }} and LOAD1: ${{ needs.deploy.outputs.load1 }}. The first Run Report shows a number for free_gb, not n/a.
  • grep -n 'until=24h' .github/workflows/prod.docs.plus.yml prints one line only, in the dangling-only 🧹 Cleanup step.
  • actionlint exits 0, with shellcheck installed.
  • The Step 4 rehearsal prints left: ace and the skip warning, and leaves no ztest-sweep image.
  • Every file in Docs to update is edited.
  • The first (build): deploy log has a buildx Total: line and no docker buildx prune failed warning.
  • The SP3 comment for that deploy shows no Rollback image … is missing warning and a free_gb of 20 or more.

Verification

Before merge:

actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml
bun run format
# then the Step 4 rehearsal

Not needed: bash .github/scripts/parse-build-trigger.test.sh and a dry_run_subject dispatch. Triage and job conditions do not change.

After merge, read the first (build): deploy (<run-id> and <job-id> of 🚀 Deploy Production). Metrics are defined in #569.

gh run view <run-id> -R docs-plus/docs.plus --json createdAt,jobs --jq '
  (.createdAt|fromdate) as $t0 | [.jobs[].steps[]?] as $s |
  def dur(re): first($s[]|select(.name|test(re))|((.completedAt|fromdate)-(.startedAt|fromdate)));
  def upto(re): first($s[]|select(.name|test(re))|((.completedAt|fromdate)-$t0));
  {M1: upto("Verify Deployment"), M3: dur("Build Docker Images"), M4: dur("disk guard")}'
gh run view -R docs-plus/docs.plus --job <job-id> --log \
  | grep -E 'Disk (before|after) prune|Total:|Keeping tags|Removed docsplus|refused|Rollback image|Skipping the image sweep|prune failed'

On run 39398067267 the first command prints {"M1":1132,"M3":513,"M4":0}, which matches the baseline row.

Accountability

  • Who. The builder reads the first (build): deploy after merge. The maintainer reviews and runs host commands if needed.

  • Record per SP3 ([Tracking] Faster, change-aware CI/CD pipeline #554), plus the deploy job id, free_gb_before, the Total: line, the count of Removed docsplus-… lines and any guard ::warning::. Run: the first (build): deploy after merge. Baseline: 14 deploys, 2026-09-22 to 2026-10-20.

    Metric Read from Baseline Expected
    free_gb Disk after prune log line or step summary 46, 32 and 28 on 2026-10-20 20 or more on every deploy. About 45 − R when the prune had to free space.
    Cache prune result Total: log line 0B on 2026-10-20 More than 0B while free space is below the target. About one record, or 0B, above it.
    M4 dur("disk guard") median 30 s, range 0-72 s 120 s or less.

    M3 (median 546 s, warm maximum 557 s, cold maximum 866 s) has no planned change. It stays within the range of its cache class. M1 (median 1090 s, p90 1386 s) rises by the M4 increase only, at most about 120 s. M2g, M2i, M5, W and M6 do not change in this issue. Do not read them for this issue.

  • Host read (read-only). Run docker image ls 'docsplus-*' and docker system df. Confirm all four images at the new last-good tag exist. Add it to the same comment.

  • Second read. After seven days or eight deploys, whichever comes first, post the free_gb values in a second SP3 comment.

  • Escalation. If free_gb stays under the target for a week, reopen the GHCR question on [Tracking] Faster, change-aware CI/CD pipeline #554 with those numbers.

  • Prune failure. On docker buildx prune failed, comment here the same day with the error line. Do not revert. Free cache by hand with docker buildx prune -af until a fix lands.

Rollback

Revert per SP4 (#554) when any of these happens:

  • the guard removes an image at DEPLOY_TAG or PREVIOUS_TAG, or prints Rollback image … is missing for an image that existed before the deploy;
  • a rollback reports Missing images for previous tag;
  • the guard step fails for any reason other than the 20 GB abort;
  • 🚨 Rollback on Failure runs after a disk-guard abort;
  • M3 is above 866 s on two deploys in a row.

Not a revert: the 20 GB abort. Free space as its error text says, then comment here and ask the maintainer for new prune values.

Docs to update

  • .github/WORKFLOWS.md, the workflows/prod.docs.plus.yml entry: add one bullet after the "Deploy guard" bullet (line 11): "Disk guard: before the build, it removes old docsplus-* tags. It keeps this run's tag, the last-good tag and every image a container uses. Then it prunes the build cache toward 45 GiB free and keeps at least 15 GiB of cache. It aborts below 20 GiB free and exports free_gb."
  • AGENTS.md §Docker, the cache-mount bullet (line 297): change "The deploy's own command" to "The deploy's command at the time". Add at the end: "Since #, the deploy prunes the build cache toward a free-space target, with no age filter." Keep the Make CI failures reach Telegram, and fix stale CI facts #557 gap-figure edit in the same bullet.
  • apps/webapp/docker/Dockerfile.bun, the comment above RUN --mount=type=cache,target=/app/apps/webapp/.next/cache bunx next build (lines 214-218 at the base; four lines after Make CI failures reach Telegram, and fix stale CI facts #557). Replace the sentence that names until=24h with: "The deploy prunes build cache toward a free-space target, oldest record first, so a recently used mount usually survives." Rewrap the comment lines if needed. Keep the other sentences and the RUN line.
  • CHANGELOG.md (root), ## [Unreleased] → ### Self-hosting: add "The production deploy now aborts below 20 GB free disk, up from 10 GB. It prunes Docker build cache toward a free-space target, not by age."

Out of scope

Read first

  • AGENTS.md, mainly §Docker, §Workflow And Review Expectations and §Simplified English Mandate.
  • .github/WORKFLOWS.md, the workflows/prod.docs.plus.yml entry.
  • .github/workflows/prod.docs.plus.yml: the deploy job (lines 420-965), mainly 🔐 Prepare Environment, the disk guard, 💾 Stash this tag as last-good and 🚨 Rollback on Failure.
  • docker-compose.prod.yml: the image: lines of the six app services.
  • .github/actionlint.yaml.
  • [Tracking] Faster, change-aware CI/CD pipeline #554 and Build the deploy images while the quality gates run #565.

Activity

  1. added this to the v2.3.0 milestone on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions