Skip to content

fix(upgrade): recover from a stale host-port reservation instead of aborting - #3438

Merged
vpetersson merged 2 commits into
masterfrom
agent/anthias-claude/9a16e1db9881
Nov 3, 2026
Merged

vpetersson merged 2 commits into
masterfrom
agent/anthias-claude/9a16e1db9881

Conversation

@vpetersson-bot

Copy link
Copy Markdown
Contributor

Fixes the upgrade failure reported in #3437.

The problem

An upgrade can abort with port is already allocated while nothing is listening on that port:

Error response from daemon: failed to set up container networking: driver
failed programming external connectivity on endpoint anthias-redis-1 (...):
Bind for 127.0.0.1:6379 failed: port is already allocated

Docker keeps host-port reservations in libnetwork's store. An endpoint that is never torn down cleanly leaves its reservation behind (daemon killed mid-upgrade, or a board that browns out while the stack is running), so the container collides with its own ghost on the next bring-up. The symptom always lands on a service that publishes a host port.

This is not the srly-ose-redis case already handled above in the same script, where a real pre-rebrand container still holds 6379. Here no container holds the port at all, so removing containers by name does not help.

Reported on the forum across three Pi 4B devices, then independently by a second user: https://forums.screenly.io/t/6688

The fix

Capture the up -d output through tee so progress still streams to the operator, and on failure check it for that specific error. If it matches, take this compose project down, which releases every endpoint it owns and removes its network, then retry the bring-up once. Any other failure aborts with its original exit code, unchanged.

The recovery is deliberately scoped to this project's own network. The workaround circulating in the forum thread is rm /var/lib/docker/network/files/local-kv.db, which drops every user-defined network on the host and takes the web UI down with it. The second reporter hit exactly that and had to repair it by re-running the script. A test now guards against anyone reintroducing it.

Testing

Three paths exercised against a stubbed docker:

Path Result
up fails with the stale reservation recovery down runs, retry succeeds, exit 0
up succeeds first time no down, exit 0
up fails for any other reason aborts with the original exit code, no down

The temp log is cleaned up on all three.

Then a real end-to-end upgrade on the Pi 4B rig with the patched script: pull, signature verification, full stack recreated, redis healthy, web UI answering 200, stale images purged, exit 0.

Four tests in tests/test_upgrade_containers_script.py (two existing, two new); the existing ordering guard was widened to assert the reboot gate sits below every bring-up rather than the only one. Full non-integration suite passes (2591 tests); ruff check, ruff format --check and mypy clean on the touched files.

Note: the original libnetwork state does not reproduce on Docker 29.6.1 (SIGKILLing dockerd with the stack up reconciles cleanly on restart), so this is written against the reported symptom rather than a live repro.

🤖 Generated with Claude Code

An upgrade could abort with "port is already allocated" while nothing
was listening on the port. libnetwork keeps host-port reservations in
its own store, and an endpoint that is never torn down cleanly leaves
its reservation behind: the daemon killed mid-upgrade, or an SD-card
board that browns out while the stack is running. The container then
collides with its own ghost on the next bring-up, so the symptom lands
on whichever service publishes a host port.

Recover by taking this compose project down, which releases its
endpoints and removes its network, then retrying the bring-up once.
Any other failure still aborts with its original exit code.

Deliberately not by deleting local-kv.db, which is the workaround
circulating in the forum thread. That file is the only record of every
user-defined network on the host, so removing it takes the web UI down
too, which is what the reporter then had to repair.

Reported across three Pi 4Bs and again on a separate upgrade:
https://forums.screenly.io/t/6688

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
Copilot AI lite review requested due to automatic review settings October 30, 2026 08:19
@vpetersson-bot
vpetersson-bot requested a review from a team as a code owner October 30, 2026 08:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@vpetersson-bot
vpetersson-bot requested a lite review from Copilot October 30, 2026 08:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Picks up 77d2c39 (#3427), which ruff-formats the Python snippet in
website/content/agents.md. That file is untouched by this PR, but the
branch predates the fix, so `ruff format --check .` failed in CI on a
file this change never modified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings November 2, 2026 17:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@vpetersson
vpetersson merged commit 580b2b4 into master Nov 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants