Repository navigation
fix(upgrade): recover from a stale host-port reservation instead of aborting - #3438
Conversation
An upgrade could abort with "port is already allocated" while nothing was listening on the port. libnetwork keeps host-port reservations in its own store, and an endpoint that is never torn down cleanly leaves its reservation behind: the daemon killed mid-upgrade, or an SD-card board that browns out while the stack is running. The container then collides with its own ghost on the next bring-up, so the symptom lands on whichever service publishes a host port. Recover by taking this compose project down, which releases its endpoints and removes its network, then retrying the bring-up once. Any other failure still aborts with its original exit code. Deliberately not by deleting local-kv.db, which is the workaround circulating in the forum thread. That file is the only record of every user-defined network on the host, so removing it takes the web UI down too, which is what the reporter then had to repair. Reported across three Pi 4Bs and again on a separate upgrade: https://forums.screenly.io/t/6688 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|



Fixes the upgrade failure reported in #3437.
The problem
An upgrade can abort with
port is already allocatedwhile nothing is listening on that port:Docker keeps host-port reservations in libnetwork's store. An endpoint that is never torn down cleanly leaves its reservation behind (daemon killed mid-upgrade, or a board that browns out while the stack is running), so the container collides with its own ghost on the next bring-up. The symptom always lands on a service that publishes a host port.
This is not the
srly-ose-rediscase already handled above in the same script, where a real pre-rebrand container still holds 6379. Here no container holds the port at all, so removing containers by name does not help.Reported on the forum across three Pi 4B devices, then independently by a second user: https://forums.screenly.io/t/6688
The fix
Capture the
up -doutput throughteeso progress still streams to the operator, and on failure check it for that specific error. If it matches, take this compose project down, which releases every endpoint it owns and removes its network, then retry the bring-up once. Any other failure aborts with its original exit code, unchanged.The recovery is deliberately scoped to this project's own network. The workaround circulating in the forum thread is
rm /var/lib/docker/network/files/local-kv.db, which drops every user-defined network on the host and takes the web UI down with it. The second reporter hit exactly that and had to repair it by re-running the script. A test now guards against anyone reintroducing it.Testing
Three paths exercised against a stubbed
docker:upfails with the stale reservationdownruns, retry succeeds, exit 0upsucceeds first timedown, exit 0upfails for any other reasondownThe temp log is cleaned up on all three.
Then a real end-to-end upgrade on the Pi 4B rig with the patched script: pull, signature verification, full stack recreated, redis healthy, web UI answering 200, stale images purged, exit 0.
Four tests in
tests/test_upgrade_containers_script.py(two existing, two new); the existing ordering guard was widened to assert the reboot gate sits below every bring-up rather than the only one. Full non-integration suite passes (2591 tests);ruff check,ruff format --checkandmypyclean on the touched files.Note: the original libnetwork state does not reproduce on Docker 29.6.1 (SIGKILLing
dockerdwith the stack up reconciles cleanly on restart), so this is written against the reported symptom rather than a live repro.🤖 Generated with Claude Code