Repository navigation
Upload the manager's container logs - #2984
Merged
Merged
Conversation
The manager's own services -- mariadb, ara-server, the *-ansible containers, the inventory reconciler -- are not kolla services. They log to the container's stdout, so neither the /var/log/kolla copy nor the container health record carries what they said. When manager-mariadb-1 stays unhealthy after manager.service is restarted, the health record shows only the probe's side of it: ERROR 1045 (28000): Access denied for user 'healthcheck'@'localhost' The containers that depend on mariadb are never started again, and the job fails waiting for ceph-ansible to become healthy. What mariadb did on that start, which is the only thing that can say why its healthcheck user was rejected, was not in the uploaded logs. Copy "docker logs --timestamps" of every container on the manager to manager-containers/<container>.log.gz, from the collect play that already uploads the kolla logs, the container health state and the validator reports. Each log is gzipped on the manager and streamed back over the existing ssh, with stdout and stderr interleaved as docker logs them. manager.service stops its containers rather than removing them, so a log spans every restart since the container was created. The step shares the play's gating: only failed builds are collected, it never fails the job, and the directory is cleared at the start of the play, since the orchestrator's home directories outlive a build. Only the manager is covered. On the other testbed hosts the kolla services log to files, which are already uploaded. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
collect-manager-container-logs
branch
from
October 29, 2026 06:14
af554cb to
c658a65
Compare
ideaship
marked this pull request as ready for review
October 29, 2026 11:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The manager's own services (mariadb, ara-server, the
*-ansiblecontainers, the inventory reconciler) log only to docker stdout, so neither the/var/log/kollacopy nor the container health record carries what they said. This addsdocker logs --timestampsof every manager container tomanager-containers/<container>.log.gzin the post-run collect play, gated like the rest of it (failed builds only, never fails the job).Motivation:
manager-mariadb-1stayed unhealthy aftermanager.servicewas restarted, with its healthcheck failing onAccess denied for user 'healthcheck'@'localhost'. Every dependent container stayed down and the deploy failed waiting for ceph-ansible. What mariadb logged on that start was not in the uploaded logs.Tested with a temporary commit (since dropped) that failed every deploy right after the manager containers became healthy, so the collect play ran against a live manager:
manager-mariadb-1.log.gzis readable and spans both the first start and themanager.servicerestart, each with its own "Initializing database files" (the data dir is tmpfs).The
zuullabel needs to be removed and re-added for a run on the final commit.🤖 Generated with Claude Code