Skip to content

Upload the manager's container logs - #2984

Merged
berendt merged 1 commit into
mainfrom
collect-manager-container-logs
Nov 2, 2026
Merged

berendt merged 1 commit into
mainfrom
collect-manager-container-logs

Conversation

@ideaship

@ideaship ideaship commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The manager's own services (mariadb, ara-server, the *-ansible containers, the inventory reconciler) log only to docker stdout, so neither the /var/log/kolla copy nor the container health record carries what they said. This adds docker logs --timestamps of every manager container to manager-containers/<container>.log.gz in the post-run collect play, gated like the rest of it (failed builds only, never fails the job).

Motivation: manager-mariadb-1 stayed unhealthy after manager.service was restarted, with its healthcheck failing on Access denied for user 'healthcheck'@'localhost'. Every dependent container stayed down and the deploy failed waiting for ceph-ansible. What mariadb logged on that start was not in the uploaded logs.

Tested with a temporary commit (since dropped) that failed every deploy right after the manager containers became healthy, so the collect play ran against a live manager:

  • deploy-current and deploy-stable: all 16 manager containers' logs uploaded, 80K gzipped in total. manager-mariadb-1.log.gz is readable and spans both the first start and the manager.service restart, each with its own "Initializing database files" (the data dir is tmpfs).
  • deploy-next failed earlier, in the manager deploy itself; the step uploaded the one container that existed (traefik) and handled the unreachable nodes without failing.

The zuul label needs to be removed and re-added for a run on the final commit.

🤖 Generated with Claude Code

The manager's own services -- mariadb, ara-server, the *-ansible
containers, the inventory reconciler -- are not kolla services. They
log to the container's stdout, so neither the /var/log/kolla copy nor
the container health record carries what they said.

When manager-mariadb-1 stays unhealthy after manager.service is
restarted, the health record shows only the probe's side of it:

  ERROR 1045 (28000): Access denied for user 'healthcheck'@'localhost'

The containers that depend on mariadb are never started again, and the
job fails waiting for ceph-ansible to become healthy. What mariadb did
on that start, which is the only thing that can say why its
healthcheck user was rejected, was not in the uploaded logs.

Copy "docker logs --timestamps" of every container on the manager to
manager-containers/<container>.log.gz, from the collect play that
already uploads the kolla logs, the container health state and the
validator reports. Each log is gzipped on the manager and streamed
back over the existing ssh, with stdout and stderr interleaved as
docker logs them. manager.service stops its containers rather than
removing them, so a log spans every restart since the container was
created.

The step shares the play's gating: only failed builds are collected,
it never fails the job, and the directory is cleared at the start of
the play, since the orchestrator's home directories outlive a build.

Only the manager is covered. On the other testbed hosts the kolla
services log to files, which are already uploaded.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship ideaship added the zuul Release the dragons, run Zuul CI label Oct 29, 2026
@ideaship
ideaship force-pushed the collect-manager-container-logs branch from af554cb to c658a65 Compare October 29, 2026 06:14
@ideaship ideaship removed the zuul Release the dragons, run Zuul CI label Oct 29, 2026
@ideaship ideaship self-assigned this Oct 29, 2026
@ideaship ideaship added the zuul Release the dragons, run Zuul CI label Oct 29, 2026
@ideaship
ideaship requested a review from berendt October 29, 2026 11:00
@ideaship
ideaship marked this pull request as ready for review October 29, 2026 11:00
@berendt
berendt merged commit edb39ac into main Nov 2, 2026
2 of 3 checks passed
@berendt
berendt deleted the collect-manager-container-logs branch November 2, 2026 06:51
@github-project-automation github-project-automation Bot moved this from New to Done in Human Board Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

zuul Release the dragons, run Zuul CI

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants