Repository navigation
feat(api): expose host system logs at /api/v2/system/logs - #3423
Conversation
Anthias had no way to read the host journal from inside the container. This adds a snapshot-only endpoint with one contract on both platforms. - Balena: supervisor POST /v2/journal-logs. No compose or privilege change -- the supervisor-api label is already present. Requests format=json so the payload matches bare metal and one normaliser serves both. - Bare metal: anthias-host-agent over hostcmd, with correlation IDs, line and byte caps, and a Redis TTL on replies. Bare-string hostcmd payloads still work, so rollout order does not matter. - Nothing in the path can block subscriber_loop: no follow on either platform, every read bounded by lines, bytes and a subprocess timeout, and every failure path still writes a reply. - Grant the host agent journal access with SupplementaryGroups=systemd-journal, and refuse the request outright when the system journal is unreadable -- journalctl exits 0 and returns the service account's own user journal otherwise. - Make journald persistent and size-capped in the Ansible role. - Redact secret-looking values (psk, passphrase, tokens) on the way out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
The SonarCloud gate failure comes from three
The new code introduces no actual vulnerabilities. CodeQL and the rest of CI pass, and the three CodeQL exception-exposure alerts were fixed in b069f54. Someone with SonarCloud access needs to mark these three findings as Safe to clear the gate. |
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
CodeQL flagged str(exc) flowing into API responses. JournalUnavailable and JournalRequestError now carry a composed user_message, which the view and serializer return instead. Supervisor and Redis transport errors are logged rather than interpolated into the response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
SonarCloud S2612 flagged the three new `mode:` values. All three match what the OS itself ships, so they are annotated rather than tightened -- but chasing the first one down found a real gap: creating /var/log/journal ourselves skips the `a+` tmpfiles line that grants the `adm` group its read ACL, so adm-based journal access silently would not work on a host where we made the directory. - Run `systemd-tmpfiles --create --prefix=/var/log/journal`, the other half of the recipe documented in journald.conf(5) - NOSONAR with reasons: systemd pins /var/log/journal to 2755 and resets anything narrower; 0755/0644 match /etc/systemd's own files Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
- redact(): the key=value pattern was quadratic. A 48 KB line (journald's default LineMax) took ~a minute and pinned a request worker; measured 6.3s at 16 KB. A lookbehind makes only a real token start a match, taking it back to linear -- 48 KB now 0.003s. - Balena truncation kept the FIRST 4 MB while the host agent kept the last. journalctl emits oldest-first, so the supervisor path was discarding exactly the entries an operator opened the log to read. Trim from the front, and bound total bytes read as well as kept. - `container` means a name on both platforms, but the supervisor only matches CONTAINER_ID_FULL, so forwarding the name returned an empty 200 on Balena. Resolve it via /v2/containerId, pass an ID straight through, and fail loudly on an unknown service. - A stopped host agent made every request block for the full 30s BLPOP. PUBLISH reports its subscriber count; zero means no reply is coming, so fail now. - journalctl output was decoded with `text=True`, i.e. the locale encoding with strict errors -- and a systemd unit has no LANG, so that is ASCII. One non-ASCII byte would raise out of process_message and kill subscriber_loop, the exact failure this feature is built to avoid. - count help_text claimed "most recent first"; entries are oldest first. It is published in the OpenAPI schema. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
They were sitting in the comment block above each `mode:`, where the analyzer does not see them -- SonarCloud kept reporting all three. Inline is also what the working examples in roles/system/tasks use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
388697b to
84190b8
Compare
Translate supervisor body-read failures into the existing 502 response and always close the response. Request full journal fields on bare metal while draining stdout and stderr into bounded buffers under one subprocess deadline; kill and reap the child on failure. Add regressions for streaming failures, long messages, output caps, UTF-8 decoding, and timeout cleanup. Existing redaction performance and newest-entry truncation regressions also pass. Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
`host_agent.subprocess` is an implicit re-export, which mypy rejects under no_implicit_reexport -- it broke run-mypy on the branch. It is the same module object as the stdlib `subprocess` the test already imports, and _run_journalctl resolves `subprocess.Popen` through its own globals, so patching the module directly is identical in effect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
The BLPOP in the bare-metal journal backend ran on the shared connect_to_redis() client, which takes redis-py's defaults: socket_timeout=5 with Retry(retries=10). A 30s blocking pop cannot survive a 5s socket read, so the socket expired first, the retry ladder re-issued the pop ten more times, and the call raised redis.TimeoutError about a minute later instead of returning None. Two things were wrong as a result: - REPLY_TIMEOUT_SECONDS bounded nothing. The real bound was the retry ladder, at roughly twice the intended budget. Measured end to end on the QA rig at 59.6s on the x86 board and 58.2s on a Pi, against a host agent that was subscribed but not answering. - The `reply is None` branch was unreachable, so that case surfaced as "Could not reach the host agent over Redis" — pointing an operator at the broker while the broker was healthy. The accurate message, telling them to check that anthias-host-agent is running and up to date, could never be reached. The path is reachable on a normal upgrade: when the container is updated before the host agent, the old agent is still subscribed, so PUBLISH reports a non-zero subscriber count and the existing fast-fail for a stopped agent does not apply. Give the blocking pop its own client with a socket that outlives the pop timer and no retries. A blocking pop is not idempotent the way a GET is — re-issuing it after a socket timeout restarts the wait from zero, which is how one bounded request became a minute-long stall. The same redis-py defaults are already documented as a trap in lib/cec_client.py and celery_tasks.py; they are right for the short round trips every other call site makes, so they are widened here rather than in the shared helper. connect_to_redis() grows a kwargs passthrough so this stays the single place that knows where redis lives; every existing caller is unaffected. Slow-but-completing reads are unaffected: the pop now waits its full budget in one attempt rather than restarting, and a stopped agent still fails in ~0.02s via the subscriber-count check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
connect_to_redis() now takes a kwargs passthrough, and lib.system_logs._reply_connection uses it to widen the socket for its blocking pop. The session-wide stub in conftest was a zero-argument lambda, so any test reaching the real helper would have hit a TypeError rather than the fake. mypy caught it as a signature mismatch; the runtime break was the same bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Raspberry Pi OS pins Storage=volatile in 40-rpi-volatile-storage.conf to keep journald off the SD card. That is a deliberate choice, not an oversight, and this role was overriding it on every install — the 99- prefix on our drop-in existed specifically to win that fight. A player writes more journal than the feature is worth paying for. Measured on idle QA boards running the normal Anthias stack: 19-28 MiB a day, roughly 7-10 GiB a year, before any real content churn. It is also the shape of write SD cards handle worst — small appends with a periodic fsync, amplified by the card's internal block management. /api/v2/system/logs never needed it. The host agent reads the volatile journal in /run just as well; verified on Pi 2/3/4/5 before the role ran, where the endpoint returned entries with persistent=false. The only thing persistence adds is history that survives a reboot, and the API already reports which one answered so a device on the default is honest about showing current-boot-only logs. So: anthias_journal_persistent, defaulting to false. When it is off the role writes nothing and leaves the image default intact — and it now removes our drop-in if it is present, so flipping the flag back actually restores the vendor setting instead of stranding the override. Creating /var/log/journal moved inside the conditional too: on an image without the Pi vendor drop-in, Debian's Storage=auto means creating that directory is itself the persistence switch. Both tunables move from vars/ to defaults/, because role vars outrank inventory and group_vars — a per-device override set the normal way would have been silently ignored from where they were. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
Issues Fixed
Anthias has no way to surface host/system logs.
bin/collect_debug.shshells out tojournalctlon bare metal, but that is a manual operator script — nothing in the app or API can read the journal.Description
Adds
GET /api/v2/system/logs: a snapshot-only journal read with one contract on both supported platforms. Nofollowon either — the bare-metal backend runs inside the host agent's single-threadedsubscriber_loop, where a streaming read would queue reboot/shutdown behind it forever, and offering a live tail on Balena alone would mean two divergent contracts.Parameters:
count(default 500, hard cap 2000),unit,container,since,until.Balena — supervisor
POST /v2/journal-logs. No compose change and no new privilege;anthias-serveralready carriesio.balena.features.supervisor-api. Requestsformat: jsonso the payload is shaped identically to bare metal and one normaliser serves both backends.Bare metal —
anthias-host-agentoverhostcmd, now accepting a JSON payload with a correlation ID alongside the original bare-string commands (so rollout order does not matter). The reply goes onto a correlation-keyed Redis list with a TTL and is collected with a boundedBLPOP— a blocking pop rather than a completion pub/sub, because it has no subscribe-before-publish race and matches theanthias.reply.<id>pattern already used on the viewer bus.Nothing in the path can block
subscriber_loop: bounded by lines, by bytes and by a subprocess timeout, and every failure path still writes exactly one reply so the container's pop resolves instead of timing out.Also here:
SupplementaryGroups=systemd-journalon the host-agent unit, plus a hard refusal when the system journal is unreadable.journalctlexits 0 for a caller outside that group and hands back the service account's own user journal with every system record missing — and under--output=jsonit does not even print its stderr hint, so there is no signal to key off. The agent asks the kernel (os.access) instead, which also honours the ACL journald grants toadm.anthias_journal_persistent. See the correction at the bottom: a player writes 19-28 MiB of journal a day, and Raspberry Pi OS pinsStorage=volatileto keep that off the SD card deliberately. The endpoint does not need persistence, so the role no longer overrides that choice. When an operator does opt in,99-anthias.confwins over40-rpi-volatile-storage.confand is followed byjournalctl --flush, because a restart alone leaves the journal in/rununtil the next boot.persistentis reported in the response rather than assumed, measured from which of/var/log/journaland/run/log/journalholds the live journal. Both cheaper probes give a wrong "yes" on hardware we ship on.psk,passphrase, tokens,Authorization) are blanked on the way out, keeping the key visible.Note for operators: persistence is opt-in (
anthias_journal_persistent: true). When enabled,SystemMaxUse=64Mis enforced by vacuuming, so on an install whose journal already exceeds it the first journald restart discards the excess. A QA x86 board had grown to 1.8 GB and came back at 56 MB. Overrideanthias_journal_max_usewhere the longer history matters more than the card.Checklist
End-to-end on the QA rig (Pi 5 / aarch64 and x86, both Debian 13), host agent and server overlaid from this commit with hashes verified in place: snapshot, every filter, parameter rejection, five concurrent requests each getting their own result, reply TTL, 2000-line read in 0.49s / 0.80s with the next host command answered ~250ms later, and redaction of a real
logger-injected line. Both boards restored to their pinned images afterwards.Verified against the real OS permission state on both boards: with the supplementary group withheld,
journalctl -u docker.servicereturns 0 lines and exit code 0; with it, 50. The agent now refuses that case with an actionable error instead of serving a silently empty view.The Balena backend was exercised over real HTTP inside the server container against a supervisor stub — correct URL, version and API key, correct body,
follow: false, identical normalised output, redaction, and the 4 MiB cap holding on a streamed 9 MB response. It has not been run against a real balenaOS supervisor; the rig has no balena device.🤖 Generated with Claude Code
Review follow-ups
Six findings from a review pass, all fixed and re-verified on the Pi 5 and x86 boards:
redact()was quadratic. A 48 KB line (journald's defaultLineMax) took about a minute and pinned a request worker; measured 6.3 s at 16 KB. A lookbehind means only a real token start can begin a match — 48 KB is now 0.003 s, with every redaction case unchanged.journalctlemits oldest-first, so the supervisor backend was discarding exactly the entries an operator opened the log to read. It now trims from the front, and bounds total bytes read as well as bytes kept.containerreturned an empty 200 on Balena. It means a name on both platforms, but the supervisor only matchesCONTAINER_ID_FULL. It is now resolved via/v2/containerId, an ID is passed straight through, and an unknown service fails loudly instead of looking like a quiet device.BLPOP.PUBLISHreports its subscriber count; zero means no reply is coming. Now 502s in ~0.02 s.text=Truedecoded journalctl output with the locale encoding and strict errors — and a systemd unit has noLANG, so that is ASCII. One non-ASCII byte would have raised out ofprocess_messageand killedsubscriber_loop, the exact failure this feature is built to avoid.counthelp_text claimed "most recent first"; entries are oldest-first. It is published in the OpenAPI schema.Chasing the first SonarCloud finding also turned up a real gap: creating
/var/log/journalourselves skipped thea+tmpfiles line that grants theadmgroup its read ACL, so adm-based journal access would silently not have worked on a host where we made the directory. The role now runssystemd-tmpfiles --create, the other half of the recipe documented injournald.conf(5).Hardware QA follow-up
Full integration run across the QA rig — x86_64, aarch64 (Pi 3/4/5) and armv7l (Pi 2, Pi 3 A+), Debian 13 and Raspbian 13 — turned up one defect in this branch, now fixed here.
The blocking pop was not bounded by its own timeout.
BLPOPran on the sharedconnect_to_redis()client, which takes redis-py's defaults:socket_timeout=5withRetry(retries=10). A 30s pop cannot survive a 5s socket read, so the socket expired first and the retry ladder re-issued the pop ten more times. Measured at 59.6s on x86 and 58.2s on a Pi against a host agent that was subscribed but not answering — roughly twiceREPLY_TIMEOUT_SECONDS, which bounded nothing. It also made thereply is Nonebranch unreachable, so the case surfaced as "Could not reach the host agent over Redis", pointing an operator at a healthy broker.The path is reachable on a normal upgrade: with the container updated before the host agent, the old agent is still subscribed, so
PUBLISHreports a non-zero subscriber count and the fast-fail for a stopped agent does not apply.The blocking pop now gets its own client — socket outliving the pop timer, no retries — verified on hardware: 30.05s and the accurate "Check that anthias-host-agent is running and up to date" message. A stopped agent still fails in ~0.015s, and slow-but-completing reads are unaffected (six queued 2000-line reads on a Pi 2, waits to 17.7s, all delivered in full).
Also confirmed on hardware, alongside 24/24 API checks on five boards:
99-anthias.confbeats Raspberry Pi OS's40-rpi-volatile-storage.conf— verified both bysystemd-analyze cat-configand by the journal physically moving out of/run.persistentmeasured rather than inferred is doing real work: a Pi 5 with 8 stale files in/var/log/journalwhile journald wrote to/runcorrectly reportedfalse. Both cheaper probes would have lied on that board./api/v2/rebootwas readable through the API afterwards, and previous-boot entries stayed reachable.redact()is linear — 1.40s for a 2000-line read containing 87 × 47 KB lines.Two notes for reviewers, neither a code change:
adm, and journald grantsadmjournal access by ACL, sojournalctl -u docker.servicereturns rows withoutSupplementaryGroups=systemd-journal. Withholdingadmas well reproduces the silent empty read exactly as described, so the grant is worth keeping as defence in depth — it just isn't what makes logs readable on a stock install.SystemMaxUsewarning is real and worth repeating in release notes: on the x86 board the first journald restart took the journal from 1.8 GB to 56 MB, discarding about 3.7 months of history.since/untilalready cover relative and absolute ranges (-5m,-2h&-1h,@<epoch>), but two sharp edges are worth knowing for any UI built on this: the API's owntimestampis not valid input (the allowlist has no.), and absolute timestamps are read in the device's local timezone while the API returns UTC — the mismatch yields a200with an empty list rather than an error. Both are left as-is here and tracked for the follow-up that surfaces this in the web UI.Correction: persistent journald is no longer enabled by default
The first version of this branch wrote
Storage=persistenton every install, overriding theStorage=volatilethat Raspberry Pi OS pins in40-rpi-volatile-storage.conf. That override was wrong. The vendor setting exists to keep journald off the SD card, and a player's journal is not small:Measured over 1h/6h/24h windows on idle QA players carrying the normal Anthias stack, before any real content churn. The bulk is our own containers:
anthias-serverandanthias-celerytogether account for roughly half the entries. It is also the write pattern SD cards handle worst — small appends with a periodic fsync, amplified by the card's internal block management.The endpoint never needed it. The host agent reads the volatile journal in
/runperfectly well. Verified on a Pi 5 left on the image default:GET /api/v2/system/logsreturns 500 entries withpersistent: false, andunit,containerandsincefilters all work — 24/24 of the API suite passes with journald writing nothing to the card. Persistence only buys history that survives a reboot, andpersistentin the response already tells a client which one it got, so a device on the default is honest about showing current-boot-only logs rather than silently truncated ones.So it is now
anthias_journal_persistent, defaulting tofalse:Storage=volatilewith the journal still in/run.99-anthias.confwhen the flag is off, so flipping it back restores the vendor setting instead of stranding our override. Verified by enabling, confirmingStorage=persistentwith the journal moved to/var/log/journal, then disabling and watching it converge back tovolatileand/run./var/log/journalmoved inside the conditional too. On an image without the Pi vendor drop-in, Debian'sStorage=automeans creating that directory is itself the persistence switch.SupplementaryGroups=systemd-journalon the host-agent unit is unaffected — that is what actually makes the journal readable, and it costs nothing in writes.Both tunables moved from
vars/to a newdefaults/, because role vars outrank inventory and group_vars: a per-device override set the normal way would have been silently ignored from where they were.