Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,16 @@ web server.
>
> - Every behaviour described here is exercised by drills against a fixture
> host. That is SYNTHETICALLY TESTED.
> - Nothing in this code has yet been run on a real host or registered with
> GitHub. That verification is the next stage, and its plan is in
> [docs/verification.md](docs/verification.md).
> - Do not run it on a machine you care about yet.
> - The full lifecycle has also run on **one real host**: an Ubuntu 24.04
> aarch64 VM with two real GitHub runners. It went from install to
> registration, jobs, pool, busy guard, reboot, deregistration and
> retirement ([docs/acceptance/2026-10-arm64.md](docs/acceptance/2026-10-arm64.md)).
> - RunnerSlice's own drills ran there as GitHub Actions jobs, on runners
> RunnerSlice installed and managed.
> - **Not verified:** x86_64, other distributions, and GitHub-hosted CI for
> this repository. GitHub-hosted CI is currently blocked by the
> organisation's billing.
> - No release exists. Use a machine you can afford to rebuild.

## Why

Expand Down Expand Up @@ -94,9 +100,10 @@ fleet with the runners idle, and register the new runner.
## Supported hosts

Ubuntu with systemd 249 or later, cgroup v2 and x86_64 or aarch64 is what
preflight accepts. **No combination has been verified with this code yet.**
[docs/compatibility.md](docs/compatibility.md) is the only list of what has
been.
preflight accepts. What is supported is narrower: **verified so far, Ubuntu
24.04 on aarch64 (systemd 255), on one VM.** Preflight warns on any systemd
other than 255. [docs/compatibility.md](docs/compatibility.md) is the only list of
what has been verified.

## Documentation

Expand Down
107 changes: 107 additions & 0 deletions docs/acceptance/2026-10-arm64.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Acceptance record: ARM64, 2026-10-15 to 2026-10-16

This is the first real-host run of the lifecycle acceptance defined in
[verification.md](../verification.md). Raw logs are kept by the maintainer
outside the repository, with credentials excluded.

## Environment

- **VM:** a disposable Ubuntu 24.04 aarch64 VM. Lima 2.2.1, Apple
Virtualization, 2 vCPU, 3 GiB RAM, 25 GiB disk, NAT networking, no shared
folders, no ports forwarded. Kernel 6.8.0, systemd 255.
- **Runners:** two repository-scoped runners, `rs-arm64-dogfood-01` (GitHub id
2) and `rs-arm64-dogfood-02` (id 22). Both serve a private dogfood
repository and form one pool. Neither was used for anything else.
- **Inventory:**
- `runners.slice`, with 100% CPU, MemoryHigh 1G and MemoryMax 1280M (from
1536M), 512 tasks, CPUWeight 50, and a deny on `1.1.1.1/32`;
- one masked directory of another service, and one protected secret file;
- the arm64 2.338.0 seed, with `apt-dependency liblttng-ust1t64`.
- **Registration and removal tokens** were obtained and entered by the
owner. They travelled through a pipe into `ACTIONS_RUNNER_INPUT_TOKEN`, and
were never shown, logged or stored.

## Commits

Each observation is recorded against the commit installed when it was made.
The installed tree was compared with `git archive` of that commit, file by
file.

| Commit | What it was used for |
|---|---|
| 6adadd3 | First install. Preflight refused the group-writable tree, which led to #16. |
| c52630f, 364a9c3 | Preflight, first apply, slice limits, deny on reload, killed download and unpack, retiring an unregistered runner. |
| f28b6a5 | The registration of runner 1, by the command it printed. |
| 9208151 | Runner 1 and runner 2 live, all dogfood jobs, pool, busy guard and reboot. |
| 2bc81c0 | Final retirement, after both runners were deregistered. |
| 7f84c01 | The #18 follow-up below. No runner was registered. |

## Dogfood jobs

All ran on the self-hosted runners, with GitHub-hosted minutes exhausted and a
$0 Actions budget in place. **Billing did not stop self-hosted jobs.**

| Run | Mode | Runner (id) | Result |
|---|---|---|---|
| 38895433749 | drills | 2 | success: shellcheck 20 clean, 436 cases, commit 9208151, tree 57413b0 |
| 38895647330 | boundary | 2 | success, 12/12 |
| 38899375262 | drills, `target=dogfood-2` | 22 | success, 436 cases |
| 38899378922 | drills, pool, concurrently | 2 | success, 436 cases |
| 38899791941, 38899805349 | boundary on each runner | 22, 2 | success, 14/14 each |
| 38899906623, 38899920558 | hold, during the busy-guard test | 22, 2 | success: not interrupted |
| 38901272065, 38901282745, 38901413328 | boundary after the guest reboot | 22, 22, 2 | success, 14/14 each |
| 38901401020 | hold, during retirement | 22 | success: retirement waited for it |

## Defects found on the real host, and fixed

Each fix landed with a drill case that fails without it.

1. **The release archive unpacked group-writable**, and root's preflight
refused it (#16).
2. **`systemctl show` prints `RestrictNamespaces=yes`, not an empty value.**
A correct host would have failed its own check (#17).
3. **The printed registration used `--replace`**, which silently takes over
another host's runner of the same name (#17).
4. **`config.sh` checks its libraries by `./bin` paths before it changes
directory.** Run from elsewhere, the check passed vacuously, with `ldd: …
No such file` (#17, `env -C`; actions/runner #945 and #4460).
5. **`config.sh remove` rejects `--unattended`** (#17).

## Follow-up: the post-acceptance review (#18), 2026-10-16

The same VM, after the acceptance had ended and the host matched its
baseline. Commit 7f84c01 (tree ae6c64a) was installed. All 49 files in the
archive were compared with the commit and are identical; root owns the whole
tree. No runner was registered. Every case used a new test-only profile,
`probe`, which was never registered, and new test-only objects. Every
temporary host change was undone. Afterwards, `/etc/passwd`, `/etc/group`
and `/etc/login.defs` matched their checksums from before, byte for byte,
and the host listing matched its state from before.

| Case | Observed |
|---|---|
| Preflight on systemd 255 | `ok systemd 255`. No "unverified" warning, because 255 is the supported version. |
| The runner directory during the unpack | 232 samples, in this order: 750 (created by apply), then 700 (for the whole unpack), then 750 (after chmod). The archive's 0775 never appeared. |
| A symlinked `bin/` in the runner directory | `check`: `cannot determine the installed runner version`. With the real `bin/`: `runner version 2.338.0`. |
| A real process running as the account (`Uid: 997 997 997 997`) | `retire` refused, named the pid, and removed nothing. |
| A process holding the account only as its saved uid (`Uid: 0 0 997 0`) | `pgrep -u`, the old check, found nothing (exit 1). `retire` refused, named the pid, and removed nothing. |
| `userdel` failing, with `/etc/passwd` locked | `userdel could not remove rs-probe (exit 1); the account is still there`. The account, group, home, runtime root, marker and unit were all still there. |
| The account removed, but an immutable file in the runtime root | `could not remove all of /var/lib/rs-probe`. The unit and the marker were kept. After `chattr -i`, the next `retire` removed the root, then the unit, and exited 0. |
| `USERGROUPS_ENAB no` | `userdel` left the group. RunnerSlice removed it as the account's own unused group (gid 987). Exit 0. |
| The group with a member, and a home not owned by the account | `userdel` exited 12. The leftover home was reported, and not removed. The group was left, because it has a member. Exit 1. On a second run, both were reported again. The group was now "without its account", and was still not removed. After a manual cleanup, as the recovery table describes, `retire` exited 0. |

Not produced on the real host: an unreadable `/proc` (root can always read
it), and a `.runner` that is a FIFO, oversized or deeply nested. Those remain
SYNTHETICALLY TESTED. The drills ran inside the VM as `nobody`: 504 cases,
0 failed. The release drill skipped, because it needs a git checkout.

## Not verified here

- Unknown job state on a real host. It cannot be produced safely as root on a
healthy host, so it remains SYNTHETICALLY TESTED.
- A misregistration on a real host. It remains SYNTHETICALLY TESTED.
- A runner self-update.
- `runner-slice status` on the real host. The VM had no authenticated `gh`, so
GitHub's view was read from the maintainer's machine instead. The command
itself was not run there.
- x86_64, and any other distribution or provider.
29 changes: 26 additions & 3 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,37 @@ actually been verified, and at what level ([verification.md](verification.md)).
| Distribution | Architecture | systemd | Status |
|---|---|---|---|
| Ubuntu 24.04 | x86_64 | 255 | **Planned** for the first verification VM. Not yet verified. |
| Ubuntu 24.04 | aarch64 | 255 | **Planned** for the ARM64 verification VM. Not yet verified. |
| Ubuntu 22.04 | aarch64 | 249 | The code RunnerSlice was extracted from has run here in production (see [field-report.md](field-report.md)). **This code** has not been run there. |
| Ubuntu 24.04 | aarch64 | 255 | **Verified** on a disposable VM (Lima, Apple Virtualization), at RUNTIME VERIFIED and REBOOT VERIFIED. See [acceptance/2026-10-arm64.md](acceptance/2026-10-arm64.md). The lifecycle was run with two real GitHub runners, ending in deregistration and retirement. One VM; no other aarch64 host verified. |
| Ubuntu 22.04 | aarch64 | 249 | The code RunnerSlice was extracted from has run here in production (see [field-report.md](field-report.md)). **This code** has not been run there. Not supported. |
| Any other distribution | | | Refused by preflight. |

## systemd: accepted is not supported

Two ranges differ, on purpose, and neither stands in for the other.

| | Versions | Meaning |
|---|---|---|
| **Supported** | 255 | Verified on a real host by the acceptance above. |
| **Accepted, unverified** | 249–254, and 256 or later | Preflight lets these through, and warns, naming the version as unverified. Nothing about them has been observed on a host. |
| **Refused** | older than 249 | Preflight refuses. |

Why the code accepts more than is supported. The floor of 249 is what the
units rely on: `ProtectHome=tmpfs` with `BindPaths`, and a slice's
`IPAddressDeny` merged into every unit inside it. The one value `check` was
found to depend on in a version-specific way is how `systemctl show` prints
`RestrictNamespaces` (`yes`, `no`, or a list). That printing code
(`src/shared/bus-print-properties.c`) was read, not run, in the v249, v252
and v255 sources and is the same in all three. That is a source review. It
is not verification, and it covers only that one property. Raising the floor
to 255 would refuse hosts that nothing shows are broken. Calling them
supported would claim what nothing shows works. So they are accepted with a
warning, and only 255 is listed as supported.

Preflight refuses:

- a distribution other than Ubuntu;
- systemd older than 249;
- systemd older than 249 (249–254 and 256 or later pass with a warning that
they are unverified);
- a host without the cgroup v2 unified hierarchy, or without its `cpu`,
`memory` or `pids` controller;
- an architecture other than the release seed's.
Expand Down
3 changes: 2 additions & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ is. It remains a candidate for its own project (decision 0004).

Only what has been verified is listed, in [compatibility.md](compatibility.md).
Preflight refuses anything other than Ubuntu with systemd 249 or later and
cgroup v2.
cgroup v2. It accepts more than is supported: any systemd other than 255
passes with a warning that it is unverified.

## Security position

Expand Down
18 changes: 12 additions & 6 deletions docs/installation.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# Installation

> **Status: pre-release.** These steps follow what the code does and are
> exercised by drills against a fixture host. They have not yet been carried
> out on a real host. That is the next verification stage
> ([verification.md](verification.md)). Until it is done, use a disposable
> machine.
> **Status: pre-release.** These steps were carried out on one real host: an
> Ubuntu 24.04 aarch64 VM ([acceptance/2026-10-arm64.md](acceptance/2026-10-arm64.md)).
> No x86_64 host has been verified. Use a machine you can afford to rebuild.

## Requirements

What preflight refuses to work without:

- **Ubuntu**, with systemd 249 or later.
- **Ubuntu**, with systemd 249 or later. Only systemd 255 is supported, that
is, verified. Preflight warns on any other accepted version
([compatibility.md](compatibility.md)).
- **The cgroup v2 unified hierarchy**, with the `cpu`, `memory` and `pids`
controllers available.
- **x86_64 or aarch64**, matching the `runner-platform` of the release seed.
Expand Down Expand Up @@ -103,4 +103,10 @@ To roll back, move the symlink back and apply again.
1. Retire every runner, deregistering each from GitHub ([operations.md](operations.md)).
2. Remove `/usr/local/bin/runner-slice` and `/opt/runner-slice-<version>`.
3. Remove `/etc/runner-slice` and the slice unit in `/etc/systemd/system`.
The slice unit is the one thing retirement leaves on purpose, because it is
shared by every runner. Once the last profile is gone, `runner-slice`
refuses an empty inventory and will not remove it for you.
4. Run `systemctl daemon-reload`.

Packages installed for `apt-dependency` stay installed. RunnerSlice never
removes packages, because other software may need them.
Loading