Skip to content

fix: correct what the real arm64 host contradicted - #17

Merged
isaacslee merged 14 commits into
fix/archive-permissionsfrom
fix/vm-findings
Oct 16, 2026
Merged

isaacslee merged 14 commits into
fix/archive-permissionsfrom
fix/vm-findings

Conversation

@isaacslee

Copy link
Copy Markdown
Member

Why

The first real-host acceptance runs on a disposable Ubuntu 24.04 aarch64 VM (Lima, Apple Virtualization, on the maintainer's laptop). It is turning the fixtures' assumptions into observations. Each assumption the host contradicted gets a commit here. This PR stays open while the acceptance continues, and its final head is the commit the acceptance record names.

Stacked on #16.

What changed

  • fix: expect what systemd prints for restricted namespaces. systemd 255 prints RestrictNamespaces=yes for a restricted unit, no for an unrestricted or unset one, and a list of types otherwise. The check expected an empty value for baseline, so a correct host failed its own check. baseline is now checked as yes, and userns (newly checked) as no.

How it was verified

On the VM. These were observed while the acceptance commit was being prepared:

  • systemctl show -p RestrictNamespaces gave yes, no, net user, and no when unset;
  • a stand-in process carrying the rendered unit's sandbox (same User, Slice, masks, ProtectHome, BindPaths, RestrictNamespaces) reported RestrictNamespaces=yes, and unshare -U inside it failed with "Operation not permitted".

Locally: check drill, 45 passed. It adds cases for a baseline runner reporting no, or net user, and for a userns runner reporting yes.

GitHub-hosted CI: BLOCKED (see #3).

Evidence level

RUNTIME VERIFIED for what systemd 255 prints. The fix is SYNTHETICALLY TESTED, until runner-slice check passes against the registered runner on the VM.

Limitations

The behaviour is observed on systemd 255 only.

Risks

None.

Compatibility impact

None.

Remaining work

Continue the acceptance after registration. Further findings will be added here as commits.

On the ARM64 acceptance VM, systemd 255 printed RestrictNamespaces=yes for
a restricted unit, where the check expected an empty value, so a correct
host would have failed its own check.

- baseline is checked as yes; userns is now checked as no
- Drill fixtures print what systemd 255 was observed to print
Copilot AI balanced review requested due to automatic review settings October 15, 2026 14:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

The printed registration used --replace, which silently takes over any
runner already registered under that name to the repository, possibly
another host's; a first registration should refuse instead.

- The registration command no longer passes --replace
- It explains that config.sh refuses on a clash, and what --replace does
@isaacslee

Copy link
Copy Markdown
Member Author

Added: f28b6a5, fix: register a runner without replacing one of the same name.

The registration command RunnerSlice printed used --replace. That silently takes over any runner already registered under the same name to the repository, which could be another host's runner. The first registration is now printed without it. If the name is already registered, config.sh refuses, and the printed note explains what --replace would do.

Verified locally:

  • The new apply-drill case fails against the previous command and passes now.
  • tests/run.sh passes: shellcheck reports 20 scripts clean, and the drills run 434 cases with 0 failed.

The acceptance subject is now f28b6a553749588a438510c1104928fce613b4fd (tree bc54b3c).

  • It is pushed to the dogfood repository as subject/<sha>.
  • The VM's installed tree equals git archive of this commit, file by file (48 files).

Also checked before the first registration, by reading the code and the host:

  • Token through the environment. Runner v2.338.0 reads ACTIONS_RUNNER_INPUT_TOKEN as the token argument and masks it as a secret. It then removes it from its own environment (src/Runner.Listener/CommandSettings.cs).
  • sudo forwarding. sudo --preserve-env=VAR passes that one variable to the runner account, and only that one. This was tried with a harmless test variable.
  • The registration does not exist yet. GitHub lists 0 runners for the repository. The runner directory has no .runner, .credentials or .credentials_rsaparams.
  • The runner files are in place. config.sh is present and executable. The directories have the profile's owners and modes. --work equals the profile's work-dir.

On the ARM64 VM, registration printed "ldd: ./bin/libcoreclr.so: No such
file or directory" three times: config.sh checks its native libraries by
./bin paths before it changes directory, so run from elsewhere the check
silently checked nothing.

- Registration and deregistration run config.sh under env -C RUNNER_DIR
- The printed note and the operations guide explain why
@isaacslee

Copy link
Copy Markdown
Member Author

Added: 9208151, fix: run config.sh from its own directory.

What happened. During the first real registration (runner rs-arm64-dogfood-01, registered by the owner), config.sh printed this three times:

ldd: ./bin/libcoreclr.so: No such file or directory

The other two lines named libSystem.Security.Cryptography.Native.OpenSsl.so and libSystem.IO.Compression.Native.so. Authentication and registration still succeeded.

What was established on the VM, and in the runner's source:

  • The libraries are present. All three exist in runner/bin/. Run as the runner account from the runner directory, ldd reports 0 unresolved dependencies for each of them.
  • The check uses relative paths. In config.sh (v2.338.0, and unchanged on main), the ldd ./bin/... checks on lines 23–37 run before cd "$DIR" on line 72. The registration ran with the working directory / (limactl shell --workdir /), so ./bin/... did not exist.
  • The check silently passed. ldd writes "No such file" to stderr, and grep 'not found' reads only stdout. The dependency check therefore passed without checking anything.
  • Reproduced:
    • config.sh --help run from / prints 3 ldd: lines;
    • run from the runner directory, it prints 0;
    • run through env -C <runner-dir>, it prints 0.
  • Known upstream: actions/runner #945 (this exact message), #2373 ("Change directories first", closed) and #4460 (open).

Conclusion. This is a working-directory problem, not a missing library. The registration is valid and was not repeated.

Fix. The registration and deregistration commands RunnerSlice prints now run env -C <runner-dir> HOME=… ./config.sh …. The note and the operations guide say why. The apply drill fails without the change, and passes now with 141 cases.

Verification:

  • tests/run.sh passes: shellcheck reports 20 scripts clean, and the drills run 436 cases with 0 failed.
  • The VM's installed tree equals git archive of the new commit, file by file (48 files).

The acceptance subject is now 92081511e1a59e982b8ee5ba017d58174a7fb9af (tree 57413b0). Evidence gathered before this commit is recorded against the commit it ran on:

  • the registration ran from the command f28b6a5 printed;
  • the earlier probes ran on 364a9c3.

On the ARM64 VM the printed deregistration ran "config.sh remove
--unattended", and the runner answered "Unrecognized command-line input
arguments for command remove: 'unattended'": remove accepts only --token,
--pat and --local.

- The printed deregistration command no longer passes --unattended
- The token still comes from the environment, which remove reads
The lifecycle has now run on one real host with real GitHub runners, so
the status lines, the verified platforms and the acceptance table say what
was shown there, at what level, and what still was not.

- An acceptance record with commits, dogfood run ids and the five fixes
- Each fixture assumption is marked as held, partly wrong or not observed
- Ubuntu 24.04 aarch64 is listed as verified; x86_64 is still not
@isaacslee

Copy link
Copy Markdown
Member Author

The ARM64 acceptance is complete. All 16 steps were run on a real host. The record is in docs/acceptance/2026-10-arm64.md (commit 01ca427).

Added since the last comment:

  • 2bc81c0 fix: deregister without a flag config.sh remove rejects. During the real deregistration of both runners, config.sh remove --unattended printed "Unrecognized command-line input arguments for command remove: 'unattended'". Runner v2.338.0 accepts only --token, --pat and --local for remove. Both removals still succeeded, because the token came from the environment.
    • A new apply-drill case fails without the fix.
    • The VM's installed tree was compared file by file with this commit before the final retirement ran on it.
  • 01ca427 docs: record the arm64 acceptance and what it verified. It changes documentation only (README.md, docs/*).

Real-host results, recorded against the commit that ran them:

  • On 9208151: two GitHub runners (ids 2 and 22) took real dogfood jobs.
    • drills: 436 cases on each runner;
    • boundary: 14/14 on each runner;
    • pool assignment observed on both runners;
    • shared ceiling under concurrent load;
    • busy guard with real hold jobs: deferred, uninterrupted, exactly one restart afterwards;
    • guest reboot, after which services, limits and filters held and boundary jobs passed.
  • On 2bc81c0: final retirement.
    • GitHub lists 0 runners.
    • The host matches its pre-test baseline, except the shared slice unit, which is the documented residue.
    • The foreign unit, the other account and the secret file are intact.

Still not verified:

  • unknown job state on a real host (synthetic only);
  • a misregistration on a real host;
  • a runner self-update;
  • runner-slice status on the host;
  • x86_64.

GitHub-hosted CI remains BLOCKED (see #3). The dogfood runs are separate evidence and do not count as that gate.

Retirement trusted pgrep, so a missing or failing pgrep read as "no
process", and it removed the unit, then the root, then the account, so a
userdel failure left a profile no run could finish.

- read every uid in /proc/<pid>/status; an unreadable table refuses
- remove the account first, then the root with its marker last, then the unit
- read userdel's outcome back; remove only the account's own unused group
- report a leftover home or group by name, and document each recovery
Preflight accepts systemd 249 and later, but only 255 has run the
acceptance. Accepting a version read as supporting it.

- preflight warns on any accepted systemd other than 255
- the documents separate supported (255) from accepted but unverified
O_NOFOLLOW covers only the last path component, so a symlinked bin/
redirected root's read of the dependency manifest. Parser errors such as
deep nesting escaped as exit 1 instead of "cannot tell".

- walk down from the runner directory with O_NOFOLLOW on every component
- any exception, and a manifest of the wrong shape, exits 2
The printed commands exported the token into the operator's shell and
relied on a later unset, and a failing gh still ran config.sh.

- print registration and deregistration as one ( ... ) block
- a gh failure ends the block before config.sh runs
tar gave the directory the archive's own mode for "." (0775) while it
unpacked, so the comment that it stayed closed to others was not true.

- unpack with --no-overwrite-dir, so the directory stays 0700 until chmod
Both guards existed with no apply-level case, so removing either passed
every drill.

- a person's account of the runner's name creates and downloads nothing
- root's temporary directory ignores an inherited TMPDIR
The retirement and reader changes were run on the acceptance VM with a
never-registered test profile, and the record must say what that covered
and what stays synthetic.

- add the follow-up table to the arm64 acceptance record
- state in verification.md which #18 cases reached RUNTIME VERIFIED
…much

An independent review of the retirement rework found that removing the
root's children one by one let rm --one-file-system empty a filesystem
mounted at work/ or runner/, and that some failures still read as success.

- refuse, removing nothing, while anything is mounted in the runtime root
  or the mount table cannot be read; rm also runs with --preserve-root=all
- write the root's marker back when the final rmdir fails
- tell a failed account lookup from an absent account, keep a group when
  accounts also come from a directory service, report an undeletable unit
The re-review of the retirement fixes found three ways the new proofs could
still read a host as safe when it was not.

- parse "passwd:files sss" and trailing comments, and treat compat with "+"
  or "-" lines in /etc/passwd as a directory service
- compare mounts against the root's canonical path, so a symlinked /var
  still finds a mount below it
- cover escaped mount paths, prefix siblings and an empty table in drills
@isaacslee
isaacslee merged commit a0c67e5 into fix/archive-permissions Oct 16, 2026
1 check passed
@isaacslee
isaacslee deleted the fix/vm-findings branch October 16, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants