Track nvx-rs host VMM supervisor bugs: a non-UTF-8 process name at a reused PID blocks reconcile, reattach, and stop #720

Bug-hunt tracking issue for the nvx-rs host VMM supervisor (launch journal, stop, reattach), added in #656, and for the host process identity through which it, the store's recovery, and nvx::host::instance::Instance examine a recorded VMM.

Baseline: origin/dev at f63b4b1f6e4eadd7cc0dab1eea8fd49d886b7549. Every permalink in the findings and in this pass's notes uses this SHA. The earlier pass, collapsed at the end, keeps its own baseline. Finding 1 is still present at the newer head 3da0cfab72182196afc551e56c30a3afb38af529, which changed none of the files that it links.

Hunt: reopened by a second time-boxed pass of about 45 minutes. It started at 01:00 −03:00 on 2026-10-22 and stopped at about 01:35, when this bug was confirmed. No repository files were modified and nothing was committed. The reproduction ran as an integration test in a scratch copy of the baseline's nvx-rs (git archive), outside the repository, under WSL 2 (Linux 6.18) with Rust 1.95.0.

Selection: the bug is in how the host examines a recorded VMM process, which the supervisor's reattach and stop rely on, so it belongs to this issue's component. No other bug-hunt issue covers it. #741 (open) covers the nightly CI reporter, and #737 (open) covers the guest managed agent's output limit. #464 covers the managed control client's pipe writes, and #733 covers sandbox resource limits. This issue's first pass had found nothing, so it's reopened and updated rather than duplicated.

Findings

• Read /proc/<pid>/stat as bytes in platform::linux::read_stat, and add a regression test with a non-UTF-8 command name.

1. A reused PID whose process has a non-UTF-8 command name blocks reconcile, reattach, and stop of a VMM that has exited

Severity: medium impact, low likelihood.

• Impact: when it triggers, every lifecycle operation of the affected sandbox fails until an unrelated process exits. For a long-lived daemon, that means until someone intervenes by hand.
• Likelihood: the PID of a recorded VMM must be reused by a process whose command name isn't valid UTF-8.
  • Linux truncates comm to 15 bytes without regard to UTF-8. So any process whose name is longer than that and has a multibyte character across the boundary qualifies, such as a localized program name.
  • Stale records themselves are routine. Any VMM that exits or crashes on its own leaves one until the next reconcile, reattach, or stop.

Where

• read_stat reads /proc/<pid>/stat with fs::read_to_string. That read fails with InvalidData ("stream did not contain valid UTF-8") whenever the command name in parentheses isn't UTF-8.
  • parse_stat never gets to look past the last ), although every field that it needs is ASCII.
• Every Linux examination of a recorded VMM goes through read_stat:
  • platform::process_start_time has these callers:
    • the store's recovery, vmm::find
    • Instance::reconcile and its recover_launch
    • serves_launch
    • the control session's connect loop. There, the error counts as inconclusive, so a session waits until its deadline instead of reporting ProcessExited.
  • open_process is behind Process::open and process::kill. The supervisor's reattach_with and Instance::stop use them.
• Both contracts say that this state can be determined:
  • Process::open documents that "the process has exited when … the process that has it started at another time". It reserves errors for when "the state of the process cannot be determined".
  • Instance::reconcile fails only on "an inconclusive check".
  • Here the line is readable, and its start time isn't the recorded one.
• aci_edge_sandboxes takes each sandbox's run state from Instance::reconcile. Its start, exec, stop, and deprovision all call it first.
• The CLI's own copy of this check isn't affected. cli::sandbox::identity::process_start_time already reads the file as bytes and parses them with linux_start_time, so nvx-rs sandbox is unaffected.

Trigger: two things happen in turn:

1. A VMM that a runtime record or a supervisor store names exits without its record being cleared. It may crash or be killed, or its guest may power off.
2. Before the next reconcile, reattach, or stop, a process whose comm isn't valid UTF-8 takes its PID.

Expected: the stat line is read, and its start time differs from the recorded one, so the VMM counts as exited:

• process_start_time returns Some(start).
• Process::open returns None.
• reconcile() returns None and clears the record.
• reattach returns Exited.

Actual: each of them fails with InvalidData until that unrelated process exits:

• reconcile() reports "cannot determine whether OpenVMM process N of sandbox S is running" and keeps the record.
• reattach maps the error to Indeterminate("the recorded VMM process N cannot be examined: …").
• Instance::stop fails through process::kill.

Reproduction: a scratch integration test, in a copy of the baseline's nvx-rs, exercises the public API:

1. It starts a live process and names it with prctl(PR_SET_NAME).
2. It writes a runtime record that names that PID with another start time, as a reused PID would have.
3. It examines the record through the public API.

The control case names the process vmm-reused-pid. The other names it aデータ処理, 16 bytes, which Linux truncates to 15 in the middle of 理.

use std::io::{BufRead, BufReader};
use std::process::{Command, Stdio};

use nvx::host::instance::{Instance, RuntimeRecord, STATE_FORMAT};
use nvx::host::platform::process_start_time;
use nvx::host::process::{BootId, Process, ProcessRef};

/// Runs a live process named `name`, as a recorded OpenVMM's reused ID would hold, and checks it.
fn check(name: &str) {
    let script = format!(
        "import ctypes, time\n\
         ctypes.CDLL(None).prctl(15, {name:?}.encode(), 0, 0, 0)\n\
         print('ready', flush=True)\n\
         time.sleep(60)\n"
    );
    let mut child = Command::new("python3")
        .args(["-c", &script])
        .stdout(Stdio::piped())
        .spawn()
        .unwrap();
    let mut line = String::new();
    BufReader::new(child.stdout.take().unwrap())
        .read_line(&mut line)
        .unwrap();
    let pid = child.id();
    let stat = std::fs::read(format!("/proc/{pid}/stat")).unwrap();
    let end = stat.iter().position(|&byte| byte == b')').unwrap() + 1;
    println!("== /proc/{pid}/stat begins {:?}", String::from_utf8_lossy(&stat[..end]));
    println!("   comm bytes: {:?}", &stat[stat.iter().position(|&b| b == b'(').unwrap()..end]);
    println!("   process_start_time({pid}) = {:?}", process_start_time(pid));
    // What the supervisor's reattach and the store's recovery hold a recorded VMM with.
    let stale = ProcessRef {
        pid,
        start_time: 1,
        boot: BootId::current().unwrap(),
    };
    println!(
        "   Process::open(pid {pid}, start time 1) is_some = {:?}",
        Process::open(&stale).map(|process| process.is_some())
    );

    let dir = std::env::temp_dir().join(format!("nvx-bughunt-{pid}"));
    std::fs::create_dir_all(&dir).unwrap();
    let instance = Instance::new(&dir, "demo");
    // A stale record of an OpenVMM that exited and whose ID this process reused: the recorded
    // start time is not this process's.
    instance
        .write_runtime(&RuntimeRecord {
            format: STATE_FORMAT,
            pid,
            start_time: 1,
            endpoint: dir.join("control.sock").display().to_string(),
        })
        .unwrap();
    println!("   Instance::reconcile() = {:?}", instance.reconcile());
    println!(
        "   runtime record kept afterwards: {:?}",
        instance.runtime().map(|r| r.is_some())
    );
    child.kill().unwrap();
    child.wait().unwrap();
    std::fs::remove_dir_all(&dir).unwrap();
}

#[test]
fn reused_pid_with_ascii_command_name() {
    check("vmm-reused-pid");
}

#[test]
fn reused_pid_with_non_utf8_command_name() {
    // 16 bytes, which Linux truncates to 15 in the middle of the last 3-byte character.
    check("a\u{30C7}\u{30FC}\u{30BF}\u{51E6}\u{7406}");
}

$ cargo test --locked --test bughunt_non_utf8_comm_v2 -- --nocapture --test-threads=1
test reused_pid_with_ascii_command_name ... == /proc/26994/stat begins "26994 (vmm-reused-pid)"
   comm bytes: [40, 118, 109, 109, 45, 114, 101, 117, 115, 101, 100, 45, 112, 105, 100, 41]
   process_start_time(26994) = Ok(Some(26512))
   Process::open(pid 26994, start time 1) is_some = Ok(false)
   Instance::reconcile() = Ok(None)
   runtime record kept afterwards: Ok(false)
ok
test reused_pid_with_non_utf8_command_name ... == /proc/26996/stat begins "26996 (aデータ処�)"
   comm bytes: [40, 97, 227, 131, 135, 227, 131, 188, 227, 130, 191, 229, 135, 166, 231, 144, 41]
   process_start_time(26996) = Err(Error { kind: InvalidData, message: "stream did not contain valid UTF-8" })
   Process::open(pid 26996, start time 1) is_some = Err(Error { kind: InvalidData, message: "stream did not contain valid UTF-8" })
   Instance::reconcile() = Err(Error { kind: Io, message: "cannot determine whether OpenVMM process 26996 of sandbox demo is running", source: Some(Error { kind: InvalidData, message: "stream did not contain valid UTF-8" }) })
   runtime record kept afterwards: Ok(true)
ok

Suggested fix: read the line as bytes, as the CLI's copy does. The smallest change keeps read_stat's String:

match fs::read(format!("/proc/{pid}/stat")) {
    // A command name may hold any bytes; only the ASCII fields after its last `)` are parsed.
    Ok(stat) => Ok(Some(String::from_utf8_lossy(&stat).into_owned())),

) is ASCII and never part of a multibyte sequence, so lossy decoding keeps the last ) and every field after it. I checked the change in the scratch copy:

• Both cases above print Ok(Some(..)), Ok(false), Ok(None), and Ok(false).
• The platform, process, instance, and store unit tests pass: cargo test --lib -- host::platform host::process host::instance host::store reports 38 passed.

For a regression test, name a child with prctl(PR_SET_NAME) as above. Alternatively, have read_stat return bytes and feed start_time_from_stat a line with invalid UTF-8.

This pass: explored and verified (no bug)

Each suspect was traced through its callers and existing tests, and then checked by reasoning or with the smallest available run.

1. Distro binding and file seals (c3bfa40, 21efc02, 8e9008e).
   • Code: DistroBinding, FileSeal, and the Windows seal.
   • provision re-verifies a layer whose link names another manifest at its resolved path, and start checks the binding at that path. The UUID is lowercased at parse time.
   • The recorded binding survives the dumps round trip, including 64-bit NTFS volume serials above i64::MAX, because Value holds integers of any size.
   • On Windows, the guarded open shares reads, so it's compatible with OpenVMM's read-only open and with other sandboxes that use the same layer.
2. Guest mapping API (f70adb9, 97c930d).
   • The path rules of Mapping::new match the decoder's, and a test pins that.
   • apply_with keeps the agent's earlier behavior: the modes, the O_EXCL | O_NOFOLLOW file target, and the remount flags.
3. CI suite selection (7ccb8d5, 9ad0468).
   • required_ci_expected_results matches the if: of each job in ci.yml: the artifact, debug-kernel, and build jobs, both OpenVMM test suites, the microVM and debug jobs, the platform jobs, and performance-gate.
   • A failed detector fails the required check.
   • The only .md files under suite input trees are READMEs.
4. Guest build dep-info check (c65bb3d, 930e3fb).
   • _dep_info_paths reads Cargo's single-rule format with \ -escaped spaces, including a literal backslash before an escaped space.
   • SOURCE_PATTERN and guest_binary_source_files agree, and ci.py embeds the pattern anchored.
5. Store and release retries.
   • The store's log-claim recheck (4fcbeb4) never sleeps past its deadline.
   • The release upload (cf8bac7) retries "release not found" only after it saw the draft without the asset.
6. Windows host platform.
   • The store's privacy checks behave as intended:
     • The owner must be the user, SYSTEM, or Administrators.
     • Unknown ACE types are refused.
     • A directory must pass entries on to its files.
   • The command-line quoting of spawn_detached follows the MSVC runtime's rules: it quotes on a space or tab, and doubles backslashes only before a quote and before the closing quote.
7. Azure runner package (build_package.py): the path validators, the overlap checks, and the 65,535-byte decoded customData limit hold.

Checked, not filed

These are low-impact, by design, or unreproduced.

• A dev push that changes only README.md or THIRD_PARTY_NOTICES.md publishes no development release.
  • Every release package ships both files (release.py#L1763, #L1799-L1800).
  • Even so, the classifier treats them as documentation (ci.py#L113), unlike LICENSE, which is a platform input.
  • The next non-documentation push publishes them, and doc/ci.md documents the rule.
• Linux seals key on st_dev. On btrfs subvolumes and other filesystems with anonymous device numbers, st_dev can change across a remount or reboot. start would then ask to provision again. That's the intended response to a binding that no longer holds. Not reproduced.
• Instance runtime records carry no boot ID, unlike the supervisor's ProcessRef.
  • On Linux, the start time counts clock ticks since boot, so the same PID and tick could recur after a reboot.
  • Not reproduced, and very unlikely.
• A Windows seal of a handle that isn't a disk file fails with Io, from code reading only. For a layer path such as NUL, seal fails with Io rather than the documented Unavailable "not found". Only the error kind of an invalid input differs.

Validation limits

• The host ran Windows 11 with Python 3.14.7 and Rust 1.95.0. Linux checks ran under WSL 2 Ubuntu (6.18.33.2-microsoft-standard-WSL2).
  • KVM, MSHV, and WHP weren't used, and no OpenVMM or guest artifacts were built.
  • Neither the microVM scenarios nor test-aci-edge-sandboxes ran.
• The reproduction exercised process_start_time, Process::open, and Instance::reconcile directly. The following follow from code reading of the call sites linked above:
  • the supervisor's reattach
  • Instance::stop
  • the store's recovery
  • the session's connect loop
  • the aci_edge_sandboxes operations
• The suggested fix was checked only in the scratch copy, against the reproduction and 38 unit tests, not the full suites.
• The guest exec driver changed often during the hunt: five follow-up fixes landed on 2026-10-20 and 2026-10-21. It was skipped because it borders Track guest managed agent bugs: output limit forwards later output after a dropped chunk #737.