nvx-rs: terminate VMs of another runtime identity on stop

aci_edge_sandboxes/README.md

@@ -641,8 +641,9 @@ Windows and `$XDG_STATE_HOME/nvx/sandboxes` (default
 
 Each sandbox's state directory is the `nvx::host::store` of its VM, which
 records the runtime identity of the VM's start: the identities of its
-OpenVMM, kernel, and initramfs, and the boot and control contract versions of
-the build that started it. Every operation finds the sandbox's VM under the
+OpenVMM, kernel, and initramfs, and the boot contract, control contract, and
+control session versions of the build that started it, which `nvx::contract`
+defines. Every operation finds the sandbox's VM under the
 backend's runtime identity, so a backend never drives a VM that another NVX
 build, or other artifacts, started.
 
@@ -662,10 +663,13 @@ build, or other artifacts, started.
 
 Without a runtime identity, `probe`, `provision`, and `start` fail with
 `backend_unavailable`, as does any operation on a sandbox whose store records a
-VM. An operation on a sandbox whose VM runs under another runtime identity,
-such as one that the build before an upgrade started, fails with
-`backend_error`, naming the parts that differ and OpenVMM's process ID, and
-leaves the VM running: stop it with the build and artifacts that started it.
+VM. A sandbox whose VM runs under another runtime identity, such as one that the
+build before an upgrade started, is never driven, since its guest might speak
+other contracts. `stop` terminates its OpenVMM through its verified process
+identity, without a control session, and reports `forced`, but no summary of
+OpenVMM's outcome report, which follows that runtime's contracts. Every other
+operation fails with `backend_error`, naming the parts that differ and
+OpenVMM's process ID, and leaves the VM running.
 
 ### How it runs OpenVMM
 
@@ -700,8 +704,11 @@ leaves the VM running: stop it with the build and artifacts that started it.
   agent kills whatever it left behind, including processes in other sessions,
   so no workload process outlives its exec.
 - **Stop** asks the guest to shut down. If the guest does not finish within
-  `stop_timeout`, OpenVMM is terminated. Everything the workloads wrote is
-  discarded, because the root file system lives in guest memory.
+  `stop_timeout`, OpenVMM is terminated. The OpenVMM of a VM of another
+  [runtime identity](#runtime-identity) is terminated without a shutdown
+  request, and the stop waits up to `stop_timeout` for it to exit. Everything
+  the workloads wrote is discarded, because the root file system lives in guest
+  memory.
 - **Deprovision** deletes the state directory. It refuses to remove a directory
   that contains unexpected files.
 
@@ -1010,7 +1017,7 @@ conversion, without the exec case, turns any `nvx::Error` into an `Error`.
 | --- | --- |
 | provision | none |
 | start | `bootMilliseconds`: time from launch to guest readiness |
-| stop | `forced`: whether OpenVMM had to be terminated; `vmOutcome`, `vmStatusCode`, `teardownComplete`: summary of OpenVMM's outcome report |
+| stop | `forced`: whether OpenVMM had to be terminated; `vmOutcome`, `vmStatusCode`, `teardownComplete`: summary of OpenVMM's outcome report, for a VM of the backend's runtime identity |
 | deprovision | none |
 
 ### Security notes
@@ -1063,6 +1070,7 @@ fake-OpenVMM lifecycle test.
   discovered artifacts.
 - `nvx.py test-aci-edge-sandboxes` runs the ignored `openvmm_e2e` tests against a real
   hypervisor with the repository's kernel and Alpine initramfs: the lifecycle,
+  reattachment and the termination of a VM of another runtime identity,
   working directories, host path mapping, network rules, private-network
   ingress, bidirectional host-loopback access, and exec environments.
   CI runs them on

aci_edge_sandboxes/src/openvmm/config.rs

@@ -27,8 +27,8 @@ pub struct OpenVmmConfig {
     pub initrd: PathBuf,
     /// The runtime identity of the artifacts, which the store records with each sandbox's VM:
     /// the identities of OpenVMM, the kernel, and the initramfs, with the contract versions of
-    /// this build. A VM that runs under another runtime identity, as after an upgrade, is reported
-    /// and neither driven nor stopped.
+    /// this build. A VM that runs under another runtime identity, as after an upgrade, is not
+    /// driven: a stop terminates its OpenVMM, and every other operation reports it.
     ///
     /// Leave it unset for artifacts located in an NVX release or repository checkout, as by
     /// [`from_release_dir`](Self::from_release_dir), [`from_repo_layout`](Self::from_repo_layout),

aci_edge_sandboxes/src/openvmm/mod.rs

@@ -32,8 +32,10 @@
 //! OpenVMM, kernel, and initramfs, which [`OpenVmmConfig::runtime_identity`] gives or the install
 //! metadata of the artifacts' release or checkout names, and the contract versions of the build
 //! that started it. A VM whose runtime identity differs from the backend's, as after an upgrade,
-//! fails every operation with [`ErrorCode::BackendError`](crate::ErrorCode::BackendError), and
-//! keeps running: only the build and artifacts that started it can drive or stop it.
+//! is never driven, since its guest might speak other contracts: **stop** terminates its OpenVMM
+//! through its verified process identity, without a control session, waiting up to
+//! [`OpenVmmConfig::stop_timeout`] for it to exit, and every other operation fails with
+//! [`ErrorCode::BackendError`](crate::ErrorCode::BackendError) and leaves the VM running.
 //!
 //! # Policy honor matrix
 //!
@@ -648,16 +650,19 @@ impl Backend for OpenVmmBackend {
 
     fn stop(&self, sandbox_id: &SandboxId) -> Result<StopResult> {
         let (mut locked, _) = self.store.lock_and_load(sandbox_id)?;
-        let running = match self.find(&mut locked)? {
+        let stopped = match self.find(&mut locked)? {
             Found::Stopped => {
                 return Err(Error::already_stopped(format!(
                     "sandbox {sandbox_id} is not running"
                 )));
             }
-            Found::Running(running) => running,
-            Found::Mismatched(mismatched) => return Err(mismatched.error().into()),
+            Found::Running(running) => locked.stop(running, self.config.stop_timeout)?,
+            // Its guest might not speak this build's control protocol, so OpenVMM is terminated
+            // through its verified process identity, without a control session.
+            Found::Mismatched(mismatched) => {
+                locked.terminate(mismatched, self.config.stop_timeout)?
+            }
         };
-        let stopped = locked.stop(running, self.config.stop_timeout)?;
         let mut metadata = Metadata::new();
         metadata.insert("forced".to_owned(), stopped.forced.into());
         if let Some(report) = &stopped.report {

aci_edge_sandboxes/tests/openvmm_e2e.rs

@@ -19,6 +19,7 @@ use std::thread;
 use std::time::{Duration, Instant};
 
 use aci_edge_sandboxes::nvx::Artifacts;
+use aci_edge_sandboxes::nvx::host::store::ArtifactIdentity;
 use aci_edge_sandboxes::nvx::resolve_guest_path as guest_path;
 use aci_edge_sandboxes::openvmm::{OpenVmmBackend, OpenVmmConfig};
 use aci_edge_sandboxes::{
@@ -35,11 +36,15 @@ fn variable(name: &str) -> String {
     })
 }
 
+/// Returns the artifacts that the environment names.
+fn artifacts() -> Artifacts {
+    Artifacts::from_dir(variable("ACI_EDGE_SANDBOXES_E2E_ARTIFACTS_DIR")).unwrap()
+}
+
 /// Returns the configuration of the artifacts that the environment names, with `state_root`.
 fn config(state_root: PathBuf) -> OpenVmmConfig {
-    let artifacts = Artifacts::from_dir(variable("ACI_EDGE_SANDBOXES_E2E_ARTIFACTS_DIR")).unwrap();
     OpenVmmConfig::from_artifacts(
-        artifacts,
+        artifacts(),
         variable("ACI_EDGE_SANDBOXES_E2E_HYPERVISOR")
             .parse()
             .unwrap(),
@@ -89,19 +94,23 @@ fn shell(nvx: &AciEdgeSandbox, sandbox_id: &SandboxId, command_line: &str) -> Ex
     run(nvx, sandbox_id, ExecRequest::command_line(command_line))
 }
 
+/// Returns the state root of the clients named `name`.
+fn state_root(name: &str) -> PathBuf {
+    env::var("ACI_EDGE_SANDBOXES_E2E_STATE_ROOT")
+        .map(PathBuf::from)
+        .unwrap_or_else(|_| {
+            env::temp_dir().join(format!("aci-edge-sandboxes-e2e-{}", std::process::id()))
+        })
+        .join(name)
+}
+
 /// Creates a client over the artifacts named by the environment.
 ///
 /// Keep `name` short: on Unix, each sandbox's control socket path,
 /// `<state root>/<name>/<32-character token>/control.sock`, must fit in 107 bytes, and the
 /// state root of `scripts/nvx.py test-aci-edge-sandboxes` takes 42 of them under `/tmp`.
 fn client(name: &str) -> (AciEdgeSandbox, Arc<OpenVmmBackend>) {
-    let state_root = env::var("ACI_EDGE_SANDBOXES_E2E_STATE_ROOT")
-        .map(PathBuf::from)
-        .unwrap_or_else(|_| {
-            env::temp_dir().join(format!("aci-edge-sandboxes-e2e-{}", std::process::id()))
-        })
-        .join(name);
-    let backend = Arc::new(OpenVmmBackend::new(config(state_root)).unwrap());
+    let backend = Arc::new(OpenVmmBackend::new(config(state_root(name))).unwrap());
     (AciEdgeSandbox::from_shared(backend.clone()), backend)
 }
 
@@ -1715,3 +1724,46 @@ fn openvmm_lifecycle_on_a_real_hypervisor() {
         ErrorCode::StaleId
     );
 }
+
+#[test]
+#[ignore = "requires a hypervisor and NVX guest artifacts; run scripts/nvx.py test-aci-edge-sandboxes"]
+fn a_vm_of_another_runtime_identity_is_only_terminated() {
+    let (nvx, backend) = client("identity");
+    let cleanup = started(&nvx, &backend, &ProvisionRequest::new());
+    let sandbox_id = id(&cleanup);
+    // A backend over the same artifacts, as after its host restarts, reattaches to the VM.
+    let (restarted, _) = client("identity");
+    assert_eq!(shell(&restarted, sandbox_id, "echo same").stdout, b"same\n");
+
+    // One whose initramfs differs, as after an upgrade, drives the VM through no control session,
+    // and its stop terminates OpenVMM.
+    let mut config = config(state_root("identity"));
+    let mut identity = artifacts().runtime_identity().unwrap();
+    identity.initramfs = ArtifactIdentity::sha256(&"0".repeat(64)).unwrap();
+    config.runtime_identity = Some(identity);
+    let upgraded = AciEdgeSandbox::openvmm(config).unwrap();
+    let runtime: serde_json::Value = serde_json::from_slice(
+        &fs::read(backend.log_path(sandbox_id).with_file_name("runtime.json")).unwrap(),
+    )
+    .unwrap();
+    let pid = support::recorded_pid(&runtime);
+    let error = upgraded
+        .exec(sandbox_id, &ExecRequest::command_line("true"))
+        .unwrap_err();
+    assert_eq!(error.code(), ErrorCode::BackendError, "{error}");
+    assert!(
+        error.message().contains(&format!(
+            "runs under another runtime identity, which differs in initramfs, so it is not \
+             driven: stop it to terminate its OpenVMM process {pid}"
+        )),
+        "{error}"
+    );
+    assert!(support::process_running(pid));
+    let stopped = upgraded.stop(sandbox_id).unwrap();
+    assert_eq!(stopped.metadata.unwrap()["forced"], true);
+    assert!(!support::process_running(pid));
+
+    // The sandbox stopped, so the artifacts that started it start it again.
+    nvx.start(sandbox_id).unwrap();
+    assert_eq!(shell(&nvx, sandbox_id, "echo again").stdout, b"again\n");
+}

aci_edge_sandboxes/tests/openvmm_fake.rs

@@ -2239,21 +2239,21 @@ fn runtime_identities_come_from_the_install_metadata() {
     );
     assert_eq!(run(&nvx, &sandbox_id, "echo before").stdout, b"before\n");
 
-    // The release is upgraded in place: the backend reads the new identity, and reports the VM
-    // that the earlier OpenVMM started rather than driving or stopping it.
+    // The release is upgraded in place: the backend reads the new identity, and does not drive
+    // the VM that the earlier OpenVMM started.
     write_checksums(&release, '4');
     for error in [
         nvx.exec(&sandbox_id, &ExecRequest::command_line("echo"))
             .unwrap_err(),
-        nvx.stop(&sandbox_id).unwrap_err(),
         nvx.start(&sandbox_id).unwrap_err(),
         nvx.deprovision(&sandbox_id).unwrap_err(),
     ] {
         assert_eq!(error.code(), ErrorCode::BackendError, "{error}");
         assert!(
-            error
-                .message()
-                .contains("runs under another runtime identity, which differs in OpenVMM"),
+            error.message().contains(
+                "runs under another runtime identity, which differs in OpenVMM, so it is not \
+                 driven: stop it"
+            ),
             "{error}"
         );
     }
@@ -2263,7 +2263,26 @@ fn runtime_identities_come_from_the_install_metadata() {
     // Back on the artifacts that started it, the backend drives the VM again.
     write_checksums(&release, '1');
     assert_eq!(run(&nvx, &sandbox_id, "echo after").stdout, b"after\n");
-    nvx.stop(&sandbox_id).unwrap();
+
+    // After another upgrade, a stop terminates the VM's OpenVMM, which it asks nothing, and
+    // summarizes no outcome report, since a report beside the VM would be the other runtime's.
+    write_checksums(&release, '4');
+    fs::write(
+        fixture
+            .state_root
+            .join(sandbox_id.token())
+            .join("outcome.json"),
+        r#"{"outcome":{"category":"success","status_code":0},"teardown":{"workers":true}}"#,
+    )
+    .unwrap();
+    let stopped = nvx.stop(&sandbox_id).unwrap().metadata.unwrap();
+    assert_eq!(
+        stopped.keys().map(String::as_str).collect::<Vec<_>>(),
+        ["forced"],
+        "{stopped:?}"
+    );
+    assert_eq!(stopped["forced"], true);
+    assert!(!support::process_running(pid));
     nvx.deprovision(&sandbox_id).unwrap();
 
     // A release without its checksum list cannot be identified.

doc/run.md

@@ -1165,10 +1165,16 @@ digests of OpenVMM, the kernel, and the Alpine initramfs, as the install
 metadata lists them, without hashing an artifact. A release lists them in its
 `SHA256SUMS`, and a checkout in the build provenance that `nvx.py` writes to
 `build/` with each artifact that it builds, and that `nvx.py download`
-installs. `exec`, `stop`, and `deprovision` drive only a VM that artifacts of
-the same identity started: one that other artifacts started, as before they
-were rebuilt or replaced, is reported, and must be stopped with the artifacts
-that started it. The store reads runtime records of its own format only, so
+installs. The transitions drive only a VM that artifacts of the same identity
+started. One that other artifacts started, as before they were rebuilt or
+replaced, may speak other contracts, so `start` and `exec` refuse it, naming
+the parts that differ, `deprovision` refuses it as running, and `stop`
+terminates its OpenVMM without a guest shutdown, waits up to `--timeout`
+seconds for it to exit, and warns that it did. A `stop` that cannot determine
+the OpenVMM process, whose terminated OpenVMM does not exit in time, or whose
+wait Ctrl-C ends, keeps the sandbox's runtime record, and the next `stop`
+retries. The store reads
+runtime records of its own format only, so
 stop every running sandbox with the NVX version that started it before you
 switch versions: a version with another format fails every transition of such
 a sandbox with a message that names the format of its record. An OpenVMM that
@@ -1298,6 +1304,15 @@ named. It checks that:
 - two overlapping `start` calls then launch a single OpenVMM process: one
   succeeds, and the other fails with `sandbox is already running or has stale
   runtime state`;
+- the `nvx-rs` CLI of an artifact root whose build provenance names another
+  initramfs, as after an upgrade, drives no VM that the checkout's artifacts
+  started: its `exec` and `start` fail, naming the initramfs and OpenVMM's
+  process ID, and its `deprovision` fails because the sandbox runs, without
+  changing the runtime record or OpenVMM process, while the checkout's CLI
+  still serves requests. Its `stop` then terminates OpenVMM without a guest
+  shutdown and warns that it did, and leaves only `config.json`,
+  `sandbox.lock`, and `openvmm.log`, without a control socket or pipe, after
+  which its `stop` finds the sandbox not running, and the sandbox starts again;
 - after OpenVMM exits without a `stop`, `exec` reports the stale runtime state
   once and clears it, after which `stop` finds the sandbox not running, and the
   state directory holds only `config.json`, `sandbox.lock`, and `openvmm.log`,

doc/usage.md

@@ -774,7 +774,10 @@ against a real hypervisor. The tests start sandboxes with `build/vmlinux` and
 the Alpine initramfs, whose userland the workloads use directly, and run
 workloads. They check the workload identity, timeouts, and cancellation, that
 guest state lasts until a stop, and that a start terminates the VM of an
-interrupted earlier start before deprovisioning. They check that each workload
+interrupted earlier start before deprovisioning. They check that a backend
+over the same artifacts reattaches to a running VM, and that one whose
+initramfs differs drives it through no control session, and only terminates
+it on stop. They check that each workload
 starts in its requested working directory, or `/` without one, and that a
 missing, non-directory, or inaccessible working directory fails the launch.
 They also map temporary host directories read-only, read-write, and denied,