Skip to content

backport(v0.5.x): migrate the data disk and env across KMS key epochs - #1574

Merged
kvinwang merged 1 commit into
release/v0.5.xfrom
backport/v0.5.x-key-epochs
Nov 13, 2026
Merged

kvinwang merged 1 commit into
release/v0.5.xfrom
backport/v0.5.x-key-epochs

Conversation

@kvinwang

Copy link
Copy Markdown
Collaborator

Backports the guest side of the 0.6.4 KMS key epoch migration to 0.5.x, so a 0.5.x guest keeps booting after a 0.6.4 KMS rotates its root or app epoch. Without it, the KMS hands the guest keys of the new epochs, which open neither the data disk nor the encrypted env.

  • KMS keys: GetAppKey takes optional root/app epochs. Requests for older epochs pin the root CA this boot established and do not extend RTMR3.
  • Data disk: the 0.6.4 LUKS2 header parser replaces the luks2 crate check. It also accepts the single dstack-kms token recording the keyslot's epochs. The disk is re-keyed to the current epochs in crash-safe steps; an epoch-0 disk is left untouched.
  • Env: opens SealedEnv (HPKE) envelopes, falls back to older epochs for a legacy env, re-seals it to the current epochs and returns it through UpdateSealedEnv, which also closes the env update window before applications start.

Not included: the 0.6.4 "initializing" disk label and historical keys for applications' GetKey.

The image must declare "sealed_env": true in metadata.json for a 0.6.4 VMM to store the migrated env (meta-dstack PR to follow); without it the guest re-migrates on every boot while the old epochs are still served.

Tests: ported 0.6.4 unit tests, including LUKS migration and crash-resume against a real cryptsetup 2.7.0 (0.5.x images ship 2.7.2), env migration, and SDK SealedEnv fixtures.

A 0.6.4 KMS answers a GetAppKey without an epoch selector with the keys of
the current epochs. Once its root or app epoch is rotated, a 0.5.x guest
gets keys that open neither its data disk nor its encrypted env.

Backport the guest side of the 0.6.4 key epoch migration:

- GetAppKey takes optional root/app epochs and returns the served epochs
  and the SealedEnv key. Historical requests pin the root CA this boot
  established and do not extend RTMR3.
- Replace the luks2 crate header check with the 0.6.4 parser, which also
  accepts the one dstack-kms token recording the epochs of the keyslot,
  and re-key the data disk to the current epochs in crash-safe steps. An
  epoch-0 disk stays untouched, so older images still open it.
- Open an env sealed with HPKE (SealedEnv) or, failing the current key, a
  legacy env with the keys of older epochs, then re-seal it to the current
  epochs and hand it to the VMM with UpdateSealedEnv, which also closes the
  env update window before applications start. A VMM without the method
  has no window; the 0.5.x VMM accepts and ignores the update.

The 0.6.4 initializing label and the application-facing historical GetKey
are not part of this backport.

Ported from v0.6.4: dstack-util/src/system_setup/{luks,sealed_env}.rs,
the env migration in system_setup.rs, crypto.rs sealed env functions,
host_api.rs update_sealed_env and the http-client StatusError.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang

Copy link
Copy Markdown
Collaborator Author

Physical TDX end-to-end test on tdxlab (Linux 6.8.0-1028-intel, QEMU 8.2.2, user-mode networking). Topology: a private dstack-vmm v0.6.4, with two dstack-kms:0.6.4 CVMs on the dstack-0.6.3 guest OS. Both KMSs use dev auth and have OS image verification on. KMS A was bootstrapped at root epoch 0. KMS B was onboarded from A with rotate=true, so it holds epochs 0–1 and serves epoch 1.

The candidate image was built from meta-dstack #85 with the dstack submodule at 04c1a05 (os_image_hash 60e34cf2…, sealed_env: true). The test app keeps a sentinel file on the encrypted ext4 data disk and reads one secret from the encrypted env.

  1. 0.5.13 with KMS A (epoch 0): boots, decrypts the env and writes the sentinel.
  2. Control: unmodified 0.5.13 pointed at KMS B: fails with Failed to open encrypted data disk.
  3. Same VM and disk upgraded in place to the candidate, with KMS B:
    • The disk is re-keyed from epochs (0,0) to (1,0) using the historical key.
    • The legacy env is opened with the epoch-0 key, re-sealed to (1,0), and sent with UpdateSealedEnv; the VMM logged stored sealed env from the guest and .sealed-env now records epoch (1,0).
    • The sentinel and the secret hash are unchanged.
  4. Reboot: only the current-epoch key request is made; there is no re-keying and no historical request. The sentinel is kept (boot 3).
  5. Switching the migrated VM back to the 0.5.13 image: the 0.6.4 VMM rejects it, because that image does not support sealed_env.

App GetKey outputs change after the rotation, as expected: historical application keys are out of scope.

@kvinwang
kvinwang merged commit 51c20c5 into release/v0.5.x Nov 13, 2026
9 checks passed
@kvinwang
kvinwang deleted the backport/v0.5.x-key-epochs branch November 13, 2026 12:39
@kvinwang kvinwang mentioned this pull request Nov 13, 2026
@kvinwang

Copy link
Copy Markdown
Collaborator Author

Re-validated on the released image dstack-0.5.14 (meta-dstack v0.5.14, os_image_hash 12f6dc37…).

Environment:

  • tdxlab, physical TDX, Linux 6.8.0-1028-intel, QEMU 8.2.2;
  • a private dstack-vmm v0.6.4 with user-mode networking;
  • dstack-kms:0.6.4 CVMs on the dstack-0.6.3 guest OS, with dev auth and OS image verification on;
  • KMS A at root epoch 0, then KMS B rotated from A (epoch 1), then KMS C rotated from B (epoch 2).

Results:

  1. 0.5.13 on KMS A: boots and writes a sentinel to the encrypted ext4 disk; the env carries one secret.
  2. Same VM moved to 0.5.14 on KMS B:
    • the disk is re-keyed from (0,0) to (1,0);
    • the legacy env is opened with the epoch-0 key, re-sealed to (1,0) and stored by the VMM;
    • the sentinel and the secret are unchanged.
  3. Reboot: only the current-key request; no migration.
  4. Same VM on KMS C:
    • the disk is re-keyed from the token's (1,0) to (2,0);
    • the SealedEnv at (1,0) is opened with the historical key and re-sealed to (2,0);
    • the data is unchanged.
  5. A second 0.5.13 VM moved straight to 0.5.14 on KMS C:
    • the disk is re-keyed from (0,0) to (2,0);
    • the legacy env search tries epoch 1 (fails, as expected), then succeeds at epoch 0, and re-seals to (2,0);
    • the data is unchanged.

Not covered on hardware: app-epoch rotation (dev auth cannot advance app epochs) and power loss in the middle of a migration (covered by the cryptsetup unit tests only).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant