guest: gate boot on GPU TEE attestation via requirements.verify_gpu

[kvinwang]

Problem

On NVIDIA images the GPU conf-compute ready state is set unconditionally by nvidia-persistenced.service (ExecStartPost=nvidia-smi conf-compute -srs 1), so CUDA work can be submitted to a GPU that never proved it is a genuine, CC-enabled NVIDIA TEE. A malicious host can attach a non-CC GPU (or leave CC mode off) and the CVM boots, gets its app keys, and runs the workload anyway.

An earlier iteration of meta-dstack#83 gated this with a dedicated nvidia-gpu-attestation.service plus Requires= drop-ins on five units — more moving parts, and the policy was baked into the image with no per-app control.

Fix

Move the whole gate into dstack-util setup (system_setup.rs) and make it an app-compose requirement:

• requirements.verify_gpu (new, defaults to true when omitted): whether an attached GPU must pass local TEE attestation before boot continues.
• In do_sys_setup, after time sync and before key provisioning:
  • no NVIDIA GPU on the PCI bus (sysfs vendor 0x10de, class 0x0300xx/0x0302xx) → skip; GPU-less CVMs are unaffected.
  • verify_gpu=true → run nvattest attest --device gpu --verifier local --nonce <32 random bytes> (5 min timeout, verifier output kept in /run/nvidia-gpu-attestation/attestation.out, 0600), then set the ready state. Any failure — a non-CC GPU, CC mode left off by the host, or a missing nvattest binary — aborts setup, so the CVM fails closed before app keys are provisioned (boot.error reported to the VMM, dstack-prepare reboots).
  • verify_gpu=false → skip attestation and set the ready state directly (best-effort: a GPU without CC mode has no ready state to set).

The companion meta-dstack change (meta-dstack#83) drops the systemd service/wrapper/drop-ins and ships only the nvattest CLI plus an ordering drop-in (dstack-prepare.service after nvidia-persistenced/nvidia-fabricmanager).

Like the other requirements fields, verify_gpu needs manifest_version: "3", so older guests fail closed instead of silently ignoring the opt-out.

Verification

• cargo test -p dstack-types -p dstack-util (new tests: verify_gpu_defaults_to_true; 23 + 34 pass), cargo fmt --check, clippy clean.
• End-to-end on a GCP a3-highgpu-1g (TDX + H100) is pending the meta-dstack image rebuild with this commit; the same nvattest invocation (nonce + local verifier) was previously verified there via the systemd variant of this gate (successful attestation in the serial log, ready state set only afterwards).

[Copilot]

Pull request overview

This PR adds a guest-side security gate that prevents GPU-enabled CVMs from continuing boot (and reaching key provisioning) unless an attached NVIDIA GPU passes local TEE attestation, controlled by a new app-compose requirement (requirements.verify_gpu, defaulting to true).

Changes:

• Add a GPU detection + attestation + “ready state” gate to dstack-util setup that runs before filesystem setup/key provisioning.
• Extend dstack-types manifest Requirements with verify_gpu plus an AppCompose::verify_gpu() accessor (default true) and tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
dstack-util/src/system_setup.rs	Adds GPU presence detection, runs nvattest (local verifier) when required, and sets NVIDIA CC “ready state” during system setup.
dstack-types/src/lib.rs	Introduces requirements.verify_gpu (optional bool) with defaulting behavior and unit tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kvinwang]

Addressed in f3a5c19 — nvidia_gpu_present() now returns Result<bool> and setup fails when the PCI bus cannot be enumerated, instead of treating it as "no GPU".

[kvinwang]

Addressed in f3a5c19 — both nvattest and nvidia-smi now run through a shared run_command() helper (tokio process + bounded timeout: 300s attest, 60s nvidia-smi), so a wedged driver cannot hang the boot.

[kvinwang]

Addressed in f3a5c19 — nvidia-smi is now invoked as /usr/bin/nvidia-smi. (PATH hijack is not really in the threat model here — the rootfs is measured and the boot PATH is systemd-controlled — but consistency with the nvattest invocation costs nothing.)

[kvinwang]

Addressed in f3a5c19 — verify_gpu() now ignores the opt-out unless manifest_version >= 3, with a test (legacy_optout). Within dstack-util this was already unreachable (verify_app_compose_policy rejects legacy manifests carrying requirements before setup_gpu runs), but the helper is now safe to call standalone.

[kvinwang]

Verified end-to-end on GCP a3-highgpu-1g (TDX + 1×H100 80GB, SPOT, project-local test image built from meta-dstack#83 + this branch via SRC_DIR:pn-dstack-guest override):

Default path (generated app-compose, manifest v2, no requirements) — attestation enforced by default, runs inside dstack-util setup before key provisioning, ~1.8s total (remote RIM + OCSP):

[   13.04] dstack-util: verifying GPU TEE attestation
[   14.82] nvattest: Attesting GPUs
[   14.83]     Measurement Result: success
[   14.84]     Attestation/Driver RIM/VBIOS RIM Cert Chains: Status: valid, OCSP: good
[   14.85] GPU attestation was successful
[   14.87] dstack-util: GPU ready state set
[   14.88] dstack-util: GPU TEE attestation succeeded

Workload container afterwards: NVIDIA H100 80GB HBM3, CC status: ON, Confidential Compute GPUs Ready state: ready.

Opt-out path (verbatim app-compose, manifest_version: "3", requirements: {"verify_gpu": false}) — no nvattest invocation, ready state set directly (~140ms):

[   11.91] dstack-util: WARN requirements.verify_gpu is false; setting GPU ready state without attestation
[   12.05] dstack-util: GPU ready state set

Container likewise sees CC status: ON / Ready state: ready.

Not exercised on real hardware: the rejection path (non-CC GPU / CC off → nvattest non-zero exit → setup bails before key provisioning). GCP a3 instances always come up with CC enabled, so this can't be provoked there; the failure propagation itself (nvattest non-zero → bail → boot.error → dstack-prepare failure/reboot) follows the same code path as a missing-nvattest bail.