From a1314a7b6bbad6f8ac21da5fb6163b876fb5b92e Mon Sep 17 00:00:00 2001 From: Hang Yin Date: Fri, 26 Jun 2026 10:48:39 +0000 Subject: [PATCH] docs: add native tee interface guide --- README.md | 8 +- docs/design-and-hardening-decisions.md | 6 +- docs/native-tee-interfaces.md | 147 +++++++++++++++++++++++++ docs/usage.md | 10 +- 4 files changed, 162 insertions(+), 9 deletions(-) create mode 100644 docs/native-tee-interfaces.md diff --git a/README.md b/README.md index b0bb0dfea..a01e88bfa 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,7 @@ Apps communicate with the guest agent via HTTP over `/var/run/dstack.sock`. Use **Reference** - [App Compose Format](./docs/normalized-app-compose.md) - Compose file specification +- [Native TEE Interfaces](./docs/native-tee-interfaces.md) - Advanced compatibility with Linux TEE devices and configfs-tsm - [VMM CLI Guide](./docs/vmm-cli-user-guide.md) - Command-line reference - [Design Decisions](./docs/design-and-hardening-decisions.md) - Architecture rationale - [FAQ](./docs/faq.md) - Frequently asked questions @@ -167,14 +168,17 @@ Yes. dstack powers production AI infrastructure at [OpenRouter](https://openrout
Can I run this on my own hardware? -Yes. dstack runs on any Intel TDX-capable server. See the [deployment guide](./docs/deployment.md) for self-hosting instructions. You can also use [Phala Cloud](https://cloud.phala.network) for managed infrastructure. +Yes. dstack runs on supported TEE-capable servers, including Intel TDX-capable hardware. See the [deployment guide](./docs/deployment.md) for self-hosting instructions. You can also use [Phala Cloud](https://cloud.phala.network) for managed infrastructure.
What TEE hardware is supported? -Currently: Intel TDX (4th/5th Gen Xeon) and NVIDIA Confidential Computing (H100, Blackwell). AMD SEV-SNP support is planned. +- **GCP**: Intel TDX (Confidential VMs) +- **AWS**: Nitro Enclaves (NSM attestation) +- **Bare metal**: Intel TDX (4th/5th Gen Xeon) and AMD SEV-SNP on supported dstack OS images +- **GPUs**: NVIDIA Confidential Computing (H100, Blackwell)
diff --git a/docs/design-and-hardening-decisions.md b/docs/design-and-hardening-decisions.md index 0e6ba9c29..81ae51ca7 100644 --- a/docs/design-and-hardening-decisions.md +++ b/docs/design-and-hardening-decisions.md @@ -24,11 +24,11 @@ The meta-dstack layer is designed to create a minimally secure image for booting ### 3. TDX Guest Driver Implementation -**Decision**: We disable the built-in TDX guest driver (`CONFIG_TDX_GUEST_DRIVER=n`) and use a custom implementation. +**Decision**: We use the in-tree confidential guest drivers and the Linux TSM report interface where they are available. For Intel TDX, the dstack kernel configuration enables `CONFIG_TDX_GUEST_DRIVER=y` and `CONFIG_TSM_REPORTS=y`. -**Rationale**: **Primary Goal** is to enable RTMR3 extension capabilities. **Custom Driver Benefits** include enhanced measurement and attestation features, better integration with our security architecture, and flexibility for dstack-specific optimizations. +**Rationale**: The in-tree TDX guest driver exposes the standard `/dev/tdx_guest` device and configfs-tsm report interface. Newer kernels also expose RTMR extension through the TSM measurement sysfs path, which dstack uses for RTMR3 runtime events. This keeps dstack aligned with the standard Linux kernel ABI while preserving measured runtime events. -**Implementation Notes**: Builds upon proven meta-confidential-compute foundation. Maintains compatibility with TDX specifications. Adds value-specific functionality without compromising security. +**Implementation Notes**: The unified dstack confidential-guest image also includes AMD SEV-SNP kernel features. Platform-specific native TEE interfaces are advanced compatibility surfaces for applications that need the kernel ABI directly. The dstack socket remains the normal application API for quotes, keys, application information, and runtime events. ### 4. Randomness Generation and Seeding diff --git a/docs/native-tee-interfaces.md b/docs/native-tee-interfaces.md new file mode 100644 index 000000000..f9beb8b2a --- /dev/null +++ b/docs/native-tee-interfaces.md @@ -0,0 +1,147 @@ +# Advanced Native TEE Interfaces in Containers + +Most applications should use the dstack API through `/var/run/dstack.sock`. Native TEE interfaces are available for advanced compatibility cases, such as unmodified binaries or libraries that already know how to use Linux TEE devices or configfs-tsm. + +The examples below use Compose syntax because dstack currently accepts Compose files for container configuration. + +## Choose an Interface + +Use native interfaces only when your application already depends on them. If you control the application code, prefer the dstack SDK or HTTP API because it also returns dstack metadata, event logs, and verification inputs. + +| Interface | Platform | Use when | +| --- | --- | --- | +| `/var/run/dstack.sock` | dstack-supported TEEs | Your application can call the dstack SDK or HTTP API | +| `/dev/tdx_guest` | Intel TDX | Existing software expects the Linux TDX guest device | +| `/dev/sev-guest` | AMD SEV-SNP | Existing software expects the Linux SEV-SNP guest device | +| `/sys/kernel/config/tsm/report` | Intel TDX and AMD SEV-SNP, when supported by the deployed OS image | Existing software expects configfs-tsm `inblob` and `outblob` report generation | + +The native report formats are platform-specific. Intel TDX returns TDX quote or report data, depending on the interface and library. AMD SEV-SNP returns an SNP attestation report and, for extended report flows, certificate data. + +## Version Availability + +The versions below refer to the dstack OS image version, not an SDK or service binary version. + +| Native interface | Available from | +| --- | --- | +| `/dev/tdx_guest` | dstack OS v0.5.0 on Intel TDX images. The v0.5.x images expose it through the bundled `tdx-guest` kernel module. The v0.6.0.a1 images and later use the in-tree Linux TDX guest driver. | +| TDX configfs-tsm at `/sys/kernel/config/tsm/report` | dstack OS v0.6.0.a1 and later on Intel TDX. It is not available in v0.5.x TDX images. | +| `/dev/sev-guest` | dstack OS v0.6.0 SEV-SNP image line and later on AMD SEV-SNP. It is not available in v0.5.x TDX images. | +| SEV-SNP configfs-tsm at `/sys/kernel/config/tsm/report` | dstack OS v0.6.0 SEV-SNP image line and later on AMD SEV-SNP, when the deployed image enables the Linux TSM report interface. | + +## Learn the Native Linux APIs + +These interfaces are Linux kernel ABIs, not dstack-specific APIs. Use the upstream documentation when you need ioctl structures, configfs file semantics, or provider-specific report formats: + +| API | Official reference | +| --- | --- | +| Intel TDX guest device | [TDX Guest API Documentation](https://docs.kernel.org/virt/coco/tdx-guest.html) | +| AMD SEV-SNP guest device | [SEV Guest API Documentation](https://docs.kernel.org/virt/coco/sev-guest.html) | +| configfs-tsm report ABI | [configfs-tsm report ABI](https://www.kernel.org/doc/Documentation/ABI/testing/configfs-tsm) | + +## Use the dstack API by Default + +Mount the dstack socket when your application can use the dstack SDK or HTTP API: + +```yaml +services: + app: + image: your-image + volumes: + - /var/run/dstack.sock:/var/run/dstack.sock +``` + +The dstack API is the normal application interface for quotes, keys, application information, and runtime events. See the [Guest Agent RPC API](../sdk/curl/api.md) for request and response details. + +## Expose Intel TDX Interfaces + +For TDX software that expects the Linux TDX guest device, expose `/dev/tdx_guest` to the container: + +```yaml +services: + app: + image: your-image + devices: + - /dev/tdx_guest:/dev/tdx_guest +``` + +For TDX software that expects configfs-tsm, mount the TSM configfs subtree: + +```yaml +services: + app: + image: your-image + volumes: + - /sys/kernel/config/tsm:/sys/kernel/config/tsm +``` + +Most configfs-tsm libraries create a report entry under `/sys/kernel/config/tsm/report`, write report data to `inblob`, and read the generated quote or report from `outblob`. + +## Expose AMD SEV-SNP Interfaces + +For SEV-SNP software that expects the Linux SEV guest device, expose `/dev/sev-guest` to the container: + +```yaml +services: + app: + image: your-image + devices: + - /dev/sev-guest:/dev/sev-guest +``` + +For SEV-SNP software that expects configfs-tsm, mount the same TSM configfs subtree: + +```yaml +services: + app: + image: your-image + volumes: + - /sys/kernel/config/tsm:/sys/kernel/config/tsm +``` + +On SEV-SNP, configfs-tsm exposes SNP report generation through the common Linux TSM report ABI. Provider-specific attributes and output files can differ from TDX. Check the library you are running for the exact files it reads. + +## Permissions + +Root containers can use root-owned device and configfs paths once they are exposed to the container. If your image switches to a non-root user, set up access before the non-root process starts. + +For device files, a root entrypoint can relax permissions and then launch the application: + +```yaml +services: + app: + image: your-image + devices: + - /dev/tdx_guest:/dev/tdx_guest + command: sh -lc 'chmod 666 /dev/tdx_guest && exec /app/start' +``` + +If the main process must stay non-root from the start, use a small root helper container to adjust the host device node before the application starts: + +```yaml +services: + tdx-device-perms: + image: busybox + volumes: + - /dev/tdx_guest:/dev/tdx_guest + command: chmod 666 /dev/tdx_guest + restart: "no" + + app: + image: your-image + user: "1000:1000" + depends_on: + tdx-device-perms: + condition: service_completed_successfully + devices: + - /dev/tdx_guest:/dev/tdx_guest +``` + +For AMD SEV-SNP, use the same pattern with `/dev/sev-guest`. + +For configfs-tsm, permissions can be more provider-specific. If your main process must remain non-root, keep the native ABI access in a small root helper process and expose only the operation your application needs over local IPC. + +## Intel TDX RTMR3 Measurements + +On TDX, RTMR3 is an append-only runtime measurement register. It is useful when a launcher measures code or configuration first, then hands permission and execution to the measured code. + +Use `/EmitEvent` through `/var/run/dstack.sock` when you want dstack's event log and verifier to replay an application runtime measurement. If your application extends RTMR3 directly through native interfaces, your application owns the event-log story for those extensions. diff --git a/docs/usage.md b/docs/usage.md index 10a101267..08f072f0d 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -56,11 +56,11 @@ For development images (`dstack-x.x.x-dev`), you can SSH into the CVM for inspec ssh root@10.0.3.2 ``` -## Getting TDX Quote in Docker Container +## Get Attestation Evidence in Containers -To get a TDX quote within app containers: +Most applications should use the dstack socket to get attestation evidence from inside a container. -**1. Mount the socket in `docker-compose.yaml`** +**1. Mount the socket in your Compose file** ```yaml version: '3' @@ -78,10 +78,12 @@ services: ```bash # The argument report_data accepts binary data encoding in hex string. -# The actual report_data passing to the underlying TDX driver is sha2_256(report_data). +# The report_data passed to the underlying TEE is padded to 64 bytes. curl --unix-socket /var/run/dstack.sock http://localhost/GetQuote?report_data=0x1234deadbeef | jq . ``` +For advanced compatibility with unmodified binaries that expect native Linux TEE interfaces such as `/dev/tdx_guest`, `/dev/sev-guest`, or configfs-tsm, see [Advanced Native TEE Interfaces in Containers](./native-tee-interfaces.md). + ## Container Logs Container logs can be obtained from the CVM's `dashboard` page or by curl: