From d0fb2563a47e54148040601719b12062138c3d19 Mon Sep 17 00:00:00 2001 From: Komh Date: Tue, 23 Jun 2026 13:50:35 +0800 Subject: [PATCH] docs: add KubeVirt USB Redirection guide --- ...USB_Redirection_for_KubeVirt_VMs_on_ACP.md | 305 ++++++++++++++++++ ...USB_Redirection_for_KubeVirt_VMs_on_ACP.md | 305 ++++++++++++++++++ 2 files changed, 610 insertions(+) create mode 100644 docs/en/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md create mode 100644 docs/zh/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md diff --git a/docs/en/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md b/docs/en/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md new file mode 100644 index 00000000..9e2e792b --- /dev/null +++ b/docs/en/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,305 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# Use USB Redirection for KubeVirt VMs on ACP + +## Overview + +ACP Virtualization can expose USB devices to a KubeVirt VirtualMachine in two different ways: + +- **USB host passthrough** attaches a USB device that is physically connected to a worker node. This is the better fit for fixed, long-lived devices and performance-sensitive workloads. +- **USB Redirection**, also called client passthrough, redirects a USB device from the workstation that runs `virtctl usbredir` into a running VirtualMachineInstance (VMI). This is the better fit for temporary, user-held, low-throughput devices. + +USB Redirection is not hardware passthrough. Traffic flows through the client, the Kubernetes and KubeVirt subresource connection, `virt-handler`, `virt-launcher`, and the QEMU usbredir channel before it reaches the guest OS. Use it for flexible access to temporary USB peripherals, not as a high-throughput storage or low-latency control path. + +Recommended use cases: + +| Use case | Recommended mechanism | Reason | +| --- | --- | --- | +| USB device is fixed on a worker node and should stay attached to one VM | USB host passthrough | Shorter path, clearer device ownership, better performance | +| User or operator needs to temporarily redirect a local USB device to a VM | USB Redirection | The device does not need to be present on a worker node | +| USB flash drive or external disk used for long-running data transfer | Prefer PVCs, image import, SCP/SSH, object storage, or host passthrough | USB Redirection adds network and userspace forwarding overhead | +| USB token, smart card reader, serial adapter, barcode scanner | USB Redirection, after device-level validation | Low-throughput interaction usually fits this model | +| Industrial control, audio/video capture, or strict timing workloads | Host passthrough or a dedicated hardware design | USB Redirection is sensitive to latency and jitter | + +## Prerequisites + +The cluster side must meet these conditions: + +- ACP Virtualization is installed and healthy. +- The KubeVirt version includes the USB Redirection subresource. KubeVirt introduced this capability in v0.44; ACP 4.4 KubeVirt builds include the `clientPassthrough` API field. +- The target VM/VMI runs on an architecture with USB support. KubeVirt rejects USB Redirection on `s390x`. +- The running VMI has `spec.domain.devices.clientPassthrough` configured. Updating the VM template is not enough for a running VMI; restart the VM so a new `virt-launcher` pod and QEMU domain are created with usbredir slots. + +The client side is the machine where the USB device is physically connected and where `virtctl usbredir` runs. For Linux clients, install: + +- `virtctl` compatible with the ACP Virtualization version. +- `usbredirect`. +- `lsusb`, usually from the `usbutils` package. + +The client must be able to reach the Kubernetes API endpoint for the target cluster and keep the `virtctl usbredir` streaming connection open. It does not need direct network access to worker nodes, the `virt-launcher` pod, the VM IP address, or the guest SSH port. SSH or console access to the guest is only needed if you want to verify the redirected device from inside the OS. + +On Debian or Ubuntu based clients: + +```bash +sudo apt-get install -y usbredirect usbutils +``` + +Managing local USB devices usually requires privileged access, so run `virtctl usbredir` with `sudo` unless a site-specific udev rule grants the required access. + +Windows clients require the Windows build of `usbredirect` and UsbDk, and `usbredirect` must be in `PATH`. Validate the Windows client path separately before using it as a supported delivery workflow. + +## Grant Minimal RBAC + +Users need read access to the VMI and `get` access to the `virtualmachineinstances/usbredir` subresource. ACP's built-in KubeVirt `admin` and `edit` roles normally include this permission; the `view` role does not include USB Redirection access. + +For a least-privilege namespace-scoped role, create a Role and RoleBinding like this: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: vm-usbredir + namespace: +rules: + - apiGroups: ["kubevirt.io"] + resources: ["virtualmachineinstances"] + verbs: ["get"] + - apiGroups: ["subresources.kubevirt.io"] + resources: ["virtualmachineinstances/usbredir"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: vm-usbredir + namespace: +subjects: + - kind: User + apiGroup: rbac.authorization.k8s.io + name: +roleRef: + kind: Role + apiGroup: rbac.authorization.k8s.io + name: vm-usbredir +``` + +Verify the permission: + +```bash +kubectl auth can-i get virtualmachineinstances/usbredir.subresources.kubevirt.io \ + -n +``` + +The expected result is: + +```text +yes +``` + +## Enable USB Redirection on a VM + +Add `clientPassthrough: {}` under `spec.template.spec.domain.devices` in the VM template. + +For a new VM manifest, include the following field. Only the fields relevant to USB Redirection are shown: + +```yaml +spec: + template: + spec: + domain: + devices: + clientPassthrough: {} +``` + +For an existing VM, patch the template: + +```bash +kubectl patch vm -n --type=merge \ + -p '{"spec":{"template":{"spec":{"domain":{"devices":{"clientPassthrough":{}}}}}}}' +``` + +Restart the VM so the running VMI receives the new device configuration: + +```bash +virtctl restart -n +``` + +Confirm that the running VMI contains the field: + +```bash +kubectl get vmi -n \ + -o jsonpath='{.spec.domain.devices.clientPassthrough}{"\n"}' +``` + +Expected output: + +```text +{} +``` + +When `clientPassthrough` is set, KubeVirt creates four QEMU usbredir slots for the VMI. To inspect the live domain: + +```bash +POD=$(kubectl get pod -n -l kubevirt.io/domain= \ + -o jsonpath='{.items[0].metadata.name}') + +kubectl exec -n "$POD" -c compute -- \ + virsh dumpxml 1 | grep -A4 ' usbredir 0951:1666 +``` + +Alternatively, redirect by bus and device address: + +```bash +sudo virtctl -n usbredir 02-03 +``` + +Keep the `virtctl usbredir` process running for as long as the device should remain attached. Press `Ctrl+C` to disconnect. If the client process exits, the client network disconnects, the USB device is unplugged, the VM restarts, or the `virt-launcher` pod is recreated, the guest loses the redirected device and you must run `virtctl usbredir` again. + +## Verify in the Guest + +Log in to the guest OS and confirm that the USB device is visible: + +```bash +lsusb +``` + +For USB storage devices used only for validation, also check block devices: + +```bash +lsblk -f +blkid +``` + +Do not mount the same storage device on the client and guest at the same time. If you must validate a USB flash drive or card reader, unmount it on the client first and prefer read-only access in the guest: + +```bash +sudo mkdir -p /mnt/usb +sudo mount -o ro /dev/sdX1 /mnt/usb +ls -la /mnt/usb +sudo umount /mnt/usb +``` + +Replace `/dev/sdX1` with the partition name shown inside the guest. + +## Limitations and Risks + +USB Redirection depends on `usbredirect`, the client OS USB stack, KubeVirt subresource streaming, QEMU usbredir support, and the guest OS driver. A device may enumerate successfully but still fail at the business-application layer because of driver, protocol, latency, or timing requirements. + +Important limitations: + +- One VMI gets four USB Redirection slots. +- The redirection session is tied to the running client process and the current VMI/`virt-launcher` lifecycle. +- Client network interruptions remove the device from the guest. +- API server, load balancer, or proxy idle timeouts may affect long-lived sessions. +- High-throughput storage, real-time control, and audio/video devices are poor fits. +- USB storage devices can be corrupted if mounted by both the client and guest. + +Validate each sensitive device model, especially USB tokens, smart card readers, serial adapters, and devices with strict timing behavior. + +## Troubleshooting + +### `Not configured with USB Redirection` + +The running VMI does not contain `clientPassthrough`. + +Check the live VMI: + +```bash +kubectl get vmi -n \ + -o jsonpath='{.spec.domain.devices.clientPassthrough}{"\n"}' +``` + +If the output is empty, patch the VM template and restart the VM. + +### `VMI not running` + +The target VMI is not in `Running` state. + +```bash +virtctl start -n +kubectl wait vmi/ -n --for=condition=Ready --timeout=180s +``` + +### `Error on finding usbredirect in $PATH` + +The client cannot find the `usbredirect` binary. + +```bash +sudo apt-get install -y usbredirect +which usbredirect +``` + +### `Could not init libusb` or permission denied + +The local user cannot take control of the USB device. + +```bash +sudo virtctl -n usbredir : +``` + +For a longer-term Linux client setup, create a site-approved udev rule for the device. + +### No free USB Redirection slot + +The VMI may already have four active USB Redirection sessions, or an old local client process may still be running. + +On the client machine: + +```bash +pgrep -af 'virtctl.*usbredir|usbredirect' +``` + +Stop only the stale process that owns the old session, then retry. + +### Client storage device does not return as `/dev/sdX` after disconnect + +Some USB storage devices may remain unbound from the client storage driver after a redirection session ends. First try a physical unplug and replug. If physical replug is not possible, inspect the local USB tree: + +```bash +lsusb -t +``` + +If the relevant interface shows `Driver=[none]`, rebind it to `usb-storage` using the interface ID shown on the client: + +```bash +sudo modprobe usb-storage +echo -n '' | sudo tee /sys/bus/usb/drivers/usb-storage/bind +``` + +The interface ID looks like `3-4:1.0`, but it must be taken from the actual client host. + +## Related Information + +- KubeVirt Client Passthrough: https://github.com/kubevirt/user-guide/blob/main/docs/compute/client_passthrough.md +- KubeVirt v1.7.0 USB Redirection validation: https://github.com/kubevirt/kubevirt/blob/v1.7.0/pkg/virt-api/rest/usbredir.go +- KubeVirt v1.7.0 `clientPassthrough` API field: https://github.com/kubevirt/kubevirt/blob/v1.7.0/staging/src/kubevirt.io/api/core/v1/schema.go +- usbredir project: https://gitlab.freedesktop.org/spice/usbredir/ diff --git a/docs/zh/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md b/docs/zh/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md new file mode 100644 index 00000000..1c10b541 --- /dev/null +++ b/docs/zh/solutions/Use_USB_Redirection_for_KubeVirt_VMs_on_ACP.md @@ -0,0 +1,305 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# 在 ACP 上为 KubeVirt 虚拟机使用 USB Redirection + +## 概述 + +ACP 虚拟化可以通过两种方式把 USB 设备暴露给 KubeVirt VirtualMachine： + +- **USB host passthrough** 将物理连接在 worker 节点上的 USB 设备透传给虚拟机。它更适合固定、长期绑定、性能敏感的设备。 +- **USB Redirection** 也称 client passthrough，将运行 `virtctl usbredir` 的客户端机器上的 USB 设备重定向到正在运行的 VirtualMachineInstance (VMI)。它更适合临时接入、由用户本地持有、低吞吐的 USB 设备。 + +USB Redirection 不是硬件级直通。数据会经过客户端、Kubernetes 和 KubeVirt subresource 连接、`virt-handler`、`virt-launcher` 以及 QEMU usbredir 通道，然后到达 guest OS。它适合临时外设接入，不适合作为高吞吐存储路径或低延迟控制链路。 + +推荐选型如下： + +| 场景 | 推荐机制 | 原因 | +| --- | --- | --- | +| USB 设备固定插在 worker 节点上，并长期绑定某台 VM | USB host passthrough | 链路更短，设备归属清晰，性能更好 | +| 用户或运维需要临时把本地 USB 设备接入 VM | USB Redirection | 设备不需要存在于 worker 节点上 | +| U 盘或移动硬盘用于长时间数据传输 | 优先使用 PVC、镜像导入、SCP/SSH、对象存储或 host passthrough | USB Redirection 引入网络和用户态转发开销 | +| UKey、智能卡读卡器、USB 串口、条码枪 | 设备级验证通过后使用 USB Redirection | 低吞吐交互通常适合该模式 | +| 工业控制、音视频采集或严格时序业务 | Host passthrough 或专用硬件方案 | USB Redirection 对延迟和抖动敏感 | + +## 前提条件 + +集群侧需要满足以下条件： + +- ACP 虚拟化已安装并处于健康状态。 +- KubeVirt 版本包含 USB Redirection subresource。KubeVirt 从 v0.44 开始引入该能力；ACP 4.4 的 KubeVirt 构建包含 `clientPassthrough` API 字段。 +- 目标 VM/VMI 运行在支持 USB 的架构上。KubeVirt 会拒绝在 `s390x` 上使用 USB Redirection。 +- 运行态 VMI 已配置 `spec.domain.devices.clientPassthrough`。只更新 VM 模板不会改变正在运行的 VMI；需要重启 VM，使新的 `virt-launcher` Pod 和 QEMU domain 带上 usbredir 槽位。 + +客户端是 USB 设备实际插入并运行 `virtctl usbredir` 的机器。Linux 客户端需要安装： + +- 与 ACP 虚拟化版本兼容的 `virtctl`。 +- `usbredirect`。 +- `lsusb`，通常由 `usbutils` 软件包提供。 + +客户端必须能够访问目标集群的 Kubernetes API 入口，并保持 `virtctl usbredir` 的流式连接。客户端不需要直连 worker 节点、`virt-launcher` Pod、VM IP 地址或 guest 的 SSH 端口。只有需要在 guest OS 内验证重定向设备时，才需要 SSH、控制台或其他 guest 访问方式。 + +在 Debian 或 Ubuntu 类客户端上： + +```bash +sudo apt-get install -y usbredirect usbutils +``` + +操作本地 USB 设备通常需要特权访问，因此除非已经通过现场 udev 规则授予了对应权限，否则建议使用 `sudo` 运行 `virtctl usbredir`。 + +Windows 客户端需要 Windows 版本的 `usbredirect` 和 UsbDk，并且 `usbredirect` 必须在 `PATH` 中。将 Windows 客户端作为正式交付路径前，需要单独完成兼容性验证。 + +## 授予最小 RBAC + +用户需要读取 VMI，并对 `virtualmachineinstances/usbredir` subresource 具备 `get` 权限。ACP 内置的 KubeVirt `admin` 和 `edit` 角色通常包含该权限；`view` 角色不包含 USB Redirection 权限。 + +如果需要按 namespace 授予最小权限，可创建如下 Role 和 RoleBinding： + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: vm-usbredir + namespace: +rules: + - apiGroups: ["kubevirt.io"] + resources: ["virtualmachineinstances"] + verbs: ["get"] + - apiGroups: ["subresources.kubevirt.io"] + resources: ["virtualmachineinstances/usbredir"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: vm-usbredir + namespace: +subjects: + - kind: User + apiGroup: rbac.authorization.k8s.io + name: +roleRef: + kind: Role + apiGroup: rbac.authorization.k8s.io + name: vm-usbredir +``` + +验证权限： + +```bash +kubectl auth can-i get virtualmachineinstances/usbredir.subresources.kubevirt.io \ + -n +``` + +预期结果为： + +```text +yes +``` + +## 为 VM 启用 USB Redirection + +在 VM 模板的 `spec.template.spec.domain.devices` 下添加 `clientPassthrough: {}`。 + +新建 VM 清单中加入以下字段。这里只展示与 USB Redirection 相关的字段： + +```yaml +spec: + template: + spec: + domain: + devices: + clientPassthrough: {} +``` + +对已有 VM，可 patch 模板： + +```bash +kubectl patch vm -n --type=merge \ + -p '{"spec":{"template":{"spec":{"domain":{"devices":{"clientPassthrough":{}}}}}}}' +``` + +重启 VM，使运行态 VMI 获取新的设备配置： + +```bash +virtctl restart -n +``` + +确认运行态 VMI 包含该字段： + +```bash +kubectl get vmi -n \ + -o jsonpath='{.spec.domain.devices.clientPassthrough}{"\n"}' +``` + +预期输出： + +```text +{} +``` + +设置 `clientPassthrough` 后，KubeVirt 会为 VMI 创建 4 个 QEMU usbredir 槽位。可通过 live domain 检查： + +```bash +POD=$(kubectl get pod -n -l kubevirt.io/domain= \ + -o jsonpath='{.items[0].metadata.name}') + +kubectl exec -n "$POD" -c compute -- \ + virsh dumpxml 1 | grep -A4 ' usbredir 0951:1666 +``` + +也可以按 bus 和 device 地址重定向： + +```bash +sudo virtctl -n usbredir 02-03 +``` + +只要设备需要保持接入，就需要保持 `virtctl usbredir` 进程运行。按 `Ctrl+C` 可断开连接。如果客户端进程退出、客户端网络中断、USB 设备被拔出、VM 重启，或 `virt-launcher` Pod 被重建，guest 内的重定向设备都会消失，需要重新执行 `virtctl usbredir`。 + +## 在 guest 内验证 + +登录 guest OS，确认 USB 设备可见： + +```bash +lsusb +``` + +如果只是临时验证 USB 存储设备，还可以检查块设备： + +```bash +lsblk -f +blkid +``` + +不要在客户端和 guest 中同时挂载同一个存储设备。如果必须验证 U 盘或读卡器，请先在客户端卸载该设备，并优先在 guest 内只读挂载： + +```bash +sudo mkdir -p /mnt/usb +sudo mount -o ro /dev/sdX1 /mnt/usb +ls -la /mnt/usb +sudo umount /mnt/usb +``` + +将 `/dev/sdX1` 替换成 guest 内实际看到的分区名。 + +## 限制与风险 + +USB Redirection 依赖 `usbredirect`、客户端 OS 的 USB 栈、KubeVirt subresource streaming、QEMU usbredir 支持以及 guest OS 驱动。设备可能可以成功枚举，但由于驱动、协议、延迟或时序要求，在业务应用层仍不可用。 + +重要限制包括： + +- 单个 VMI 最多有 4 个 USB Redirection 槽位。 +- 重定向会话绑定客户端进程以及当前 VMI/`virt-launcher` 生命周期。 +- 客户端网络中断会导致设备从 guest 中消失。 +- API server、负载均衡或代理的 idle timeout 可能影响长连接。 +- 高吞吐存储、实时控制、音视频设备不适合该机制。 +- 存储类 USB 设备如果被客户端和 guest 同时挂载，可能造成数据损坏。 + +对于敏感设备型号，尤其是 UKey、智能卡读卡器、串口适配器以及有严格时序要求的设备，需要按型号验证。 + +## 故障排查 + +### `Not configured with USB Redirection` + +运行态 VMI 未包含 `clientPassthrough`。 + +检查 live VMI： + +```bash +kubectl get vmi -n \ + -o jsonpath='{.spec.domain.devices.clientPassthrough}{"\n"}' +``` + +如果输出为空，patch VM 模板并重启 VM。 + +### `VMI not running` + +目标 VMI 不处于 `Running` 状态。 + +```bash +virtctl start -n +kubectl wait vmi/ -n --for=condition=Ready --timeout=180s +``` + +### `Error on finding usbredirect in $PATH` + +客户端找不到 `usbredirect` 二进制文件。 + +```bash +sudo apt-get install -y usbredirect +which usbredirect +``` + +### `Could not init libusb` 或权限不足 + +本地用户无权接管 USB 设备。 + +```bash +sudo virtctl -n usbredir : +``` + +对于长期使用的 Linux 客户端，可为该设备配置经过现场审批的 udev 规则。 + +### 没有可用 USB Redirection 槽位 + +VMI 可能已经存在 4 个活跃 USB Redirection 会话，或客户端上仍有旧进程占用会话。 + +在客户端机器上执行： + +```bash +pgrep -af 'virtctl.*usbredir|usbredirect' +``` + +只停止确认已经失效的旧进程，然后重试。 + +### 断开后客户端存储设备未恢复为 `/dev/sdX` + +部分 USB 存储设备在 redirection 会话结束后，可能没有重新绑定到客户端存储驱动。优先尝试物理拔插。如果无法物理拔插，检查本地 USB 树： + +```bash +lsusb -t +``` + +如果对应接口显示 `Driver=[none]`，可使用客户端上显示的接口 ID 重新绑定到 `usb-storage`： + +```bash +sudo modprobe usb-storage +echo -n '' | sudo tee /sys/bus/usb/drivers/usb-storage/bind +``` + +接口 ID 类似 `3-4:1.0`，但必须以实际客户端主机上看到的值为准。 + +## 相关信息 + +- KubeVirt Client Passthrough: https://github.com/kubevirt/user-guide/blob/main/docs/compute/client_passthrough.md +- KubeVirt v1.7.0 USB Redirection 校验: https://github.com/kubevirt/kubevirt/blob/v1.7.0/pkg/virt-api/rest/usbredir.go +- KubeVirt v1.7.0 `clientPassthrough` API 字段: https://github.com/kubevirt/kubevirt/blob/v1.7.0/staging/src/kubevirt.io/api/core/v1/schema.go +- usbredir 项目: https://gitlab.freedesktop.org/spice/usbredir/