How do I set up shared GPU resources in virtualized environments?

Setting up shared GPU resources in virtualized environments takes planning, compatible hardware, and the right software stack. Here is how to approach it.

1. Evaluate your requirements

• Workload type: AI/ML training, graphics rendering, video transcoding, or VDI each demand different GPU profiles.
• Performance targets: Define the compute, memory, and latency each VM needs.
• Density: How many VMs or users will share each physical GPU?

2. Choose suitable hardware

• GPU selection: Data center GPUs with hardware virtualization support — NVIDIA A100, H100, L40, or AMD Instinct MI200/MI300 series. Workstation cards like the RTX 6000/8000 Ada Generation work for smaller deployments but lack SR-IOV and ECC on some models.
• Server platform: Verify PCIe lane count (x16 per GPU), power budget (300–700 W per card), and cooling capacity.
• VRAM sizing: Total frame-buffer must cover the sum of all vGPU profiles plus hypervisor overhead.

3. Verify hypervisor and GPU compatibility

• VMware vSphere: vGPU (requires NVIDIA license) or DirectPath I/O (passthrough only).
• Microsoft Hyper-V: Discrete Device Assignment (DDA) for passthrough; no native vGPU equivalent.
• Citrix Hypervisor (formerly XenServer): vGPU support with NVIDIA GRID drivers.
• KVM/QEMU: PCI passthrough, mediated devices (mdev) for NVIDIA vGPU, or AMD SR-IOV virtual functions.

Install the GPU driver that matches your hypervisor version and guest OS. Mismatched versions are the most common cause of “GPU not found” errors.

4. Pick a virtualization technology

• NVIDIA vGPU (GRID): Time-sliced or MIG-backed profiles on supported GPUs. Requires per-GPU license.
• AMD MxGPU: SR-IOV-based hardware partitioning on Instinct MI-series. No per-GPU license fee.
• Mediated devices (mdev) on KVM: Lets you carve an NVIDIA GPU into vGPU-like profiles without GRID license on some consumer cards (GeForce/RTX), but with driver lockout caveats.
• Container-level sharing: NVIDIA Container Toolkit + Kubernetes device plugin for workload-level sharing on a single GPU.

PCI passthrough assigns the whole GPU to one VM — use it when a workload needs exclusive access, not for sharing.

5. Configure the hypervisor

VMware vSphere

1. Install the NVIDIA vGPU Manager VIB on each ESXi host.
2. In vCenter, create vGPU profiles (e.g., A100-8C, A100-4C) under the GPU device.
3. Edit each VM: Add PCI Device → select the vGPU profile → reserve all guest memory.

Microsoft Hyper-V

1. Enable DDA in the VM settings: Add Hardware → Physical GPU → select the GPU.
2. Disable dynamic memory for that VM.
3. Install the NVIDIA/AMD driver inside the guest OS.

KVM/QEMU (libvirt)

1. Enable IOMMU in kernel cmdline (intel_iommu=on or amd_iommu=on).
2. Bind the GPU to vfio-pci via modprobe.d.
3. For mediated devices: echo "nvidia-XX" > /sys/class/mdev_bus/.../mdev_supported_types/.../create then attach the mdev UUID in the VM XML (<hostdev mode='subsystem' type='mdev'>...</hostdev>).
4. For AMD SR-IOV: enable VFs in the driver (echo 4 > /sys/bus/pci/devices/.../sriov_numvfs) and pass each VF to a VM.

Citrix Hypervisor

1. Install NVIDIA GRID host driver via XenCenter or CLI.
2. Enable vGPU on the pGPU, choose a profile.
3. Assign the vGPU to VMs in their GPU properties.

6. Kubernetes GPU sharing (optional)

For containerized AI/ML workloads:

1. Install the NVIDIA device plugin from the official release:

   kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.0/nvidia-device-plugin.yml

2. Request GPUs in pod specs:

   resources:
     limits:
       nvidia.com/gpu: 1

3. For fractional sharing, use the nvidia.com/gpu resource with the nvidia.com/gpu.count extended resource (requires device plugin 0.14+ and --mig-strategy=single or time-slicing config).
4. Operators like Kubeflow or the NVIDIA GPU Operator automate driver, plugin, and MIG setup.

7. Monitor and optimize

• Host level: nvidia-smi, dcgmi, or rocm-smi for real-time stats.
• Cluster level: Prometheus + dcgm-exporter or k8s-device-plugin metrics → Grafana dashboards.
• Watch for memory fragmentation on time-sliced vGPUs; switch to MIG or larger profiles if VMs stall.
• Set QoS: vSphere shares/reservations/limits, Kubernetes resource quotas, or cgroups cpu.shares on the host.

8. High availability and backup

• vSphere HA restarts VMs on another host — ensure the target host has an identical GPU and vGPU Manager version.
• Kubernetes: use node affinity/taints for GPU nodes, pod disruption budgets, and the GPU Operator’s node feature discovery for automatic labeling.
• Back up VM XML/libvirt configs, vSphere host profiles, and Kubernetes manifests. GPU driver state is not portable across reboots.

9. Stay current

• Update GPU drivers, hypervisor patches, and container tooling on a tested cadence. NVIDIA vGPU releases often change supported profile matrices.
• Track licensing changes — NVIDIA vGPU moved to subscription; AMD MxGPU remains perpetual.

10. Licensing and cost

• NVIDIA vGPU: per-GPU, per-year subscription (CCU or vPC/vApps editions).
• AMD MxGPU: included with Instinct GPU purchase.
• Factor in server rack space, power, cooling, and hypervisor licensing (vSphere Enterprise Plus for vGPU).