Publish Date: September 28, 2026
Executive Overview
Enterprise data centers are experiencing unprecedented capital expenditure reallocation driven by the demands of generative artificial intelligence and high-throughput model inference. Physical infrastructure budgets increasingly prioritize specialized hardware accelerators—specifically enterprise-grade graphics processing units (GPUs). However, recent empirical software engineering and cloud infrastructure studies reveal an acute economic paradox: enterprise data center GPUs frequently operate at or below 50% average Streaming Multiprocessor (SM) utilization. In rigid bare-metal architectures, dedicating discrete physical servers populated with multi-GPU arrays to single-tenant models results in stranded accelerator capacity, squandered thermal and power envelopes, and inflated total cost of ownership (TCO).
Broadcom’s technical performance engineering publication, co-authored by Lan Vu, Hari Sivaraman, and Uday Kurkure, directly addresses this capital inefficiency. Part 2 of the AI performance series, titled “Maximizing GPU Utilization for Best Total Cost of Ownership,” delivers an empirical architectural evaluation of accelerator allocation mechanisms within VMware Cloud Foundation (VCF) 9.1. By benchmarking Llama 3.1 8B (FP4) inference workloads across server configurations equipped with NVIDIA RTX 6000 Pro GPUs, the study demonstrates that rigid bare-metal allocations yield sub-optimal utilization (~34% to 38% SM occupancy). Through dynamic virtual machine rightsizing, Multi-Instance GPU (MIG) partitioning, time-sliced vGPU scheduling, and offline inference batching, VCF 9.1 drives GPU SM utilization up to 94% while expanding system token throughput by up to 10x over isolated baselines.
This technical advisory provides an enterprise architecture evaluation of the GPU allocation mechanisms, microarchitectural scheduling models, and platform consolidation strategies detailed in the publication. It analyzes the trade-offs between DirectPath I/O (passthrough) and NVIDIA vGPU (time-sliced vs. MIG), evaluates empirical throughput scaling across varying tenant densities, explores cross-industry enterprise deployment scenarios, and assesses alternative infrastructure hosting models. Furthermore, this evaluation delivers a critical perspective on accelerator virtualization penalties, NUMA interconnect dependencies, and multi-tenant security boundaries, concluding with strategic recommendations for enterprise Chief Technology Officers (CTOs) and infrastructure platform architects.
Features
Overcoming accelerator underutilization without sacrificing workload isolation or deterministic service-level agreements (SLAs) requires fine-grained control over GPU hardware allocation. The technical publication details core platform capabilities, architectural allocation models, and empirical benchmark topologies within VMware Cloud Foundation 9.1:
-
Dual Accelerator Allocation Mechanisms (DirectPath I/O vs. NVIDIA vGPU): VCF 9.1 provides two foundational methodologies for binding physical GPU silicon to virtual machine compute runtimes:
-
DirectPath I/O (Physical Passthrough): Bypasses hypervisor mediation to map an entire physical GPU directly into a single virtual machine’s PCI address space. DirectPath I/O supports multi-GPU aggregation per VM (essential for large-scale distributed training and massive LLMs requiring model parallelism) and provides near bare-metal execution speed. However, it imposes rigid 1:1 hardware dedication, preventing GPU sharing across multiple independent workloads.
-
NVIDIA Virtual GPU (vGPU): Employs hypervisor-mediated virtualization to allocate a full physical GPU to a VM or partition a single physical GPU across multiple independent VMs. Each virtual machine receives a dedicated, hardware-isolated slice of GPU frame buffer (HBM/VRAM), enabling dense multi-tenant hosting.
-
-
Taxonomy of vGPU Virtualization Modes (Time-Sliced vs. Multi-Instance GPU): The vGPU architecture splits into two distinct operational paradigms designed for different workload characteristics:
-
Time-Sliced vGPU: Multiple virtual machines share physical GPU compute cores (Streaming Multiprocessors) using a temporal hypervisor round-robin scheduler while maintaining dedicated VRAM allocations. This mode is optimal for workloads with variable utilization profiles, interactive developer environments, or day/night shifting tasks (e.g., daytime desktop virtualization paired with nighttime batch inference processing).
-
Multi-Instance GPU (MIG vGPU): Leverages physical silicon partitioning native to modern NVIDIA architectures (such as Ampere, Hopper, and Blackwell). MIG physically carves a single physical GPU into up to seven independent GPU instances, each possessing dedicated SM compute slices, memory controllers, and isolated high-bandwidth memory bandwidth. MIG provides strict hardware Quality of Service (QoS), eliminating noisy-neighbor compute contention across concurrent virtual machines.
-
-
High-Speed Hardware Interconnect and Fabric Integration: To prevent virtualization abstractions from throttling inter-accelerator communication, VCF 9.1 validates platform-native acceleration fabrics:
-
NVIDIA NVLink and NVLink Switch Support: Facilitates high-speed, direct GPU-to-GPU memory addressing across physical sockets, preserving high communication bandwidth for tensor-parallel model execution.
-
RDMA over Converged Ethernet (RoCE) and InfiniBand: Bypasses host CPU cycles to stream data directly into GPU memory across high-speed network fabrics.
-
GPUDirect Storage (GDS): Enables direct DMA transfers between NVMe-backed software-defined storage (vSAN ESA) and GPU memory, eliminating intermediate host system RAM copies during large model checkpoint loading.
-
-
Empirical Workload Consolidation and SM Utilization Benchmarks: The Broadcom performance engineering team conducted controlled scaling experiments evaluating Llama 3.1 8B (FP4) inference on physical hosts populated with four NVIDIA RTX 6000 Pro GPUs:
-
Baseline (Config 1 – Siloed Bare-Metal Emulation): Four independent ESXi hosts, each running a single VM assigned four GPUs via passthrough, processing 18 queries per second (QPS) per VM. The GPUs operated with severe underutilization, yielding ~34% to 38% GPU SM occupancy and an aggregate cluster throughput of 2,328 tokens/second.
-
1:1 Consolidation (Config 2 – Single Host, Dedicated 1-GPU VMs): Four 1-GPU VMs consolidated onto a single physical server chassis. System inference throughput quadrupled to 9,314 tokens/second, while individual GPU SM occupancy increased to 49%, freeing three physical host chassis for other business workloads.
-
MIG Partitioning (Config 3 – Single Host, 8x MIG VMs): Allocating a partial GPU profile (MIG 2-48c, representing half of a physical GPU’s compute and memory) across eight concurrent VMs on a single host doubled system throughput again to 18,621 tokens/second, pushing active SM utilization to 79%.
-
Offline Batching Optimization: Running an offline, non-latency-sensitive batch inference profile on the 8x MIG configuration saturated hardware pipelines, achieving 23,365 tokens/second (a 10x throughput uplift over Config 1) and driving GPU SM utilization to 94%.
-
Benefits
Transitioning private cloud AI infrastructure from dedicated bare-metal allocations to hypervisor-orchestrated GPU consolidation delivers measurable financial, operational, and architectural advantages:
-
Drastic Reduction in AI Infrastructure Total Cost of Ownership: Enterprise GPUs represent significant capital expenditures, with high-end server nodes costing upwards of hundreds of thousands of dollars. Dedicating entire multi-GPU physical nodes to inference workloads that utilize less than 40% of their SM compute cycles wastes capital and facility resources. By consolidating four independent GPU workloads onto a single host chassis, enterprises can reclaim up to 75% of physical server footprint requirements, substantially cutting CapEx, data center rack space, and ongoing maintenance fees.
-
Substantial Compression of Power and Cooling Overhead: Data center power density is rapidly becoming the primary constraint on enterprise infrastructure expansion. High-density GPU nodes draw between 5 kW and 10 kW per rack. Consolidating underutilized workloads onto optimized physical hosts reduces the total physical node count, directly lowering megawatt power consumption and facility heat rejection requirements while advancing corporate Environmental, Social, and Governance (ESG) mandates.
-
Guaranteed Multi-Tenant Isolation and Zero Noisy-Neighbor Friction: In traditional bare-metal multi-tenancy, sharing a physical GPU across multiple processes requires running shared Linux containers or OS-level software schedulers. A memory leak or kernel panic in one process can crash adjacent workloads. Utilizing NVIDIA MIG profiles within VCF 9.1 provides hardware-enforced memory and compute boundaries; virtual machines run with independent operating systems, dedicated memory controllers, and isolated fault domains, ensuring complete tenant isolation.
-
Operational Agility via Dynamic Accelerator Right-Sizing: Unlike physical bare-metal hardware where GPU bindings are fixed by PCIe cabling and chassis topology, VCF allows platform teams to dynamically reassign and re-profile accelerator capacity. A physical server can host eight compact MIG virtual machines during daytime transactional business hours to support low-latency microservices, and reconfigure to execute large distributed batch jobs overnight, maximizing asset utilization across a 24-hour cycle.
-
Unified Enterprise Lifecycle Management and Availability: Hosting GPU workloads within VMware Cloud Foundation brings accelerators under enterprise IT governance. AI virtual machines inherit native hypervisor management workflows, including automated health monitoring, vSphere Distributed Resource Scheduler (DRS), predictable vSphere Lifecycle Manager (vLCM) patching, and integration into enterprise backup and disaster recovery pipelines, eliminating operational silos between platform engineers and data science teams.
Use cases
The GPU consolidation and virtualization strategies detailed in Part 2 apply across diverse enterprise operating environments and industry verticals:
-
Enterprise Financial Services Real-Time Fraud Inference: A multinational retail banking corporation operates real-time transaction scoring microservices across multiple internal banking divisions. Each department historically maintained dedicated GPU-accelerated physical servers running compact fraud-detection models. Because transaction volumes fluctuate, average GPU utilization hovered at 30%. By transitioning to VCF 9.1 and applying NVIDIA MIG 2-48c profiles, platform architects consolidate multiple divisional inference engines onto a single dual-socket server equipped with four NVIDIA GPUs. Hardware SM utilization increases from 35% to 78%, transaction scoring sustains strict sub-10ms response latencies, and the bank decommissions six physical server chassis.
-
Healthcare System Digital Pathology and Diagnostic Imaging: A regional hospital network hosts clinical diagnostic imaging models that analyze high-resolution MRI scans and X-rays alongside conversational patient-intake chatbots. The diagnostic imaging models require rapid inference during clinical consultations, while conversational intake exhibits bursty, intermittent utilization. The infrastructure team deploys VCF 9.1, assigning dedicated MIG profiles to the imaging models to guarantee deterministic execution speed, while conversational agents share secondary GPUs via time-sliced vGPU. This hybrid configuration delivers rapid clinical diagnosis while optimizing hardware spend across medical facilities.
-
High-Throughput E-Commerce Visual Search and Batch Catalog Indexing: A global e-commerce retailer operates visual search models that assist online shoppers during daytime hours, alongside massive background catalog indexing pipelines that process millions of newly ingested product images daily. Platform engineers leverage VCF’s dynamic scheduling: during peak daytime shopping traffic, physical GPUs are partitioned across multiple time-sliced VMs to serve real-time user visual queries. During off-peak overnight hours, platform orchestration shifts compute capacity to offline batch workers, pushing GPU SM utilization to 94% and indexing entire merchant catalogs without procuring dedicated batch processing clusters.
-
Sovereign Cloud Internal Developer Platform for Polyglot AI Teams: An aerospace engineering organization provides internal AI development environments for hundreds of software engineers fine-tuning custom code-generation and engineering documentation LLMs. Rather than granting developers direct access to expensive bare-metal GPU nodes, platform engineering exposes standardized vGPU and MIG templates via VCF Automation. Software engineers self-service pre-configured, isolated virtual machines with exact GPU fractional allocations (e.g., 1/4 or 1/2 of a physical GPU). Resource quotas automatically reclaim idle instances, preventing single developer squads from monopolizing hardware accelerators.
Alternatives
When architecting private cloud infrastructure to support enterprise artificial intelligence workloads, technology leaders must evaluate multiple accelerator allocation and hosting methodologies:
-
Alternative 1: Virtualized GPU Consolidation via VCF 9.1 (Recommended Standard): The primary architectural model advocated in the publication pairs hypervisor virtualization with fine-grained GPU allocation (DirectPath I/O, MIG vGPU, and time-sliced vGPU). This model maximizes hardware utilization (driving SM occupancy up to 94%), preserves strict enterprise tenant isolation, enables dynamic workload scheduling, and unifies private cloud operations under standard vCenter and VCF Operations interfaces. However, it requires software licensing investments in the NVIDIA AI Enterprise and VMware Cloud Foundation stacks, and demands platform engineering expertise in GPU profile management.
-
Alternative 2: Physically Dedicated Bare-Metal AI Deployments: Under this legacy model, organizations deploy bare-metal server chassis (such as standalone NVIDIA DGX or OEM GPU servers) running vanilla Linux operating systems, assigning entire multi-GPU physical nodes to individual workloads. While this approach eliminates all hypervisor overhead and provides direct physical access to hardware registers, it guarantees severe economic inefficiency. Average GPU SM utilization remains constrained below 50%, vast amounts of server CPU and memory sit permanently idle, and data center real estate and electrical capacities are quickly exhausted.
-
Alternative 3: Bare-Metal Kubernetes with Container-Level Slicing (e.g., KubeShare, Volcano): Enterprises can deploy raw container orchestrators directly on bare metal, using open-source or software-based container GPU schedulers to share physical accelerators across container pods. While this avoids hypervisor licensing and provides a container-native API interface, it introduces operational and security risks. Software-level container slicing lacks the hardware-enforced memory isolation provided by physical MIG, leaving clusters vulnerable to memory corruption or noisy-neighbor contention. Furthermore, it discards foundational virtualization resilience features, such as automated host failover and dynamic resource rebalancing.
-
Alternative 4: Public Cloud Hyperscaler GPU-as-a-Service (e.g., AWS EC2 P-instances, Azure ND-series): Rather than procuring and virtualizing physical GPUs on premises, an organization can spin up GPU-accelerated compute instances in public clouds on an hourly consumption basis. While this approach eliminates physical data center facility constraints and hardware supply-chain delays, it incurs substantial operational cost premiums for sustained, steady-state enterprise inference workloads. In addition, constant data transmission to public clouds generates significant network egress fees and risks violating stringent data sovereignty and compliance mandates governing sensitive corporate data.
Alternative perspective
A critical analysis of GPU consolidation and accelerator virtualization across modern enterprise private clouds requires examining the operational trade-offs, architectural constraints, and hidden complexities inherent in shared hardware models:
-
The Reality of Fractional GPU Profile Rigidities: While NVIDIA Multi-Instance GPU (MIG) provides hardware-enforced isolation, it enforces rigid, static silicon partitioning schemes. A physical GPU cannot be carved into arbitrary compute and memory slices; administrators must conform strictly to supported factory profiles (e.g., 1g.10gb, 2g.20gb, 3g.40gb, or 4g.48gb). If an enterprise workload requires slightly more memory than a specific MIG profile provides, the administrator is forced to allocate the next larger tier, immediately stranding unallocated memory. Furthermore, dynamically changing MIG profiles on a running host requires terminating all active virtual machines bound to that physical GPU, introducing operational friction for continuous production environments.
-
Interconnect Bottlenecks and NUMA Topology Traps: Achieving the high throughput numbers documented in the study requires physical server hardware to align with high-bandwidth interconnects (such as PCIe Gen 5 and NVLink). In multi-socket enterprise servers, physical GPUs are wired directly to specific CPU Non-Uniform Memory Access (NUMA) root complexes. If a virtual machine is scheduled with virtual CPUs and memory residing on NUMA Node 0, while its assigned vGPU or passthrough adapter resides on NUMA Node 1, all DMA transfers must traverse inter-socket interconnects (AMD Infinity Fabric or Intel UPI). This cross-socket traversal induces microsecond memory latency penalties that can degrade token generation rates, requiring strict, disciplined vNUMA and PCIe topology mapping.
-
The Overhead of NVIDIA Software Licensing Stacks: Achieving fine-grained GPU sharing within VMware Cloud Foundation is not a pure hypervisor feature; it requires the deployment of NVIDIA Virtual GPU (vGPU) software and NVIDIA AI Enterprise (NVAIE) licensing subscriptions. These software tiers introduce recurring annual operational licensing expenditures, require dedicated enterprise license servers (such as NVIDIA DLS or CLS), and introduce operational failure modes if local license tokens expire or become unreachable, which can cause the hypervisor to throttle GPU clock speeds to baseline frequencies.
-
The Skillset Divide Between Virtualization Ops and Data Science Teams: Effectively operationalizing GPU consolidation requires bridging deep cultural and technical gaps. Virtualization administrators understand hypervisor resource pools, HA policies, and ESXi kernel parameters, but are rarely familiar with machine learning frameworks, CUDA compute libraries, TensorRT-LLM runtimes, or token generation batch dynamics. Conversely, data science teams understand FP4 quantization and prompt context lengths, but struggle with virtual distributed switching or host NUMA balancing. Without dedicated, cross-trained platform engineering squads, organizations risk misconfiguring GPU environments, resulting in either unharvested capacity or unstable production models.
Final thoughts
Part 2 of Broadcom’s performance engineering series delivers a vital, empirical corrective to the prevailing assumption that artificial intelligence workloads mandate dedicated, single-tenant physical hardware. With enterprise GPUs representing one of the most capital-intensive investments in modern data center history, allowing accelerators to operate at sub-50% utilization is an unsustainable economic liability.
By establishing that VMware Cloud Foundation 9.1 can virtualize and partition physical GPUs via DirectPath I/O and NVIDIA vGPU (MIG and time-sliced modes)—driving hardware SM utilization up to 94% and expanding system token throughput tenfold—the study proves that private clouds can deliver near bare-metal computational efficiency while preserving the multi-tenant isolation, agility, and governance required by modern enterprise IT.
To successfully operationalize these GPU optimization capabilities, enterprise CTOs, infrastructure architects, and platform engineering teams should implement the following strategic recommendations:
-
Audit and Baseline Active GPU Utilization Fleets: Immediately deploy infrastructure monitoring tools to measure real-world GPU Streaming Multiprocessor (SM) utilization, memory occupancy, and query latency across all production and staging AI servers. Identify underutilized bare-metal or single-tenant nodes operating below 50% capacity as primary candidates for consolidation.
-
Standardize on Multi-Instance GPU (MIG) for Latency-Sensitive Inference: For multi-tenant production inference environments hosting models that require less than half of a physical accelerator’s resources, enforce NVIDIA MIG profiles (such as MIG 2-48c). Hardware partitioning guarantees deterministic compute latency and eliminates noisy-neighbor resource contention across tenant virtual machines.
-
Deploy Time-Sliced vGPU for Asynchronous and Shifting Workload Enclaves: Leverage time-sliced vGPU profiles for developer sandboxes, batch data preparation, and environments with distinct temporal operational peaks (e.g., sharing GPU compute between daytime interactive analytics and overnight model training).
-
Consolidate and Batch Non-Latency-Sensitive Workloads: For offline data processing, bulk document embedding, and non-real-time agentic indexing tasks, configure inference batching pipelines. Squeezing maximum batch concurrency into consolidated MIG virtual machines pushes hardware pipelines to peak saturation, capturing up to 94% SM utilization.
-
Enforce Strict NUMA and PCIe Hardware-Aware Provisioning: When provisioning GPU-accelerated virtual machines in VMware Cloud Foundation, validate that virtual CPU reservations, system memory allocations, and physical GPU PCIe devices reside within the identical host NUMA node, eliminating inter-socket interconnect latency and maximizing end-to-end token generation throughput.