Publish Date: September 25, 2026
Executive Overview
Enterprise data center modernization initiatives routinely confront a daunting operational hurdle: the physical migration of thousands of mission-critical workloads across disconnected network and virtualization topologies. Historically, executing large-scale data center consolidations or platform migrations into software-defined private clouds required costly maintenance windows, complex application re-architectures, and protracted coordination between application owners and network engineers. The requirement to re-IP virtual machines upon migration historically introduced severe operational friction, breaking hardcoded database connection strings, disrupting Active Directory service principals, and jeopardizing customer-facing service-level agreements (SLAs).
Broadcom’s technical architectural publication, authored by Cassio Homem, Technical Lead at Broadcom, delivers an authoritative operational blueprint addressing this modernization bottleneck: Part 3 of the comprehensive series on VMware Cloud Foundation (VCF) Operations HCX 9.1, titled “Extending Networks and Migrating Workloads.” Building upon prior guidance detailing appliance installation, Network and Compute Profiles, Site Pairing, and Service Mesh instantiation, this installment evaluates the core runtime mechanics of workload evacuation into VMware Cloud Foundation 9.1. It details the procedural implementation of Layer 2 Network Extensions, evaluates the transport underlay verification pipelines, and analyzes the taxonomy of HCX migration types—specifically contrasting Replication-Assisted vMotion (RAV), Bulk Migration, and traditional live vMotion across throughput, downtime, and rollback dimensions.
This technical advisory provides an enterprise architecture evaluation of the workload mobility mechanisms, underlay transport prerequisites, and migration scheduling models detailed in the publication. It examines the operational protocols of HCX Diagnostics and Transport Analytics, analyzes Layer 2 broadcast domain stretching across NSX Virtual Private Clouds (VPCs) and overlays, evaluates large-scale batch migration via Mobility Groups, and weighs architectural alternatives to HCX. Furthermore, this assessment delivers a critical perspective on “trombone routing” packet latency, Seed Checkpoint storage overhead, virtual hardware locking traps, and WAN link stability, concluding with strategic recommendations for enterprise Chief Technology Officers (CTOs), infrastructure platform architects, and migration reliability engineers.
Features
Eliminating migration-induced application downtime while safeguarding data integrity requires coordinating low-level network encapsulation, pre-flight WAN telemetry, and asynchronous storage replication engines. The publication outlines key structural capabilities, diagnostic procedures, and operational workflows embedded within VCF Operations HCX 9.1:
Pre-Migration Underlay Validation and Health Telemetry: Before initiating virtual machine data transfers, HCX 9.1 enforces rigorous pre-flight validation to prevent failed migrations and aborted cutovers:
HCX Data Plane Diagnostics: Executed directly from the source HCX Manager across the established Service Mesh. The diagnostic suite tests all foundational network ports and protocols (including TCP 902 NFC, TCP 8000 vMotion, and TCP 31031/32032 vSphere Replication), verifying that perimeter and intermediate firewalls permit inter-site control and data plane communications without packet drops.
Transport Analytics Engine: Interrogates the HCX WAN Uplink, dynamically measuring available bandwidth, round-trip latency, and packet loss against the minimum requirements of each specific migration mechanism. The engine renders an explicit services matrix displaying green validation status banners only when underlay metrics satisfy the operational criteria for cold, vMotion, Bulk, or RAV pathways.
Layer 2 Network Extension (Broadcast Domain Stretching): HCX Network Extension allows virtual machines migrated into VCF 9.1 to retain their original IP addresses, MAC addresses, and default gateway configurations, eliminating the requirement for post-migration re-IP schemes:
Extensible Source and Target Matrix: The framework bridges heterogeneous network topologies. Supported source networks include legacy vSphere Distributed Switch (DVS) port groups, standard NSX overlays, NSX VLANs, and NSX VPCs (on NSX 9 sources). Supported destination networks within VCF 9.1 encompass modern NSX VPCs, NSX Overlays, and NSX VLAN-backed segments.
First-Hop Gateway Mapping: During network extension, administrators map the source subnet’s default gateway IP and prefix length to a designated First-Hop Router within VCF 9.1 (such as an NSX Tier-1 Gateway or an NSX VPC Router), preparing the target fabric for eventual localized default gateway cutover.
Granular Taxonomy of HCX Workload Migration Types: VCF Operations HCX 9.1 delivers a multi-tiered migration taxonomy tailored to diverse application SLA requirements, network capacities, and rollback strategies:
Replication-Assisted vMotion (RAV – Zero Downtime, Parallel Scale): The recommended standard for enterprise-scale live migrations. RAV combines the zero-downtime cutover mechanics of vMotion with the parallel, asynchronous disk seeding of vSphere Replication. It supports up to 1,000 concurrent migrations per HCX Manager (via KB 373010 tuning) and up to 200 concurrent disk syncs per Interconnect (IX) appliance. Requires a stable network with $\ge 150\text{ Mbps}$ bandwidth (when WAN Optimization is asserted) or $250\text{ Mbps}$ (without WAN Optimization), $\le 0.1%$ packet loss, and $\le 150\text{ ms}$ latency. Cutover switchover occurs serially per Service Mesh to prevent network saturation.
Bulk Migration (Scheduled Low Downtime, High Parallelism, Preserved Source): Functions as an asynchronous replication process rather than a physical relocation. Base disks are synchronized in the background over TCP 31031 (or TCP 32032 for encrypted VMs) while the source VM remains fully operational. Cutover occurs during a defined maintenance window where the source VM is gracefully powered down, final delta changes synchronize, and the destination VM powers on. Bulk Migration tolerates degraded WAN links (requiring only $50\text{ Mbps}$ bandwidth and withstanding up to $1.0%$ packet loss). Crucially, the original source VM is retained in a powered-off, disconnected state, providing an instantaneous fallback and rollback plan.
Standard vMotion (Zero Downtime, Serial Execution): Moves virtual machines across sites with zero downtime over TCP 8000. However, VMs queued in a Mobility Group are processed strictly serially (one at a time per Service Mesh), and concurrent disk replication is unsupported, making it unviable for large multi-terabyte clusters.
Cold Migration: Designed strictly for powered-off workloads, transferring virtual disks over the VMware NFC protocol (TCP 902).
Resilient Data Staging and Seed Checkpoint Recovery: To insulate migrations against transient network failures, HCX 9.1 incorporates Seed Checkpoint protection. If a Bulk or RAV migration experiences a dropped WAN connection or an administrative cancellation, the target storage disks are not wiped. Subsequent migration attempts detect the existing target data blocks, reusing transferred data and resuming delta synchronization without restarting replication from zero.
Orchestrated Batch Migration via Mobility Groups: VCF Operations HCX unifies fleet-level migration governance through Mobility Groups:
Granular Cutover Scheduling: Administrators group application tiers (e.g., web, application, and database nodes) into unified Mobility Groups, scheduling switchovers to execute at precise dates and times to align with corporate change windows.
Lifecycle and Security Tag Preservation: Advanced switches allow administrators to carry over NSX security tags, retain original virtual MAC addresses, and selectively orchestrate post-migration VMware Tools upgrades.
Hardware Version and EVC Governance: The interface surfaces explicit controls for virtual hardware upgrades and Per-VM Enhanced vMotion Compatibility (EVC), enabling platform teams to prevent irreversible VM format upgrades that would prohibit backward migration to legacy clusters.
Benefits
Transitioning to VCF Operations HCX 9.1 for cross-cloud workload mobility delivers substantial operational, strategic, and financial advantages across enterprise IT organizations:
Elimination of Application Re-IP and Re-Architecture Costs: Re-IPing an enterprise application catalog is historically one of the most expensive and error-prone operational undertakings in data center history. By stretching Layer 2 broadcast domains across the HCX Service Mesh, workloads move into VCF 9.1 while retaining identical IP addresses, subnets, and MAC addresses. Enterprise applications, DNS pointers, external firewall rules, and API endpoints continue functioning seamlessly without requiring code changes or application downtime.
Compression of Enterprise Data Center Evacuation Timelines: Traditional physical-to-virtual (P2V) conversions and cold-migration windows limit migration velocity to dozens of VMs per weekend. Utilizing Replication-Assisted vMotion (RAV) and Bulk Migration within HCX 9.1 allows platform engineering teams to replicate hundreds of virtual machines in parallel during normal business hours. Organizations can execute full data center evacuations in weeks rather than months, accelerating the decommissioning of legacy physical facilities and eliminating costly colocation lease penalties.
Deterministic, Predictable Migration Success via Transport Telemetry: Attempting to execute live vMotion across unvalidated WAN links frequently leads to migration timeouts, VM freezes, and corrupted execution states when underlay latency spikes. Incorporating native Transport Analytics and HCX Diagnostics provides platform teams with mathematical proof that underlay bandwidth, packet loss, and firewall ports conform to vendor requirements before a migration group is launched, driving migration success rates near 100%.
Risk-Free Rollback Pathways for Mission-Critical Applications: Migrating mission-critical tier-1 databases into next-generation infrastructure creates executive anxiety regarding unexpected performance regressions. By utilizing Bulk Migration, the original source virtual machine remains physically preserved in an un-deleted, disconnected state on the legacy cluster. If an application encounters an unexpected incompatibility within the target VCF 9.1 environment, operations teams can immediately power the legacy VM back on, establishing a low-risk rollback strategy without requiring long recovery cycles.
Reclamation of Platform Engineering Bandwidth: Coordinating mass infrastructure migrations manually consumes thousands of hours of systems administration time. Automating workload discovery, replication scheduling, network stretching, and cutover execution through HCX Mobility Groups transforms manual migration marathons into automated, policy-driven background workflows, allowing platform teams to focus on platform engineering and private cloud governance.
Seamless Multi-Generational Platform Bridging: VCF Operations HCX functions as a universal architectural bridge, abstracting underlying hypervisor and hardware generations. Organizations can migrate workloads seamlessly from legacy vSphere 6.5, 6.7, 7.x, or 8.x environments directly into modernized VMware Cloud Foundation 9.1 clusters without intermediate hypervisor stepping-stone upgrades, eliminating technical debt in a single operational step.
Use cases
The workload mobility and network extension capabilities of HCX 9.1 apply across diverse enterprise operating environments and transformation initiatives:
Global Financial Institution Data Center Consolidation: A multinational commercial bank operates 4,000 virtual machines across two aging colocation facilities scheduled for lease termination. The application portfolio contains complex core banking databases and transaction settlement services that cannot tolerate extended downtime or IP address modifications. The infrastructure team pairs the colocation sites with a new, sovereign VCF 9.1 private cloud using VCF Operations HCX. They stretch 120 production VLANs onto NSX Tier-1 routers and execute parallel Replication-Assisted vMotion (RAV) across structured Mobility Groups. The entire 4,000-VM estate is evacuated over four consecutive weekends with zero downtime, eliminating $1.2M in upcoming data center lease renewals while preserving PCI DSS compliance.
Healthcare System Migrating Electronic Health Records (EHR): A regional healthcare provider manages Electronic Health Record (EHR) databases, medical imaging PACS systems, and clinical telemetry applications across a legacy vSphere 7.0 environment. To satisfy strict patient safety mandates, clinical databases must maintain a verified, instantaneous rollback capability during migration to a modernized VCF 9.1 cluster. Platform engineers leverage HCX Bulk Migration: base disks are seeded in the background over several days without impacting live hospital operations. During an approved 15-minute maintenance window, the source EHR database is gracefully shut down and brought online in VCF 9.1. Because the source VM is retained in a powered-off state, clinical leadership achieves the peace of mind required to execute platform modernization.
Enterprise SaaS Platform Transitioning to NSX Virtual Private Clouds (VPCs): A digital software provider operates continuous deployment pipelines across disparate private cloud clusters. The company standardizes on VCF 9.1 to take advantage of multi-tenant NSX VPCs for automated developer isolation. Using HCX Network Extension, the platform team stretches legacy Distributed Virtual Switch (DVS) port groups directly into modern target NSX VPC subnets. Using Mobility Groups, engineering teams migrate microservice containers and backing database virtual machines into their dedicated VPC envelopes, preserving network connectivity throughout the phased modernization.
Air-Gapped Sovereign Cloud Evacuation and Disaster Staging: A national defense agency operates an isolated, air-gapped data center supporting sovereign engineering simulations. Strict security mandates prohibit external SaaS tooling and require mathematical verification that inter-site communications remain contained within private, encrypted fiber links. The agency deploys VCF Operations HCX 9.1 entirely on-premises, using the WAN Optimization appliance to maximize throughput across a bandwidth-constrained 150 Mbps dedicated link. Transport Analytics verifies that the underlay meets the $\le 0.1%$ packet loss threshold, allowing the agency to replicate classified workloads into a hardened VCF 9.1 enclave without violating national security protocols.
Alternatives
When planning cross-data-center workload mobility, application evacuation, and infrastructure modernization, technology leaders must evaluate several alternative methodologies:
Alternative 1: Platform-Native Workload Mobility via VCF Operations HCX 9.1 (Recommended Standard): The primary methodology detailed in the publication leverages HCX 9.1 to stretch Layer 2 networks, validate underlay transport paths, and execute parallel zero-downtime migrations via RAV and Bulk Migration. This model maximizes migration speed, eliminates application re-IP requirements, provides deterministic rollback options, and automates fleet batching through Mobility Groups. However, it requires deploying and managing the HCX appliance ecosystem (HCX Manager, Interconnect, Network Extension, and WAN Optimization appliances) and demands network bandwidth planning.
Alternative 2: Native Cross-vCenter vMotion (Without HCX): Organizations can utilize native vSphere Cross-vCenter vMotion (XVM) via the vSphere Client or PowerCLI. While this approach avoids deploying dedicated HCX virtual appliances and requires no additional software licensing, it is operationally limited. Native XVM does not provide Layer 2 network extension (mandating either application re-IPing or manual, physical network stretching via dark fiber/EVPN), lacks parallel replication seeding (processing VMs serially), cannot tolerate packet loss above 0.1%, provides zero Seed Checkpoint recovery if a transfer drops, and offers no integrated rollback preservation.
Alternative 3: Storage-Level Array Replication and LUN Remounting: Under this legacy infrastructure model, an enterprise replicates entire storage LUNs from source storage arrays (e.g., Pure Storage, NetApp, Dell EMC) to target arrays across dedicated Fibre Channel or iSCSI links. During the cutover window, storage arrays are unmounted from the legacy vCenter, presented to the new VCF cluster, and virtual machines are manually re-registered into inventory. While this approach moves raw storage blocks efficiently, it represents a blunt, high-downtime instrument. It requires all VMs on a given datastore to migrate simultaneously, cannot execute live zero-downtime migrations, introduces severe risk of VM identifier collisions, and fails to stretch networks without physical networking re-cabling.
Alternative 4: Backup and Disaster Recovery Restore (e.g., Veeam, Commvault): Enterprises can execute migrations by taking a final image-level backup of virtual machines using third-party backup software and restoring the images into the target VCF 9.1 environment. While this approach utilizes existing backup infrastructure and requires no specialized migration appliances, it imposes significant application downtime. Restoring hundreds of multi-terabyte virtual machines sequentially from backup storage takes days, prolonging maintenance windows and leaving applications offline for extended periods.
Alternative perspective
A critical evaluation of large-scale workload mobility and Layer 2 network extension requires examining the systemic technical debt, network traps, and operational liabilities that can arise during migration projects:
The Trap of Perpetual Layer 2 Network Extension and “Trombone Routing”: While stretching Layer 2 networks via HCX Network Extension provides a clean, zero-downtime migration mechanism, it is designed strictly as a temporary transition tool. If an enterprise leaves extended networks active long after virtual machines have migrated, the network architecture suffers from severe traffic hairpinning (“tromboning”). Inbound packets targeting a migrated virtual machine inside VCF 9.1 must still enter the legacy data center gateway, traverse the HCX Layer 2 tunnel across the WAN, reach the VM, and hairpin back across the tunnel to egress the legacy firewall. This dynamic introduces ongoing latency penalties, consumes WAN bandwidth, and creates a fragile physical dependency on the legacy facility that prevents its decommissioning. Platform architects must enforce strict post-migration gateway cutover runbooks to un-stretch networks promptly.
The Risk of Irreversible Virtual Hardware Locking: The publication issues an explicit warning regarding HCX advanced migration settings, specifically
Upgrade Virtual HardwareandDisable HCX Per-VM EVC. While upgrading virtual hardware to version 21 or aligning with modern CPU instruction sets is technically beneficial for VCF 9.1 performance, applying these upgrades during the migration permanently locks the virtual machine to the target platform. If an application encounters an unexpected software bug post-migration, the virtual machine cannot be migrated back to the legacy vCenter running older ESXi hosts. Platform engineering teams must resist the urge to combine hardware upgrades with migration cutovers; hardware modernizations should be deferred to a separate, post-migration change window once operational stability in VCF 9.1 is proven.The Storage Overhead and Disk Lock Considerations of Seed Checkpoints: While the Seed Checkpoint capability is an invaluable resiliency feature that prevents restarting multi-terabyte disk transfers after transient WAN drops, it introduces storage capacity overhead. If a series of test migrations are aborted or abandoned by application teams without executing an explicit clean-up or reverse migration, target vSAN datastores can accumulate orphaned, partially replicated virtual disk files. In capacity-constrained VCF workload domains, unmanaged seed checkpoints can quietly consume terabytes of expensive flash storage, necessitating disciplined storage auditing within VCF Operations.
The Reality of High-Churn Databases and vMotion Stun Cycles: While Replication-Assisted vMotion (RAV) promises zero-downtime migration for tier-1 workloads, migrating exceptionally high-churn memory workloads (such as write-intensive transactional databases or massive in-memory caching layers) introduces real-world physical limits. If a database’s memory page write rate exceeds the available bandwidth of the HCX WAN tunnel, the vMotion engine will struggle to converge during the final synchronization phase. The hypervisor may be forced to prolong the VM “stun” cycle to transfer final memory pages, inducing temporary microsecond-to-millisecond application latency stalls that can trigger database failover events. High-churn workloads must be migrated during scheduled periods of low transactional activity, accompanied by temporary write-throttling where necessary.
Final thoughts
Cassio Homem’s technical publication provides an indispensable operational roadmap for enterprise platform architects navigating the transition to VMware Cloud Foundation 9.1. In an IT landscape where data center lease deadlines, hardware refresh cycles, and infrastructure consolidation mandates drive urgent modernization initiatives, workload mobility can no longer be approached through manual, high-downtime migration methods.
VMware Cloud Foundation Operations HCX 9.1 establishes an enterprise-grade migration fabric that decouples application availability from physical data center location. By delivering robust underlay diagnostics, resilient Layer 2 network extensions, parallel disk replication through RAV, and deterministic rollback safety via Bulk Migration, HCX empowers infrastructure teams to execute large-scale platform migrations with speed, confidence, and minimal business disruption.
To successfully operationalize HCX 9.1 and execute seamless workload migrations into VMware Cloud Foundation 9.1, enterprise CTOs, infrastructure platform architects, and migration engineering teams should implement the following strategic recommendations:
Institutionalize Mandatory Pre-Migration Underlay Validation: Mandate that migration runbooks include the execution and sign-off of HCX Diagnostics and Transport Analytics prior to scheduling any Mobility Group. Verify that available WAN bandwidth, latency, and packet loss satisfy the exact minimum requirements for the selected migration type before committing to change windows.
Establish Phased Layer 2 Network Extension Decommissioning Deadlines: Treat HCX Network Extension strictly as an ephemeral bridge rather than a permanent network architecture. For every stretched subnet, establish a formal gateway migration schedule to cut over default gateway routing to native NSX Tier-1 or NSX VPC gateways, systematically eliminating trombone routing and enabling the physical decommissioning of legacy data center networks.
Standardize Migration Types Based on Application SLA Profiles: Align migration methodology with business criticality: enforce Replication-Assisted vMotion (RAV) for tier-1 workloads requiring absolute zero downtime, and standardize on Bulk Migration for general-purpose workloads to capitalize on parallel throughput, seed checkpoint reuse, and preserved source rollback protection.
Enforce Strict Environmental Decoupling During Cutover: Prohibit the activation of
Upgrade Virtual HardwareandDisable HCX Per-VM EVCduring initial migration execution. Maintain source-compatibility baselines throughout the cutover window to preserve immediate backward mobility, scheduling virtual hardware and VMware Tools upgrades only after the application passes post-migration performance validation.Automate Large-Scale Batch Migration via Mobility Groups: Group multi-tier application dependencies (web, app, database) into cohesive, synchronized Mobility Groups. Coordinate switchover schedules with business stakeholders, automating the final cutover sequence to execute during off-peak hours while monitoring real-time replication health via VCF Operations dashboards.