Windows Server 2012 Storage Live Migration

Windows Server 2012 brought new levels of mobility to the virtual environment. This mobility extends beyond the previous live migration capability, which was limited to migration within a cluster with shared storage. Windows Server 2012 introduces migration of virtual machines (VMs) between any Windows Server 2012 Hyper-V hosts, standalone or clustered. The cluster is no longer a mobility boundary, so enterprises have complete flexibility. Often the requirement is to move not the VM but rather its storage, something that was possible prior to Windows Server 2012 only after shutting down the VM.

Windows Server 2012 supports three main types of storage for VMs: DAS; SAN-based (typically connected via Fibre Channel or iSCSI); and—new to Windows Server 2012—support for Server Message Block (SMB) 3.0 file shares, such as those hosted on a Windows Server 2012 file server or any NAS/SAN that has SMB 3.0 support. Windows Server 2012 storage live migration allows the storage used by a VM, including the VM's configuration and virtual hard disks (VHDs), to be moved between any supported storage, with zero downtime to the VM. Migration to a different folder on the same disk, between LUNs on the same SAN, from DAS to SAN, from SAN to an SMB file share—if the storage is supported by Hyper-V, then VMs can be moved with no downtime. Note that storage live migration can't move non-virtualized storage, so if a VM is using pass-through storage, then it can't be moved. The good news is that with the new VHDX format (which allows 64TB VHDs), there's no reason to use pass-through storage, from either a size or performance perspective.

The ability to move the storage of a VM at any time, without affecting the availability of the VM, is vital in two key scenarios:

How Storage Live Migration Works

The mechanics behind Windows Server 2012 storage live migration are quite simple but provide the most optimal migration process. Remember that the VM isn't moving between hosts (although you can use shared-nothing live migration to accomplish that); only the storage moves from a source location to a target location.

Storage live migration uses a one-pass copy of VHDs. The pass works as follows:

The actual storage live migration process is managed by the Virtual Machine Management Service (VMMS) in the parent partition. However, the heavy lifting of storage live migration is performed by the VMs' worker process and the storage virtualization service provider in the parent partition. The mechanism for the storage copy is just an unbuffered copy operation plus the additional I/O on the target for the mirroring of writes during the copy. In reality, the additional I/O for the ongoing writes is negligible compared with the main unbuffered file copy. The path used is whichever path exists to the target: iSCSI or Fibre Channel for a SAN target, whichever network adapter or adapters have a path to the share for SMB. Any underlying storage technologies that optimize performance are fully utilized. If you're copying to or from SMB and using NIC Teaming, SMB Direct, or SMB Multichannel, then those technologies will be used. If you're using a SAN that supports offloaded data transfer (ODX) and you're moving a VM within a LUN or between LUNs, then ODX will be used, meaning that the move will use almost no load on the host and will complete very quickly.

The SAN ODX scenario is the best case. For all other situations, it is important to realize exactly what an unbuffered copy means to your system. The unbuffered copy is used because during storage live migration, you don't want to use a large amount of system memory for caching of data on a virtualization host.

Performing a copy can cause a significant amount of I/O load on your system for both reading the source and writing to the target. To get an idea, try manually creating an unbuffered copy on your system by using the Xcopy command with the /J switch. This creates a similar load to what a storage live migration would inflict on your system, again considering that the ongoing mirrored writes are negligible. Therefore, consider moving a VM between folders on a local disk (likely to be a worst-case scenario). The data would be read from and written to the same disk, causing a huge amount of disk thrashing; it would likely take a long time and would adversely affect any other VMs that use that disk. If the source and target are different storage devices, then the additional load won't be as severe as a local move—but must still be considered.

Moving a VM causes nothing Hyper-V–specific about the disk I/O, which is the same as for any data-migration technology (although other technologies might not have capabilities such as ODX when a SAN is involved). Ultimately, the data must be read and written. This doesn't mean that you shouldn't use storage live migration, but it does mean that you should plan carefully when you use it.

You probably won't want to perform the migration during normal working hours because of the possible adverse effect to other loads. I suspect this is why no automated storage live migration process is part of the Dynamic Optimization in System Center Virtual Machine Manager (VMM) 2012, which rebalances VMs within a cluster. If you detect a large I/O load on a storage subsystem in the middle of a weekday, the last thing you want to do is add a huge extra load by trying to move things around. The best option is to track I/O over time, then move the VM's storage at a quiet time—a task that is easy to script with PowerShell or to automate with technologies such as Microsoft System Center Orchestrator 2012.

Configuring Storage Live Migration

If you have installed the Hyper-V role on your server, you are all done. No specific configuration is needed to use storage live migration; it just works. As previously stated, storage live migration uses whichever path exists to communicate with the source and target storage, and it is enabled by default (in fact, you can't disable it). The only configuration is that you can set how many simultaneous storage live migrations are allowed. To do so, use the Hyper-V Settings action. In the Storage Migrations area, set the desired Simultaneous storage migrations number, as Figure 1 shows.

Figure 1: Setting the Number of Simultaneous Storage Live Migrations

You can also configure this setting by using PowerShell:

Set-VMHost -MaximumStorageMigrations 

You only need extra configuration if you're using SMB storage for the migration target and are initiating the migration remotely, either through Hyper-V Manager or PowerShell. In other words, you aren't running the tools on the actual Hyper-V host. This type of remote management is the preferred approach for Windows Server 2012; all management should be performed remotely, using PowerShell or from a Windows 8 machine.

When you configure SMB storage for use with Hyper-V, you need to set several specific permissions, including giving administrators full control to create a VM on SMB or to move to SMB as part of a storage live migration, as their credential is used. As I explain in the article "Shared-Nothing VM Live Migration with Windows Server 2012 Hyper-V," remotely initiating a shared-nothing live migration requires the configuration of Kerberos constrained delegation on each Hyper-V server. The Microsoft Virtual System Migration Service requires this configuration because by default, a Windows server cannot pass a credential that is being used on the server to another server. Doing so would generally be bad from a security perspective but is exactly what we need here and is acceptable in this specific, scoped context:

To enable this scenario, you must enable Common Internet File System (CIFS) constrained delegation for each Hyper-V server to the various SMB file servers. This task is a simple one:

Figure 2: Enabling Kerberos Constrained Delegation to the File Servers for CIFS

You can now remotely trigger storage live migrations, even to SMB storage.

Performing Storage Live Migration

Now that the environment is ready for storage live migrations, all that is left is to perform them. Storage live migrations can be triggered through Hyper-V Manager or through PowerShell. You have two options when performing a storage live migration. You can move everything to one location, or you can choose different locations for each item that is stored as part of a VM (i.e., one location for the configuration file, one for the snapshots, one for smart paging, one for VHD1, one for VHD2, and so on), as Figure 3 shows.

Figure 3: Selecting Items to Move

This is not a problem when using graphical tools but adds an interesting aspect when using PowerShell.

Start by using Hyper-V Manager to perform the move. Doing so helps you understand the available options:

To perform the storage live migration from PowerShell, use the Move-VMStorage cmdlet. If you're moving everything to a single location, simply pass the VM name and the new target location with the DestinationStoragePath parameter. (Note that a subfolder with the VM name is not created automatically. If you want the VM in its own subfolder, you need to specify that as part of the target path.) Here's an example:

Move-VMStorage -DestinationStoragePath  -VMName 

If you want to move separate data to different locations, the process is more complicated. Instead of using DestinationStoragePath, use the SmartPagingFilePath, SnapshotFilePath, and VirtualMachinePath parameters to pass the location for the smart paging file, snapshots, and VM configuration, respectively. For the VHDs, use the Vhds parameter. However, you can have more than one VHD per VM—in fact, you can have hundreds of them—and PowerShell doesn't like an arbitrary number of parameters. Therefore, to pass the VHDs' new location, you need to create a hash value for the SourceFilePath and DestinationFilePath for each VHD, and then place them into an array, which is passed to the Vhds parameter. Pleasant!

The following example moves a VM with three VHDs, a smart paging file, configuration, and snapshots. Note you don't need to move all elements of a VM; you only need to specify the pieces that you want to move. Other unspecified elements stay in their current location. Note that in the example, the hash values (value pairs) use curly brackets { } whereas the array uses parentheses ( ).

Move-VMStorage -VMName  -SmartPagingFilePath d -SnapshotFilePath  -VirtualMachinePath  -Vhds @(@{ "SourceFilePath " = "C:vmvhd1.vhdx "; "DestinationFilePath " = "D:VHDsvhd1.vhdx "}, @{ "SourceFilePath " = "C:vmvhd2.vhdx "; "DestinationFilePath " = "E:VHDsvhd2..vhdx "}, @{ "SourceFilePath " = "C:vmvhd3.vhdx "; "DestinationFilePath " = "F:VHDsvhd3.vhdx "})

When the storage live migration is initiated, it runs until it's finished, no matter how long that might take. As the administrator, you can cancel the storage live migration manually by using the Cancel move storage action. Rebooting the Hyper-V host also cancels all storage live migrations. You can see the progress of storage live migrations in the Hyper-V Manager tool or by querying them through Windows Management Instrumentation (WMI):

PS C: > Get-WmiObject -Namespace rootvirtualizationv2 -Class Msvm_MigrationJob | ft Name, JobStatus, PercentComplete, VirtualSystemNameName       JobStatus      PercentComplete VirtualSystemName        ----       ---------      --------------- -----------------        Moving Storage Job is running 14          6A7C0DEF-9805-4242-92F9-98E6F...

Migrate Responsibly

Storage live migration is a great new capability for Hyper-V, if you use it wisely. The feature gives organizations new flexibility in implementing new storage without affecting the availability of services. You can even use it to rebalance storage subsystems with uneven loading—but be sure to plan your migrations to minimize I/O impact.