Using Persistent Memory Devices for Virtual Machine Storage

Persistent memory devices such as VNDIMM and NVDIMM-N are like DIMM RAM, except that they provide persistent storage. These devices retain their contents, even when power is removed. As such, persistent memory devices are sometimes used as an alternative to SSDs when extremely low-latency storage access is required.

While persistent memory devices have obvious benefits for workloads running on physical servers, they have also long been of interest for use in virtualized environments. While it is possible for a virtual machine to take advantage of persistent memory devices, there are a few things that admins have to consider prior to attempting to leverage such devices.

1. Compatibility

The first thing that you have to think about if you are considering using a persistent memory device in a virtualized environment is compatibility. You need to make sure that both the hypervisor and the guest operating system can use the persistent memory device. Depending on the software you use, an upgrade may be required.

Consider what would be involved in providing persistent memory device access to a virtualized server running on Microsoft’s Hyper-V. Microsoft began supporting persistent memory devices in Windows Server 2016 (and Windows 10). However, Windows Server supports only specific types of persistent memory, including NVDIMM and JEDEC-compliant NVDIMM-N devices.

Another important thing to be aware of is that even though Windows Server 2016 supports these devices, Windows Server 2016 Hyper-V does not. If you want to expose persistent memory to a Hyper-V virtual machine, your host will need to be running the Windows Server 2019 version of Hyper-V.

It is also worth noting that persistent memory devices can be used only in conjunction with Generation 2 virtual machines. Generation 1 VMs are not supported for use with persistent memory devices.

2. Functional Limitations

Before you commit to using a persistent memory device in association with a virtual machine, it is extremely important to consider whether using such a device will adversely impact the virtual machine. In some cases, using a persistent memory device is absolutely the right choice. In other cases, though, loss of functionality at the virtual machine level might make using a persistent memory device unacceptable.

This of, course, raises the question of what types of functionality you might expect to lose. Again, this varies by hypervisor, but let’s use Microsoft Hyper-V as an example.

Microsoft Hyper-V treats persistent memory devices as being local to the host. This is similar to the way that Hyper-V treats pass-through storage as host-specific, although there are key differences between the two technologies. As is the case with pass-through drives, virtual machines that use persistent memory devices cannot be live migrated to another Hyper-V host. Additionally, Hyper-V is unable to perform storage migrations for virtual machines that use persistent memory devices. These limitations will likely go away at some point, but it may take a few years.

Using persistent memory devices also inhibits your ability to create production checkpoints of a virtual machine. This is not to say that you will be unable to create a production checkpoint, but rather that your production checkpoints will not include the persistent memory state.

3. Special Requirements

One more important consideration is whether your hypervisor requires anything special to make a persistent memory device available to a virtual machine.

For the most part, Microsoft Hyper-V can be regarded as storage-agnostic. Hyper-V virtual machines store their data within virtual hard disks, and, with a few exceptions, Hyper-V does not really care where those virtual hard disks reside. In my own organization, for instance, I have Hyper-V servers that store virtual machines on internal direct-attached storage (DAS), but I also have a couple of Hyper-V servers that store virtual hard disks on iSCSI-attached external storage arrays. Normally, you can store virtual hard disks wherever you want, so long as you adhere to Microsoft’s stated best practices.

The same cannot be said for persistent memory devices. You can’t just place a virtual hard disk onto a persistent memory device and expect Hyper-V to be able to use it. Virtual hard disks residing on persistent memory devices must a xz  dhere to a specific file format (VHDPEM instead of the usual VHD or VHDX).

In addition, you will have to equip the virtual machine with a virtual persistent memory controller, and then connect the persistent memory controller to the VHDPEM-based virtual hard disk. It's an easy, three-step process, but it is quite a bit different from the way you normally associate a virtual disk with a Hyper-V virtual machine. You can find the exact steps involved in the process here.