4 Common Misconceptions about NVMe over Fabric

There is little doubt that Non-Volatile Memory Express (NVMe) over fabric will eventually become the de facto standard for high performance storage in the enterprise data center. Even so, the technology is still relatively new, and, as with any other new technology, there are some common misconceptions about its use. Here are four of the biggest misconceptions about NVMe over Fabric (or NVMeOF, as it is also known).

1. NVMe over Fabric is a single, distinct technology.

When it comes to enterprise storage, technologies often come in multiple flavors. For example, storage arrays are sometimes connected to servers through a 10 gigabit Ethernet (10GbE) connection. The physical 10 gigabit connection can be made using twisted pair (10GBase-T) or by a more commonly used SFP+ connector.

Similarly, NVMe Over Fabric comes in multiple flavors. Specifically, there is NVMe over Fibre Channel, NVMe over Remote Direct Memory Access (RDMA) and NVMe over TCP.

2. NVMe over Fabric is largely hardware-agnostic.

There is a bit of truth in this one, since NVMe over Fabric can be used over a TCP connection. Even so, NVMe over Fabric isn’t as adaptable as older standards such as iSCSI.

The iSCSI protocol will work with almost any IP network, so long as TCP ports 860 and 3260 are open. Likewise, the iSCSI protocol doesn’t really care about the storage hardware that makes up the iSCSI target. However, NVMe over Fabric is different: It has been specifically designed for optimal storage performance. As such, the fabric has to be fast enough to meet the protocol’s demands. Similarly, NVMe over Fabric does not work with HDDs. The use of flash storage is an absolute must.

3. NVMe over Fabric is just a faster alternative to iSCSI.

One of the more common misconceptions regarding NVMe Over Fabric is that it really isn’t all that different from iSCSI. However, the differences between iSCSI and NVMe Over Fabric are considerable.

iSCSI (and quite a few other popular data transport protocols) is based on the SCSI command set, which has been around for nearly half a century. Conversely, the NVMe standard was built from the ground up and is far newer than the SCSI standard.

One of the biggest differences between the NVMe standard and the SCSI standard is the way that its queues work. The SCSI specification allows for a single command queue. This queue can hold up to 32 commands. In contrast, the NVMe standard allows for 65,000 separate queues to be created, and each one of those queues can hold 65,000 commands. This means that, in theory, the queues could collectively hold 4.225 billion commands.

Another absolutely crucial difference between NVMe Over Fabric and iSCSI is that iSCSI is designed to be multi-point but NVMe over Fabric is not. The NVMe over Fabric standard assumes that there will be a one-to-one relationship between initiators and targets. That simply is not the case with iSCSI.

This is an important distinction because many highly available systems work by attaching multiple nodes to a common cluster shared volume via iSCSI or some other multi-path protocol. This gives each node in the cluster access to the same storage volume. NVMe over Fabric simply was not designed to be used in this way.

This does not mean that NVMe over Fabric is not suitable for use with highly available resources. It just means that some architecture modifications will be required. One option is to implement a multi-hop topology, although that approach will undoubtedly increase latency. Another possible option is to architect shared-nothing clusters. A shared-nothing cluster would eliminate the need for a cluster shared volume, but would presumably increase storage costs since each cluster node would require its own dedicated copy of the data. The use of software-defined storage is another model to consider.

4. NVMe over Fabric should be deployed right away.

Given the enormous performance gains that NVMe over Fabric can potentially deliver, it's natural to want to take advantage of the technology as soon as possible. Even so, it is worth noting that NVMe over Fabric is still in its infancy. Although there are plenty of vendors that are currently offering NVMe over Fabric solutions, these solutions are not yet 100% standardized. There is no guarantee that one vendor’s NVMe over Fabric hardware will work with hardware made by another vendor. Even so, the next year is likely to see the beginnings of mainstream NVMe over Fabric adoption, and that will help to drive hardware standardization.