Increased network traffic between the nodes in a Shared VHDX and Hyper-V Cluster

Q. I'm using Shared VHDX with my Hyper-V cluster, and I've noticed increased network traffic between the nodes in the cluster. Why?

A. Cluster Shared Volumes were introduced back in Windows Server 2008 R2. They permit a LUN to be accessed by all nodes in the cluster simultaneously by allowing direct I/O from all nodes to data while any metadata changes are performed by the coordinator node for the specific CSV. The metadata requests are sent over the cluster network to the coordinator node. It's possible for nodes to lose direct access to storage, but then they'll go into redirected mode and send all I/O over the cluster network to the coordinator mode, and thus continue to read and write data.

Shared VHDX allows multiple virtual machines to use the same VHDX file across multiple nodes. So it's necessary to orchestrate the shared access.

How does this work?

The coordinator node sends all I/O to the shared VHDX. This, in turn, requires all VM-hosting nodes that are not coordinator nodes to send their I/O to the coordinator node.

For example, if the storage is on a Scale-out File Server (SoFS) is accessed via SMB 3, then all nodes will be directed to the coordinator node in the SoFS for the specific share hosting the Shared VHDX. If a server has a 8 Gb connection to a SAN and a 1 Gb connection to other nodes via the cluster network, then this could affect storage performance. You will definitely see increased cluster traffic. 

When a Shared VHDX is used --  even if nodes in a cluster have direct connectivity to the storage hosting the Shared VHDX -- the actual I/O is sent over the cluster network to be performed by the coordinator. It is therefore very important to carefully design the networking in the cluster when using Shared VHDX.

Note this ONLY applies to the Shared VHDX traffic, all other I/O is sent direct to the storage.

Some great additional information can be found at the following: