Windows Storage Spaces Direct: Delimiting Volumes vs. 3-Way Mirroring

One of the most fundamental elements of storage management is resilience. It’s a given that hard disks are going to unexpectedly fail from time to time, so it’s critically important to use the available resiliency features to make sure that workloads are not disrupted when hardware failures happen. For organizations that use Windows Storage Spaces Direct, the go-to resiliency mechanism is three-way mirroring. Three-way mirroring ensures that a Storage Spaces Direct deployment includes three complete copies of your data, meaning that a volume can withstand two concurrent node failures. But, even though three-way mirroring does a good job of protecting Storage Spaces Direct volumes, there are times when you may be able to achieve even better protection by delimiting your storage volumes.

Before I get into a discussion of delimiting storage volumes, there is one simple, but important, concept that I need to explain--the idea that Storage Spaces Direct can arrange your data across the physical disks in some pretty creative ways.

Imagine that you have a small Storage Spaces Direct cluster consisting of three nodes. If you were to enable three-way mirroring for a volume, then each node will obviously contain a copy of the volume.

Now imagine that you have a Storage Spaces Direct cluster consisting of six nodes, and that you have enabled three-way mirroring for a particular volume. Logically, it may seem as though volume replicas would reside on three of the six nodes, leaving the other three nodes for something else. However, that isn’t what usually happens.

Even though Storage Spaces Direct is designed to ensure data resiliency, it was also designed with performance in mind. Storage Spaces Direct strives to strike a good balance between performance and data resiliency.

So, with that in mind, let’s go back to my earlier question of what would happen if you enabled three-way mirroring for a volume residing within a six-node Storage Spaces Direct cluster. If Windows were to place a volume replica on three of the six nodes, that arrangement could indeed function as a three-way mirror. But consider performance for a moment. In this example, read and write operations are only being distributed across three cluster nodes.

For the sake of illustration, let’s say that the volume in question is larger than any one physical disk. The volume’s size mandates striping the volume across multiple physical disks. However, there is no rule requiring all of the members of a stripe set to reside on the same physical server. As such, Storage Spaces Direct will usually attempt to evenly distribute the disks among the cluster nodes.

For the sake of example, let’s say that a volume spans two disks and has been three-way mirrored. Within a six-node cluster, the replicas of disk 1 might reside on nodes 1, 3 and 5, while the replicas of disk 2 reside on nodes 2, 4 and 6. I’m over-simplifying things here, but I’m trying to illustrate a concept rather than delving into a literal discussion of how Storage Spaces Direct clusters allocate data. Regardless, the arrangement that I just described allows read and write operations to be distributed across all six nodes, not just three. Each node in this example is being treated equally, as opposed to some nodes being overworked while others remain idle.

This all sounds great, but there is one major shortcoming associated with this architecture. The volume can survive only two node failures, even though it spans six nodes. The reason for this is that none of the nodes (in this example) contains a full copy of the volume. The volume is being stored across multiple nodes.

This is where the concept of delimited allocation comes into play. As you may have already guessed, delimited allocation allows you to reduce the number of nodes used to accommodate a particular volume. The advantage to doing so is that when properly implemented, delimited allocation makes it possible for a volume to survive a three node failure.

The disadvantages to using this technique are that the administrator becomes responsible for balancing the data, as opposed to Storage Spaces Direct doing it automatically. Delimited allocation also results in additional physical storage consumption.

As you can see, delimited allocation can certainly help you to improve data resiliency. Even so, it should only be used in Storage Spaces Direct clusters with six or more nodes, where three-way mirroring is being used. It is also worth noting that delimited allocation cannot be used on clusters that use parity or mirror accelerated parity for resiliency.