Microsoft Multipath I/O for iSCSI

Many businesses, regardless of size, are looking to SANs to address increasingly demanding storage needs. SANs offer flexibility for a variety of common infrastructure scenarios, including database and email servers, common file storage, and virtualization. SANs are incredibly popular when fault tolerance is a requirement, allowing quick recovery from disk or server failure. SANs can be built using a variety of technologies, ranging from DAS, to Fibre Channel, to incredibly popular iSCSI networks. Although many SAN architects and administrators focus on building a fault-tolerant disk subsystem, or clustering the servers in front of them, it isn’t uncommon to find attention to the actual connections to the SAN neglected, with basic configurations that have single points of failure or less than optimal overall performance caused by bottlenecks and misconfiguration.

Complicating matters is the fact that many SAN vendors provide their own device drivers and management software designed to work with their equipment—but that the OS can’t take true advantage of. Often, SANs built using equipment from multiple vendors must use generic drivers and might lack end-to-end management. 

To address these problems, Microsoft built support for Multipath I/O (MPIO) in Windows Server, which is designed to help businesses build highly available, fault-tolerant SAN configurations. As an additional benefit, MPIO can improve performance depending on your SAN equipment and overall configuration. In this article, I describe some of the features of MPIO in Windows Server 2008 R2, and I provide general recommendations for leveraging this powerful feature in your environment.

MPIO Basics

Before going into detail about the features of MPIO in Windows Server, it’s necessary to cover a few basics about the available configuration options, including the benefits of each option. Note that some of these options might not be available to you, depending on the type of SAN you have, as well as the support for MPIO available from the manufacturer of the components that it consists of. Server 2008 R2 supports the following six MPIO configurations:

Failover. The simple Failover configuration, also known as Fail Over Only, requires two or more paths from the server to the disks whether DAS, via host bus adapters (HBAs) in a Fibre Channel system, or NICs and paths in an iSCSI SAN. The SAN administrator will select one path as the primary communication path and each additional path as failover paths. Each failover path has a preference assigned to it, and each path is used in turn from the most preferred to the least preferred when the primary path fails. When the primary path is restored, the SAN administrator must manually configure the system to use it, switching back from the failover path in use. 

Failback. The second configuration option, called Failback, is somewhat related to the Failover option. Like Failover, a primary path is defined; when it fails, communication is routed over alternative paths in decreasing order of preference. However, unlike Failover, communication is routed back over the primary path when it’s restored. Failback is typically used when a primary communication path is faster or has fewer devices between the server and the disk subsystem than Failover paths and therefore has fewer points of failure. It should be noted that Failover and Failback operations aren’t necessarily instantaneous, and there might be momentary disruptions in service when communication paths are switched. Although many applications won’t suffer from momentary disruptions, high-performance applications such as database servers and heavily used email mailbox servers might see even a momentary disruption as a disk failure, which could cause unintended consequences such as server cluster node failovers on connected servers. For this reason, Failover is typically preferred over Failback unless the differences between the primary and alternative paths are marked. Figure 1 shows an example of Failover and Failback configurations in an iSCSI SAN deployment.

Figure 1: iSCSI MPIO Failover and Failback

Round-Robin. When a server has two or more communication paths, the SAN administrator can choose to leverage them in a Round-Robin configuration. In this configuration, if a path fails, it ceases to be used by the server and is dropped from the round-robin pool of available paths until communication is restored. The advantage of this configuration is that requests are sent over multiple paths to the disk subsystem, which can improve performance. This configuration doesn’t take into account the performance characteristics of each path, the complexity of the requests, or a queue of outstanding requests on a path, if any. To address potential performance issues, a SAN administrator should use this configuration only if all communication paths are equal.

In addition, it can be assumed that all requests will likely be equivalent and there will be no queue of outstanding requests on any path greater than on any other path (which can happen if there are switches or routers on the path). If these assumptions hold true, a SAN administrator might still fail to see an increase in overall performance if the disk subsystem simply takes all requests into a single queue for processing regardless of the number of paths they travel over, and if the time it takes to process each request is greater than the time it takes a request to travel over any individual path. Performance can also degrade in a round-robin configuration if there’s a failure in a component on a path, as well as if failover is configured at the component level, which results in lower performance. Figure 2 shows a typical round-robin configuration in a Fibre Channel SAN. A variation on Round-Robin configuration, called Round-Robin With Subset, is one in which one or more paths are set aside for failover in decreasing order of preference. When all round-robin paths become unavailable, the highest preference failover path available is used until one or more paths in the round-robin configuration are restored.

Figure 2: Fibre Channel MPIO Round-Robin

Least Queue Depth. The next configuration available to the SAN administrator is called Least Queue Depth. It requires drivers and components in the SAN to be able to report the number of outstanding requests for each path. MPIO will route proportionately more requests over the path with the least number of outstanding requests. This configuration doesn’t require (or benefit from) all paths having equal performance characteristics or every request being similar in complexity. In fact, this configuration is designed to work well with uneven loads. This configuration doesn’t have explicit failover paths defined, either. If a path is unavailable, it’s simply removed from consideration.

Weighted Path. The next configuration available is called the Weighted Path. Each path is assigned a weight, and among the given available paths, MPIO selects the path with the least weight.

Least Blocks. The final configuration available is called Least Blocks. MPIO routes requests over the path with the least number of pending requests.

Installing and Configuring MPIO

MPIO is an optional feature in Server 2008. You can install it from Server Manager, or you can enter the following command on a command line:

ocsetup MultipathIo /norestart

When the feature is installed, a new tool called MPIO is added to the Administrative Tools folder. Also installed is an executable called MPclaim.exe. Although MPIO is easy to install, further configuration is highly dependent on your type of SAN and the equipment that it’s comprised of. The reason for this is that although Server 2008 R2 ships with support for what’s called a Device-Specific Module (DSM), you’ll most likely need to get a DSM from your vendor(s). DSMs can be loaded from the MPIO tool. Before you proceed to load DSMs and configure MPIO, I recommend that you consult documentation from your SAN equipment manufacturer, because it’s easy to make mistakes that can result in corrupt or lost data.

I recommend using MPIO with iSCSI, because MPIO is supported natively in Server 2008 R2 and doesn’t require you to load a custom DSM. In addition, iSCSI is becoming the SAN of choice for many enterprises because of its flexibility and low cost of entry.

MPIO and iSCSI

Before you can use MPIO with iSCSI, you need to discover existing multi-paths to iSCSI Targets. This can be done by launching the MPIO tool, selecting the Discover Multi-Paths tab, selecting the Add support for iSCSI devices check box, and clicking the Add button. Note that this will cause your system to restart. You can also discover iSCSI multi-paths from the command line, again with a reboot, by typing the following command:

MPclaim -r -i -d "MSFT2005iSCSIBusType_0x9"

Obviously, for this command to work, you need to have support for multiple paths from your iSCSI Initiator (your server) to your iSCSI Target(s). This is most simply achieved by using multiple NICs to connect to your iSCSI-based SAN, with IP addresses on unique subnets. You can verify that your iSCSI Targets can be managed by MPIO by typing the following command:

MPclaim -s -d

Figure 3 shows this command’s output. You can get more information about any iSCSI Target managed by MPIO by specifying the disk number at the end of the command—for example, MPclaim -s -d 0. 

Figure 3: Verifying that iSCSI Targets can be managed by MPIO

After you discover possible multi-paths, you can configure them using the iSCSI Initiator client, which can be launched from the Administrative Tools folder on the Start menu, or by typing iscsicpl from the command line. You should already have targets listed under the Discovered targets section of the iSCSI Initiator Properties applet. To configure MPIO for a target, select the target and click the Connect button to launch the Connect To Target dialog box. In the dialog box, select the Enable multi-path check box and then click the Advanced button. In the Advanced Settings dialog box, configure the alternative path to your iSCSI Target; then, click OK to exit and click OK again to exit the Connect To Target dialog box. Repeat these steps for every alternative path to your iSCSI Target.

Figure 4: Device properties for MPIO-enabled iSCSI Target

After you’ve added all your paths to your iSCSI Target, you can view the details by clicking the Discovered target and clicking the Devices button. Figure 4 shows an iSCSI Target that’s represented as Disk 1 on the server, with two paths to it. Clicking the MPIO button launches the Device Details dialog box, from which you can modify the load balance policy, as Figure 5 shows. The default policy for iSCSI multi-paths is Round Robin, but you can pick from the other configurations supported by MPIO, with the exception of Failback. As Figure 6 shows, selecting a Path Id and clicking the Details button will show you details of the path to the iSCSI Target, such as the IP address and port of the iSCSI Target Portal. The Edit button launches a dialog box that you can use to specify the path type (active or standby) and weight (preference) each path has for MPIO configurations that let you specify active and standby paths, as well as a weight or preference for each.

Figure 5: MPIO device details

Recommendations

Every SAN configuration is unique, whether it’s the hardware used, the nature of the requests made by servers connected to it, or both. MPIO provides a means for you to build high availability and, depending on your SAN configuration, potentially improve performance. Because each SAN is unique, it isn’t possible to provide detailed recommendations for all situations, but some high-level guidelines exist.

The first rule is that you should always make certain that there are multiple paths to your SAN from your servers, to ensure availability of services when components fail—MPIO is a means to accomplish this. The second recommendation is that wherever possible, you should use a MPIO configuration that takes advantage of the multiple paths to improve performance (for iSCSI MPIO, this is typically Round-Robin, the default). The third recommendation is that you should thoroughly test MPIO before putting production data into your SAN or running production applications. You can test MPIO by pulling out network cables from NICs dedicated to iSCSI connections, as well as by shutting down Fibre Channel switches and so on, to ensure that there’s no disruption in access to data. The last recommendation is the most important: Work with your SAN vendor to get your vendor’s recommendations for configuration of MPIO with Server 2008 R2 and the vendor’s equipment. Many vendors provide extensive documentation for free on their website—a simple search will typically find this information.