Load Balancing Your NICs

No more NIC-bottleneck blues

Redundancy permeates every nook and cranny of the computer industry, and new technologies increase fault tolerance and accommodate 24 X 7 availability. At the same time, processor speed and memory size continue to increase at a blinding rate. Despite these innovations, the network interface card (NIC--i.e., the physical network adapter) remains a weak link for many servers. The NIC often throttles performance and acts as a point of failure.

You can install a second NIC in the server to help alleviate these problems. However, this configuration is less than ideal because you must place each NIC in that system on a separate subnet. As a result, you must fragment your network to accommodate your server's NICs.

What if you combine two or more NICs to create a high-speed virtual NIC that you access from one subnet? In this month's review, I'll focus on three software-based solutions that combine multiple LAN connections (i.e., ports) in one system to create a high-performance data pipe that provides output two to four times faster than systems with one LAN connection. These super NIC arrays accomplish this increased speed by directing output through any available NIC port. In all three products, a network device interface specification (NDIS) driver balances the load of the multiple LAN connections.

The benefits of using super NIC technology include increased throughput performance and stable load balancing of requests and the subsequent output. Super NICs also provide fault tolerance, so if one LAN connection fails, the system removes that connection from the NIC array and dynamically balances the load among the remaining connections. When the failed connection comes back online, it dynamically rejoins the NIC array.

The Test Environment
I tested all three software packages using a brand-name server with four 200MHz Pentium Pro processors, 2GB of RAM, 512KB of Level 2 cache, and two 4.2GB hard disks. I configured the test server with an Adaptec ANA-6944A/TX quad-port NIC that uses only one PCI slot but provides four separate LAN connections.

Next, I set up four 166MHz Pentium client systems each with an Adaptec 10/100 NIC and at least 64MB of RAM. I created a workgroup running TCP/IP and attached the four workstations and the test server to a Compaq NetIntelligent switch.

Before testing each product, I installed Windows NT Server, Enterprise Edition (NTS/E), Service Pack 3 (SP3), and the Microsoft NDIS hotfix on each system. This hotfix is a post-SP3 patch that corrects a memory leak in Microsoft's NDIS driver that can result in a blue screen of death. This memory leak often occurs when you use intermediate (layered) NDIS miniport drivers under NT Server. The layered drivers typically provide additional functionality, such as the virtual NIC feature in the products I reviewed for this article. You can download the free NDIS hotfix for Intel and Alpha systems from Microsoft's FTP site (ftp://ftp.microsoft.com/bussys/winnt/winnt-public/fixes/usa/NT40/hotfixes-postSP3/ndis-fix).

Until Service Pack 4 (SP4) arrives, I recommend applying the NDIS hotfix each time you apply SP3, even if you aren't loading an intermediate driver. Averting the blue screen of death is worth the extra time you spend installing the fix.

The Tests
To test the three NIC load-balancing products, I evaluated several criteria, as Table 1, page 74, shows. Specifically, I considered each product's price and which NICs you can use with each product. I identified which protocols and which operating systems (OSs) each product supports. I looked at each product's ability to automatically remove a failed LAN connection from the NIC array and automatically rejoin an operational LAN connection to the NIC array. I identified which products support segmenting a virtual NIC into two or more separate LAN connections (segmenting lets you achieve greater throughput and fault tolerance for various departments or users within an organization). I also evaluated whether each product includes a built-in monitoring tool and whether each product includes built-in alert options, such as email, Simple Network Management Protocol (SNMP) messaging, or paging. Finally, I considered each product's installation and configuration process, documentation, and technical support.

After I evaluated each product's features, I used the Web Capacity Analysis Tool (WCAT) that comes with Microsoft Windows Internet Information Server Resource Kit to test the products. WCAT runs scripts that generate various workloads from client servers to a specific server. I used the four client systems to run the workload against the 4-way server running the NIC load-balancing products. Each client simulated 250 simultaneous user streams. I ran two baseline tests and then ran the same test to measure each product's ability to balance the workload.

To measure the I/O, I installed Network General's NetXray on a separate system. NetXray showed data transfer per second for each NIC via its unique media access control (MAC) address and by IP address. Finally, I used Artisoft's ConfigSafe 1.0 after each test to restore the system settings to the original configuration, and ran a batch file to clear the system cache so that the cached data didn't skew the test results.

Balance Suite for Windows NT 2.5
The first product I reviewed was NSI Software's Balance Suite for Windows NT 2.5. The product comes on one CD-ROM with a thin user manual. The manual lists NICs that NSI has qualified for use with Balance Suite. This list does not include all the NICs Balance Suite supports--only the NICs that NSI recommends for use with Balance Suite. Although my NIC (Adaptec's ANA-6944A/TX Quartet) wasn't on the list, I was able to use it with Balance Suite.

To install the software, I inserted the CD-ROM in my system and followed the easy-to-use installation wizard. The wizard gathers data about where to install the product, asks whether you want to view the Read Me file, and opens the Network applet in Control Panel so that you can see the IP configuration. The Balance driver appears to NT as a new virtual adapter and disables the bindings between existing adapters and TCP/IP. The setup file pulls existing TCP/IP settings into an IP configuration file that you can print for reference during the setup process. By offering to open the Read Me file and Network applet, the wizard saves you from opening these resources manually and provides the information you need to immediately configure the new virtual adapter.

The next step is to add the NSI Balance Network Protocol, a process identical to adding any protocol in NT. I selected the Protocols tab from the Network applet in Control Panel, clicked Add, Have Disk, entered the path to where I installed the Balance Suite software, and clicked OK twice to finish installing the protocol driver. After I installed the protocol driver, the TCP/IP Properties dialog box appeared and displayed two choices: Obtain an IP address from a Dynamic Host Configuration Protocol (DHCP) server or Specify an IP address. I selected the second option, manually entered the IP address and subnet mask for the new NSI Balance Network Adapter, and clicked Yes to restart the machine.

When the system restarted, Balance Suite automatically loaded and started operating. The software automatically binds all NICs (or in my case all four ports on my multiport NIC) installed in the system to the NSI Balance Network Adapter. Next, I configured the protocol by accessing the Protocol tab in the Network applet. I clicked Properties for the NSI Balance Network Protocol to display the NSI Balance Settings dialog box. From this dialog box, you can configure each NIC individually.

Balance Suite selects the Enable Redundancy check box in the NSI Balance Settings dialog box by default, which includes each LAN connection in the virtual pipe. The software lets you specify how often the system checks each LAN connection and how many times it tries to make a LAN connection before failing. Finally, you can select one or more network types (i.e., IP, IPX, and NetBIOS/IPX) that you want to monitor and load balance. In addition to binding NICs for use with the NSI Balance Network Protocol, you can unbind NICs so that you can use them for applications other than Balance Suite's super NIC data pipe.

I pinged the Balance Suite virtual NIC IP address several times to ensure I had properly set up the software. The pings were successful, and the virtual NIC worked without any noticeable problems.

With the high-performance virtual NIC in place, I started each WCAT client and started the test on the WCAT controller. Using NetXray, I was able to see the test immediately start generating load against the virtual NIC. I toggled between viewing the data as filtered by IP address and as filtered by the NIC MAC address. Throughout my test, the virtual NIC operated without any problems, and each LAN connection shared the load balancing. The virtual NIC performed flawlessly and handled all incoming and outgoing data transfer, achieving a maximum throughput of 247 megabits per second (Mbps) when I combined all four ports on my NIC into one data pipe.

Next, I removed one of the LAN connections from the network switch, and within 1 second, the Balance Suite monitor updated the statistics and graph windows, showing that this connection had failed. The other three LAN connections balanced the remaining load without any noticeable difficulty. I plugged the cable for the failed connection back into the switch, and the software automatically rejoined the LAN connection within a few seconds. I continued removing cables from the switch one, two, and three at a time, and the software continued to process the data with little interruption and generally at the same high speed.

As Screen 1 shows, Balance Suite's internal monitor displays the type and pattern of activity on the network segment associated with the NSI Balance Network Protocol driver. You can also use the monitor to verify that your system is optimally distributing IP and IPX traffic among the available LAN connections. Although the monitor is an interesting feature, I ran into several problems. For example, you can't select individual LAN connections and add them one at a time to the monitor, as you might add counters to NT's Performance Monitor. Also, you can't adjust the statistics window at the bottom of the monitor, which prevents you from viewing all the LAN connections and their data at once without scrolling. (Balance Suite 2.6, which arrived too late for review, lets you check the software's performance using Performance Monitor.)

Balance Suite was easy to configure and performed well in my tests, but it has several shortcomings. For example, you can't use Balance Suite to segment the virtual NIC into more than one high-performance LAN connection. This limitation restricts midsized and large businesses that want multiple departments to share a server but want to keep traffic separate. The monitor tool interface, which required me to scroll down to view the fourth port on my multi-NIC, also needs work: NSI needs to reconfigure the interface to display more than three NICs at a time. Finally, the version I tested didn't offer any notification alerts, such as SNMP, pager, or email (version 2.6 includes over 40 SNMP traps). Most companies producing these types of load-balancing products provide notification so that you know when the high-performance NIC array has failed. You can download an evaluation copy of Balance Suite from NSI's Web site and try it in your environment, but I recommend you consider waiting to purchase the product until NSI fixes some of these deficiencies.

NIC Express 1.01
NIC Express 1.01 from IP Metrics provides load balancing and failover in a hardware-independent environment. The product uses an NDIS intermediate driver that performs all functions in NT's kernel mode. The system requirements are relatively small, consisting of 2MB of hard disk space and 32MB of RAM.

NIC Express came on one 3.5" disk, with a 14-page setup guide and a small (but complete) online Help file. However, now IP Metrics ships the product on a CD-ROM complete with additional documentation and two instructional videos in AVI format.

After I installed SP3 and the NDIS hotfix, I inserted the NIC Express disk into the system to install the software. Whereas the other two products I reviewed install virtual adapters, NIC Express installs a new protocol and adapter, but displays only the protocol to the user. I opened the Network applet in Control Panel and selected the Protocol tab. Next, I clicked Add, Have Disk, and typed the path to the disk to add the NIC Express protocol service. I selected the NIC Express from IP Metrics option, and clicked OK.

Next, the NIC Express Setup dialog box appeared, which let me select available NICs on my system to include in the NIC array. Because I had four ports to choose from on my Quartet NIC, I selected the first port from the Available Adapters dialog box and clicked Add. The software prompted me to provide a name for the new array, which I called NTLABS.

The next prompt asked me whether I wanted to use the protocol information for the port I selected as the primary address for the NIC array. I selected Yes to set the primary IP address of the system I wanted to advertise to clients. If you select No, you can add other NICs and set the IP address for the NIC array later. When I clicked OK, the system bound the NIC and automatically rebooted the system. After the system rebooted, I returned to the Network applet in Control Panel and selected the Protocol tab. Next, I selected TCP/IP and clicked Properties. I reviewed the configuration settings for the NIC Express Virtual Adapter, which displayed the Quartet's four ports as one NIC. I clicked OK twice to close the windows and was ready to test the virtual NIC.

I pinged the NIC Express virtual NIC IP address to ensure that I had properly set up the software. The pings were successful, and the virtual NIC worked without any noticeable problems.

NIC Express' monitoring tool, as Screen 2 shows, displays the LAN connections in the array and the Cumulative or Per Second statistics for individual LAN connections or the array as a whole. You can access the monitoring screen from within the Protocol tab in the Network applet.

Next, I opened the Network applet, selected the Protocol tab, selected the NIC Express Transport, and clicked Properties to display the NIC Express Status dialog box. Color coding in the Status dialog box lets you view the status of each LAN connection instantly. NIC Express tracks five states for each LAN connection: The adapter is working properly, the adapter has stopped working for the first time since the server booted, the adapter has failed at least once since the last boot, the adapter is currently down and has been down before, and the adapter has failed more than three times in 1 hour (the default). To reset a LAN connection's state after three failures requires a system reboot.

With the high-performance virtual NIC in place, I started each WCAT client and started the test on the WCAT controller. Using NetXray, I was able to see the test immediately start generating load against the virtual NIC. I toggled between viewing the data as filtered by the IP address and as filtered by the NIC MAC address. Throughout my test, the virtual NIC operated without any problems. Although the input on the primary NIC was high initially, the software eventually load balanced all the NICs equally. The virtual NIC performed flawlessly and handled all incoming and outgoing data transfer, achieving a maximum throughput of 237Mbps when I combined all four ports into one data pipe.

Next, I removed one of the LAN connections from the switch. Within 2 seconds, NIC Express identified and removed the failed LAN connection from the NIC array and began load balancing the incoming test data to the remaining three connections. When I reconnected the cable for the failed connection to the switch, the software took about 2 seconds to rejoin the connection into the array and balance the data across all four NICs again. No matter how many LAN connections I disconnected, the software continued to service the incoming network requests as long as at least one connection was available.

With NIC Express, you can create multiple network segments. To set up two segments, I selected the Setup tab in the NIC Express dialog box. I then selected NIC 3 and NIC 4 and clicked Remove to remove them from the NTLAB segment. Next, I placed my cursor in the drop-down box next to the Add button and entered EDITORS for the name of the new segment. I selected each NIC in the Available Adapters dialog box, and clicked Add, Yes to include them in the new segment. Finally, I clicked OK, Finish, Yes to reboot the system.

NIC Express supports SNMP traps, which notify you when a LAN connection fails or comes online, but I was disappointed that the software didn't offer any other alert tools, such as pager or email notification.

NIC Express was easy to install and configure, and I like the ability to create multiple segments with the system's available NICs. Despite its lack of notification tools, NIC Express is a flexible tool to increase the throughput and fault tolerance of your network servers. Who can argue the price of the software when you can increase your server's performance two or three times? You can download a working, 30-day trial version of NIC Express from IP Metrics' Web site, and see your server's throughput and fault tolerance increase.

2

Duralink Port Aggregation 3.0
Duralink Port Aggregation 3.0 is Adaptec's NIC load-balancing offering. Unlike the other two products, Duralink is compatible with only Adaptec's single-port, dual-port, and quad-port NICs.

Installing Duralink requires that you first remove drivers for any Adaptec NICs you've already installed in the computer. This step seemed unnecessary, but having little choice, I opened the Network applet in Control Panel, clicked the Adapters tab, selected each of the Quartet's four ports, and clicked Remove. After I removed all the connections, I closed the Network applet and restarted the system.

I opened the Network applet again, selected the Adapters tab, and clicked Add. From the Select Network Adapter dialog box, I clicked Have Disk, inserted the Duralink disk, and entered the driver installation path. I selected Adaptec PCI Ethernet Adapter from the Select OEM Option box, and clicked OK.

After I installed the NIC, the Adaptec New Hardware Found dialog box opened so that I could configure the new Duralink driver. I left the Autodetect Default Connection type and clicked OK to open the Duralink Port Aggregation dialog box.

You can group Adaptec NICs into separate subnets by selecting two or more NICs from the Available Adaptec NIC Ports dialog box. I selected two of the ports, clicked Add, and typed the group name, NTMAGLAB, in the Groups dialog box to finish creating the new group. I then created a second group by selecting the other two ports, clicking Add, and typing in the second group name, EDITORS, in the Groups dialog box. As Screen 3 shows, separating the ports into two groups resulted in two segments that each group can access. Finally, I clicked OK twice and then Close.

After I created the groups, I set the TCP/IP addresses for each group and put them on separate subnets using the TCP/IP Properties dialog box. I then clicked OK, Yes to restart the system.

You must use the Adapters tab of the Network applet in Control Panel to manage Duralink. I selected the Adaptec Duralink adapter, clicked Properties, then selected the Status tab to display the Status dialog box. I clicked the plus symbols next to NTMAGLAB and EDITORS, and the two groups I set up earlier appeared. The Bytes Received and Bytes Transmitted boxes show the data for groups or individual LAN connections, depending on which one you click.

I pinged the Duralink primary IP address to ensure that I had properly set up the software. The pings were successful, and the virtual NIC was ready for more thorough testing.

Users familiar with NT's Performance Monitor will appreciate an object that comes with the software that lets you track the performance of Duralink-enabled ports or groups. To use the object, I opened Performance Monitor from the Administrative Tools menu, clicked the Object box, and selected EMPCI. Using this object, you can monitor two counters: Bytes Received/sec and Bytes Sent/sec for each instance.

I used the object to monitor the NTMAGLAB group by selecting all the Performance Monitor instances for this group, highlighting the Bytes Received/sec and Bytes Sent/sec counters, and clicking Add. Finally, I clicked Done, and Performance Monitor began tracking the statistics for the two LAN connections and virtual connection for the NTMAGLAB group.

I started each WCAT client and started the test on the WCAT controller. Using NetXray, I was able to see the test immediately start generating load against the virtual NIC. I toggled between viewing the data as filtered by IP address and as filtered by NIC MAC address. Throughout my test, the virtual NIC operated flawlessly, and each LAN connection shared the load balancing. The virtual NIC achieved a maximum throughput of 248Mbps when I combined all four ports into one data pipe.

Next, I removed one of the LAN connections from the Compaq switch, and within 1 second, Duralink identified and removed the failed adapter from the NIC array and began load balancing the incoming test data to the remaining three LAN connections. When I reconnected the cable for the failed connection to the switch, the software automatically rejoined the LAN connection to the NIC array within 1 second and balanced the data across all four connections. No matter how many LAN connections I disconnected, the software continued to service the incoming network requests as long as at least one connection was available.

Duralink supports SNMP remote management through the use of agents to provide notification in the event of a failure. I prefer to have other notification tools such as email or pager to ensure timely notification. The software's limitation of supporting only Adaptec NICs might hinder the wide acceptance of Duralink in the market. With the huge variety of NIC suppliers and manufacturers, potential customers have a lot of other solutions to choose from.

Duralink presented no real problems during installation or configuration. However, uninstalling the Quartet adapters already installed on my system and then reinstalling these same adapters for use with Duralink was frustrating. In contrast, the object for use with NT's Performance Monitor to monitor the status of the NIC array was a great low-overhead solution.

If you already have Adaptec NICs or are considering purchasing these NICs, check out Duralink. You can download a free 30-day trial version of the software from Adaptec's Web site.

The Decision from the Bench
After spending many weeks reviewing these three products, I came to the conclusion that they were all excellent. Setting up each product was easy thanks to wizards and comprehensive documentation, and each product was ready for testing within 10 minutes after I started the installation process.

Each product did a great job of load balancing the various test loads I ran against them. In all three products, the NICs accepted the load and then distributed it within the virtual NIC.

NSI's Balance Suite currently doesn't let you create more than one segment at a time, although the company reports it will include this feature in a future version. Also, the Balance Suite monitor tried to fit too much data in one window, and the version I tested only let me view data for three NICs simultaneously. Finally, the version I tested didn't offer any SNMP alerts, although NSI has added over 40 SNMP traps to version 2.6.

IP Metrics' NIC Express had almost everything you want from a software NIC load-balancing solution. I prefer to have the monitor window separate from the Networking applet so that I can track the NIC data and minimize the window. NIC Express' performance was the worst of the three products I tested, but it was still acceptable.

The only criterion that really separated any of the three solutions was whether the product supported generic NICs. Adaptec's Duralink Port Aggregation software works with only Adaptec NICs, while the other two products work with most NIC brands. I also discovered that when I configured a four-port virtual NIC and wanted to divide it into 2 two-port segments, I needed to reenter the IP addresses and subnet mask data for the new segments.

You can't go wrong in choosing Balance Suite, NIC Express, or Duralink Port Aggregation. The performance you'll gain is well worth the investment, although your results will vary depending on the configuration of the system you use. When every ounce of performance is required, it makes sense to remove the bottleneck of a sub-$100 NIC. Your servers and customers both will thank you.