Techniques to Speed Up Your NT Network

LEARN HOW TO ADD NETWORK THROUGHPUT TO YOUR RESOURCE AND APPLICATION SERVERS

In the October 1996 Lab Guys column, "Overloading Your Server withMulti-Homing," John Enck and I raised the issue of putting multiple IPaddresses or multiple NICs in your Windows NT server: Does multi-homing work? Weinvited readers to submit ideas about how to make multi-homing work, because theWindows NT Magazine Lab ran into serious operational difficulties.Thanks to everyone who suggested ideas or echoed the Lab's concerns. We're happyto report that we have a solution--not the solution, mind you--but onepossible solution. Let's look at some issues the Lab faced and how we resolvedthem.

The Problem
One problem is how to increase network bandwidth for application servers. Wetest servers of all sizes in an enterprise environment designed to mimiccorporate networks. We need to know two things: 1) that we can build fat datapipes into our test servers for thousands of users on one network, and 2) thatnetwork I/O is not a bottleneck in our test environment.

You can take two approaches to network administration: Spend a lot of moneyand buy the biggest, fastest hardware; multiple Class-C IP licenses; and soforth, or make do with what you have and hope your users are patient. The Labaimed somewhere in the middle: We limited IP addresses, servers, and networkcards, but we used some expensive equipment.

Our setup is not unique. We need to run one NT domain, using TCP/IP,Dynamic Host Configuration Protocol (DHCP), and Windows Internet Name Service(WINS) for up to 50 workstations (simulating 500 to 2000 users) and 10 serverson 4 independent subnets, while maintaining trusts with other domains and accessto our 10Mbps corporate network and the Internet. We need to operate on100Base-TX so that our virtual users don't get bogged down by thin network pipesand we get a clear picture of server performance. Compared with a corporateorganization that has 2000 client workstations, 100+ servers, and who knows howmany separate networks, the Lab's setup is like a walk in the park.

The Solution
I won't describe all the things the Lab tried, but I will mention two thingswe learned not to do:

*Don't assign IP addresses to two cards in the same server on the samesubnet (e.g., 205.200.45.50 and 205.200.45.51) without subnet masking. Such anarrangement hopelessly confuses NT, even if you run Multi-Protocol Routing(MPR), because the OS doesn't know which card to use for which packets.

*Don't cascade hubs. You can chain only a limited number of 10Base-T hubs,you cannot chain Class-1 100Base-TX repeaters at all, and you can cascade onlytwo Class-2 hubs. If you cascade hubs, you end up with contention between ports;that is, systems (IP packets) earlier in the chain get priority over later ones.Instead, use switches, which boost network performance in other ways (e.g., addfull-duplex capabilities and high-speed backbones).

What should you do? Your solution depends on your situation. Ourlimitations dictated our course of action. Figure 1 depicts our logical networklayout, and "Windows NT Magazine Lab's Enterprise TestingEnvironment" details our resource and testing requirements. Working closelywith engineers from Adaptec and Cisco Systems, who each supplied a piece to ourpuzzle, we set up our network solution as follows:

*We had only one Class-C license, so without setting up a proxy server(also an option), we divided our 256 available addresses among four IP subnets(one resource and three client virtual LANS--VLANS, same physical network,different collision domains).

*To accommodate four separate networks talking to the same resource server,we needed a router. We used a server running Microsoft's NT MPR software (builtinto NT 4.0; available for NT 3.51 from Microsoft's Web site athttp://www.microsoft.com). Using an NCR Globalyst S40 with two 133MHz Pentiums,128MB of RAM, three 2.1GB Fast and Wide SCSI-2 drives, and an Adaptec Quartet4-port 100Base-TX full-duplex NIC, we installed MPR and DHCP services to routeamong our four networks and assign IP addresses to all clients and resourceservers.

*Because our setup was a domain, we needed a Primary Domain Controller(PDC). We used a Digital Equipment Prioris HX 5133DP, with two 133MHz Pentiums,64MB of RAM, two 2.1GB SCSI-2 drives, and a single Adaptec 100Base-TX NIC. Thissystem also functioned as the WINS server for NetBIOS name resolution (ourtesting software referenced machines by name).

*We set up five 100Base-TX hubs for our network: one 8-port Netelligentfrom Compaq and four 12-port hubs from Adaptec. We also needed a fast Ethernetswitch--we started with a 6-port Netelligent 100-TX switch from Compaq, but weoutgrew it. So we acquired a Cisco Catalyst 5000--an expensive piece ofnetworking hardware (with a base price of $11,000), but worth the price if youwant the power. For our test environment, we needed expandability, flexibility,and performance. The Catalyst 5000 came with one supervisor card (with two100Base-TX ports) and three line cards (12 ports each), giving us a total of 38configurable full-duplex switch ports. We separated 32 of these ports into 4VLANs, giving us 4 essentially independent 8-port switches. (We could have used4 separate, smaller boxes, but the Cisco gave us more flexibility).

Making It Work
Now that you know about the physical components, let's look at how we putthem together to set up our multi-homed network. We used TCP/IP, MPR, DHCP, andWINS to implement our solution. Let's look at what you need to know about usingthese services in a multi-homed environment.

TCP/IP. TCP/IP presents some unique problems (e.g.,multi-homing, security, administration) compared to IPX/SPX. So, the logicalconclusion is don't use TCP/IP! Unfortunately, the Lab doesn't have thatluxury. Microsoft based NT on TCP/IP and included IPX/SPX for compatibility withNetWare, but IPX/SPX is better than TCP/IP for certain things. For example, ifwe set up our testing environment with IPX, we wouldn't need multiple networks.(However, we couldn't access the Internet without an appropriate TCP/IP proxyserver.) To increase server network throughput, we'd just add NICs and make suretheir IPX network numbers didn't conflict. But we couldn't do that, so off wewent.

In the October 1996 Lab Guys column, we suggested using a proxy server as afirewall between you and the outside world so that you can use any address rangeyou want. Another option is IP subnet masking--breaking your network intological segments within a single address range. If you have more than 256computers, you need more than one Class license, but you can still group usersinto separate networks going into the resource servers to manually load balance.

Setting up subnet masking is as easy as entering a single number into theSubnet Mask field on the IP Address tab in TCP/IP Properties (as shown in Screen1). If you have only one logical network, you enter a 0 in the rightmost (last)octet field of the subnet mask. (An octet is an 8-bit identifier--an IP addresshas four octets that define its value. You can use all four octets to define asubnet mask, but we used only the last octet to keep things simple.) If you wanttwo segments, enter 128; for four segments, enter 192; for eight segments, enter224. Let's review how to determine these values.

The 8 bits of the octet define 256 possible logical network segments, butnot all of them are usable. When the destination IP address of an IP packet getsXORed (XOR is the logical Boolean operation defined in Table 1A) with the subnetmask (which happens regardless of whether you have multiple networks), thedestination network is calculated. If you have multiple networks, the subnetmask tells the packet which one to go to, based on the IP address.

To break the 256 possible IP addresses into two segments (0 through 127 and128 through 255), you enter 128, which is binary 10000000, in the last octetfield of the subnet mask. The 1 in the most significant bit (MSB) position ofthe subnet mask's last octet tells the protocol that the MSB of the destinationaddress carries an identifier (0 or 1) that defines the network segment.

Thus, the two logical network definitions are 10000000 and 00000000 for thetwo address ranges. The MSB of the final calculated address (after the XORoperation) defines which network to route to. The remaining 7 bits define theaddress within the network. Table 1B, gives an example calculation.

A value of 0 in the last octet of a subnet mask signals one logicalnetwork, and all 8 bits define the IP address (instead of the MSB and 7remaining bits defining the network number and address, respectively). If youwant four networks, you need four sequential identifiers: 00, 01, 10, and 11. Totell IP that the two MSBs are identifiers, you need to set the subnet mask to196--binary 11000000.

We needed eight subnets--four for the testing environment and four for therest of the Lab--resulting in a subnet mask of 255.255.255.224 (224 is binary11100000). This subnet mask passes the packet to one of eight networks andstrips away the first three octets in the address.

The IP addresses assigned to the card (in this case, statically assignedaddresses for the ports in the MPR server because the MPR and DHCP server needsto know the numbers for the NIC that does the assigning) determine which logicalnetwork they are on. In the Lab, we broke up our eight networks as addresses 0through 31, 32 through 63, 64 through 95, 96 through 127, 128 through 159, 160through 191, 192 through 223, and 224 through 255. You cannot use the startingor ending addresses in each range, either via static assignment or DHCP, becausethe TCP/IP specification reserves them for special use (the first address in therange is reserved for broadcasts to the rest of the world; the last is reservedfor broadcasts to the rest of your internal network). So our NCR server with the4-port card had X.X.X.65, X.X.X.97, X.X.X.129, and X.X.X.161 assigned to it(leaving the other address ranges for future use), defining it in networks 3, 4,5, and 6. When each port gets a DHCP request, the subnet mask defines whichnetwork the address belongs to; all networks are logically separate, just as ifthey were separate Class-C ranges.

MPR. Microsoft's MPR is a networking service that lets youroute IP packets between multiple logical networks, collision domains, andaddress ranges, or route network data between networks running differentprotocols (e.g., TCP/IP vs. IPX/SPX), different network hardware (e.g., 10Mbpsvs. 100Mbps), and disparate wiring types. You can set up an NT server withmultiple NICs to function as the pathway joining your different networks.Because TCP/IP or IPX/SPX is just the underlying transport protocol, MPR enablesdifferent NT domains or networks to communicate with one another as if they wereone big happy network.

MPR includes several services you need to install: DHCP (BOOTP) Relay Agent(if you plan to route DHCP requests), Routing Information Protocol (RIP) forTCP/IP, and RIP for NWLink IPX/SPX-compatible transport (depending on whichprotocols you're running). Installation is easy enough: Go to the Control PanelNetwork applet, open it (or right-click your Network Neighborhood icon andselect Properties), click the Services tab, click Add, select the service, andhit OK (install one service at a time). When you exit the Network applet, NTwill want to restart--let it.

Next, you need to do some basic configuration (you can't configure thesoftware components for DHCP Relay Agent or the RIP for IP services). Bring upthe Network applet again, go to Protocols, double-click TCP/IP to bring upTCP/IP Properties, and select the DHCP Relay tab, as shown in Screen 2. ClickAdd and enter the addresses for the cards and IP addresses you assigned.

One last check box you need to select is Enable IP Forwarding, which youfind on the Routing tab of TCP/IP Properties. If your system has more than oneNIC or multiple addresses assigned to one NIC, IP Forwarding lets your systemreceive packets via one NIC and retransmit them out another, or take packetsfrom one network (or subnet) and forward them to another. People disagree aboutwhether to enable this setting on nonrouting servers and workstations, but youneed to enable IP Forwarding on your MPR system--if you have only one NIC andexperience IP problems, turn IP Forwarding off.

NT will want to reboot again; let it. Then ping from one card to another todetermine whether the cards can talk over the cables; next, ping to and fromanother system. IP routing should be working fine at this point. If not, thereare a couple of possibilities you can try. First, create an lmhosts file. Thistext file resides in the %systemroot%system32driversetc directory andis a list of NetBIOS names and their associated IP addresses. If you create thisfile, put each system on a separate line with the IP address in the first columnand the name in the second. For more information, see Microsoft Windows NTServer Resource Kit: Windows NT Server Networking Guide. The lmhosts filedefines your PDC, MPR Server, and any other important servers with staticallyassigned IP addresses. You need to put this file on each system listed in thefile, so that they all know where they are and don't use the router to identifyeach other. If it still doesn't work, you might need to set up routing tables onthe MPR server, which you can do from the command line with the route command(refer to the NT Server networking guide for a detailed discussion).

If you also want to access the Internet, you need to run Domain Name System(DNS) somewhere. Either point your MPR system to your DNS server and Internetgateway, or install DNS on your router and configure your IP settingsaccordingly (DNS server IP address and domain name).

DHCP. DHCP (also installed from the Services tab in theControl Panel Network applet) lets you dynamically assign IP addresses toservers and workstations, rather than manually administer a huge number ofstatic addresses across your enterprise. When you set up a new workstation orserver, you simply tell it to use DHCP; when it comes online, it broadcasts arequest for an address assignment. The first DHCP server to respond tells thenetworked systems about the gateway, the WINS server, and other relatedservices.

To use DHCP on multiple networks or subnets of the same physical network,you must either set up a separate DHCP server for each network or cover them allwith one multi-homed server. Using a multi-homed server is an extension ofcreating multiple DHCP scopes (ranges of addresses) under a single Class-B orClass-C license.

For the Lab's purposes, dedicating four servers to take care of IPaddresses didn't make sense--and we certainly didn't want to manage all theaddresses on our own. Using the server with the multi-port NIC and MPR toservice DHCP requests did make sense. As the MPR server, each port onthe card already required a static address, and the subnet masking broke ourClass-C range into eight logical groups.

NT is intelligent in the way it handles DHCP. A server with one NIC can bea DHCP server for one large range of addresses or several small ranges withexclusions. By combining DHCP with multiple NICs and subnet masking, you canservice DHCP-enabled systems across multiple subnets or even across multipleClass-C IP licenses. If you need to service multiple subnets, properly settingthe DHCP scopes lets NT match the scope to a specific port. However, you don'thave to follow our example of using multiple NICs for a DHCP server. If you areusing standard network routers, you can set up a DHCP server with one NIC andmultiple scopes, and use the DHCP Relay Agent to route DHCP requests from thedifferent logical--or physical--networks. This setup lets you use one DHCPserver for your entire enterprise, because each router forwards its appropriatesubnet information along with the DHCP request. This way, systems across yournetwork will not be assigned the wrong addresses from the wrong subnets.

In the Lab, our eight-segment network required eight DHCP scopes (fourscopes per server; one scope can cover multiple subnets, but you must excludeany reserved or statically assigned addresses). Screen 3, shows the four scopesfor our second DHCP server. You create these scopes with the DHCP Manager appletinstalled with the DHCP service (double-click the defined server to set up a newscope). We defined the DHCP scopes on 66 through 94, 98 through 126, 130 through158, and 162 through 190, and we excluded 66 for the domain controller and WINSserver. (We could have started the scope at 67 to accomplish the same result.)Table 2 lists other DHCP scope configuration settings we used.

With these scopes defined and the addresses 65, 96, 129, and 161 staticallyassigned to the card, by a little sleight of hand, NT can assign a request thatcomes in over a particular card to an address from the correct matching pool.For example, if a request comes into the port at address 65, it gets an addressfrom the 66 through 94 range, not the 130 through 158 range.

We used physical wiring segmentation (creating separate collision domains)to logically delineate the Lab's network. This method works well if you want tooptimize your network throughput, but the hardware costs extra money. You canset up a flat network using hubs instead of switches between the MPR system andthe clients, and even put systems from disparate subnets into the same hub (youcould no longer use DHCP), but you will end up with significantly increasedpacket collisions, lower throughput, and decreased performance. The betterchoice is to use fast switches to physically separate the logical networks andkeep like-addressed systems on the same hubs. You can combine the twoapproaches, based on your priorities in the tradeoff between cost andperformance.

Figure 1, shows the Lab's network, with links from the DHCP/MPR system tothe four VLANs on the switch, hubs that handle the workstations, and directswitched connections for loaded resource servers. We'd get the best performanceif every computer had a switched port (and money was no object). The arrangementwe set up allows for the greatest expandability and flexibility, without a hugecost.

The physical segmentation of the network solves one problem: a DHCP clientrequest that arrives at the wrong card and causes the system to receive thewrong address and end up on the wrong network. If all systems were on the samephysical LAN (attached to the same hubs or the same switch), you couldn'tpredict the behavior of the DHCP server. By physically separating theworkstations and servers according to the networks they belong on, DHCP canassign addresses as I described previously, and you don't have to worry aboutrequests ending up in the wrong place.

One minor problem we experienced deserves mention. On a network, the firstDHCP server that responds to an address assignment request gets the prize. TheLab's network has two DHCP servers (covering two different IP ranges)--one forthe 10Mbps side and the other for the MPR/DHCP system. We experience occasionalrandom address assignments: A machine mistakenly gets an address from the100Mbps side when it is plugged into the 10Mbps side because the MPR system isthe first to respond (this situation always occurs for requests off VLAN 1, theone connected to the outside world). Setting up another MPR system to act as abroadcast packet filter between the 10Mbps and 100Mbps networks or using adedicated network router are the only solutions we've found to this problem. Bythe way, if you double-click a particular scope in the DHCP Manager tool, youcan see all the systems with registered IP addresses on that subnet.

WINS for Multiple Subnets. WINS, which handles NetBIOS nameresolution for NT (i.e., WINS lets NT systems see each other by name no matterwhat network they are on), is another service that you install from the Servicestab of the Control Panel Network applet. The WINS Manager applet installed withthe service (shown in Screen 4) lets you manage WINS Server properties. However,WINS can be complicated to set up and make functional--debugging the system tookus a while.

At first, we had difficulties putting WINS and DHCP on the same system, sowe separated them onto two servers: the MPR system (NCR) and the PDC (DigitalEquipment). As the WINS Help files state, WINS will not work on multi-NIC (ormulti-homed) systems. We weren't able to properly ping across the router usingNetBIOS names. The MPR system could ping the other clients, but the clients hadtrouble doing the reverse (pinging worked--but very slowly).

After you've installed the service, you can configure the primary WINSserver and secondary or replication servers. Go to the WINS Manager applet underAdministrative Tools, and open Add WINS Server under the Server menu. Enter theIP address for the system (in this case, the statically assigned address of ourPDC) in the dialog box, and press Enter--you're finished. The system is nowavailable for WINS name registration--or you can initiate scavenging (i.e.,force the system to go out and look for machine names) under the Mappings menuto speed the process.

Assigning gateways is an issue that covers several of the topics discussedhere. Gateway assignments affect what happens to lost packets, Internet access,and name resolution. For example, if you want to get from your desktop computeronto the Internet, you need to know who and where the gateway is. If you areusing DHCP, the DHCP server tells your computer how to route the packets.

But you have to tell the DHCP and WINS machines where to find the gateways.For name resolution, tell the DHCP server the WINS server's IP address (even ifthey are on the same machine). Tell the WINS server that the MPR system is thedefault gateway, and use the IP address of the port or VLAN that's attached tothe outside world. Make the default gateway for each network segment on the MPRserver point to the system that functions as your Internet gateway (or if yourMPR server is your gateway and DNS machine, use the appropriate VLAN port's IPaddress). Table 3 shows the address assignments that we used for the four LabVLANs. By the way, nothing prevents one physical server from functioning as PDC,DHCP server, DNS, and MPR; because the Lab environment was so complicated, wedecided to separate them--which also solved some intermediary problems andsimplified troubleshooting.

Performance note. WINS takes forever to configure: Hours passbefore the WINS server builds complete browser lists. After each system isregistered in the domain and given an IP address, you'll wait awhile before thesystem is added to the list and made available in the Network Neighborhood(although you can manually force access using the Run dialog: \machinename).You can try to force the WINS Manager to go out and scavenge for names, but thisaction takes just about as long to rectify as waiting for it to happen on itsown. About the only way to speed this process is to configure a secondary WINSserver, which not only distributes the load for managing the names, but providesfault tolerance if the primary server fails (which otherwise stops name accessfrom one computer to another on the network).

Multiple Cards
Extending network bandwidth in your server has some tradeoffs. Extraoverhead and I/O processing (e.g., interrupt handling) come into play when youadd cards. Each PCI NIC you add is another interrupt that the CPU must handle toservice I/O requests. More CPU interrupts mean slower processing.

Adding bandwidth increases the amount of information the system mustprocess. Additional bandwidth consumes extra resources such as memory and CPUtime--often requiring that you augment them to maintain your desired performancelevel. As your system handles more data, you need to look at your disk subsystemas the next potential I/O bottleneck.

All you can do is hope that putting the extra cards into your applicationserver doesn't slow it and negate the benefit you get by increasing networkthroughput. You can help this problem somewhat by using full-duplex cards inyour servers and giving them direct connections to your fast Ethernet switches.In our environment, we saw drastic improvements in performance by adding networksegments to the database servers under test, such as a quad-processor PentiumPro system going from a maximum of about 100 transactions per second to morethan 1800.

Wrapping Up
What you have in the end is a resource server, with a nice fat data pipeinto it and your users segmented into multiple logical networks to optimize I/O.You can accomplish this functionality with 10Mbps hardware instead of 100Mbps,but you will still want to use switched 100Mbps hardware for the backbone(10Mbps hubs or switches with 100Mbps uplinks).

For example, if you have a SQL server that needs more network bandwidth,you can segment your network, separating your users into discrete groups tobalance your load. Put one or more extra cards in your SQL server, assign themthe appropriate addresses (you don't have to use DHCP) to match your groupings,add some switching or routing hardware, and away you go. Keep in mind the packetcollision considerations, and you have successfully increased your bandwidth.

Don't forget to download and install Service Pack 2 (SP2) for NT 4.0. Thisupdate fixes several bugs in NT's networking components, such as duplicateaddresses from DHCP, IPX performance issues, and NetBIOS session conflicts. SP2also adds the ability to handle BOOTP DHCP requests directly from the clients.SP2 states that you must reapply it if you change or reload any networkservices; otherwise, the system uses the wrong libraries from the NTdistribution CD-ROM. (For more information and some caveats about SP2, seeJonathan J. Chau, "Service Pack 2," and Mark Minasi, "Recoveringfrom a Network Disaster," March 1997.)

See Also "Enterprise Testing Environment"