Digital Clusters for Windows NT

What are 99.9% PC/LAN server up-time and availability worth to you? More to thepoint, can you afford to bet your business on Windows NT?

Many companies have their LAN, databases, and all other business functionson NT systems. But companies such as financial institutions question whether NTis ready for prime-time, mission-critical applications. When you rely oncomputers for your accounting, product development, human resources management,data management, and now sales through the Internet, your systems must beoperational 24 hours a day, seven days a week. Failure is not an option.

Clustering, which has been around in Unix and VMS for more than 10 years,is one technology for achieving near 99.9% server up-time. By letting youduplicate a mission-critical system, this technology guarantees availability, soyou can bet your business on your OS.

Now clustering is coming to NT. Although this technology is not on thegrand scale of its Unix or VMS predecessors, clustering offers functionalityheretofore unknown to PC operating systems and represents a big step for NTtoward availability worthy of those major-league, mission-critical enterpriseapplications. By having two computers instead of just one to support a task, youdouble your chances for meeting the goal of 99.9% server up-time.

Clustering 101
Before I get into the specifics of Digital's cluster solution, let meexplain some clustering terminology: load balancing, primary server, failover(or secondary) server, failover, and failback. You can set up each server sothat all five terms apply to it.

For example, suppose you use SQL Server for your accounting and orderfulfillment departments and you have two databases that you want to protect byimplementing a cluster. In a single-cluster environment (two servers), you canmanually load balance--divide the work between the two servers--byinstalling SQL on both machines. Make one the accounting database's primaryserver--the system with principal ownership and management responsibilityfor a resource--and the other system the ordering database's primary server.Then, set up each system to have a primary disk (or disks) on the shared storagearray (a chassis housing shared disk drives where cluster software stores andshunts data between systems). This disk will serve as the database device. Sofar, this configuration is no different from setting up two independent servers,except that the shared disks are on a subsystem physically connected to bothservers.

Now, you set up the cluster by configuring each machine to be the othermachine's failover (secondary) server--the system that will inheritownership and responsibility for a resource--to the other. So when one system(the primary server) goes down, it will fail over--relocate clusterservices or resources from the faulty system to the operational one. Itsresources move to the failover server, and the service (such as a database)keeps running. When the primary server comes back online, the service will failback--automatically migrate cluster resources from the failover server backto the primary server.

The failover server is not just a cold standby server (as with Novell): Theserver performs meaningful work and provides more than disk-mirroring orsingle-system availability through hot-swappable disks. The open architecture ofboth the software and the off-the-shelf hardware means that you havescaleability built in. You can add disk storage almost ad infinitum andfunctionality with more CPUs and peripherals such as printers and tape drives.

Digital's Configuration
Digital Clusters for Windows NT is two servers, a network connection,cluster software, and an external disk array with SCSI adapters. (Although the1.0 product release supports hardware-based RAID, the Digital BA356 storagesubsystem doesn't. A future product release will have a built-in RAID 5controller. Also, version 1.0 does not support software-based RAIDthrough NT.) A key feature of this clustering solution (Digital will contributethis feature to Microsoft's Wolfpack standard--Mark Smith explains Wolfpack in,"Closing In on Clusters," page 51) is that Digital's clustering canuse off-the-shelf hardware for disks, network cards, and SCSI controllers.

Digital will officially support only its listed hardware (AlphaServer 1000,1000A, 400, 2000, 2100, 2100A, 4100, Prioris ZX Pentium, Prioris ZX Pentium Pro,Prioris HX, Prioris XL), but the software works on other systems, too. You canuse any two servers running NT Server 3.51 with Service Pack 4, but theclustered CPUs must be the same. You can't mix Intel with Alpha because ofdifferences in how the NT File System (NTFS) handles file tags (information onpermissions, groups, etc.) and page logs on Intel and RISC platforms. The twoclustered systems don't have to be similarly configured (one can be a dualPentium and the other a quad Pentium Pro), but on each machine, you have toinstall the same software (SQL Server, Oracle7 Workgroup Server, or any otherapplication) you intend to fail over from one system to the other.

The disk array is a BA356, which is part of the Prioris kit you buy fromDigital, without disk drives. This standard external storage chassis has amultichannel-capable, Fast and Wide, differential SCSI-2 backplane­you canhave as many SCSI channels on it as you have drives and controllers in your twoservers. You can set up the disk array to be either in the middle of the SCSIchain between the two servers or at the end. Where you put the array depends onwhether you leave the terminators installed and whether you use what Digitalcalls a trilink SCSI adapter. This adapter is a Y connector from thedisk array to the two servers. You can order a standard cluster kit from Digitalthat comes with cables, terminators, and an Adaptec 2944W Fast and Widedifferential SCSI-2 controller for each server.

The network connection is just a medium for a heartbeat between the twomachines. The heartbeat lets each machine know the other is alive. If onedisappears, the failover begins, and the remaining system takes over allassigned functions.

This connection can either be through a dedicated direct connection with abasic 10Mbit Ethernet card, or you can go through your usual high-speed LANconnection. Beware of using your usual LAN, because your domain controller andcompetition for your Ethernet media can introduce extra delays that can add tothe 20- to 30-second failover time. Also, a failure in the part of your networkbetween the clustered machines will initiate a failover: Each cluster machinewill think the other is dead, so the clustering software on each server willdrop ownership of the disks and leave them offline to prevent data corruption.Digital recommends a direct, standalone connection between the two servers forbest performance.

The cluster software is where all the magic occurs. This software acts as ashim--new code that the software adds without disrupting existing OS code. Thesoftware provides the means for the SCSI drivers and the network layers in theOS to carry out the clustering capabilities. The software also has anadministration tool for setting up drives, failover scripts, and othercharacteristics of the cluster (such as its network alias and administratorlogin and password). The logic behind the cluster's operation is complex, butthe user and administrator aspects are simple.

The Technology
Let's get down to the business of understanding how Digital's clusteringworks. To the user, a cluster of two computers and a disk array appear as onecluster alias with shares. A new network path for Digital Clusters for WindowsNT shows up in your browse list, as you see in Screen 1. Users connect to thealias instead of directly to each server. For the users, that's all there is toit--they won't even know the names of the two servers.

Digital Clusters for Windows NT follows ideas such as objects and groupsthat are already in NT. An object refers to a server, a disk, thecluster alias, or failover scripts and shares. Groups are where you assignobjects that will fail from one system to the other. Although these groupsaren't the same ones you need in user administration, the concepts are the same.For example, you create a cluster group, assign it a primary drive, create theshares, and assign applications such as SQL Server. You assign script objects tocluster groups, and these objects control what happens when a system fails.(This idea can be confusing because you assign the script to the serverwith primary control, but the script runs on the server that remainsafter a failure!)

The cluster wizardry is in the dynamic link libraries (DLLs) you install onthe server and client. On the server side of a basic cluster, you have a SCSIstorage shim and a network shim. The SCSI shim lets the two servers reside onthe same physical SCSI bus without arguing over bus and disk ownership. Theservers can't share the drives simultaneously, otherwise the servers wouldcorrupt the data. Instead, primary server failure causes the disks to gooffline, and control shifts from one server in the cluster to the other. Thenetwork shim on the servers lets you create the cluster alias that usersreference through the new cluster domain.

On the client side, a DLL lets the system see the alias and treat bothservers as one. Without the client software, you can see only individualservers, and the cluster alias doesn't appear in your network browse list whenyou try to connect to its network shares.

Failover Manager software on each server monitors the other and managesaccess to shared resources. The Failover Manager uses the network and storageshims and the Cluster Failover Manager Database (CFMD) to orchestrate thepolicies regarding the failover of cluster groups (and included objects). Figure1 shows the Failover Manager architecture. Each machine contains complementaryinformation that lets the Failover Manager manage the process of movingresources back and forth.

Another component of the cluster is the failover script. A script is acommand that refers to the primary server but runs on the failover server.Although you don't use a scripting language (you enter commands as you do from acommand prompt), you can initiate multiple actions when a server fails or comesback online. One example is a netsend command that issues a network message toall users about the system failure (or system recovery). You can execute anapplication that performs certain administrative functions. This application canbe useful for failing over applications that the clustering software doesn'tsupport, or failover IP addresses if you are running Microsoft's InternetInformation Server (IIS) or Exchange on the cluster. Screen 2 shows the scriptadministrator.

Installation and Configuration
When you buy Digital's cluster kit, you get the cluster server and clientsoftware, SCSI adapters, cables, terminators, and external storage cabinet; youmust buy the servers, the disk drives, and NT Server separately. Although youhave several server options, make sure your configuration lets you manually loadbalance your cluster and the configuration has enough power to handle thesecondary load when one system goes down: Don't use a single-processor 100-MHzPentium system as the failover for a quad-processor 200-MHz Pentium Pro.

Setting up a cluster is easy; configuring actions upon failure such as SQLdatabase failover is more complicated. To help, Digital provides a GUIadministration tool that lets you set up your cluster objects and groups, createfailover scripts, etc. Screen 3 shows the GUI tool. NT's standard File Manager(which you can access from the Tools menu) lets you establish network shares forwhole drives or directories. All file system attributes and security remainintact, and you can manage them as you do for any nonclustered drive.

Making failover groups, assigning drives, and entering scripts are allpoint-and-click (and drag) operations, with little typing necessary even for thescripts. For example, when you install a drive, you tell the cluster which busthe drive is on (because your server now has more than one SCSI bus), assign thedrive to a primary server and a failover server (and run Disk Administrator fromthe primary server to format it, etc.), and put the drive in a group. Thecluster drives must be on a bus separate from the drive or drives containing theOS and application software.

The GUI administration tool presents three views of cluster resources:system, cluster, and class. The system view lets you see the cluster from aphysical hardware perspective (system names, SCSI adapters, and disks). Thecluster view shows you the cluster from a failover group perspective (definedgroups, included disks, applications, etc.). The class view presents the clusterfrom the perspective of available cluster objects, without regard to physicallocation and grouping, and shows all objects such as group lists, SQL objects,and scripts.

You can use the GUI administration tool to perform manual failover foradministration purposes when you are servicing the machine and need to take itoffline, and for manual failback. You can disable failover and failback entirelyif you expect the system to be up and down several times in a short period.

You can install client software to support automatic failover for anyWindows for Workgroups, Windows 95, and NT client. Both the server and clientcomponents support common protocols (TCP/IP, Internet PacketeXchange--IPX/Sequenced Packet eXchange--SPX, NetBEUI) with hooks for SimpleNetwork Management Protocol (SNMP) server/cluster management through Digital'sServerWORKS Manager 2.0. This capability gives you failover of NTFS shares, SQL6.5 and Oracle7 Workgroup Server 7.1 and 7.2 databases, and any applicationsthat you launch or close with a script. Note, however, that Digital clusteringdoes not support failover for DOS, OS/2, or Mac OS clients. Users on suchsystems can still connect directly to the servers and access network shares, butthese users will have to manually reconnect to the remaining server after aprimary failure.

You can upgrade an existing server installation for clustering or startfrom scratch. Each server has its own primary disk for the OS, applications, andso forth, and only data is on the shared drives. Whether you upgrade or set up anew cluster, you need to install the SCSI controllers in each system, set up aseparate direct network connection (recommended over using your LAN) between theservers, and connect the storage subsystem. The software uses InstallShield onboth the server and the clients, so installation is easy; Digital provides freesoftware licenses for as many clients as you need.

When you set up a cluster, it has its administrator login and password,separate from the domain accounts (although you can match your domainadministrator login, the separate login and password add a level of security).However, all domain user accounts still function, and each machine has itsadministrator login.

Application and database failover configuration depends on the program,such as SQL Server 6.5 or Oracle7 Workgroup Server 7.1. SQL 6.5 includes storedprocedures that support clustering, so you can set up a primary database serverand a failover directly through the database application.

This first product release of Digital Clusters for Windows NT does have afew hitches. For clustering to work on the client side, users mustaccess the server through a network redirector modified by a network shim (whichis a kernel-mode component). So after you install the cluster server and clientsoftware, for the failover to function, users must enter the common clustername. The client Name service intercepts user universal naming convention (UNC)requests for cluster resources and sends them to the server Name service, whichtranslates the cluster alias into the UNC server address. The server Nameservice passes the UNC server address for the cluster member that is exportingthe resource back to the client Name service, which in turn passes the UNC backto the client redirector.

This process lets a user connect to a share through an alias rather thandirectly to a specific server. If users access a server resource directly(through the Connect Network Drive option and browse list) and the system fails,they lose the connection as if no cluster existed. The only way around thisproblem is to educate the users not to connect directly to the servers.

Performance
You can perform manual load balancing by assigning specific database work orfile services to specific servers and disks. Digitial Clusters for Windows NT1.0 doesn't support dynamic load balancing (where the cluster uses any availablehardware for processing overflow, regardless of the administrative setup).However, you will see dynamic load balancing within the next couple of years andautomatic IP failover for Web and email applications in version 1.1. You canscript any applications that the cluster software doesn't directly support, buthere are some caveats: If you are clustering an application server, you have todesign applets (that a script executes) to manage your applications during afailover. Even then, users can still experience service interruptions. For now,you are better off leaving any compute work on your clients and data on thecluster servers.

When a failure occurs, users will experience a 20- to 30-second delay duringthe failover (you can adjust this setting on the server). Local applicationswill keep running on the client, and a well behaved, or cluster-aware,application will at most give a message such as "Network connection notready: retry?"

Windows NT Magazine Lab ran several tests using a Prioris ZX5133MP/4(quad processor 133-MHz Pentium) and a ZX6166MP/2 (dual processor 166-MHzPentium Pro). In our tests, SQL failed over without complaint, and a testapplication (from a Digital demo CD) halted for about 30 seconds and thencontinued. We didn't test performance because the machines weren't set up forit. Digital says failover time will be the same no matter what hardware you runthe cluster on.

So what data is lost during a failure? It depends on what you're doing. Ina database, the application might roll back the logs to reconstruct the data,and you will lose any cached information from a query (especially if you're inthe middle of a query at failure). An application such as Microsoft Word (if runfrom the client only) will pause and perhaps display the "retry connection"message; however, if you're in the middle of a save, you can lose your changes.If Word runs from the server, the program will crash entirely.

Other problems occur with nonsupported applications on the clusteredservers. If a client runs an application connected to a cluster resource thatcontains open files or named pipes--an interprocess communication mechanism--afailover will break the pipe. Reads or writes to the cluster resource will failif the client application doesn't handle the I/O properly. The individual clientapplication is responsible for handling the error, then closing and reopeningthe file or named pipe on the failover server. A properly written programrunning on a client connected to the cluster alias automatically performs thesesteps, transparently to the end user. This sequence of steps maintains filesystem integrity.

The clustering software automatically disconnects users from the failedserver and reconnects them to the remaining server, so failover and failbackdon't affect the user much. For situations other than those described above,Digital makes no provisions for preserving application context. You may getdumped out of the application and have to reconnect to your file, but thissituation is no worse than when a single server goes down--it's just a whole lotfaster and doesn't require manual administrator intervention.

Wolfpack
Digital is working closely with Microsoft and other vendors to develop aclustering standard code named Wolfpack. In fact, Digital and Microsoft areworking so closely that Wolfpack and even NT Server will directly incorporatesome of Digital's concepts. Digital's products will include all the Wolfpackstandards and some of its own enhancements. Figure 2 illustrates Digital'sWolfpack strategy. Digital will provide a wizard to help its NT clustercustomers migrate to Wolfpack after its release. If any functionality inDigital's product doesn't make it into Wolfpack, Digital will provide thatfunctionality as a low-cost add-on called the NT Cluster Plus Pack.

Digital's goal in this cooperation is to avoid leaving customers out in thecold if they go with Digital's first release. Digital Clusters for Windows NT1.0 will interoperate and scale according to the Wolfpack APIs and standards, socustomers will be able to upgrade cleanly--without losing their investment inexisting hardware and software.

What makes Wolfpack so special when clustering has been around for so long?The PC/LAN environment has never had such functionality in a nonproprietary,hardware-independent, and inexpensive standardized technology, which is whatWolfpack will be. Other availability solutions such as Novell's NetWare SFT IIIrequire specific software (and sometimes hardware) that not everybody supports.When you own the OS (e.g., NetWare, VMS), you can easily design whatever youwant into it and be proprietary about the technology. Standardizing andsupporting end users in any configuration they want is not easy if you also wantto provide an upgrade and product-interoperability path. Wolfpack aims toachieve this goal, and Digital is betting heavily on its success.

Present and Future
Digital is following the Unix and VMS path toward full NT clusterfunctionality. Digital's VMS clustering product is the roadmap for its NT-basedclustering products. Digital Clusters for Windows NT 1.0 is the first productrelease. Digital will introduce RAID support and ServerWORKS Manager 2.0integration in a third-quarter update. Version 1.1 will include NT 4.0 supportand IP failover for Lotus Notes, Microsoft Exchange, and IIS.

Digital is aiming its NT-based clustering at mission-critical environments,such as finance, medical, and utilities, which need near 99.9% server up-time.In an operating environment that needs basic failover capabilities for SQLServer and file services, Digital has hit its mark. Digital's next challenge isto deliver products that address the remaining issues of dynamic load balancingand IP failover.

However, phase 1 is an excellent start. Because Digital Clusters for NT ishardware independent and relatively inexpensive when compared to other NThardware vendor solutions or Novell, it's already a major step toward reachingthe Wolfpack goals.