Look, Ma-No ICA!

How Motorola Deployed an RDP-Only Thin Client Solution

Windows NT Server 4.0, Terminal Server Edition (WTS), is a good start to a server-based networking environment, but most companies also need helper software to flesh out the support capabilities (e.g., client-side device support, sound, load balancing) that WTS lacks. Most companies choose Citrix MetaFrame. However, MetaFrame is expensive and requires purchase of the entire product package. So when implementing a thin-client solution for its manufacturing plants, Motorola's Semiconductor Products Sector (SPS) broke with convention and chose a less expensive and more modular option: Network Computing Devices' (NCD's) ThinPATH suite of WTS helper software.

SPS is Motorola's semiconductor design and manufacturing division; according to Motorola, the 15,000 SPS employees represent about one-tenth of the company's employees worldwide. The division's Bipolar Manufacturing Complex (BMC) is a set of factories on three sites in and around Mesa, Arizona. The BMC operates around the clock (i.e., two 12-hour shifts) and employs approximately 850 factory workers, 50 office-support personnel, and 10 fabrication-support personnel. BMC factory-floor workers need continuous access to operation instructions and chip cookbooks, which contain step-by-step recipes (i.e., instructions) for each stage of the semiconductor manufacturing process. These instructions and cookbooks reside in an Oracle database application running on a VMS server at each factory site.

PC's aren't the best clients for the SPS factory floors. The factory floors are clean rooms: Anyone entering the room must undergo a rigorous and expensive cleaning process. Keeping nonfactory workers (e.g., PC repair folk) off the floor is desirable, but PCs—with moving parts that can malfunction—aren't low-maintenance. The PCs' internal fans can create dust particles and blow them throughout the factory. Also, space is at a premium on the floor, and whereas terminals are potentially light enough to mount on a wall, PCs aren't.

SPS already used a server-based computing method to deliver recipes to the factory floors. The BMC's Mesa factory used 115 X terminals to connect technicians to the manufacturing database on the VMS server. (Figure 1 shows the original arrangement.)

The X terminals on the factory floor connected to the VMS server, then to the production databases, through the X-11 protocol on a 10Mbps Ethernet network. Through the VMS server, clients could access printers that attached directly to the IP network. This configuration worked, but it had problems. First, because of the X-11 protocol's high bandwidth requirements, connections were slow and were sure to get slower as the BMC increased the terminal count to 250 on the same 10Mbps network. Second, Motorola is designing the next generation of its factory-operations applications for Windows rather than UNIX. To make that software rollout easier to manage, SPS needed a platform that could handle access to both OSs; NCD's ThinPATH suite of helper software fit the bill.

Divide and Conquer
Duane Gordon, senior NT engineer for SPS, had only a few months to manage the switch to WTS—requirement gathering began in June 1999, and the cutover date was September 1999—so he divided deployment tasks between implementing the servers and deploying the terminals on the factory floor. Heading up the server-implementation team with Gordon were systems analyst Peter Zeiszler; Thin Client Computing's Steve Greenberg, who provided outside expertise in server-based computing; and Gary Colvin, then systems engineer for NCD's western region. B.J. Hiatt (the group lead for BMC) and the Mesa Fab Support Group were in charge of terminal deployment.

Server implementation. The rollout wasn't simply a Windows version of the existing X network. Gordon's team kept the VMS server and Oracle application in place, but used one of two inhouse terminal emulation packages, depending on a worker's position in the manufacturing process, on each WTS machine to provide client access. Placing the Window Manager on the WTS servers instead of the slower X terminals improved performance, because the RDP protocol uses less bandwidth than the X protocol does. But users needed—and got—more than Windows-based access to existing applications. The new WTS machines support Hummingbird's Exceed 6.1; Microsoft Word Viewer, Excel Viewer, and PowerPoint Viewer for Office 97; and Adobe Acrobat 4.0 Viewer. Client access to Microsoft Internet Explorer (IE) 5.0 was also crucial because Motorola operational requirements include support for viewing Web-based documents. Factory-support personnel can use such documents to provide factory workers with up-to-date documentation in a standard format (i.e., HTML), but the VMS system didn't provide a client-side browser. The new application suites also support browser-based email (i.e., Netscape's Messenger Express) and multimedia training videos in .avi format delivered from BMC's Tempe site to Mesa factory workstations across a 45Mbps Synchronous Optical Network (SONET) ring that connects the BMC factory sites.

Because of application incompatibilities, not all the servers in the server farm can run all applications simultaneously. For example, the Tempe Final Manufacturing (TFM) facility combines two fabrication areas, both of which needed access to a manufacturing application that pointed from the farm's WTS machine to a port on a UNIX middleware system. The application couldn't point twice to the same port from the same server, so the team installed the application on two servers, one for each instance of the application. Gordon's group then used NCD's ThinSTAR Management Service (TMS) to statically load-balance the terminals, based on terminal IP address, across the two servers.

To provide fault tolerance, the server team clustered the NT 4.0 file and print servers and domain controllers but didn't need to cluster the WTS machines. (Factory-floor workers use stateless applications, so even if a WTS machine crashes while a user is writing a record to the database, only that record is lost. The user can reconnect to the server farm and restart the session on the next available server.) The team used NCD's ThinPATH Load Balancing to load-balance the WTS machines. Network administrators manage the WTS machines from an NT server running ThinPATH Manager's ThinPATH Configuration Tool 1.01 and TMS 2.14.

Gordon originally planned to have six servers in the server farm to support the floor's 250 clients. However, after the farm was up and running, monitoring showed that the servers performed well with as many as 125 clients running on each, so Gordon reduced the number of servers in the farm to three. The servers are performing fine, and Gordon has no immediate plans to build the farm back to its originally planned proportions. (If users need more server capacity, Gordon can easily use the images he's already created to build additional servers.)

Load-balancing the servers had an advantage other than preventing server overload: flexibility. Earlier in the year, SPS moved its data center to a new building across campus. Moving a server means taking it offline for a day as someone packs up the server, moves it, reassembles it in its rack, reconnects it to the network, and returns it to the server farm. By load balancing the servers, SPS was able to support users without any interruption during the move.

Terminal deployment. Hiatt's terminal-deployment team originally installed NCD ThinSTAR 200 terminals with NCD's locally installed X client (as well as RDP support) so that workers could use the same terminals to connect directly to the VMS machine and the WTS machines. (When ThinSTAR 400s became available, the team upgraded the terminals to that model.) The team gave each terminal a dedicated IP address and a generic user account. (Workers don't have personal user accounts because a user's identity is less important to the applications than the user's location in the manufacturing process.) Using TMS, the team remotely edited the terminal configurations to stop using the X client to connect to the VMS machine and to start using RDP to connect to the WTS server farm. Figure 2 shows an approximation of the final clients-and-servers arrangement. ThinSTAR terminals on the factory floor connect to the WTS server farm, which in turn connects to the VMS server.

Shaking Out the Bugs
Implementing the new terminals and software wasn't a trouble-free process. Some bugs appeared during deployment.

First, the new implementation had a serious problem with profile corruption. Because of a problem in the way that WTS handled profiles, the user profile binary (i.e., ntuser.dat) sporadically became corrupted (WTS Service Pack 5—SP5—subsequently fixed this problem but wasn't available at the time.) When a user with a corrupted profile attempted to log on to a server, the server blue-screened and crashed. Because the servers were load-balanced, when the user made another attempt to log on, the next server crashed—and so on until all the servers in the farm had crashed. One user with a corrupted profile could bring down the entire server farm in the time it took to make three aborted logon attempts.

To resolve this problem, the team hand-tuned each profile separately. Using the name associated with each profile, administrators logged on, created a new user profile, and logged out to save the changes. After recreating every profile, the team copied the profiles to the local profiles directory (i.e., %systemroot%ProfilesSessionIDprofile) on each terminal server. To finish the process, the team configured WTS client accounts to get user-profile information from the local profile path rather than from a profile server, thus forcing WTS to apply local profiles instead of roaming profiles. This solution worked because the required user-profile settings were location-specific rather than user-specific. In other words, users logged on with a machine account rather than unique user accounts, so the deployment team could set up user account names for locations rather than for users, and the profiles didn't need to roam. After the team arranged the profiles on one server, the team could use Symantec's Norton Ghost to duplicate that server and ensure that the settings remained consistent across all servers in the farm.

Although this solution worked, it wasn't ideal because network administrators needed to copy any profile changes to every server in the server farm for load balancing to work properly. For the remaining parts of the rollout at the two remaining BMC factory sites, Greenberg plans a different strategy: a profile replacement application. He likes Tricerat Software's Desktop 2000, which doesn't depend on profiles. Instead, the software replaces the usual NT shell with a shell that exposes only applications that an administrator explicitly chooses. Because Desktop 2000 lets you expose applications on a per-user or per-terminal basis, it should work well for the factory floor.

Second, the solution to the profile problem led to another problem: the Ghosted servers weren't showing up properly in the TMS Terminal Server Administrator window. (The management window is in TMS rather than in WTS's Administrator window.) The servers appeared but were grayed out (i.e., unavailable). Running Browstat forced an update to the WINS database, thus making the servers available, but another glitch existed. Although the server-implementation team changed the SIDs on the Ghosted servers, performance-monitoring tools couldn't distinguish among the servers until the team manually edited each server's Registry to change the machine names.

Third, the X client caused a few headaches. Initial ThinSTAR 200 connections used the terminal's X client rather than RDP so that factory workers could continue working with their VMS applications while Gordon's team set up the WTS machines. SPS had some concerns (e.g., about speed) with the X clients; NCD resolved these problems with new builds, but the scenario still wasn't as seamless as RDP connectivity turned out to be. Now that the X client resides on a terminal server, SPS is having problems with VMS responsiveness: The VMS server seems to be flooded. So far, SPS hasn't been able to determine whether the problem lies with the network, the VMS server, or the WTS machines.

Let's Do It Again
The BMC implementation was the first WTS system that SPS implemented, but it wasn't the last. Gordon's group has since rolled WTS out to the other BMC factory sites, implementing NCD ThinSTAR 400 terminals as the standard client because those terminals have higher video resolution than the ThinSTAR 200s and support multimedia delivery (with the help of the ThinPATH tools). Several other Motorola factories also plan to convert to the WTS and NCD software solution for application delivery. As Gordon says, "The reduced bandwidth requirements of Terminal Server [WTS] let us administer the system either through Motorola's intranet or through dial-up or high-speed remote access—a huge benefit when supporting a 24 х 7 solution." Motorola is also using Citrix MetaFrame thin-client technology to deliver NT/UNIX interoperability to engineers who work from home.

According to Gordon, the thin-client user experience has been as positive as the administrator experience, and independent evidence suggests that he's right. During an airplane flight after researching this case study, I found myself sitting next to an engineer who works for Motorola. We got on the subject of the security folks' responsiveness to letting engineers access needed tools. "No problem," he said happily, "we get anything we need. For example, we just got this really cool capability where we can dial in from home and use our applications on our home computers—well, running on the servers at work, but I can see what's happening on my home computer. It's great."