Eli Lilly Automates Its NT Performance Monitoring

Making the most of the support staff's time

Eli Lilly and Company needed a way to automatically monitor the performance of its fast growing Windows NT environment. "Our NT environment had grown so fast, but unfortunately our support staff numbers weren't keeping up," said Eric Frash, NT systems analyst, from the company's Indianapolis, Indiana, headquarters. "We had little time to monitor performance and do capacity planning, and we were constantly in firefighting mode. Problems arose and management asked how performance had been last month." Explosive growth was just one reason Frash and the NT administration group implemented an automated performance monitoring solution (for Frash's thoughts on this project, see the sidebar, "An Interview with Eric Frash").Founded in 1876, Lilly is among the best known pharmaceutical firms in the world. With more than 28,000 employees worldwide, the Fortune 500 company sells its products in 150 countries.

The performance of Lilly's enterprise systems is crucial to the company's continuing success. Lilly's enterprise computing environment is a mix of new and legacy systems running in multiple locations over heterogeneous LAN and WAN network topologies. The Lilly NT administration group manages about 10,000 NT accounts in North America, South America, and Puerto Rico. The enterprise typically has between 400 and 800 server connections running simultaneously via drive maps and at least 200 open files at peak usage. A typical NT server at Lilly is a Compaq ProLiant 4500 or 5000, with 128MB of RAM, two internal 2.1GB drives, and a RAID-5 array. About 80 servers run NT Server 3.51, Service Pack 4. Lilly's enterprise management toolbox includes Avalan's Remotely Possible/32, Microsoft Systems Management Server (SMS), Microsoft Perfmon, the Microsoft Resource Kit 4.0, and HP OpenView.

Lilly considered building custom performance monitor solutions inhouse and using other vendors' tools. Frash scrutinized several automatic performance monitoring, alerting, and event management solutions. In the end, Lilly chose Heroix Corporation's RoboMon NT (for information about RoboMon NT, see the sidebar, "Automated Performance Monitoring with RoboMon NT"). "We chose RoboMon NT because of its established performance rules for NT, its cross-platform compatibility, and its long history," Frash said.

After deciding on RoboMon NT, Frash mapped out a two-phase implementation plan. He began by selecting seven servers to monitor with the software. "After we identified the problem-child servers, we installed RoboMon NT on seven of our most heavily used servers and on the desktops of our tier 2 and 3 support staff," said Frash. "Our support staff can use the RoboMon Event Monitor to receive important information about how a server is behaving."

The NT administration group then configured RoboMon NT to take an hourly performance snapshot of each server and concatenate the data in a file that is closed out at month's end. The software runs in an alert-only mode, and the support staff acts immediately on the alerts it receives. Now, the support staff can analyze the data to fine tune the configuration, chart historical performance, and use solid information for capacity planning. Frash and his team can also implement a second phase of deployment for writing custom rules, testing the automatic fault correction features, and integrating RoboMon NT with other tools, if needed.

In addition to answering the corporate need to collect and archive historical performance data, the software brought other, more immediate benefits. Lilly's enterprise management is now more proactive than reactive. The support staff can identify and resolve many problems before they become issues to users. For example, RoboMon NT has identified memory leaks in certain services and charted memory usage. According to Frash, the support staff has used performance data to determine whether a particular server requires additional RAM. Frash can also identify over-utilized servers and distribute the load from these servers to other servers. The support staff can act on disk space availability alerts and contact users to advise them to delete or archive unnecessary files. However, RoboMon NT has not supplanted judicious use of NT's Perfmon. Frash explained that administrators continue to use Perfmon on a case-by-case basis to perform detailed investigations.

Lilly's NT administration group is in Phase 1 of the RoboMon NT implementation. "We have customized some of the built-in rules, and we plan to write our own as we get more familiar with the tool," Frash said.

The jury is still out concerning how the product will ultimately contribute to enterprise management, but Frash is confident it will play a significant role. "Any tool that frees up our valuable time is a good thing."

SOLUTION SUMMARY
Eli Lilly and Company faced explosive growth in its Windows NT enterprise computing environment and an insufficient number of support staff to manage it. The company needed to automatically monitor the performance of this environment. Lilly considered building an inhouse solution, but ultimately decided on Heroix Corporation's RoboMon NT. The software hasn't replaced the use of NT's Perfmon but lets the support staff proactively respond to problems.

RoboMon NT lets NT administrators take hourly performance snapshots of seven NT servers and archive the performance data in a monthly log. The software runs in alert-only mode and notifies the appropriate personnel when it encounters a problem. Now the support staff can respond to problems as they occur and use the historical information that RoboMon NT provides for capacity planning.

Lilly is looking for ways to further implement RoboMon NT in its NT environment. The company's NT administration group plans to customize some of the software's built-in rules and write rules to better fit the company's needs.