Case Study: Mission-Critical Scalability

A Boston-based medical group trusts Caché to save lives

For hundreds of years, physicians have scribbled notes and orders on paper. Partners HealthCare System, a group of Boston hospitals and medical offices, wanted to computerize physicians' medication and treatment orders to modernize this tradition.

Problem
The updated system had to be able to make cost recommendations and provide doctors with warnings based on a patient's medical history. The system needed to be able to track doctors' locations and notify them about patients in critical condition, so the system had to be available 24 X 7. Partners needed the system to save lives and improve customer service.

Evaluation
InterSystems' Caché was automatically a front-runner in Partners' search for a relational database solution because one of Partners' member hospitals was already using an InterSystems database. "We knew that InterSystems' technology has a proven track record in clinical environments," said Steve Flammini, director of application development for Partners. "We looked at Oracle, and although it can probably scale, it isn't as cost competitive as Caché. In the healthcare arena, the data models are complex and lend themselves to an object-oriented style rather than a relational style. The number of tables and joins we require is a serious challenge to a database solution, and Caché is more elegant than Oracle in defining complex data structures.

"We use Microsoft SQL Server for non-mission-critical data because Microsoft's products integrate fairly easily," continued Flammini. "But when we require scalability and transaction processing for high-availability, mission-critical data in complex data models, we use Caché. Caché is ODBC compliant, so it supports relational tools."

Solution
Partners implemented a multi-tier client/server architecture for all the applications that the organization develops inhouse. This architecture lets Partners slim down the client side of the inhouse applications and gives Partners the ability to run more thin clients from one Windows NT Server 4.0, Terminal Server Edition system. Partners uses Compaq as its primary server vendor, and each NT-based Caché database server provides between 45GB and 100GB of storage. In the middleware tier, applications run business and clinical rules, such as retrieving patient medical records and discovering clients' medication conflicts. The Caché middleware scales well from doctors' offices to hospitals, and performance is excellent.

Partners' Caché-based applications run on 60 servers. This setup takes advantage of the Caché database server's key capability to partition and load balance across multiple servers. Caché achieves this capability by creating a map of database partitions that resides in each Caché server's memory. For example, if a client connects to and queries server A, the server consults its database map to answer the question, Who has the data the client is looking for? If the required data resides on a partition in peer server C, the map arbitrates a request to server C to acquire the data and pass it through server A to the client destination.

In addition, Partners has implemented multiple 2-node clusters for database servers. Each of Partners' servers has a corresponding shadow server that provides full failover clustering. The shadow server contains a complete copy of the database partition, which lets administrators perform database maintenance, such as defragmentation and realtime backups, without taking down the server. Administrators can fail over the primary server to the shadow server, defragment the primary server, then failback the primary server. The primary server synchronizes the journals from the shadow server and, just minutes later, the primary server is up and running.

Results
InterSystems' Caché technology let Partners roll out a physician order entry, which is Partners' primary medical record system. This technology also let Partners create a high-performance Web application that provides a centralized clinical data repository of lab records, which clients can see as one view. In addition, Partners used Caché to create an enterprisewide directory and paging application to quickly notify a doctor about a critical patient situation.

"We've been rolling out computers and users by the yard," Flammini said. If Partners' DBAs run out of space on a server, they can dynamically move the data to another server. "We've moved data to a new location, recompiled the database map, let the servers synchronize database maps, and we're up in 7 minutes. We move data live, which dramatically reduces downtime."

Partners' administrators partition data in 2GB to 16GB chunks for easy patient record recovery. Caché's partitioning capability isn't automatic—the technology relies on the DBA to determine how Caché partitions data. For example, if you have 3 million patient records, you can partition records 0 to 999,999 on server A, records 1,000,000 to 1,999,999 on server B, and records 2,000,000 to 3,000,000 on server C. Or you might partition the records by date, using higher disk densities and fewer spindles.

Future
"Although we can scale to more than 80 servers, our network isn't a lot of fun to set up and manage. We want to consolidate the data onto 15 to 18 servers, and we want to maximize I/O throughput for interactive transaction work. We're well over the 5000 simultaneous-users mark," said Ethan Fener, Partners' manager of systems programming. Partners plans to centralize the 60 servers' storage onto a Storage Area Network (SAN). This setup will consolidate backups of 60 separate DLT drives into one tape library attached to a SAN and save administrators a considerable amount of time.

In addition, Partners plans to use thin-client/server technology to manage application deployment. "Refreshing a new application on 26,000 PC desktops is ridiculous. We'll be using Terminal Server in a huge way," Fener said.