Data Replication, Automatic Failover Drive University's Search for New Storage Solutions

For a relatively small university in rural Kansas, Pittsburg State University is amazingly progressive. Over the past several years, it has consistently worked to improve its facilities and academic offerings. Tim Pearson, assistant director in charge of IT infrastructure and security, has made it his mission to ensure that the technology also keeps pace, including--but not limited to--searching for new storage solutions with a current focus on automatic failover and data replication.

Most recently, Pearson set his sights on upgrading a pair 8-year-old EqualLogic SANs, which the university was using for both backup and storage at two sites on campus. While the system was still functional, Pearson had started to notice a degradation in performance. The issues typically involved bugs in the firmware that caused the SAN system to perform poorly or required restarting. There were times that the university’s systems were offline for several hours at a time—something he considered unacceptable. Pearson also said the quality of the support had become an issue.

In his search for new storage solutions, Pearson wanted to move away from time-based replication to synchronous replication to ensure multiple, up-to-date copies of data. He tried making it work with the EqualLogic storage arrays, but they simply didn't have enough horsepower to handle the task. Until he could find a replacement, Pearson had to revert to time-based replication by dividing the virtual machine workload into three categories: data that changes frequently, data that changes once in a while, and data that almost never changes.

The EqualLogic arrays continued to perform well with backups. Each night, the system would perform compressed deduplicated backups on two separate arrays in two different buildings on campus. A third site at sister university Wichita State--connected to PSU via a robust, reliable, low-latency network--houses the third backup copy.

But when Pearson began to see the spinning disk failure rate increase, he knew it was time for something new.

“The arrays got to the point where I was scared to death that we were going to have spinning disks fail in rapid enough succession that the redundancy scheme on the SAN wouldn’t be able to tolerate it and we’d lose data,” he said.

Upgrade Specifications

Pearson assembled a team consisting of network administrators and engineers, system administrators and data center administrators to develop requirements for new storage. The most important features were reliability and robustness. More specifically, the team wanted to be able to replicate data in real time with automatic failover between on-campus locations, at  minimum, and all the way to Wichita State if possible. The requests for proposal (RFP) required external iSCSI or NFS connectivity to enable PSU’s other stand-alone Unix and Linux hosts to use the new storage.

Because the university’s IBM BladeCenter blade servers were reaching end of life, Pearson wanted to find a way to include new compute on the RFP, as well. The specs for the compute layer requested enough horsepower to run critical workloads in either of the university’s two data centers.

As a public university, budgets are always tight, so Pearson didn’t know if he would have enough money to get the computer power he needed along with the storage capabilities. After some research, his team came across federal grant money available for universities that educated nurses in rural areas. PSU has a robust School of Nursing, so it was able to join with other universities in a similar situation to apply for and receive the grant.

Eventually, the team settled on the Hedvig Distributed Storage Platform running on HPE Apollo 4200 servers. After running the numbers, the team chose to implement the system in its own data centers instead of opting for a cloud-based system. The system was installed this past spring.

With the new system, all three locations—the two on campus and one at Wichita State—have nodes consisting of two HPE Apollo 4200 servers loaded with spinning disks and solid state drive storage. Wichita State also has additional compute capacity so it can run critical functions like DNS, domain controls and Active Directory.

“That way, if our campus were to become isolated from the internet completely, we would still be able to run critical services, such as the cloud-based learning management system our students rely on,” Pearson said.

For now, Pearson still performs traditional backups on the original EqualLogic arrays. However,  he said, when the university’s support contract is up next year, he will find a new way to ensure reliable backups. Hedvig’s snapshot scheduling and maintenance tool, currently under development, will be a consideration, but Pearson said he is not sure which way he’ll go yet.

“We’ve had some input into the tool, so we’re interested to see where Hedvig takes it, but it will depend on whether or not it’s mature enough at the time,” he said. “If it is, we may forego traditional backups in favor of Hedvig’s point-in-time snapshots replicated to the three locations.