Kitselas First Nation Improves Backup and DR After Disaster

When a burst water pipe flooded the small offices of Kitselas First Nation, the community panicked. The offices, which housed all of the small British Columbia nation’s files, were at risk. Everything the community needed to manage its finances, programs and services, such as healthcare and education, were stored there.

The flooding occurred during Labour Day weekend in 2019. The weekend started out as many holiday weekends do, with staff at home. Don Agnew, IT manager for the 700-person community, noticed on Friday afternoon that he couldn’t communicate with the servers. He assumed there had been a minor power failure. As the weekend continued without communication being restored, he decided to go to the facility on Sunday afternoon. He was greeted there by water that was inches deep in some locations. A plumbing pipe had burst, and the water had been on the ground for days. Agnew’s office desk and server rack were covered by the debris of the collapsed ceiling above them.

At the time, Kitselas First Nation’s only storage systems were onsite. The IT environment included a Windows backup on an external hard drive, which sat in Agnew’s closet at home and was later found to be malfunctioning.

Thinking quickly, Agnew grabbed the drivers from the servers so he could put them in a borrowed server to save whatever data he could. In the final accounting, Agnew was able to restore much of the data but lost about two months’ worth of health data and some virtual machines.

A New Backup and DR System

The flooding steeled Agnew’s resolve to get Kitselas First Nation’s backup and disaster recovery into better shape.

In addition to acquiring strong onsite and offsite backup, Agnew had to make sure that all data remained within Canada’s borders for regulatory purposes. He worked quickly, choosing to standardize on Scale Computing HC3, hyperconverged infrastructure software that combines servers, storage and virtualization. In addition to Scale Computing HC3, Agnew deployed Acronis technology for data protection, disaster recovery, long-term retention, threat mitigation and cloud backup.

Today, Kitselas First Nation is in a much better position to survive a disaster. Its five main sites now run on three Scale Computing HC3 units and 20 virtual machines. For more peace of mind, Agnew also installed a NAS drive in a building across the parking lot from its main building, where daily and hourly backups occur.

Primary offsite backup takes place at Agnew’s house, which houses an HC3. “I wanted one near me so if there was in incident, I could run a few VMs for critical functions like finance and take my time cleaning up the network,” he explained. Secondary offsite backup uses Google Cloud running Acronis.

The system backs itself up by taking periodic snapshots. Depending on the workload, snapshots might be taken every three hours or once daily. For good measure, the system also is set up to back up half of all data every Saturday to a Google Drive in Quebec. The other half is backed up on Sunday.

While the backup and DR system might seem like overkill to some, Agnew insists that it’s not. “Users don’t even realize when things are backing up,” he said. “And if I need to grant access for contractors, I can fire a VM up for them and eliminate it when they are finished. It’s so much easier.”

The Importance of DR Capabilities

Within the past six months, Agnew has seen firsthand why it’s so important to shore up disaster recovery. One of the other reserves in the area was hit by ransomware. The reserve had to pay $80,000 to get the data back.

To protect against these types of cybersecurity threats, Agnew is considering adding Scale Computing’s Business Resilience System running Acronis Cyber Protect Cloud.