For Brown Bag Films, Reliable Storage Isn’t Child’s Play

Remember "The Magic School Bus?" Of course you do; it was probably part of your childhood. It’s back in production today with new episodes. But producing the wildly popular kids' TV show--which wraps science lessons in an adventurous, animated story--isn’t child’s play. It requires a lot of bandwidth, a lot of scalable storage, and, most importantly, reliability and manageability. Brown Bag Films, an animation studio based in Toronto that produces The Magic School Bus: Rides Again and many other shows, knows this well.

Until recently, each post-production team had its own storage setup directly attaching its Mac Pro or high-end PC to an eight bay-SCSI array running RAID 5. While the performance was acceptable, it had other problems. An unfortunate situation a few years ago brought that point home when one of the arrays overheated to the point where it degraded the drives, causing two drives to fail simultaneously. Although it could have been worse, the staff lost three or four days of work.

That incident was the last straw, causing the company to look for a more reliable storage system—one that provided better visibility, manageability and disaster recovery capabilities.

“The storage we had up until then was disparate and hard to back up, so we wanted a solution that would give us the performance of a direct attached array,” explained Chris Brown, Brown Bag Films' senior systems engineer. “We also wanted to centralize the storage back at the data center so it would be accessible to every edit team.”

After considering and rejecting block storage, object storage and network attached storage, Brown settled on Qumulo File Fabric (QF2) for its flexibility, scalability and features like continuous replication and analytics. 

Today, Brown Bag Films uses QF2 in both its production and post-production workflows. It runs eight QC104 units, each providing 104TB of raw HDD capacity and 6.2TB of raw SSD capacity, in a cluster. The clusters are connected to each via a 40 gigabit connection. The post production staff connects to the storage via 10G fiber cabling to each desktop, while the production system is attached by a RJ45 1G copper link.

It’s much easier to monitor performance and pinpoint potentially troubling issues with the new solution, Brown said. While the previous RAID array-based system did provide rudimentary email alerts, they often came too late. In one case, Brown said, by the time he got an alert, two disks had already failed.

The Qumulo cluster also includes built-in analytics that provide Brown’s team with critical information.

“If a user is complaining of slow speeds, we can quickly see how many IOPS he’s using and his throughput speed, and troubleshoot the problem quickly,” he explained. Brown says the analytic capabilities also are helping the team better predict capacity requirements for future productions by analyzing the capacity and throughput of current productions. Those numbers then help the company better forecast the budget for individual productions.

The QF2 solution also addresses one of the most important requirements of the entire project—avoiding data loss.

Before moving to the new system, Brown Bag Films didn’t have a traditional backup system to disk or tape. Instead, it relied on incremental backups. Brown was looking to change that by moving a solution that provided a backup option as well as a disaster recovery option. That made the continuous replication feature of the Qumulo solution a game-changer. With this feature, the company maintains a primary cluster in Toronto, with a secondary cluster in a disaster recovery site a bit to the north of Toronto. The secondary cluster runs QF2 on four HPE Apollo 4200 Gen 9 servers that hold about 450 TB. The system replicates all activity to the second site, which is configured as a read-only device unless it’s needed. 

“If we lost our total facility, our cluster in our DR site could be used immediately as a primary cluster and we could have it back online in minutes,” Brown said.

Just in case, Brown’s team also takes hourly snapshots of all production and post-production workflows, along with weekly snapshots.

As the business continues to grow, Brown says he can see transitioning from the typical 2K projection into 4K or even 8K projection. While that move would clearly benefit Brown Bag Films' clients, it would also greatly increase performance requirements.

“I can see us creating new storage workflows that incorporate more flash disk and getting that throughput higher,” he said. “But our Qumulo cluster is very scalable, so we can just add nodes on the fly.”