Cloud Journey: Moving a Complex Analytics App into the Cloud

Moving workloads into the cloud is sometimes simple. If you can lift and shift your on-premises server into an IaaS compute platform, good for you. In other situations, migrating a workload to the cloud is much more challenging. For example, if your workload involves multiple layers of infrastructure and software, or requires a nuanced and complicated provisioning process, your cloud journey may be much harder.

Bill Schneider, senior principal engineer at Optum, recently discussed his team’s experience addressing these challenges in a session at Interop 2020 titled “Our Cloud Journey Migrating a Legacy Reporting Application to AWS.” Schneider explained how he and his colleagues moved an on-premises analytics app--or most of it, at least--into the AWS cloud, why they chose the cloud architecture they did, and which benefits they’ve enjoyed since moving the workload off-premises.

The App: A Healthcare Data Analytics Tool

The app that Schneider and his team develop and manage, known as OPA, helps Optum’s clients manage healthcare data. It allows them to find and analyze patient data and correlate it with factors like healthcare prices and the quality of care received. The app is a public-facing resource that is used by Optum’s clients, not just by internal Optum employees.

Originally, OPA was an on-premises app that consisted of four main components. The first was flat files that stored raw data created or collected by Optum. From there, the data passed into an Oracle data warehouse, where it was processed and restructured. After data transformation completed, the data moved to a data mart, which was also supplied by Oracle. Finally, a MicroStrategy analytics application pulled the data from the data mart in order to process user queries.

Transitioning to the Cloud

Today, OPA runs almost exclusively in the cloud. There is still an on-premises component because the data pipeline begins with data stored locally, but that is uploaded into AWS S3 buckets, where data is stored before being ingested into AWS Redshift. From Redshift, the data can be queried by MicroStrategy application instances hosted in the AWS cloud.

Traffic to the entire cloud-based pipeline is managed via a single Elastic Load Balancer (ELB), which routes traffic to different MicroStrategy instances. By having just one ELB, Schneider and his team are able to simplify certificate management. They also avoid having to update DNS records when they add a new client for the OPA app.

Why the Cloud Journey?

The on-premises architecture that originally powered OPA was subject to several limitations. Beyond the cost of licensing Oracle software, Schneider said, the app suffered from scaling challenges because it is “heavily dependent on in-memory caches to make the reporting responsive to end users.” As a result, to support additional clients, Schneider’s team would have had to add more servers to make more memory available.

Migrating OPA to the cloud offered a solution to both of these challenges. By using AWS Redshift instead of Oracle’s data warehousing and data mart tools, Optum reduced its software licensing costs. At the same time, the cloud makes it much easier to allocate more infrastructure resources to application instances in order to ensure adequate performance, even in situations where the app is handling as many as a billion rows of data.

A third advantage that emerged out of the app’s migration to the cloud is fully automated orchestration. Schneider and his team implemented AWS Step Functions to call code hosted in AWS Lambda to handle provisioning and management tasks. “Each step function represents a high-level process, like ‘do a data look-up’ or ‘deploy a release,’” Schneider explained. “Those step functions then call a series of Lambdas, each of which does an independent task,” like setting up AWS infrastructure or security groups.

By allowing application instances to deploy automatically, this approach virtually eliminates the need to spin up new application environments manually. Schneider said that this represents a tremendous improvement in efficiency over the on-premises version of the application, for which the team used IaC tools, like Puppet, to orchestrate some management tasks but otherwise depended on manual workflows.

Cloud Journey Challenges

OPA’s journey to the cloud was not without some bumps in the road. One key challenge, Schneider said, was finding ways for app developers to connect securely to the cloud-based environment from the on-premises workstations where they did development work. Optum originally had to create special firewall exceptions for this purpose, but now takes advantage of AWS’s SSM service, which lets engineers execute commands on machines in the cloud over HTTPS.

MicroStrategy’s application, which came into being long before cloud computing became the norm, was also not natively designed to run as part of a highly dynamic application stack. That created some hiccups, Schneider said, but his team was able to collaborate with the vendor to resolve them.

Technical debt within OPA, too, has become a more pressing challenge for Optum since moving the application to the cloud. In the cloud, technical debt not only poses performance challenges, but also comes with direct cost implications. Going forward, Schneider said, a key priority for his team is to continue to find and fix technical debt with OPA to make sure that the app is as efficient as possible from a cost standpoint as well as a performance one.

Conclusion

The cloud isn’t just for simple legacy apps that you can lift and shift directly into a hosted VM. Nor is it only for cloud-native apps that are born in the cloud from the start. As Optum’s story shows, you can take complex, multi-layered on-premises applications and move into the cloud in a way that allows you to take full advantage of the cloud’s scalability and automation features.