IT Modernization Project Taps Red Hat Ansible, Hashicorp

With a mission to help countries of the Asia-Pacific region tackle poverty and build sustainable infrastructure, the Asian Development Bank relies as heavily on technology as it does on its thousands of employees.

Since its inception in 1966, ADB has continued to push the boundaries of how its workforce engages technology. That led to a decision in 2016 to improve mobility and cloud-based collaboration – an effort that would be appreciated when the COVID-19 pandemic hit in 2020, forcing a shift to remote work.

ADB’s initial IT modernization project involved a transition from Lotus Notes to Office 365 for collaboration, the replacement of desktops with laptops, and a switch from BlackBerry to iOS devices. The organization also replaced its secrets engine, Lotus Notes ID Vault, with the open-source version of HashiCorp Vault for storing passwords, API keys, tokens, and certificates.

After completing the project, ADB leaders realized that employees continued to struggle with collaboration, data management, and general efficiency. A second, larger IT modernization initiative was launched a few years later. “We generated about five Eiffel Towers a year in paper consumption,” said Krista Camille Lozada, ADB’s senior IT specialist for innovation and engineering. “We are a development bank, but we couldn’t walk the walk.”

Related:Technical Debt and Modular Software Architecture

Then the pandemic hit, which put ADB’s capabilities to the test. While the organization adapted to many of the challenges, employees still experienced plenty of issues. Spotty connectivity, network slowness and latency, security, and a complex disaster recovery situation complicated everything.

Luckily, ADB reacted quickly. The bank tripled its investment in IT modernization from $6 billion to $20 billion practically overnight.

ADB’s had ambitious goals for its new IT modernization project:

Fast-tracking Automation

The IT modernization project involved automation improvements.

After deciding to standardize on Microsoft Azure, ADB’s first step in advancing its automation capabilities was to settle on the right tools. The team settled on three systems: the agentless Red Hat Ansible IT automation engine, HashiCorp Packer for building VM images, and HashiCorp Terraform for developing and manipulating infrastructure.

All three tools have roots in infrastructure as code, a method that manages and provisions infrastructure through code via a graphical user interface instead of through more manual processes. Adopting these types of tools was important, because they foster collaboration and reusability, Lozada said.

“We wanted to write our code in a way where everything is modular,” she explained. “For example, in Terraform, there would be one module representing one piece of infrastructure or resource, so there is one declarative module to say, ‘This is a SQL Server in Azure,’ or, ‘This is a Windows VM in VMWare.’ The modules are like Lego blocks.”

At the same time, the team opted to upgrade from the open-source versions of its HashiCorp tools to the enterprise versions, which offered broader features and support. Terraform Enterprise, for example, lets developers ensure that all Terraform runs use ADB modules. The modules have all the security protocols already embedded in them, Lozada noted.

Terraform is also an important infrastructure-building tool for ADB. Everything the organization needs to spin up is done in Terraform. The team uses Ansible for configuration management and other IT automation tasks like application deployment and orchestration. Lozada put it this way: “If ADB was building a restaurant, the construction workers would be Terraform and the people running the restaurant would be Ansible.”

As the project came together, the team began to see benefits from how automation built on itself. For example, the IT team built a playbook detailing the care and feeding of servers, but when the security team adopted security orchestration, automation, and response (SOAR) to identify zero-day vulnerabilities, the IT team had a brainstorm. “Since we already had a mechanism for patching servers, and the SOAR team now wants us to patch servers when it has identified something, why not have SOAR automatically remediate it by calling an Ansible API?” Lozada said. “We already did the work, so it was an easy call.”

With automation efforts under control, it was then time to pursue a serious move of ADB’s data centers to the cloud. While ADB had originally planned to set up a new data center using Azure in the Singapore region closer to its Philippines headquarters, the spread of COVID-19 demanded a different strategy. The new data center is virtual, based on Azure and built by ADB using Terraform, Ansible, and Microsoft Azure DevOps Server. While complex with more than 2,000 assets, the virtual data center was up and running in less than five days. The ADB team had all assets configured in less than 45 days.

In addition, the data center lets users spin up a new VM in less than two minutes.

Disaster Recovery Goes to the Cloud

ADB has used complicated disaster recovery processes that produce unsatisfactory results. For example, the team runs DR procedures or activations only twice a year, but it’s never seamless because dependencies or updates had not been done correctly. “There were always bits and pieces missing because we forgot something,” Lozada said. “But the biggest issue we had is that because knowledge is siloed, one team may have updated something, but other teams would have no way of knowing that.”

The team decided to standardize, automate, and centralize its disaster recovery processes, all without compromising security or resiliency. The project started by selecting a secondary site. It was timely, since ADB’s Manila-based headquarters is overdue for an earthquake. As a temporary measure before moving DR to Azure, the team chose a site in Valencia, Spain. And instead of relying on a vendor-developed DR offering, the ADB team chose to develop its own, relying on techniques like data replication. Eventually, all disaster recovery will be in the cloud, using cloud-native, immutable infrastructure.

In addition to moving disaster recovery to the cloud, ADB’s IT modernization project seeks to improve data management, governance, and storage. “We know we have the data, but we repeat the same mistakes again and again because a lot of the knowledge is left in somebody’s inbox and we never find it,” Lozada said. “There is no concept of the single version of truth.”