How DevOps Practices Failed in the Face of 2020’s Crises

The DevOps approach to software delivery has been around for more than a decade. A majority of organizations have embraced DevOps practices, and 60% report continuous deployment rates of multiple times per day. You might think, then, that the typical organization today is well positioned to deliver software early and often, and to operate in an agile fashion in response to major challenges. The experience of 2020 suggests otherwise.

2020 ushered in a series of crises--including but not limited to the pandemic--that posed unprecedented DevOps challenges. And while some DevOps teams rose to the challenge, the results were generally lackluster. Most organizations struggled to deliver new software quickly and fix problems fast, which casts doubt upon the extent to which DevOps practices have really taken root within the typical organization.

2020: Three Major Challenges

During the course of 2020, development and IT teams faced three major types of challenges that, in theory, DevOps practices would have helped to solve.

1. COVID testing and vaccination

The most obvious crisis was the COVID-19 pandemic, and the new applications that organizations were forced to push out in response to it on short notice.

In the spring of 2020, developers across the world were tasked with building apps that would help schedule COVID tests. The results were not exactly impressive. The most famous effort, which was a site envisioned by Google’s sister company, Verily, barely materialized and never scaled in the way its backers promised. Many state governments eventually got their acts together and built relatively user-friendly COVID testing hubs, like California’s and New York’s, but these mostly didn’t mature until several months into the pandemic.

That pace doesn’t reflect the velocity that DevOps practices are supposed to enable, especially when dealing with a relatively simple app that reports testing availability. These apps are not trivial to build, to be sure. But it’s not as if they are tracking shuttles flying into space. Building an app that reliably tracks testing sites and directs people to them shouldn’t take months if you are an agile organization that understands DevOps and continuous delivery.

More recently, developers at government agencies and private companies have rushed to deploy apps for coordinating vaccination appointments. Many of these have also not worked as well as one would hope. From Florida to California, they have been prone to bugs, adding another hurdle to the already problematic rollout of vaccines.

The good news is that the apps have, at least, shown signs of continuous improvement. Take New York State’s vaccination app, for instance, which was an SRE’s nightmare when it first appeared in mid-January: Pages took minutes to load, and the site spewed error codes all over the place. Within a few days, however, it was clear that developers had begun pushing out new versions with performance enhancements. For example, whereas the app originally displayed full details for every vaccination site, even if the site had no open appointments, a later release limited the display to sites with availability appointments. By reducing the amount of content that the app needed to serve with each request, this change seemed to improve performance by quite a bit.

2. Unemployment software platforms

The sharp increase in unemployment that resulted from the pandemic foisted another sudden challenge upon developers and DevOps teams: The need to handle unprecedented levels of traffic for online unemployment systems.

Engineers had an advantage on this front in that the apps already existed. They didn’t have to build them from scratch, as they did with most COVID-related apps.

Nonetheless, the surge in demand for unemployment websites caused many to crash, often for extended periods. These technical issues led to severe real-life disruptions for jobless people counting on unemployment systems to see them through a period of personal financial hardship.

Over time, matters improved, but only somewhat. The unemployment sites of states like Michigan and Kansas were still crashing sporadically as of 2021, which suggests that the developers and IT teams behind them have not risen to the challenge as well as one would hope. In a perfect world, DevOps teams would have used the experiences of the past year to figure out why unemployment apps can’t handle volume consistently and fix them, but that has not happened.

3. Voting software

For voting software platforms, too, 2020 was not a banner year.

The most notable fiasco was the app that Iowa Democrats used to report caucus results in February 2020. Apparently due to a lack of testing, the app struggled to report data completely.

When you’re deploying an app that will see massive traffic during its first day in production, and you can’t really count on being able to fix issues that may arise during that period, you’d ideally use DevOps techniques like shift-left testing to make sure the app works reliably. But that doesn’t appear to have happened.

And then there was the controversy in November over Dominion’s voting software. It’s hard to chalk that up as a failure of DevOps, especially because there is no evidence that the software actually failed to do its job. Still, perhaps one takeaway here is that maintaining an audit trail and continuous visibility is critical. Without those, it’s hard to convince regulators and users to trust your app.

Conclusion

What does all of the above say about the state of DevOps practices? In short, despite all the hype and celebration of DevOps’ success, DevOps has a long way to go to deliver the real-world results that it promises in theory.

To be sure, not every app that developers rolled out or scaled up in response to the crises of 2020 was a failure. Some worked reasonably well, and others improved over time. Yet, the fact that things generally didn’t go well is a sign that organizations still struggle to deliver software with the velocity and agility that DevOps is supposed to enable.

Admittedly, many of the apps described above were built and maintained by government agencies, which aren’t exactly known for being agile or cutting-edge from a technological perspective. But when even a parent company of Google struggles to release an app fast, you have to wonder whether DevOps practices are really delivering.

2020 would have been an even worse year if DevOps didn’t exist. But there remains a wide gap separating DevOps in theory from DevOps in practice, a reality that has become all too clear in our present era of cascading crises.