The Pandemic Tests the Limits of Internet Infrastructure Supply Chains

One of the many things the COVID-19 pandemic has exposed has been the unpreparedness of supply-chain strategies for a crisis so prolonged and of such scale. That applies to digital infrastructure.

Thanks to their early focus on automation and a discipline of maintaining capacity well ahead of demand, operators of some of the largest web-scale networks have had no trouble absorbing the massive increases in traffic driven by stay-at-home orders around the world. But during the recent weeks, they’ve gotten more up close and personal with the limits of their hardware supply-chain strategies than ever before.

Hardware lead times have been greatly expanded, and companies have been adjusting their procurement strategies on the fly to ensure they have enough capacity in their data centers to remain ahead of demand.

The men in charge of network infrastructure for Verizon Media, Uber Technologies, and Akamai Technologies discussed supply chain issues they’ve been facing Tuesday in a webinar organized by Kentik, a San Francisco-based provider of network monitoring tools.

Verizon Media Adjusts to Longer Lead Times

The infrastructure team of Verizon Media, the Verizon Communications media business whose brands among others include Yahoo!, TechCrunch, Engadget, AOL, and HuffPost, uses both custom and off-the-shelf hardware.

But its data center equipment lead times have gone up across the supply chain. What used to take two weeks to get now takes four, six, and in some cases 16 weeks, Igor Gashinsky, Verizon Media’s chief network architect, said.

In response, the team has been looking to expand the pool of vendors for various parts of its infrastructure while also planning for longer timeframes.

“We definitely see if we have [vendor] alternatives and if they’re easy to use,” Gashinsky said. “We just extend our planning cycle.”

Instead of its typical approach of staying three months ahead of demand, the buffer is now six months. “That’s just what you have to do,” he said.

Uber Puts a Magnifying Glass to Its Supply Chain

The network infrastructure team at Uber, which has seen a steep drop in usage of its ride-hailing app accompanied by a spike in usage of its food-delivery service Eats, is also seeing delays in the hardware manufacturing process, Jason Black, head of global network infrastructure at Uber Technologies, said.

“Anything coming out of Asia, I’m seeing that there is a slowdown,” he said.

The response “really does come down to building ahead.” He’s been having weekly calls with parts suppliers across the supply chain, “down to the chip level,” to understand its status.

Uber uses a hybrid-cloud strategy, relying on a combination of public cloud and its own data centers. The company uses custom white-box servers in its data centers, but it’s also now working on its own white-box network switch design, Black said.

“I do look across the entire supply chain to see where the holdups are,” looking to onboard additional vendors for some components.

The hybrid-cloud strategy has been helpful in preventing the sluggish supply chain from affecting Uber’s ability to support the changes in demand. The company’s been able to shift workloads from on-prem data centers to the cloud when it’s not able to add data center capacity quickly enough, Black said.

Akamai Gains Greater Understanding of Its Supply Chain

Akamai Technologies, which operates one of the world’s largest content delivery networks – it has points of presence in about 4,000 locations across 1,000 cities – has also seen delays in its compute and networking hardware supply chain.

One positive impact has been that the team now understands its supply chain better than before, Christian Kaufmann, senior director of network services at Akamai, pointed out.

Like others, his team has been discussing various mitigation scenarios with its vendors. For some components, the answer has been to build a bigger on-hand inventory, and for others to find more vendors.

Outside spending more time in meetings “it’s not particularly bad,” Kaufmann said.

Now, as China is starting to open back up, the supply chain issues are “getting slowly better,” he said.

More Transparency and Shorter Chains Needed

A change in supply chain strategies will likely be one of the pandemic’s big structural outcomes for the digital infrastructure industry – and other industries.

“There’s definitely going to be a lot of lessons learned in supply chain management,” Verizon’s Gashinsky said. The needed changes that are clear already have to do with transparency and length of the chains.

“We definitely need more visibility in the supply chain,” he said.

Supply chains will also have to be “shortened,” Gashinsky said. Getting a piece of hardware made involves so many interdependencies – where different parts are manufactured in different locations (even different countries) and product shipped back and forth for assembly – that even a small snag at one stage of the process can cascade throughout the chain.