UK Data Center Operators Say Prolonged Lockdown Would Risk Outages

Data centers keeping the economy running through the lockdown might start to break down if they keep the skeleton crews they adopted to deal with the coronavirus outbreak for long, industry representatives in the UK have warned.

Fear of systems failure possessed data center operators late in March, as they paired back crews and suspended maintenance regimes built over decades by an industry obsessed with making sure digital services would always be available no matter what disasters befell the world around them.

The rest of the country came to depend on that digital infrastructure suddenly and almost universally, just as the crisis forced those running it to adopt emergency measures themselves to stop the virus that causes the disease COVID-19 spreading among them.

Running on skeleton crews, they had cut any maintenance not deemed critical. Thursday, as the government extended its lockdown another three weeks, and after officials predicted the national economy would slump 35 per cent another three weeks after that, the data center industry was asking how long it could carry on before the nation's data infrastructure started to fail.

Emma Fryer, associate director for data centers at techUK, an industry association data center operators have rallied around, told DCK that, having concluded that the worse might happen, they had started trying to determine when.

"If you are reducing maintenance to essential-only, then there's going to come a point when you need to do it," she said.

"If you don't put oil in your car at the right time, you can stick it in a month later, and it will make no difference. If you leave it a year, until you've driven 25 thousand miles, your car's going to go very badly wrong. It's just obvious," Fryer said.

Skeleton Crews

Typical data centers, and those that house communications hubs on the backbone of the country's internet, have been running with skeleton crews – up to two-thirds fewer staff than usual – according to a preliminary analysis tech UK published last week.

"It is unlikely that this can be sustained long term without increasing risk of outages or operational failures," the report said. It was based on a survey of 30 companies operating 100 data centers across the country.

On Tuesday, in its most recent analysis, the group said the greatest risk in the short and medium term was that skeleton crews could lead to systems breakdown.

Operators that housed their administrative offices in the same buildings as their data centers sent about 93 percent of their staff home. Operators that spoke with DCK said their network and computer systems administrators were doing their work remotely, from home, as were their sales, marketing, finance, and management.

"We've been able to successfully pivot to having about 85 per cent working from home,” Jack Bedell-Pearce, CEO of 4D Data Centres, told us. "We've still got staff on site 24-7. But that really is the bare minimum required to keep the data center open and to make sure all the facilities are kept running.

"A lot of companies are putting a change freeze in with regard to IT infrastructure. So, a lot of change projects have been put on hold. Just so everything can remain stable.

"In the long term this is not sustainable because equipment needs to be upgraded, and businesses will grow or contract, and they have to change their IT systems accordingly," said Bedell-Pearce.

Finding New Ways to Work

The engineering director of a global data center operator, who asked not to be named, said data center operators were being forced to put some construction projects on hold, even as customers demanded more capacity to deal with their own floods of online users.

The lockdown is not sustainable for data centers, just as it’s not sustainable for the whole economy, he said. Yet, industry had feared for worse.

"There was a little panic in the first couple of weeks of this that we weren't going to be able to cope,” the engineering director said.

"That may well have been correct a couple of weeks ago, but the resolve of people has come to the fore in realizing there's no quick way for us to get around this,” he continued. “We can't just work it out and go and fix whatever the problem is. We have to work out another way of doing it.”

He, like Bedell-Pearce and other data center operators, said the remote-working methods they had introduced and the reduced foot traffic would lead to permanent changes in business practices when the crisis passed. Many operators had stopped the customers who kept computer systems in their data centers from making anything but emergency visits to tend to them. They were all doing more remote-hands services.

Zahl Limbuwala, executive director of strategy at CBRE, one of the UK's largest data center outsourcers, said he had accelerated the company's plans to automate operations at its facilities.

"The pandemic has given everyone the realization that things like remote access of building management systems and so on is an important thing they should have, not just when access is challenged, or personnel numbers are challenged, but in general. It could help not only in this situation but also in cost reduction," he said.

But business is slowing for those whose customers can’t operate during the lockdown.

Redcentric, a managed services provider, told investors on April 3 that profits were going up and it had been doing emergency work providing network systems for health services. But it expected sales to fall as lockdown hardships forced its customers who own businesses to collapse. Redcentric was also apparently affected by the strain a surge in demand has put on the infrastructure of Microsoft Azure.

The UK government’s Department for Digital, Culture, Media, and Sport had been consulting techUK while trying to determine what it must do to help keep the country's data infrastructure running through the crisis. It asked techUK how the industry worked, the association said this week. Operators had been working out what to say.

"We are still understanding the impact," said Allan Bosley, information manager at Ark Data Centres, said.

"We are currently assessing the medium and long term impacts,” he said. “I think that's undecided. It would be very difficult to make decisions about the medium and long term on the basis of three weeks of lockdown.”