Bare-Metal Cloud Firmware Security Fail Isn’t Limited to IBM – by Far

Tools used to manage bare-metal cloud environments can be used to attack data centers and are often overlooked, experts say, with IBM being one recent victim.

Security vendor Eclypsium reported last week that the Cloudborne vulnerability could be used by attackers to change a rented bare-metal server’s firmware to allow them to attack whoever uses the machine next.

One of the cloud providers that used the vulnerable baseboard management controller firmware by Supermicro was IBM Cloud, which wasn't careful about wiping the firmware between customers, John Loucaides, VP of engineering at Eclypsium, told Data Center Knowledge. But the problem could happen with any cloud provider, he added.

"This is really a broader industry concern about the firmware layer being effectively ignored by almost everybody.” If IBM can miss this, anyone else can, too. "IBM missed this – and missed this for quite a while. And there are a lot of smaller providers out there that don't have the resources that IBM has."

To protect their infrastructure, Loucaides said, data center managers should ensure their equipment hasn't been tampered with and that all patches are properly applied. Then, clean servers carefully after use by every customer. "Normally, in the reclamation process, you'd wipe the machine from the operating system level," he said. "Think about doing that from the firmware level as well."

Data centers should also make sure basement management controller passwords and logs are cleared. "You don't want them to be seeing the logs of whatever the previous person was doing."

While this vulnerability was in the baseboard management controller, Eclypsium has discovered other, similar vulnerabilities in other firmware.

What Should Cloud Users Do?

There are steps users of services like IBM Cloud can take to protect themselves.

For example, they can check their firmware version and see if there are known vulnerabilities, Loucaides recommended, or even install the firmware themselves and then doublecheck that the installation has gone through and wasn’t blocked by any malware.

Of course, if there's malware in the firmware, it can lie about its version number and about success of a new installation. "It's not that they can't do that, but it's harder," he said.

IBM Says No Known Client Impact

Eclypsium notified IBM about the problem in September. IBM announced last week – some six months later – that it is now erasing all BMC firmware logs, regenerating passwords, and reflashing the firmware between customers, calling this a "low-severity" vulnerability.

"We are not aware of any client or IBM data being put at risk because of this reported potential vulnerability, and we have taken actions to eliminate the vulnerability," Faye Abloeser, director of communications for IBM Cloud, told us. "Given the remediation steps we have taken and the level of difficulty required to exploit this vulnerability, we believe the potential impact to clients is low."

One of Several Warnings

Eclypsium isn't the only security vendor to point out firmware security problems, including those in baseboard management controllers.

“Our team uncovered BMC vulnerabilities earlier this year and reported that they could easily be exploited for malicious purposes," Nicolas Waisman, VP of security consulting at Cyxtera Technologies, told us.

Once a server was compromised, if there was a network connection, attackers could get to it. Waisman suggested that data center managers could add another layer of protection by isolating systems at the network level. "In our research, we were able to mitigate the risk of inbound calls to the BMC and lateral movement using a software-defined perimeter solution," he said.

The underlying problem is that data center security is focused more on the operating system level and on applications security. "They're ignoring the hardware," Chris Rouland, co-founder and CEO at Phosphorus Cybersecurity, an Atlanta-based vendor specializing in securing firmware, said.

But with the management features available in motherboards today, it's like having a whole other computer sitting underneath the operating system level. "And if that computer is not up to date, all the investment you've made in securing the OS goes out the window," he said.