Hackers Exploiting SaltStack Vulnerability Hit Data Centers

After infrastructure automation software vendor SaltStack released a patch for two critical vulnerabilities last Wednesday, hackers acted quickly to reverse engineer the exploit and attack vulnerable data centers.

Olle Segerdahl, principal security consultant at F-Secure who originally found the vulnerabilities, posted a warning on Thursday: "Patch by Friday or compromised by Monday."

He may have been too optimistic in hoping his advice would be followed, since companies began getting hit over that weekend.

The Ghost blogging platform went down on Sunday, May 3, after attackers used the vulnerability to gain access to its infrastructure, and it took the organization 13 hours to get back up and running again.

The platform says it has over 2 million installs and claims Tinder, OkCupid, NASA, Square, DigitalOcean, Cloudflare, DuckDuckGo, and Mozilla among its customers.

Other companies hit include LineageOS, an open-source Android distribution, which saw its website, wiki, download portal, mail, and other services go down on May 3.

Software maker Xen Orchestra was hit the same day and decided to stop using SaltStack altogether for the time being.

Hackers had taken advantage of the vulnerability to disable Xen’s firewalls and install cryptocurrency mining software on virtual machines. The company saw that some services became unreachable while others showed high CPU usage.

"At this stage, it was clear we were caught in a broad attack affecting tons of people," the company said in a statement. "We believe the initial assessment that 'thousand of infrastructures targeted' might be an understatement. Good news in this case, was the very generic payload, built probably really quickly to generate money fast without being stealthy."

Most embarrassingly, DigiCert, a major digital security company and a leading issuer of digital certificates, had to take down a certificate transparency log server.

"The key used to sign SCTs [signed certificate timestamps] may have been exposed via critical SALT vulnerabilities," the company said in a statement.

Relatively Light Damage

These victims may actually have been lucky, because the crypto mining malware that was used in these attacks was relatively easy to detect and did relatively little damage.

"But a more stealthy attacker could do something less obvious," F-Secure's Segerdahl told DCK. "They could install a backdoor to find out what server it is, who it belongs to, if there's a database of sensitive information to steal or an organization they can perform a ransomware attack against."

Data center operators use SaltStack, and so do data center customers, using it to manage servers or virtual machines in colos and public clouds, he said. It can be used to monitor servers, shut them down, run administrative commands, or install new software.

"If these servers are connected to the internet, they can pull down any software and install it," he said. "If they're not connected to the internet, the central management framework can push new files to the servers."

A Chance Discovery

A SaltStack user had hired F-Secure to check the software for vulnerabilities, Segerdahl said. What he found were security problems that had been present in the code going back at least five years.

"Before the patch was out last week, everyone who was running SaltStack was vulnerable to the issue," he told us.

The software is open source, so it's possible that others had found the vulnerability earlier but kept it to themselves.

"This issue is quite simple to understand and create an exploit for," Segerdahl said.

Potential for Damage: 10 Out of 10

One of the vulnerabilities allows attackers who connect to the request server port of the SaltStack master controller to bypass all authentication controls and gain full root access to both the master and the "minion" agents on the VMs and servers it controls. The second vulnerability he found enables directory transversal, allowing access to the entire file system of the master server.

According to Akamai, the targets attacked included web servers, Confluence instances, Docker instances, databases, and crypto mining software. The malware used included a remote access tool and the ability to disable firewalls and AppArmor. One variant of the malware included a worm to attack machines not using Salt.

Security researcher Tai Groot, who's been tracking development of malware that exploits the SaltStack vulnerabilities, said attackers were getting cleverer.

"There have been at least five different versions of the salt-store payload, each more advanced than the last," he wrote. "There are additional backdoors installed, and private keys are getting stolen right and left."

According to SaltStack, the two vulnerabilities that were disclosed and patched last week were rated critical, with a score of 10 in the Common Vulnerability Scoring System – the highest possible.

‘SaltStack Is Really Popular’

F-Secure's Segerdahl searched the internet for instances where the SaltStack master controllers were exposed and found 6,000 masters.

"From experience with our clients, SaltStack is really popular," he said. "And that's what we saw when we scanned the internet. It's popular with the big cloud vendors. People using Amazon AWS and running a lot of virtual machines seem to like this kind of service to keep track and configure those servers. We saw it across big and small cloud providers, and we also saw indications that it was used to manage services in data centers and hosting companies."

The 6,000 SaltStack masters he found were just the ones directly exposed to the internet. The ones that weren’t could still be vulnerable if attackers got into them by moving laterally through data center environments.

"If someone was to compromise any of the servers of the customers in the data center, then they could gain access to the network and to the SaltStack server," he said.

According to Alex Peay, SaltStack's  senior VP of product, the company has hundreds of paying customers – as well as an unknown number of customers who have downloaded the open source tool.

"Users do not have to register with SaltStack," he told DCK, adding that the 6,000 Salt masters found by F-Secure represented "a very small portion of the install base."

How to Guard Against Attacks on SaltStack?

The first step is to patch.

Some data centers don't do automated patching because they are worried about patches breaking critical services and causing downtime. Some installed the software without using a package manager that handles automatic patching.

Those data centers should immediately install the patches, Segerdahl said.

If that's not possible, because, say, they are running older, unsupported versions of the software and cannot update, then they should add some layers of protection by doing things like creating a white list of permitted inbound connections or using VPNs to connect the masters to the machines they control.

Having extra security is good in any case, he added. "There might be other vulnerabilities in the future."

SaltStack’s Peay said patches were available for the current supported versions, Salt 3000 and Salt 2019, as well as unsupported versions going back to 2015.

"Providing patches to these older versions of Salt ensures that our Salt Community can get the patch onto their systems as fast as possible without needing a major upgrade," he said. "Of course, we recommend that users of Salt maintain their instance and upgrade to a supported version as soon as possible."