7 Takeaways: Supply-Chain Attack Hits SolarWinds CustomersMitigation Advice Includes Immediate Updates and Scans for Signs of Compromise
Any cybersecurity professionals hoping for a quiet December to wrap up the annus horribilis starring COVID-19 got an unexpected surprise on Sunday - details of what is shaping up to be the worst hack attack campaign of the year (see: SolarWinds Incident Response: 4 Essential Security Alerts).
"We have identified a global campaign that introduces a compromise into the networks of public and private organizations through the software supply chain," FireEye CEO Kevin Mandia announced in a Sunday blog post. "This compromise is delivered through updates to a widely used IT infrastructure management software - the Orion network monitoring product from SolarWinds. The campaign demonstrates top-tier operational tradecraft and resourcing consistent with state-sponsored threat actors."
Left unsaid in Mandia's statement was that FireEye was one of the victims of the campaign against an unknown number of SolarWinds customers, which include hundreds of the world's largest companies and government agencies, including the U.S. National Security Agency.
Here are seven takeaways from the attack campaign and imperatives for customers, based on still-emerging details.
1. Full Scope of Attack Campaign Unclear
How recently information surrounding this campaign has been identified and publicly released is highlighted by the fact that, when FireEye issued its alert on Sunday on what it's dubbed the SUNBURST backdoor, the alert included a link to a piece of Trojanized code - a SolarWinds Orion software update - that was still live.
"We have been advised this attack was likely conducted by an outside nation state and intended to be a narrow, extremely targeted and manually executed attack, as opposed to a broad, system-wide attack," SolarWinds says in a Sunday security alert.
But as highlighted by the firm's use of the word "likely," as of yet there's no full picture of how damaging this campaign might have been and whether it remains live.
FireEye, for example, reports in a Sunday blog post that it has detected signs of potential compromise "at multiple entities worldwide" and that "the victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East." It also expects that more victims will be uncovered and says it has been directly notifying all suspected victims it finds.
SolarWinds, based in Austin, Texas, counts numerous large organizations as customers, and it reported 2019 revenues of nearly $1 billion. On Wednesday, the company's board announced that it has tapped Sudhakar Ramakrishna, who formerly led Pulse Secure, to serve as CEO. The company trades on the New York Stock Exchange under the SWI ticker symbol and has a $6 billion valuation.
Among its many customers, SolarWinds lists all five branches of the U.S. military, the Pentagon, State Department, NASA and National Security Agency, as well as 425 of the 500 largest publicly traded U.S. companies.
"SolarWinds currently believes the actual number of customers that may have had an installation of the Orion products that contained this vulnerability to be fewer than 18,000" of its more than 300,000 customers, the company says in a form 8-K filing to the U.S. Securities and Exchange Commission on Monday.
While the full nature of the attack and compromise remains unclear, the company says that it is investigating an attack against its Microsoft Office 365 instance which resulted in its email being accessed by outsiders, and says "other data contained in the company’s office productivity tools" may have been exfiltrated by attackers.
The U.S. Cybersecurity and Infrastructure Security Agency, or CISA, on Sunday ordered all civilian federal agencies to take action over the SolarWinds hack, via Emergency Directive 21-01, which "calls on all federal civilian agencies to review their networks for indicators of compromise and disconnect or power down SolarWinds Orion products immediately."
Chris Krebs, the former director of CISA, which is devoted to ensuring the safety of America's systems, networks and critical infrastructure, has cautioned that details of the campaign are likely still being unearthed (see: Analysis: Does Krebs' Firing Leave US Vulnerable to Attack?).
As news breaks about what looks to be a pretty large-scale hack, I have the utmost confidence in the @CISAgov team and other Federal partners. I'm sorry I'm not there with them, but they know how to do this. This thing is still early, I suspect. Let's let the pros work it.— Chris Krebs (@C_C_Krebs) December 13, 2020
2. Hitting FireEye - Attackers' Undoing?
The attackers' decision to target FireEye may have led to their undoing because the cybersecurity firm appears to have first detected the campaign.
Last week, Mandia warned that his company had suffered a breach, leading some industry watchers to criticize the firm for its security defenses having failed.
This week, however, FireEye stands as the victim that apparently unmasked its attackers, while fellow victims had not.
"For those giving FireEye hell last week and poking fun and jest at their data breach because they were a security company, do you think your threat model would have protected against this?" tweets Dave Kennedy (@HackingDave), the founder and CEO of consultancy TrustedSec and the former CSO of financial and retail technology giant Diebold.
The bad news, however, is that this campaign appears to have run for at least nine months, based on the Trojanized software first having appeared in March. The backdoored code appears to give attackers direct access to any customer's network, and is still being used by an unknown number of SolarWinds customers (see: Hacked: US Commerce and Treasury Departments).
"If the compromise of FireEye and the other targets began in Q1 or early Q2, the attackers have had many, many months to collect everything," tweets the operational security expert known as The Grugq. "That’s pretty bad, to use the jargon."
In all seriousness, if the compromise of FireEye and the other targets began in Q1 or early Q2, the attackers have had many many months to collect everything. That’s pretty bad, to use the jargon.— thaddeus e. grugq (@thegrugq) December 14, 2020
Hackers appear to have enjoyed complete access to SolarWinds' software development environment. "Although we do not know how the backdoor code made it into the library, from the recent campaigns, research indicates that the attackers might have compromised internal build or distribution systems of SolarWinds, embedding backdoor code into a legitimate SolarWinds library with the file name SolarWinds.Orion.Core.BusinessLayer.dll," Microsoft says in a customer alert issued on Sunday, referring to the .dll file that FireEye has dubbed SUNBURST. "This backdoor can be distributed via automatic update platforms or systems in target networks seen globally since March."
3. Update Orion Immediately - and Again Soon
SolarWinds says versions 2019.4 HF 5 through 2020.2.1 of its Orion platform were Trojanized. Those versions had been released from March through June, and some customers continue to use them.
The SolarWinds security advisory released Sunday urges all customers to ensure they are running the most recent version of Orion.
"We are recommending you upgrade to Orion Platform version 2020.2.1 HF 1 as soon as possible to ensure the security of your environment," it says. "The latest version is available in the SolarWinds customer portal."
SolarWinds says that, on Tuesday, it plans to issue another hotfix - 2020.2.1 HF 2 - which "replaces the compromised component" and also "provides several additional security enhancements."
For customers that cannot immediately update, SolarWinds has issued suggested workarounds, which include "having your Orion platform installed behind firewalls, disabling internet access for the Orion platform and limiting the ports and connections to only what is necessary."
4. Antivirus: Run Full Scans Now
In the wake of the attack details coming to light, security vendors have begun updating their antivirus and endpoint detection and response software with signatures designed to detect the malicious backdoor code.
Microsoft, which has dubbed the backdoor "Solorigate," says it has updated all of its antivirus products and recommends that all organizations that use Windows Defender Antivirus - on Windows 10 and 8.1 - or Microsoft Security Essentials on older systems immediately run a full scan. It's also released detection updates for cloud-based security information and event manager tool Azure Sentinel.
5. Indicators of Compromise: Keep Tracking
All SolarWinds users should also track current and forthcoming indicators of compromise, or IOCs, tied to the campaign, says incident response expert David Stubley, who heads Edinburgh, Scotland-based security testing firm and consultancy 7 Elements.
Such indicators can be used for spotting and blocking future attack attempts and can also be used for reviewing network logs for signs of intrusion.
"Given the type and context of the compromised components, it would be prudent to use the published IOCs to gain assurance over the current status of your network," Stubley tells Information Security Media Group. "However, it is worth noting that the situation may evolve as more comes to light. As such, it will be necessary to keep an eye on new IOCs as they become available."
Already, SolarWinds, FireEye and Microsoft are among the organizations that have issued IOCs and mitigation recommendations that organizations can use to block future attacks. Experts also recommend that all customers use the IOCs to review their logs for signs that they may have been compromised.
Microsoft also recommends that organizations "block known C2 endpoints" - referring to known command-and-control IP addresses - included in its IOC list.
6. These Attackers Were Sophisticated - for Real
Breached businesses stating that they were hacked by sophisticated attackers has been a cybersecurity cliche, beloved by businesses attempting to blunt the blame for what experts say are too often suboptimal defenses and a failure to block what turn out to be unsophisticated social engineering attacks.
But in this case, the attackers really do appear to have run a very advanced attack, some experts say.
Nick Carr, a security and intelligence researcher for Microsoft who was one of more than three dozen individuals FireEye thanked for working around the clock to help investigate the attack campaign, says that this "massive software supply chain intrusion" was "the most carefully planned, complex espionage I’ve ever helped uncover."
Carr previously served as the director of FireEye's advanced practices unit, which reverse-engineers and researches attacker tradecraft.
One measure of attackers' sophistication is that they subverted SolarWinds' own update mechanisms to spread a Trojanized version of the Orion software.
"Targets with auto-updates enabled could have been breached automatically - assuming the threat actor so deciding, they naturally had to specialize towards specific targets, considering how many of them suddenly became available," tweets Brussels-based security and privacy expert Lukasz Olejnik.
Theoretical supply-chain compromise via widely used IT management software like SolarWinds might well be the biggest hack of the year, maybe not merely this year alone. https://t.co/e5AfviujLP pic.twitter.com/AtYvkroKhQ— Lukasz Olejnik (@lukOlejnik) December 14, 2020
This apparent "supply-chain compromise via a widely used IT management software … might well be the biggest hack of the year," or perhaps even of recent years, he adds.
While that might sound like a sensationalist take, many experts are cautioning that, because this attack campaign involves supply chains and because it appears to have run for months, many more victims may have yet to be identified.
On the other hand, attackers may have limited their intrusions to the biggest, juiciest of targets - not just FireEye, but also the Treasury and Commerce departments, with the latter notably comprising the U.S. National Institute of Standards and Technology.
7. Supply Chain Compromise: Holy Grail
All signs so far suggest that the attackers enjoyed complete and unfettered access to SolarWinds software and distribution mechanism.
If these tactics sound familiar, it's because they recall the 2017 campaign involving NotPetya, in which alleged Russian hackers Trojanized accounting software called M.E.Doc to add backdoors. The software was then automatically distributed to all users, after which it was used to install fake ransomware called NotPetya, which began wiping systems not just in Ukraine but globally, taking down such organizations as Danish shipping giant Maersk and FedEx's international courier delivery subsidiary TNT Express (see: NotPetya Patient Zero: Ukrainian Accounting Software Vendor).
For the SolarWinds campaign, The Washington Post reports, investigators are eyeing the Russian hacking group known as Cozy Bear, aka APT29, as the culprits . For comparison's sake, the U.S. government has indicted six GRU military intelligence hackers - acting as part of a group known as Sandworm, aka TeleBots, Voodoo Bear and Iron Viking - with having perpetrated the NotPetya attacks.
The Russian Embassy in Washington on Sunday issued a statement denying any Russian government involvement in the attacks.
Regardless, nation-state attackers and crime gangs alike have long known that, if they can compromise a managed services provider, or any organization that pipes software directly into customers' networks, then they can potentially unlock an easy way to remotely compromise numerous organizations (see: Texas Ransomware Responders Urge Remote Access Lockdown).
"Supply chain is how you impact the many versus the few," says TrustedSec's Dave Kennedy. "One thing is clear: MSPs, cloud providers, software development organizations … You all are on notice."