Baltimore Ransomware Carnage Compounded by Local StorageAuditor Reveals Lack of IT Policies Ensuring Employees' PCs Centrally Backed Up
More evidence that too many organizations still fail at disaster planning: Baltimore says its ransomware recovery has been hampered because there was no IT policy ensuring that all PCs were being centrally backed up. As a result, numerous files that were crypto-locked in its May 7 attack have reportedly been lost forever.
The city's outdated approach to backups came to light last week in a city council committee meeting, when Baltimore City Auditor Josh Pasch said that the IT department couldn't document whether it was meeting goals, including modernizing mainframe applications and making more data accessible to residents via the city's website, in part because required information had been stored only an employee's hard drive that was compromised by the ransomware attack, the Baltimore Sun reports.
"'Ransomware' puts new requirements on backup; instead of recovering a few files one may have to recover many systems and applications at the same time."
"One of the things I've learned in my short time here is a great number of Baltimore city employees store entity information on their local computers. And that's it," Pasch told city councilors, the newspaper reports.
Ensuring that all city systems get backed up would seem to be a no-brainer IT policy. City officials didn't immediately respond to a request for comment about why it was not already in place. But the Baltimore Sun reports that city IT Director Frank Johnson, who was put on leave following the ransomware attack, promised in a written response to the audit findings to revamp the city's data storage practices.
Fallout Cost: $18 Million
The lack of mandatory, centralized backups is just the latest chapter in the unfolding Baltimore ransomware saga. To its credit, the city opted to not pay a ransom to attackers, who demanded 13 bitcoins - then worth about $100,000 - to decrypt all of the files, which had been forcibly encrypted using RobbinHood ransomware.
But recovery has been costly. The city estimates that the incident will cost $10 million in recovery and digital forensic expenses, and another $8 million in lost revenue (see Baltimore Ransomware Attack Costing City $18 Million).
Hence the longstanding warning over ransomware: Invest in preparation, or risk paying much more later, if not risking bankruptcy.
Maintaining proper backup and recovery operations remains critical to recovering from ransomware. Because many ransomware strains can crypto-lock all network shares mounted to a PC, or systems connected to a server, organizations also need to ensure they're storing recent backups offline, so these backups do not also get crypto-locked by malware.
But backups are only part of the challenge. Organizations also need to ensure they have well-designed restoration processes in place, ideally to be able to rapidly restore all systems in just days.
"'Ransomware' puts new requirements on backup; instead of recovering a few files one may have to recover many systems and applications at the same time," information security expert William H. Murray writes in a recent SANS Institute newsletter. "In light of these new requirements, one should revisit one's backup strategy before a ransomware attack. However, the availability of safe backup trumps the speed of recovery."
Prep Now or Pay More Later
Escaping ransomware unscathed best starts by planning ahead. Of course, keep systems up to date with the latest versions and security patches, to eliminate known flaws. Also ensure anti-virus software has the latest signatures.
Where ransomware is concerned, keeping lots of good, up-to-date backups also remains essential. "The best way to limit the damage from ransomware is to maintain and verify current backups of valuable data," researchers from Secureworks Counter Threat Unit write in a blog post. "CTU researchers recommend that organizations employ a 3-2-1 backup strategy to ensure successful restoration of data in the event of a ransomware attack."
The 3-2-1 Backup Strategy
An effective backup and recovery program is one that not only ensures everything is being safely backed up, but also that it can can be restored in the required timeframe, and that this all gets regularly tested and verified.
Ransomware-wielding attackers, for example, have long targeted organizations with time-sensitive data. The calculus is simple: Even if an organization can recover data, it might not have time to undertake a full recovery, which can often require weeks, if not months. Hospitals, for example, may be predisposed to quietly pay for a decryptor, in the hope that it can get systems restored in just days. And that's why some organizations - inside the healthcare sector and out - have been quietly stockpiling bitcoins in case this happens (see Ransomware Extortion: A Question of Time).
But solid IT solutions exist to help solve this challenge, provided they're deployed in advance of an attack.
For example, the city of Sparks, Nevada, had working backups that allowed it to recover from a 2015 ransomware outbreak that first hit its police website. While the city was able to restore its emergency systems within hours, restoring the city's geographic information system - "the lifeblood of any city's operations" - from LTO3 backup tapes, each storing 700 GB of data, took much longer, writes Steve Davidek, the city's IT manager, in StateScoop.
"We had to get all of the necessary tapes from an off-site vendor in order to feed them into the tape library one by one - a painfully slow task," he writes. "It was two weeks before our GIS data was completely back online, as huge image files needed to be restored from small, slow tape backups."
Based on lessons learned from the incident, Davidek says the city has now tapped a vendor to provide managed detection and response. In addition, the city revamped its backup and recovery controls, relying now both on better tape backups as well as the cloud. "We can restore virtual machines via the cloud, and we use higher-capacity tapes to make it quicker and easier to restore data," he writes. "In fact, I now believe our critical server infrastructure can be up and running within an hour."
How many other organizations can say the same?