Jeremy Kirk: Hey there, thanks for stopping by to listen to the first bonus episode of The Ransomware Files podcast. This podcast tells the stories of IT professionals who are fighting back against cybercriminals who extort organizations with malicious software called ransomware. It’s one of the greatest crime waves to ever hit the internet.
If you’ve listened before, you know that the main episodes go pretty deep into the details of a ransomware attack and interviews those in trenches. But there’s often material left over that’s quite interesting. I’m going to occasionally publish shorter bonus episodes with these stories, and this is one of them. It is a companion one to Episode 5, which covered how the REvil ransomware gang exploited flaws in software made by American software company Kaseya. If you haven’t had a listen to that one yet, check it out. And as far as this bonus episode goes, hopefully this morsel will keep you going while I work on a longer episode on something that I’m pretty excited about. Thanks for listening.
Software supply chain attacks are one of the most sneaky and unnoticeable ways in which an organization can get hacked. In these types of attacks, someone manages to tamper a component of a software program, and it ends up getting distributed to anyone who uses that software or service. And the effects can be not unlike that of an industrial fishing trawler scraping the bottom of the oceans.
In July 2021, one of the largest supply-chain style attacks on record occurred. It involved a type of ransomware that was made by a group that called itself REvil. Attackers managed to discover and exploit several vulnerabilities in software made by Kaseya. Kaseya develops the Virtual System Administrator, which is remote monitoring management software.
VSA is used by lots of companies, including managed service providers that help other companies by managing their IT systems for. Both managed service providers and vendors that make remote management software have been on the radar of cybercriminals, nation states and ransomware operators for a long time. They’re attractive targets because hacking one entity could open the door to the network of a whole lot more companies.
In early April 2021, researchers with the Dutch Institute for Vulnerability Disclosure discovered around seven vulnerabilities in the VSA product and warned Kaseya. Kaseya had fixed some, but not all of the issues, including a really nasty remote authentication bypass flaw in the on-premises version of VSA.
On July 2, 2021, REvil struck. It seeded a bogus software update for the VSA that actually was ransomware. Exploitation of the vulnerabilities allowed the attackers to distribute ransomware to up to 60 managed service providers. By attacking those systems, REvil was able to then distribute ransomware to the clients of those MSPs. You probably know the rest of the story: Nearly 1,500 organizations around the world were affected, including grocery stores in Sweden, pharmacies in that country and small businesses. REvil asked for $70 million as a ransom.
One of those affected was a company called Hoppenbrouwers in the Netherlands. It specializes installing climate control, lighting, sprinklers, security systems and solar installation - any sort of technical or electrical system that would go in a building. It has been around for more than 100 years. It has 17 offices across the Netherlands, employs around 1,600 people and has more than a quarter of a billion euros in annual revenue. So how severely was Hoppenbrouwers affected?
Marcel de Boer: It was close to 100%.
Kirk: That’s Marcel de Boer. He is Hoppenbrouwers’ financial director. Hoppenbrouwers was unique in the sense that its infection with the REvil ransomware wasn’t because it was using a managed service provider that used VSA, which was the case for most others affected. It just used the on-premises version of Kaseya’s VSA to manage laptops and servers. But Hoppenbrouwers recovery was fast and complete, and the company is actually working on a book describing its experience with a hope of helping others who have faced ransomware. Marcel says the company first noticed hiccups with its systems on the evening of Friday, July 2, 2021, European time.
de Boer: Yeah, it was July 2 on our side of the world. It was the beginning of the evening when one of our people tried to log into the system and couldn't get in. So he called the IT department. They looked into it and it happened to be a problem with ransomware, so that was when all the alarms went off.
Kirk: Hoppenbrouwers had been using VSA on some 1,600 laptops that were used by its employees out in the field. It also had it installed on 150 servers. VSA is very powerful software because if you control VSA, you can do just about anything to an endpoint - deploy patches, control access, service helpdesk tickets and more. That’s what makes it so useful for admins, especially in companies such as Hoppenbrouwers where so many people are out and about.
Marcel says VSA has served its purpose for the company but was getting long in the tooth. The company was shifting to a Microsoft stack, and part of that transition was replacing VSA with Microsoft’s InTune, which is mobile device and application management software. Lest to say REvil accelerated those plans. In fact, Kaseya’s VSA was out the following week.
Given that it was nearly completely encrypted by REvil, Hoppenbrouwers had a pretty severe situation on its hands. But it was ready going into the fight. It had cyber insurance. It had backups. And it had a CEO named Henny de Haas who was very determined to get the company back on its feet. Marcel says incident response folks that had been called in said it generally takes about three weeks on average before companies are back in business given the damage. But Henny said Hoppenbrouwers needed to open. By Monday. Keep in mind the attack occurred on Friday evening. Marcel says:
de Boer: He didn’t know what he was saying.
Kirk: But amazingly, it was going to prove possible. Marcel says it was an all-hands-on-deck effort. More than 200 people came on site to the company’s premises on that weekend to help with the restoration effort. They recalled all of the laptops from the field and established a protocol for checking those machines. By 9 PM on July 3, 90% of the laptops had been returned to home base and 80% had been checked. Hoppenbrouwers’ cyber insurance arranged for an IT security consultancy and managed security services provider called Northwave to come in as well. What proved to be the foundation for their recovery was… drumroll please… (sound of drumroll) backups!
Just two months earlier, Marcel says the company just put in place HPE's Nimble Storage product. Nimble retains snapshots. Analysts checked the snapshots to ensure there were no signs of infection, and they were clean.
de Boer: And there we had, maybe a bit of luck with the storage systems. We managed to get back a snapshot of that Friday afternoon.
Kirk: The snapshots helped speed up recovery enormously. They also needed to restore about 150 servers. Those machines included file servers, its enterprise relationship management system and servers that ran applications the company uses to draw technical installations. The snapshots restored some 150 servers with 75 terabytes of data in just minutes. But Marcel says there was a small problem: the snapshots only went back to the previous Friday around noon. When Hoppenbrouwers' employees came in on Monday, as far as their work was concerned, they knocked off at midday on Friday. Granted, this is far from the worst problem an organization has had with ransomware. But it still meant people had to reconstruct their work and try to remember what they’d done. And do it again. Although the snapshots were a huge step forward in the recovery process, it would not have been possible to be back in business on Monday without the help of Hoppenbrouwer employees. Communicating with them was key in the recovery process, so they filmed a webinar in the company’s studios on the weekend.
In the first webinar, Henny stands with a slide clicker in his hand. Marcel is to his left. He looks down, shifts around and moves his water glass slightly. He begins speaking.
de Boer speaks to employees in Dutch
Kirk: Of course, Henny is speaking in Dutch, but you can hear it. Henny starts speaking and then he chokes up. He says it's an emotional moment. He says that the night before, the company became the victim of an online attack; that they don’t know who did it. But he says that the Hoppenbrouwers was prepared. And with help of its employees they were indeed up and running on Monday like Henny had declared.
They also had contact with REvil a day after the attack. Marcel says it was standard procedures for Northwave - which was the incident response firm helping Hoppenbrouwers - to reach out to the attackers and see what they want. REvil had asked for $70 million to decrypt all victims by supplying a universal decryption key. The group claimed that more than one million systems were infected, but it didn’t appear who the victims were. But the gang’s customer support reps - which would walk victims through the negotiation process for a decryption key - appeared overwhelmed and couldn’t really handle the flood of victims contacting them. Here’s Marcel.
de Boer: So, they asked for $50,000 in monero. And Northwave tried to discover if the attackers had exfiltrated data out of the system. The answer to that was a little vague. We stopped that negotiations. We were already recovering from our backups by that time.
Kirk: As you heard before, Hoppenbrouwers was close to transitioning away from Kaseya’s VSA when the attack happened, which was unfortunate for the company. After the attack, Kaseya immediately shut down the software-as-a-service version of VSA, and along with the Dutch research team and other computer emergency response organizations, warned companies like Hoppenbrouwers that were running the dangerously vulnerable on-premises version. Kaseya also made efforts to contact every customer, even those that weren’t affected by the ransomware attack. Marcel says a Kaseya account manager reached out in the week after the incident and tried to make an appointment with the board of Kaseya and the board of Hoppenbrouwers, but…
de Boer: But that appointment has never happened. I mean the guy was too busy.
Kirk: Marcel says by that point, the company wasn’t even using Kaseya anymore and had moved to Microsoft inTune. But Marcel has a thought. He thinks Kaseya should have told its customers that its software was vulnerable soon after it found out about the software flaws. Then, those customers could decide on their own whether it was worth the risk of continuing to run it while Kaseya engineered patches.
de Boer: We were very disappointed they didn’t warn us beforehand.
Kirk: Marcel raises interesting points about how software vendors handle vulnerabilities. Information about unpatched software vulnerabilities often becomes public before a patch is available. That can occur for different reasons. Maybe attackers are already exploiting it in the wild, making the flaw what’s known as a zero-day vulnerability. Maybe whomever found the bug decided to publicize it before there was a patch.
In the case of Kaseya’s VSA, the Dutch researchers privately informed the company. Between early April, when the researchers did that, and July 2, when the massive attack happened, there were no reports of exploitation of the flaws. At that point, it still was a secret.
Vendors don’t have a lot of great options when they’re told their software has a dangerous flaw aside from quickly engineering a patch. Vendors will often issue advisories if the issue has become public and tell their customers to implement mitigations that reduce the risk until a patch is ready. But what if the issue isn’t known publicly yet, is easy to exploit and is highly severe? Do you just keep quiet about it? Or do you tell your customers to shut down the software, but not really tell them the full story? And if that course of action is taken, what if some customers don’t get the message and keep running the software? Kaseya faced all of these questions, and kept quiet.
This story has an ugly ending. REvil used some of the same flaws discovered by the Dutch researchers in its attack. We still don’t know how they did that. It might mean the group, or people working for the group, independently found the flaws. Regardless of how all that went down, Marcel says Hoppenbrouwers would have gladly shut down the VSA in advance to spare themselves the stress of a ransomware attack.
de Boer: We would have taken it down. It happens a lot: if there is a problem with software, it is fixed before you know it... there is a lot happening that we don’t know. In this case, it was quite a big problem.
Kirk: The Nimble snapshots were critical to Hoppenbrouwers' recovery, but Marcel says the company took many more steps after the attack to become more resilient. The incident response firm and consultancy Northwave is now monitoring Hoppenbrouwers' network and endpoints 24/7. It also changed how it manages privileged accounts, including requiring special approvals to access certain powerful accounts. The problem of third-party supply chain risk is vexing. But Marcel has a warning.
de Boer: You have to be prepared and you have to also be aware of the supply chain vulnerability. If you have a supplier who is critical to your company, and that supplier is hit by ransomware, then you also are in trouble.
Kirk: This bonus episode of The Ransomware Files was written, researched, edited and produced by me, Jeremy Kirk. The production coordinator is Rashmi Ramesh. The Ransomware Files theme song is by Chris Gilbert of Ordinary Weirdos Records.
If you enjoyed this episode of The Ransomware Files, please share it and leave a review. It will help keep this project going. The series has its own Twitter handle @ransomwarefiles, which tweets news and happenings about ransomware. I’m on Twitter @jeremy_kirk. If you would like to participate in this project or have an idea for it, please get in touch. The project is looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.
