3rd Party Risk Management , Access Management , Cloud Access Security Brokers (CASB)
The Ransomware Files, Episode 4: Maersk and NotPetyaMalware Disguised as Ransomware Nearly Sank Logistics Giant Maersk
What if malware disguised as ransomware destroyed every copy of a company's Active Directory except for one?
That's exactly what happened to global shipping and logistics company Maersk on June 27, 2017. Maersk was one of dozens of organizations crippled by the NotPetya malware in one of the strangest and most devastating global cyberattacks.
Gavin Ashton was Maersk's identity and access management service owner at the time.
"We talk about milestones and project plans and three-, five-year plans," Ashton says. "And the thing about ransomware, or extortion, or whatever you want to call it these days, is it doesn't really care about any of that. It could literally strike this afternoon. That was our wake-up call."
Bharat Halai was Maersk's former head of identity and access management. The attack knocked out all of Maersk's copies of Active Directory. Halai's quick thinking uncovered the last remaining uncorrupted copy in Lagos, Nigeria, which had experienced a wide area network outage.
"I asked the head of IT and said, 'Can you just make sure you call every single site and ask them if any of them had a WAN outage at any point since the NotPetya event or before?'" Halai says. "Fortunately, the head of IT came back and said, 'Yes, we have one site that has had a WAN outage before the 27th of June.' I thought, 'Great.' Now, first thing to do, is back that baby up quick, quick before it goes."
In this episode of "The Ransomware Files," Ashton and Halai explain how the dedication and tenacity of the team at Maersk brought the company back from an IT systems meltdown. They also explain how they rectified weaknesses in Maersk's identity and access management systems that had caused NotPetya to spread so quickly.
"The Ransomware Files" is a podcast miniseries available on Spotify, Apple Podcasts, Google, Audible, Stitcher and more. I'm speaking with those who have navigated their way through a ransomware incident to learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed. But there's invaluable knowledge gained. There should be no shame in getting infected, and it's important to share the lessons.
If you enjoyed this episode of "The Ransomware Files," please follow it on a podcast platform and leave a review. Also, the show has a Twitter handle, @ransomwarefiles, that tweets news and happenings about ransomware.
If you would like to participate in this project and tell the information security community about your organization's brush with ransomware, please get in touch with me at email@example.com or direct message me here on Twitter. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.
Speakers: Gavin Ashton, former identity and access management service owner, Maersk; Bharat Halai, former head of identity and access management, Maersk; Jeremy Kirk, executive editor, Information Security Media Group.
Production Coordinator: Rashmi Ramesh.
"The Ransomware Files" theme song by Chris Gilbert/© Ordinary Weirdos Music.
"Soviet March" by Shane Ivers.
Music by Uppbeat. (Tracks and license codes here.)
- ABC News, "Petya cyber attack: Cadbury chocolate factory in Tasmania hit by ransomware," June 28, 2017.
- CSO Online, "Petya ransomware and NotPetya malware: What you need to know now," Oct. 17, 2017.
- Data Breach Today, "To Prevent Another WannaCry, Microsoft Patches Old OSs," May 15, 2019.
- Dragos, "Spyware Stealer Locker Wiper: LockerGaga Revisited," March 2020.
- Fortinet, "Key Differences Between Petya and NotPetya," July 9, 2017.
- Gigamon, Exorcising the Ghost in the Machine, January 2022.
- I, Global Intelligence for Digital Leaders, "Maersk: Springing back from a catastrophic cyber-attack," August 2019.
- Gavin Ashton, "Maersk, Me & NotPetya," June 2020.
- Malwarebytes, "Petya - Taking Ransomware To The Low Level," July 16, 2021.
- Securelist, "Schroedinger’s Pet(ya)," June 27, 2017.
- Securelist, "ExPetr/Petya/NotPetya is a Wiper, Not Ransomware," June 28, 2017.
- U.S. Department of Justice, "Six Russian GRU Officers Charged in Connection with Worldwide Deployment of Destructive Malware and Other Disruptive Actions in Cyberspace," Oct. 19, 2020.
- The Washington Post, "NSA officials worried about the day its potent hacking tool would get loose. Then it did," May 16, 2017.
- Wired, "The Untold Story of NotPetya, the Most Devastating Cyberattack in History," Aug. 22, 2018.
- Wired, "The Leaked NSA Spy Tool That Hacked the World, March 7, 2018.
Jeremy Kirk: You described in your blog post that you're sitting in a glass wall meeting room, having a meeting at 10 am on June 27, 2017. Describe for me, what that day was like, what the meeting was about, and what happened next.
Gavin Ashton: Well, so I wondered how long it would take -- the hair's already going up on my arms. So it was an entirely normal, pedestrian sort of day, you know…
(Gavin talking in background)
That is the voice of Gavin Ashton. Gavin is an expert in identity and access management. Almost five years ago, Gavin had a role in one of the most devastating, strangest and unprecedented global cyberattacks that's ever occurred. The incident is known by the ransomware - or at least purported ransomware - that caused it, which was called NotPetya. Gavin worked for the Danish global shipping giant Maersk.
Ashton: This is something I described to customers of mine now: we talk about milestones, project plans and three-five year plans, etc. And the thing about ransomware or extortion - whatever you want to call it these days - it doesn't really care about any of that. It could literally strike this afternoon. And that was our wake-up call.
Kirk: Maersk was one of several multinational companies that spent hundreds of millions of dollars recovering after NotPetya struck. And it was just one of dozens of organizations that were affected. NotPetya pretended on the surface to be ransomware, but it actually wasn't. Although it displayed a ransom note that would indicate it was just encrypting files, it actually wrecked the master boot record of computers.
The master boot record is the first sector of a PC's hard drive that the computer looks to before loading the operating system. There's no recovery from that sort of digital vandalism. This malware was designed to destroy. We’ll get into a bit of the who and the why later. But NotPetya was extremely successful at destroying IT systems with breathtaking speed. And one component of that success involved taking advantage of weak identity and access management controls.
Two decades ago, identity and access management was a back office IT function that no one really thought about except for the person or two who maintained whatever system was in use. Now the systems that broker access are among the most important ones to defend against intrusions, ransomware attacks, and in the extreme cases as in this one, a cyberattack by a nation state that whether intended or not, got completely out of hand.
This is The Ransomware Files. I'm Jeremy Kirk.
In this podcast mini-series, I'm speaking with those who have navigated their way through a ransomware incident and learn how they fought back and what tips they can pass on to others. No ransomware infection is ever welcomed, but there's invaluable knowledge gained. There should be no shame in getting infected, but it's important to share the lessons.
In June 2020, Gavin published an eloquent and highly detailed blog post describing his experiences at Maersk. He started working at Maersk in 2015 as its identity and access management subject matter expert. The piece was an in-depth look at why Maersk was so affected by NotPetya, this toxic, self-propagating malware. Gavin described how NotPetya took advantage of identity and access management weaknesses and also how Maersk eventually improved. The piece was remarkable for its frankness, openness and the fact that Gavin put his name on it. Tech is an industry where most people are reluctant to talk about former employers for fear of retribution or legal trouble. To his surprise, the piece captured an audience, and Maersk didn't bother him about it.
Ashton: When I posted that post, the reaction kind of blew up. And it was my two minutes of stardom in the infosec world. But I'd had a very normal career - and have a very normal career, really - but I started off as a system admin in a local company. I found identity and security pretty interesting.
Kirk: Gavin ended up at Maersk after cycling through some partner and in-house roles, honing his skills around Microsoft's identity and security stack. The Maersk role, which was in a city called Maidenhead about an hour west of London, was an outstanding opportunity. Maersk is one of the largest shipping and logistics companies in the world, and it has more than 80,000 employees in 130 countries. Its operations are a crucial part of the global supply chain.
The job was a step up for Gavin.
Ashton: Maersk was the first organization I'd worked at where you really felt the values - they're still majority family owned. The stuff that Maersk does, the history…you go back to the Second World War, where they're ferrying, picking up people out of the water and stuff. I mean, the stories of Maersk are just huge. They're a great organization.
Kirk: Bharat Halai hired Gavin. At the time, Bharat was Maersk's service owner for directory services, which included Active Directory, the all-important directory that's critical for authenticating Windows users. His role also included purview over Forefront Identity Manager, or FIM, and Microsoft Identity Manager, or MIM.
Bharat Halai: I was quite lucky that Gavin was one of the first CVs that came up. And I knew that he came from a well-respected consultancy within the U.K. So when he came through, I spoke to him, he was very well spoken. I guess the rest is history because he had a good technical understanding of FIM and its management.
Kirk: Bharat says that FIM and MIM are good for what their missions are: identity management. But neither are great at access management. And if you roll back the clock five years ago, the tooling for controlling and monitoring privileged access was less refined. Gavin says the scene at Maersk was what you'd expect. The principle of least privilege wasn't generally followed. The least privilege principle dictates that user and admin accounts shouldn't have access to more resources than needed. For example, service accounts shouldn't be used across multiple applications and end user accounts don't need administrative privileges everywhere. At Maersk, some server administrators had access to huge numbers of other systems, Gavin says. There were some vague security baseline policies, but they weren't consistent and not really followed. All of it meant trouble if a bad actor gets hold of a highly privileged account. There were funding challenges as well, as Gavin says IT was viewed as a cost center to be minimized. He had been pushing for privileged access management controls - referred to as PAM - which is tooling that can provide visibility and tighter management of who has access to what.
Ashton: I guess anybody from the identity and security PAM field would sort of recognize very typical behaviors. For a lot of organizations, security isn't the first priority. It's delivery of the business, and getting those containers around, allowing people to submit and process their orders - the business end of IT. And you typically find in many organizations that security is something that is not an afterthought, but it's definitely not a main priority. And I found that that it was the story at Maersk as well. They had differing levels of maturity when you looked at different layers of the IT stack. In the days gone by, you operated in a four-walls environment, you had your data center, your office block, and you'd focus on your network layer, on your firewalls and you keep yourself in your little box - that doesn't really work anymore, which was one of the things that NotPetya really highlighted.
Kirk: The demonstration of why that doesn't work anymore was far from subtle. It started mid-morning on June 27, 2017. Gavin was in a meeting.
Ashton: I mean, if we make a Hollywood movie of this…it'd be screens and sequences…you look down a row of desks that go bump, bump, bump, bump, bump down the row. They go black, and you get a pretty little red ransomware message. But the interesting thing wasn't message itself, because you think "Oh, okay, so we've got some sort of issue locally." It was the news that came in over the next couple of hours or so. "Oh, okay. It's not just Maidenhead, it's not just Europe, it's not just America... Okay, this is global. Oh, and it's not just desktops, it's servers as well. Okay, which servers? Oh, all of them." Nothing quite prepares you for that news.
Kirk: The NotPetya attack occurred on the same day that Bharat was conducting the first service review with a Polish company that supported Maersk's identity management setup. That included the company's support of Active Directory, MIM, FIM and also Azure AD, which is Microsoft's cloud offering for Active Directory. Maersk had recently put that into the mix as well. Bharat had booked a room that was just behind Maersk's command-and-control center for the meeting. The command-and-control center had big screens that monitored the shipping company's IT operations around the world.
Halai: We were sitting in that room, we were just talking away. And suddenly, I heard some very large footsteps; some very loud footsteps, should I say. And it was my peer who basically ran the desktop, the workstation services, for Maersk. And I knew that if she was walking that fast and that loud, something must be wrong. So she asked us to clear the room, so we cleared the room and went downstairs to the canteen. And we continued our search review, and it went on for a good hour or two. I had no laptop in front of me. So the service review was finished, I thanked the vendor, they came back upstairs with me to go grab their bits and bobs - by this time, something had gone wrong. You could see that people were concerned, you could see that something had happened. This was probably half-eleven, 12 o'clock on the 27th. Something had gone wrong.
I asked them, "What's happened?" And I said, "well, we seem to have some issues. We've got system offline." And I thought nothing of it. I thought "okay, well, I'm sure AD's okay." Thought nothing of it, and carried on my usual business.
It turned out that wasn't the case. And there had been something that happened and you can see in the command-and-control center that the screens had gone black and you had the red writing of the ransomware. At that time, I didn't know it was ransomware. All I knew was that there was something there that could have been a failure of some sort. It never occurred to me, at that time, that it could be ransomware, but a few hours later, it turned out to be ransomware. So that's how the day started. It was 12-1 o'clock on the 27th.
Kirk: Exactly what had struck Maersk was still unknown. But it was spreading incredibly fast. Within two hours, almost every domain-joined service or device was gone - from laptops to workstations to virtual machines to physical servers. And what about Bharat's 180 or so domain controllers and Active Directory that he initially thought were probably fine? Yep. Gone. As Gavin put it in his blog post: Maersk had just been sent back to the Dark Ages.
A little over a year later, the head of technology for Maersk, Adam Banks, relayed a tally of the damage to a publication called I - Global Intelligence for Digital Leaders, which is a publication sponsored by Fujitsu. Adam said that 49,000 end-user devices were destroyed, 1,200 applications were inaccessible and 1,000 were destroyed. Also, some 3,500 of 6,200 servers were completely wrecked.
Why was whatever was happening so virulent and so destructive? NotPetya was one potent piece of engineering. To get into its lineage, it has a similarity to a ransomware called Petya, which appeared around March of 2016. Like Petya, NotPetya had its own bootloader and kernel. Once it hit a system, it would write over the master boot record with its own custom boot loader that then loaded a small malicious kernel. But rather than encrypting files, it actually just encrypted the master file table. Now the master file table is what keeps track of what files are on a system and the metadata of those files. Encrypting that, of course, made the files inaccessible.
NotPetya had several features that caused it spread globally quickly. One, it was a self-propagating worm. Once it infected a system, it sought out other systems to infect. One of those methods used a software exploit known as EternalBlue, which was developed by the U.S. National Security Agency. Somehow, the agency lost control of this cyber weapon. It was leaked in April 2017 by a group calling itself the Shadow Brokers. To this day, we still don’t know who that group is or how they came to be in possession of one of the NSA's most powerful hacking tools.
EternalBlue exploited a vulnerability in Microsoft's Server Message Block protocol, which is used for accessing files on other systems and connecting to printers. EternalBlue proved so fruitful for the spy agency that one U.S. government official told the Washington Post that it was like fishing with dynamite. EternalBlue was also used by the infamous WannaCry ransomware, which struck just a month before NotPetya. But using EternalBlue wasn’t NotPetya's only trick. It also used another exploit called EternalRomance, which also targeted the SMB protocol. Also in its toolkit was a modified version of Mimikatz, which pulls authentication credentials from the memory of Windows systems. Between exploitation and credential harvesting, NotPetya was razor sharp spear.
So, how did this end up on Maersk's systems? NotPetya was initially slipped into a software update for accounting software that's used in Ukraine called M.E. Doc. It's developed by small company called the Linkos Group. The software is used by any business that does business with Ukraine. So when the malicious update went out, NotPetya was unknowingly installed by those who used M.E. Doc. It crippled Ukrainian organizations, but they were far from the only ones affected.
FedEx, pharmaceutical company Merck, and even a Cadbury chocolate factory in Tasmania were hit. But due to NotPetya's extremely effective propagation, there even more further afar victims. One of those was the Heritage Valley Health System in Sewickley, Pennsylvania. NotPetya jammed up systems that held patient histories and lab records, and admin systems were unavailable for a month.
U.S. investigators believe NotPetya was created by Unit 74455 of the Russian Main Intelligence Directorate, also known as the G.R.U. The group is known to computer security researchers by a cluster of names Sandworm Team, Telebots, Voodoo Bear and Iron Viking. In October 2020, the U.S. Department of Justice charged six G.R.U officers, who were all men in their late 20s and early 30s, with creating malware, including NotPetya. The journalist Andy Greenberg of Wired wrote an excellent book on the Sandworm group and the immense cyber destruction it wrought with NotPetya.
There have been questions on if those who created NotPetya intended for it to cause so much collateral damage. Obviously, it was intended to hurt Ukraine, which it certainly did. After invading and annexing Crimea in 2014, Russia has been engaged in an ongoing deadly conflict and for all practical purposes, also taken a chunk of Ukraine's eastern territory. And as of early 2022, Russia appears to be poised to invade the country. As part of that buildup, we've already seen aggressive moves on the cyber front, including the deployment of malware that - get this - overwrites the master boot record, wipes files and displays a fake ransomware note.
But NotPetya didn't spare Russia. It hit organizations there too. That included Rosneft, which is Russia's largest crude oil exporter; Evraz, a mining and steel manufacturer; and Sberbank, one of the largest banks in Russia. Perhaps NotPetya's creators didn’t anticipate just how effective the malware would be. The indictment against the six G.R.U members claims that they celebrated the deployment of NotPetya, but unfortunately doesn't say how investigators learned that.
What some victims saw was a ransom note in red that said "Oops your important files have been infected." It went on: "Perhaps you are busy looking for a way to recover files, but don't waste your time." The note asked for $300 in bitcoin as a ransom. But it was a ruse. When an organization gets infected with ransomware, they're usually assigned some sort ID so the attackers can identity the victim and provide the right decryption key. But NotPetya just created random data for that identifier, and it essentially was useless. It was impossible to decrypt the data.
However, all of this - how NotPetya spread, the Russian involvement and the fact this was fake ransomware - was unknown at the time to Maersk. Gavin says 2017 was a year that no one was really ready for.
Ashton: What surprised the whole world that year was WannaCry and NotPetya. Up until that point, you'd had major names go down every couple of years with some big event - maybe a Yahoo or a Sony, etc. But they'd been isolated events that would happen every couple of years. What WannaCry and NotPetya did was really set that bar of 'this is how it's done'. And although they weren't ransomware, they were sort of malware dressed as ransomware. If you draw a graph, it's then a vertical line from that point to today. And it wasn't something that you were prepared for.
Kirk: Maersk needed to get containers moving again. Where do they even start? The answer is the same place where the attackers often start: Active Directory. But Gavin says recovery plans often assume something is still available and working to serve as a basis for rebuilding. In this incident, though, Maersk had been fried.
Ashton: This wasn't isolated to Maersk and was true in other organizations too - your recovery plan makes an assumption about availability. So, AD is designed to be a highly resilient system and if you're global, you've got hundreds of domain controllers. Well, you might lose a country, but you'll still have AD. In this situation, where you've just lost everything, what do we do now? We were lucky we found a way to bring AD back. But there were organizations out there who were not able to do that, so this can really be pretty existential.
Kirk: NotPetya had wiped Maersk's Active Directory out. Bharat had one idea that worth trying, but it was still a long shot.
Halai: I asked the head of IT, "Can you just make sure you call every single site and ask them if any of them had a WAN outage at any point since the NotPetya event or before? And so he got his local IT guys to call out to all branches and find out. And it just so happened that Nigeria had experienced a WAN outage. Lagos experienced a WAN outage."
Fortunately, the head of IT came back and said, "Yes, we have one site that has had a WAN outage before June 27." I said, "Great! Can you ask them if they haven't already powered down the devices. "Ask if they experienced the same issues we did." But lo and behold, they had not experienced the same issue. I thought, "Great, now the first thing to do is to back that baby up quick, quick before it goes."
Kirk: Then, Bharat says it was a matter of getting that data back to headquarters in Maidenhead.
Halai: Unfortunately, none of the guys from Lagos had a visa. But we had the Central Africa head based in Ghana. He flew from Ghana to Nigeria, picked the disk up and and flew it to us in Maidenhead. In parallel, we had one of his team members uploading the backup to a safe location to see if I could download it as well. But due to connectivity issues, it was difficult. And it actually took more than 10 hours to do the upload, by which time he had arrived in Heathrow. And if you know where Maidenhead is - I was on the M4 for half an hour. So he got a taxi and came to Maidenhead with the drive and handed it over to me and my esteemed AD architect colleague.
Kirk: With that in hand, things were looking better. But one move Maersk made before the incident turned out to help quite a bit, given the Active Directory problem. Around two weeks before the NotPetya attack, Maersk had moved to Azure AD Single Sign-On with password hash sync. What that meant is even though most on-premises Active Directory copies had been wiped out, people could still authenticate into cloud-based services such as Office 365. They could still use email.
One large task was how to move all of these compromised systems to a good state. Maersk had been on Windows 7. But Gavin said there was no point to restoring infected systems back to that O.S, so the company decided to upgrade systems to Windows 10.
So, tell me about the sausage factory.
Ashton: What could you possibly not know about just those words alone?
Kirk: The "sausage factory" was the term that Gavin and others used for how to move large swathes of compromised systems into a clean state using a tiered access model and privileged access management controls. Tiered access essentially means making sure that highly privileged accounts are separated from high risk zones. Part of that task was also ensuring that what Maersk needed from a business perspective was prioritized for restoration.
Ashton: That's the "sausage factory". So you want to start off with, let's get AD in check. So whatever you want to do, there's multiple lists out there for things to do after a cyberattack to help protect Active Directory and recover your trust in the directory. But then the issue will be, well, domain controllers and AD assets are fine, because that's a fairly limited amount of stuff. What about these hundreds or thousands of servers? What do we do there? And you go, "okay, right." So what are your business critical processes? Don't forget about IT, walk across the business island and say, "right, you guys, which processes can this business not live without, what do we actually need on day one of recovery of a BCDR process?" And then it's a case of mapping those processes back to systems that actually matter. So whatever those systems might be, doesn't really matter. So those systems, there are going to be ones that you say, "okay, these ones are going to be things that we actually have to get across as quickly as possible." But the rest is still a threat. Until you've actually put the controls you need around your whole environment, you're still kind of at risk, because you're providing lateral access paths.
Kirk: Moving away from on-premises software to the cloud has strong security advantages. But Gavin says there is always going to be a minority percentage of systems that, for whatever reason, won't be moved. But there are ways to deal with those risks.
Ashton: At that point, it's a case of how we set up ring fences around these things. How can we isolate them from an identity perspective, making sure that we're controlling those access paths into those administrative systems that underpin those services. And you're then into this scenario, where we're really trying to make AD as small as possible. So if you think of AD today in a typical organization, we've got lots of old servers and stuff, pretty big attack surface. So as much as you can, minimize that and commoditize how that's secured as well. I used to hear a lot about "stop the rot." People in the business spinning up systems and not managing them particularly well - that's really going to get you wiped out. So trying to commoditize how AD is secured, let's control the access paths into those systems, let's not allow service accounts to sign in interactively. Let's do the principle of least privilege, let's use tools like whatever your PAM of choice is. And just not leave AD quite sort of open.
Kirk: Bharat says that prior to the attack, Maersk was using some best practices, such as using separate accounts for, say, logging onto Outlook and one as domain admin. But post NotPetya, Maersk also tried to ensure the infrastructure was as segregated as possible.
Halai: One of the things we did was not just separate the accounts, but separate the infrastructure as well. So not physical separation, not network segmentation, but logical separation. And that was something that Microsoft developed, called the tiering model, which I think was quite new in the time when we were hit. When we look to the first 90 days after the attack, one of the things we did was to also split as much as possible the ability for domain controllers to have admin rights to servers and workstations. If you think about it, why does the domain administrator need to go to manage all of the administration on the servers and the workstations that's within the domain? Not really required. And then, as an admin, how am I accessing the domain admin from my workstation? Okay, well, if I'm logging on to my workstation with my standard account, and then logging on to a remote desktop session to a domain controller, then potentially that's an attack vector for my password to be compromised for the domain admin account. Because you could have listeners on there, some other USB stick reader, keyboard reader on the back of my keyboard. I wouldn't know that, right?
Kirk: In June 2021, Bharat wrote this on LinkedIn:
"4th year anniversary of NotPetya hitting, pretty much to the minute. Testimony to those who spent months recovering and the military-like discipline. Time to reflect on lessons learnt and continuing to ensure that business and individuals stay protected from people who are hell bent on disruption."
He tagged Gavin and 19 other people. The comments are telling of the gargantuan task of rebuilding Maersk, and the resilience and fortitude of those who helped.
Stephen Valasco, a former senior application manager at Maersk, wrote: "Ahh, good times. Emptying the snacks aisle of the local Sainsbury's and working together 24 hours a day to get things up and running. A unique moment to have experienced, and fantastic people to work together with."
Dave Lanagan, who work with service management maturity and capability build, wrote: "We all made a great team, watching the sun come up in Maidenhead for the second time without barely leaving the office. And that was just the start: Kanban boards, WhatsApp, setting up crisis teams for different services. Those first 48 hours had some critical decisions to be made that were bold and brave, and the team faced up to them. I'll never forget the welfare team protecting people for working hours, meals, drinks, rest. They were sensational even if a few of us were ducking and diving them to just try and get in a few more hours."
Mandy Sunner, a former lead business analyst, wrote: "Hell is bent on destroying good. Love your disciplined approach and calmness, Bharat.
In your LinkedIn post, you wrote testimony to those who spent months recovering and the military-like discipline. And then Mandy Sunner, she wrote a little bit further down. She said, "Love your disciplined approach and calmness." Tell me about those components, like the discipline and then also the calmness, because you seem like a really calm guy. You seem like you'd be somebody who's quite cool under a lot of pressure.
Halai: From a young age, I would say that I've been working under pressure as it were. Like most people bullied at school, and then suddenly hit puberty to become six feet tall with muscles, bullies don't tend to bully you. So I think I learned some resilience there. But yeah, there were not many people that really lashed out, went crazy or got really emotional about the situation. Yes, we had some frustrations. If some vendors weren't doing what you wanted them to do, you could shout or cry about it, but it wouldn't solve anything. And I think that's the thing that really pulled us together. Everyone else was calm around you, so why would I get worked up? I know that there was a lot pressure for the first few days and weeks on me to get Active Directory, because if Active Directory wasn't back, the Windows servers weren't coming back. Workstations weren't coming back. So one of the things that the architect and I kept telling ourselves throughout the whole thing was "It is what it is." And I wouldn't say I'm a spiritual Hindu, I am a practicing Hindu, but as you know, Hinduism is generally quite a peaceful religion. And that was just the mantra that was in my head all the time.
Kirk: There were stressing dynamics that resulted from the attack. Remember, nearly all of Maersk's Windows systems were gone. That also meant payroll. Bharat says that there were questions as to whether everyone was going to get paid due to the outages.
Halai: We didn't know if we'd get paid or not for this at the end of the day, right? You think about it: if your company has lost all of its IT, how would you know, sitting in an office in Maidenhead, whether you're going to get paid or not? HR did a phenomenal job of making sure that we got paid. That was around the end of the month, and we would have usually been paid by that time, but you didn't know if your salary was coming in the following month. But they knew that we were pulling out all the stops. Maersk is a family-owned Danish business and fortunately, there were reserves there.
Kirk: For Gavin, the incident did take its toll over the long term. There were lots of vendors involved. There were passionate discussions over which type of privileged access management technology to use and why. Discussions about prioritization. Disagreements. Deloitte, one of the largest technology consultancies in the world, was heavily involved. Not every decision went Gavin's way.
Ashton: I just had a difference of opinion. I've got a huge admiration for the stuff that actually happened. As I say, this is stuff that normally in an organization, you talk and talk and talk about for years, and years and years. And all of a sudden, it was all just happening. So, it wasn't a simple question at all. And ultimately, the right things did happen in the end. But my personal reaction to all this was, I was sort of desperate, just as desperate as anybody else, to see the right things happening. And after two years, I was burned out to a crisp.
Kirk: He quit Maersk. He acquired a motorhome and took his wife and three kids on the road throughout France, Spain and Portugal for six months. But something weird and disturbing kept happening.
Ashton: I left Maersk to just have a break. And I found myself thinking about it all the time. Even if I got another job, I was still talking and thinking about Maersk constantly. And I went to get counselling. And it turns out, that was grief. I was grieving my old job that I missed, that had been torn away from me. I never thought I could grieve over a job. That's ridiculous.
Kirk: That's fascinating. What did you feel like was unresolved?
Ashton: Well, grief is the word. I can only imagine… I've not lost either of my parents yet... But when you lose a loved one or a close loved one, or a pet? I don't know, it's that feeling of grief, like something had gone away. And I hadn't processed it. I hadn't acknowledged it in my own mind. I really loved that job and the people I worked with and the stuff we were doing. It was a really cool job. And it suddenly wasn't there anymore. I guess I'd been so busy for those two years with the recovery process and then building it back stronger, I hadn't really had time to process anything. And only after I left and we took six months out to go travelling, I felt into quite a dark spot.
Kirk: Time has passed and Gavin's over it. He now works at Microsoft. Amid all of the great advice in his blog post about Maersk, there's one piece that’s not only funny but powerful:
I like this: "Business continuity plans are vital. It's obvious when you say it, but seriously, at whatever level of the organization you are, there are things you can do to plan for the worst. No, literally the worst. No, worse than that. I mean, the absolute worst you can possibly think of, plan for that, because when it all goes, bang, you will seriously thank yourself."
That is wonderful advice.
Ashton: I wish I followed it all the time.
Kirk: If you enjoyed this episode of The Ransomware Files, please share it. Also, The Ransomware Files now has its own Twitter handle @ransomwarefiles, which tweets news and happenings about ransomware. And I'm on Twitter @jeremy_kirk. If you would like to participate in this project or have an idea for it, please get in touch with me. My direct messages are open on Twitter and I'm easy to find on LinkedIn. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.