News
Anatomy of a Ransomware Attack: Immutable Cloud Blob to the Rescue!
Ever watch your IT infrastructure start disappearing in front of your very own eyes amid a ransomware cyber-attack?
IT pro and ransomware expert Dave Kawula has been there. But in this case, Kawula, managing principal consultant at TriCon Elite Consulting, reported that the organization he was helping managed to recover its data, thanks to off-network immutable cloud storage that allowed for recovery of deleted resources.
Kawula shared his story with co-presenter John O'Neill Sr. in a recent multi-part online summit held by Virtualization & Cloud Review, titled "Top Security & Ransomware Threats," now available for free on-demand viewing.
It happened on a Monday morning, and before it was over the victimized organization suffered a complete loss of on-premises infrastructure and backups. However, thanks to some lucky Azure Blob Storage backups, it eventually achieved a 100 percent complete data recovery.
Here's the story.
One Monday Morning
"This tip is one that I kind of got a little bit of PTSD from, John, because it's happened so many times over and over again. And it starts off with a little story here. And the story starts off with one Monday morning. This particular organization that we were working with, I got the phone call at about, I think it was about 9:30 a.m. from the director of IT for this organization. And he's like, 'You know what, the servers are down, you know, something's not right. The network's not behaving properly.' And so it's 'Okay, let's see what's going on.' I was on my way, heading over to my desktop, I received another phone call -- even a more panicked phone call -- from him this time. And he's like, 'You're not gonna believe this. We just got hit by ransomware.' And I said, 'Well, how do you know that you got hit by ransomware?' And he said, 'They printed ransom notes off on every single one of the corporate printers. And it looks like they have pretty much everything.'
[Click on image for larger view.]
Turn Everything Off
"And so you know, the first tip that I have for you, if you're ever in this same situation, and this really saved us -- and I'll explain this in a minute -- is you need to shut everything off. You have no idea where the bad guys are, turn it off, turn everything off, because if it's off, they can't keep hacking. Turn off your firewall, turn off your servers, turn everything off. And then what we're going to do is we're going to bring things back on in a staged fashion and see what's going on.
The Payload
"And one of the things that we quickly learned with, with this particular attack, the payload that was used to deliver the ransomware, John, was actually set on a log-on trigger on the servers. And so this wasn't just ransomware, this was a full-on hack. They got admin-level access, and I'll explain how they got that in a minute.
"And so we actually got to watch our infrastructure start disappearing in front of our very own eyes."
Dave Kawula, managing principal consultant, TriCon Elite Consulting
'Play a Little Funny Game on Us IT Pros' (anyone for some PacMan?)
"But they had full domain admin-level access to the infrastructure. And so what they did is they thought they'd play a little funny game on us IT pros, which is, 'I'm going to go set a scheduled task that triggers on log on and wait 60 seconds for you to log on and see that your data is fine. But the moment that you start logging into your servers, I'm going to start PacMan and everything.' And so we basically got to watch our volumes either disappear and get deleted, because there's deletion strings in most modern ransomware. And so if it sees the word 'backup' in a folder or other things, it'll just start deleting or ransoming. And so we actually got to watch our infrastructure start disappearing in front of our very own eyes. And once we figured that out, it was like, 'Whoa, okay, stop logging into servers to see if they're okay.' First thing that you need to do is, if they're in a virtual format, power up the virtual hard disks offline, because maybe the payload hasn't been delivered.
[Click on image for larger view.]
On the Edges of COVID-19
"And so luckily for us, John, in this particular example, we had a 4TB file server that we didn't log into that had been shut off. And we were actually able to recover all of the file server data for this particular customer without having to go through any type of recovery efforts. And you're thinking to yourself, now, 'Okay, well, since this is an admin-level compromise, okay, well, where exactly did that come from?' Well, it's very simple. This attack, it came on the edges of COVID-19 hitting. IT teams were pushed to get VPN setup right now. We said, 'All employees home right now.' And they were working from the home networks, their kids were home from school, everybody was home, and the home networks became very insecure.
"And so what ended up happening was there was a malware payload on one of the desktops that was taken home. And there was admin credentials that were cached on there, because somebody had logged in with full domain admin credentials to do workstation work, which we'll talk about as a big no-no. They were able to harvest those credentials. And they were able to come back into the network and know they were in the network; it doesn't even matter how long they were in for once they had admin level rights, they were able to do what they were able to do. Because we didn't have Zero Trust, we didn't have multi-factor, we didn't have any of that with this particular customer that we were working with. So we were able to save the file server, but no domain controllers, nothing like that. Those were all taken out, John.
'You're Going to be Next in Line'
"Luckily for us, we had some immutable storage that was up inside of the cloud. And so in this case, I understand that, for some of you that are listening, as Azure Blob is not fully immutable, you've got to go to AWS for some cloud stuff to do that. But we had some storage up with a separate key that was stored off-network that the bad guys couldn't get to. And we were actually able to recover some backups in the cloud, bring some domain controllers back online and bring some services back online to get this customer back up and running. But this particular example was was one where we had to take, and we had to do, a cloud recovery. And we're talking about weeks to get everything back online, really. We got payroll and a couple other services back online right away. And so this was a situation that, if you're not looking forward and protecting yourself, you're going to be next in line for one of these attacks."
Of course, Kawula and O'Neill Sr., who is chief technologist at AWS Solutions, touched on many other tips in their presentation, having to do with Advanced Persistent Threat Protection, Remote Desktop Protocol (not the best choice), remote access, training, Multi-Factor Authentication, the Zero Trust model and more.
The Trouble with Cyber Insurance
For example, O'Neill Sr. weighed in on ransomware cyber insurance, something the duo has addressed before.
In this session, O'Neill Sr. was asked about three things people need to be paying attention to regarding cyber insurance.
"Well, the first thing is, is everybody needs to realize, cyber insurance is now the equivalent of insurance for things like hurricanes, or floods in flood zones and all that," O'Neill Sr. said. "And if you've ever bought a house that's in one of those areas, you know, how much different the requirements are for that kind of insurance, versus your standard homeowners thing. And this is the same deal with cybersecurity insurance, it's totally different than a lot of your other business insurance.
"Because, well, let's talk about that Kaseya breach that we just talked about. Think about the billions, possibly trillions of dollars, when you look at it across the entire impacted landscape, that that cost -- no single insurance company could absorb that hit. So they spread it out. And they have a marketplace. And they kind of spread it amongst different insurers that have different appetites, capabilities and that sort of thing. But what they're realizing is that it's happening more and more, and they're having to pay out. And they don't like that.
"Insurance companies are for-profit businesses. So they've started to really look at ways to mitigate their risk, and work things in their favor."
John O'Neill Sr., chief technologist, AWS Solutions
"Insurance companies are for-profit businesses. So they've started to really look at ways to mitigate their risk, and work things in their favor. And one of those comes through the the MFA attestation that we talked about before. And you know, attestation, fancy word for, you're going to give your word that you use MFA. And guess what, your word is good enough on the front end to get the insurance policy in place, and for you to pay them your bill. But if you ever have a breach, and you expect them to pay out, they're gonna come in and verify that what you attested to, what you gave your word on, is actually what was in place. And if it wasn't in place, even though it may have had nothing to do with the attack you were breached by, you're gonna get denied, because it was a requirement of the policy. You didn't meet the requirement. Case closed.
"And the third point, and Dave, you know, this one, and we've lived through it with people that didn't know how this works, you better have a slush fund because they don't pay in advance. So when you're ransomed or when you're hit, and you have to start paying money -- and I'm not talking just the money for the keys or whatever. I'm talking the money to keep your employee employees getting a paycheck because they can't do any work right now -- they're down. I'm talking about the money to pay your consultants to come in and help and that sort of thing. You have to pay all that out of pocket, and then go back to the insurance company and ask to be reimbursed. So you better have a little bit of slush fund available because otherwise, you got to come up with money out of nowhere."
As noted, the half-day summit in which they were presenting is available for on-demand viewing, along with replays of the other two sessions, slides used by all the presenters and other resources. But live attendance provides extra benefits, like a Q&A opportunity with presenters (and a chance at a drone, laptop, action camera or other cool prize), so keep these upcoming events in mind: