Is Your SOC Ready For an Incident?

This week, I had the opportunity to participate with our Professional Services team in our, SimSpace's, first in-person team assessment, Live Fire Exercise, since before the pandemic. It was not exactly my role, as a sales engineer, but the team needed some help and I wanted to see how they do what it is they do.

The Live Fire Exercise (LFE), is a scheduled opportunity to put a security team through their paces and assess how they do their job. In this case, we took 4 shifts of an incident response team for a Fortune 50 retailer, for 16 hours on 2 days, through the worst shifts of their life, hopefully. We succeeded in 10 phishing attempts that led to exfil over C2 channels, got blocked with Follina by their EDR, succeeded with our crafted zero-day ransomware, and succeeded with a web shell to the DMZ web server. The red team (I was not on the red team) was very mean. The exchange server and domain controller were completely compromised and the exchange database and AD configs and SAM files were all successfully exfiltrated.

As someone who has been shot down multiple times (simulated) in exercises, I think this is a good thing, to get attacked over and over again in a short window.

The joy of this was that the incident response teams had their own EDR/SIEM in our completely safe range. So, there was no chance of us hitting anything in their production environment.

What was the point of all of this?

First, story time. January of 2012, I went on my first off-station mission in the C-130. I was one of 3 copilots on the plane. We were going to Ft Bragg, NC to support "Large Package Week" (I didn't name it that, the Army did). It is a regularly held training exercise where the 82nd airborne division invades the woods of North Carolina and goes to war for a week or two. "Large Package Week" usually kicks off with an airborne infiltration, with dozens of aircraft airdropping troops, equipment, and vehicles. Nothing like following a few miles behind a plane dropping a firetruck, another dropping a front loader, and another dropping a couple howitzers. We dropped a couple of the humvees. And everyone made several trips to drop off the thousands of paratroopers. Then we would spend the week practicing our tactics while resupplying them during the ground portion of their exercise (the part they actually cared about). We would also exfiltrate "prisoners", "evacuees", and the wounded (no quotes, people got hurt out there). That was the point of the firetruck, so we could land on the dirt strip runway.

"Cool story, Josh. What does this have to do with what you were talking about before?"

The guys on the ground would have observers who graded how they operated. We, the aircrew flying the planes, would also have observers in the plane with us, grading us. During that week, we were at war, too (but in the Air Force version, we sleep in hotels and the army guys sleep in the field. It's a rough life but somebody's got to do it). We would get graded as a whole AF contingent for process, planning, execution, and general success. We would also get graded as crews. We took lots of extra pilots, so everyone could get an opportunity to fly in the seat on one day or another. The feedback we got from these exercises was more valuable than gold (or gasoline in 2022).

In the Air Force, we had a term for the coordination and interaction of the aircrew members on a plane, or between planes in formation. It is called Crew Resource Management (CRM). CRM is hugely important. Even in pilot training when there were only 2 pilots, the student and the instructor, and maybe another plane in formation, there was still a grade for CRM. If the pilot does something without telling the other pilot, it can get very confusing and dangerous in the air. It's important to verbalize and show who has control of the yoke/stick, especially when passing control back and forth. Imagine having two steering wheels in your car, and they both work. You can't have two people trying to control things. Similarly, in the C-130, I had loadmasters in the back of the plane who are loading/unloading things on the ground, and in charge of dropping things out during flight. They need me to fly in a specific way to do the things they need to do for the mission. Similarly, they can't do their actions without coordinating with the front of the plane. Nobody is more important than the other, we are all part of the crew and success lies upon the crew being in sync and doing each of our own parts effectively and coordinating with each other clearly, from planning to execution to debrief (which then feeds back into future planning).

Well, an incident response team, SOC, or red team are all a "crew". There are different participants who all have a role to fill. Even if you have 2 SOC analysts of equal status, you don't want them to do double work, so they need to coordinate and deconflict. Someone needs to track the whole investigation process. Someone else might search for threat intelligence. Someone might start reverse engineering found binaries. Someone else might work to decrypt encrypted files. Someone else might work to restore backups. But this all doesn't just happen on its own. The team has to coordinate. Often, a lead will delegate out tasks and responsibilities. That is also a key of CRM.

In the Air Force, we had standardized checklists, and we had standardized forms for our mission plans, so everyone knew where to find the information they needed. We also planned, then briefed, then executed, and debriefed, every single flight. You didn't step onto the plane if you weren't at the briefing (generally). That process, even outside of the plane, on the ground, before the mission has even begun, is also part of CRM.

And during these exercises, the aircrew would get graded on their skills (perfect timing to the target, perfect aim with the airdrop, etc.), but would also get graded and evaluated on their CRM. The CRM aspect was the more important evaluation. Usually, we were pretty stinking good at the skills. What causes crashes and mishaps is usually poor CRM, even for a short amount of time. And bad CRM or bad processes are hard to see from the inside, usually. This is why we have evaluators and observers watching and grading us during exercises. Because we have to identify and remediate problems to keep people alive.

So, let's return to the cybersecurity incident response team. The organization has been through a penetration test, a vulnerability assessment, and audits. So, they're good to go and ready for an incident, right? We checked that people are qualified and trained, right? Maybe. But, how do we check that the team is any good together? Or that the procedures, playbooks, ticketing systems, incident tracking tools will be effective in a real incident? Umm, guess you have to wait until a bad attack and see how things go, right?

Personally, I'd rather know that my SOC is garbage long before the rest of the world knows because the threat group posted about it to their social media and is holding our data for ransom.

But, how do you do that?

Make everyone get certs!!

Right?!

Maybe no.

What if we could attack them with a zero-threat environment and see how they do?

Well, that's where SimSpace comes in. We build ranges that look, feel, and smell like a real enterprise network. And we can size them from something small (think an app developer startup), to one of the largest banks in the world, to a whole government entity. We have all of those. But wait, there's more. On the user endpoints, we have emulated user interaction. Not a packet creator or a pcap replay. We have automation (think of NPCs) where users login based on realistic profiles (office workers who check in at 8 and log out at 5, take lunch breaks, send emails, open emails, post to blogs, search social media, make spreadsheets and documents, upload the documents to the file share, download files from the file share, open email attachments, forward emails, delete emails, etc.) This creates realistic network traffic, and realistic processes and footprints all over the endpoints and servers. We also have a neat chimera attack engine that uses real malware, phishing emails (the ones that the emulated users will open and execute, just like Carl always does), and more.

The really great thing about those aircrew exercises we went on, the planes were tricked out just like we would have them "downrange" in a deployed environment. Because realism matters. Just like that, we can make our range feel just like what the SOC team uses on a daily basis. We can install your SIEM, your EDR, your SOAR solution, your firewalls, your routers and switches. We can also add virtualized OT/ICS, or use the virtual bridge to connect appliances to the range servers and include real technology in the range.

Well, this week, the customer got their eyes opened. They have things to improve about their process and their policies. They also learned a lot about their security tools. They had issues with some of their tools in the range. We set it all up pretty fast and there are things that will be improved for next time. But, the veterans, the analysts who remember when Windows was new, said that this was the first training, and the first range, where they got to use their actual tools. Those who have done similar exercises on other ranges in the past said that they usually got whatever tools existed in the range. If you don't know Splunk, you're becoming an expert in a week, only to use something completely different when you get back to work. Hope your queries work the way you want...

It was exciting to see. I was an observer and passed notes to the evaluators. I got to really see the product I'm selling. And it got me thinking about all the exercises I did in the military and how much we learned every time. I'm excited to give others that opportunity with their cybersecurity teams.

If you're curious how it works, here's a nice little demo video. But I'd love to show you, just message me and we'll set it up.