Welcome back to the monthly TCE Strategy newsletter! From CrowdStrike to CDK to UHG to AT&T, there is a lot to talk about this month in the world of cybersecurity. Let’s see how this month’s cybersecurity news can help us make better decisions about what is Secure Enough for us, the companies we work for, and our families.

The whole world was CrowdStruck

As I was meandering through the Denver airport the day after the CrowdStrike apocalypse, I couldn’t help but laugh at the “blue screen of death” on one of the 8.5 million broken Windows computers around the world, as this one has a screen that was at least 7 feet high and 12 feet wide. Advertising at its finest:

Friday July 19th was a challenging day for the whole world. Airline traffic was snarled. A large handful of 911 centers went offline, making this a life safety event. Most large companies went to corporate triage mode, as over half of Fortune 500 companies use CrowdStrike as their antivirus provider. If you had a Windows machine using CrowdStrike for antivirus protection that was turned on between 4:09-5:27 UTC on July 19th, your computer looked like the one above. Details on the timeline of events are here and a wonderful video on what CrowdStrike did to brick 8,500,000 computers is here, but the Cliff Notes version is this: CrowdStrike is an antivirus tool, and to do its job it needs to monitor essentially everything going on with a computer. In order to do this, it needs access to essentially everything on a computer. This is called “kernel” access, and you can think of the kernel as the one ring to rule them all on your computer. Because CrowdStrike has kernel access, the possibility exists that a faulty update file can corrupt the computer’s kernel, and when that happens, the computer stops doing anything to prevent further damage from occurring. That’s what the “blue screen of death”, or BSOD, is on Windows computers. All computers have a BSOD, but Linux machines’ kernel crash screens are black and Macs are pink, so I suppose that Macs have a PSOD instead of a BSOD.

CrowdStrike pushed an update file that was full of zeros instead of actual data. The CrowdStrike application wasn’t written to know what to do with a data file that didn’t have any actual data in it, so it caused an error. Because CrowdStrike has kernel access, that error crashed the whole computer instead of just the CrowdStrike application. 

The reasons for how all this was allowed to happen are well explained in the video I provided a link to above, but the most immediate root cause is that CrowdStrike had a horrible lapse in their own quality controls that sent out the wrong update to the whole world, and they chose to send it in one giant push as opposed to starting with a small subset of non-critical companies (read: not the airlines or 911 centers) and then waiting a few hours to make sure that no one made a huge error. From CrowdStrike joining the S&P500 on June 24th to CrowdStrike causing what is likely the largest computer disruption in history, they have had quite the month.

Our technology infrastructure is fragile, and incidents such as this underscore just how dramatic an impact a simple mistake can make. There isn’t much in the way of takeaways around this event, as we as consumers of these products have little influence on how these products are run by their parent companies day-to-day. Hopefully enough people will vote with their wallets and consider CrowdStrike competitors for antivirus products that it will drive better practices across the industry to prevent issues like this in the future.

CDK took the #2 largest ransomware payment record from UHG

We spoke at length about the CDK cyberattack that took down over 15,000 auto dealerships across the USA. New information came out this month that CDK almost certainly paid cybercriminals $25 million dollars for the keys to decrypt their data. For a company that sells cybersecurity products to dealerships to “stop cyberattacks in their tracks,” this is equally ironic as the CNA Financial breach, where a company that sells cybersecurity liability insurance policies paid a $40 million ransom back in 2019, which is $49 million in today’s dollars. The takeaway for this is simple: offline backups. Offline backups are the last line of defense against ransomware attacks, and it’s obvious that CDK (and CNA Financial) didn’t have them. Offline backups are a pain in the rear end to do, provide little value for non-cybercriminal attacks, but are the best, last line of defense against cybercriminals. The Minneapolis Public School District was hacked last year, but did not pay a ransom, likely because they had offline backups.

UHG (Change Healthcare) breach losses go from bad to worse

We covered the UHG (Change Healthcare) breach in the April TCE Strategy newsletter, and UHG announced in April that they spent $872 million on the remediation (including their $22 million ransomware payment). $872 million sounds like a big number, but it is dwarfed by their new estimate of $2.3-$2.45 billion in total cybersecurity remediation costs. Even for a company as large as UHG, cybercrime can have a very meaningful impact on the bottom line of a company.

AT&T has all kinds of cyber issues

Essentially all customers of AT&T were swept up as part of the Snowflake breach, and data about “nearly all of AT&T’s wireless customers and customers of mobile virtual network operators” are now in the hands of cybercriminals. This will likely lead to a new wave of targeted phishing campaigns, SMS texting attacks, and other targeted campaigns to get AT&T customers to give up sensitive data about their usernames/password, financial information, or access to their computer. Please be careful of any emails, text messages or phone calls that you receive from someone claiming to help you. They may be a wolf in sheep’s clothing.

Until next month, stay safe!