Recall is Microsoft's "MCAS Moment"

Since Microsoft announced Copliot+/Recall, I've been scratching my head wondering how this got past their security and privacy teams. Anyone who has done a modicum of offensive cyber work (or even those who have defend against it) knows it's ripe for abuse. I hoped to hear from some of friends who work in security at Microsoft that things weren't as bad as they appeared. But as Kevin Beaumont highlighted, they've been notably absent from social media since the announcement.

Yesterday, Zac Bowden revealed in a fantastic article that even internal employees were limited in their ability to test Recall.

When I read this, it was like a lightbulb went off. Previously, I was befuddled that Recall was being released with so many obvious security issues (some of which Microsoft announced they're changing). Then I remembered reading the book Flying Blind by Peter Robison and it hit me: Recall is Microsoft's MCAS.

Now all analogies are imperfect, some more so than others. But in both Recall and MCAS, features were developed in secret and hidden even from those within the organization. In both cases, there were people in the organization who could have said "here's how to do it safely." And in both cases those people appear to have been intentionally removed from the process, likely because product managers were focused on a singular goal. Unfortunately, in both cases that goal wasn't security/safety.

There are certainly lessons here, ironically from Microsoft's own SDL. One of the first principles of the SDL is "Design – ensure that the design doesn’t naturally allow attackers to easily gain unauthorized access to the workload, its data, or other business assets in the organization." You can't convince me that happened here during the design phase. In fact, when Microsoft is discussing wholesale security architecture changes from announcing the product on May 20th to today (June 7th), it's clear that security design and threat modeling weren't at the top of anyone's mind.

For what it's worth, while I think the feature itself is still questionable from a privacy and liability standpoint, the changes Microsoft announced for Recall are likely going to be effective in blocking the majority of attacks identified thus far. I say "likely" because none of us raising some the initial concern about Recall have our hands on the changes announced today.

One of the major changes announced is that Windows Hello will be required to enable Recall at all. Additionally, "proof of presence is also required to view your timeline and search in Recall." Finally, snapshots are now encrypted on disk and "Recall snapshots will only be decrypted and accessible when the user authenticates. In addition, we encrypted the search index database."

Side note: replacing the scary term "screenshots" with the much less scary term "snapshots" is an obvious attempt to confuse less technical consumers. Whoever made this decision should be ashamed of themselves.

This change in architecture parallels the MCAS story in so many places. Like Recall, MCAS was designed without clear notification to the rest of the organization, let alone the product's implications. Just as Recall was initially deployed without encryption of its database and screenshots (I refuse to call these snapshots), so too was MCAS deployed without redundant angle of attack sensors. In the case of MCAS, that meant that the failure of a single sensor could result in catastrophe. In the case of Recall, lack of encryption meant that an administrator on the machine could trivially view the user's snapshots.

Boeing failed to train pilots on the existence of MCAS. Additionally, those involved in assessing the safety of MCAS relied on faulty assumptions of how long it would take a pilot to respond to an unsafe condition. Similarly, in the initially announced release of Recall, Microsoft engineers relied on flawed assumptions that data would be safe as long as only the currently logged in user could read it. But of course that includes malware running under the user context - something security teams obviously have in their threat model but developers seldom do. The new changes requiring Windows Hello and proof of presence should mitigate this problem with Recall, similar to how pilot training and system redesign has helped mitigate risks with MCAS.

There are certainly lessons to be learned from Recall that will doubtless be captured in software engineering and business case studies. But the core of both Recall and MCAS is that getting involvement from across your teams is critical (actually listening to them doubly so). In the case of MCAS, the product team allowed planes ordered with only one angle of attack sensor to "push to prod." Microsoft pushed Recall to preview (Copilot+ NPU-enabled laptops don't ship until later in June) without ironing out obvious security issues. Luckily Microsoft had both the time and good sense to take a step back and get security engineers involved.

Thankfully, unlike MCAS Recall hasn't killed anyone (yet). Given the continued involvement of stakeholders from the frankly brilliant security people at Microsoft (many of whom I'm proud to call friends) I'm much more confident in Recall's direction. I'm one of the many people who has ordered a new Copilot+ laptop, and while I did so primarily for security research, I'm also interested in trying out the feature. While it doesn't feel like something I need, I said the same thing about a smart phone many years ago (and was wrong then).

#HugOps to all my security friends at Microsoft. We all know this wasn't on you.