What All Facebook's F8 Augmented Reality Talk Really Means

AR & VR Takeaways from F8

Nearly three years ago, Facebook made a big bet on virtual reality with its purchase of Oculus, creating big buzz around future computing platforms. Over the past couple days, it’s done the same for augmented reality with a series of announcements at F8. I already shared a few of my thoughts on this topic on Bloomberg TV, but wanted to dive a little deeper into Facebook’s latest VR and AR developments and their significance.

AR

Facebook's new mobile AR platform allows you to place digital objects atop the real world

Ever since Facebook’s MSQRD acquisition (reported to be between $70 million to $120 million) over a year ago for its AR masks, we’ve been waiting to see how they would leverage the company’s technology and open it up as part of its larger AR strategy. (Snapchat acquired Looksery a year earlier for a similar purpose.)

Now we know, with the unveiling of the Camera Effects Platform, developers can build AR tools for Facebook’s in-app camera. At F8, Zuck outlined three categories of AR:

1) Information - Adding information atop the real world (e.g., directions over the street in front of you)

2) Digital Objects - Adding digital objects onto the real world (e.g., a virtual post-it note on the fridge)

3) Enhancement - Enhancing real objects with reviews, novelty filters, edits, etc.

Now that the developer community has a new technology to build on top of the world’s largest social network, here’s how I see them running with it:

• Selfie masks and filters: Developers will create their own selfie masks and tap into Facebook’s data (e.g., a user's location, the objects they’re filming) and depth-sensing technology. Some of these masks and filters will draw information from other apps too as we saw with examples from Manchester United and Nike.
• Games: There’s all-new possibilities now that you can turn any flat surface into a game board to play an interactive AR game. Facebook is using a state-of-the-art 3D mapping technique, SLAM (simultaneous localization and mapping), to recognize flat planes in the real world environment and superimpose digital objects on top of them. This leverages the inside-out tracking technology that is being developed at Oculus.
• Graffiti: There will be strong appetite for drawing animated AR artwork on building walls or tables, and allowing people to add their personal tag to the real world around them. You could even leave virtual Post-It notes in relevant places (e.g., an AR sticky note on the fridge reminding your partner to pick up more milk).
• “Business AR”: Businesses will be excited by the potential AR applications at their physical locations. How cool would it be to see social recommendations of your friends' favorite dishes at a restaurant you're visiting overlaid on top of the menu?
• AI + AR: Developers building apps for Facebook can now tap into a user's precise location, utilize FB's object recognition and depth detection to create various AR novelty effects. Facebook is using convolutional neural networks to detect what's going on in a photo or video, live in real time, making it possible to suggest contextual AR effects for that particular scene.

Being first can be important to developing a head start on developers wanting to build AR effects. And it’s clear by opening up their AR platform to outside developers, Facebook is going for scale. They don’t need to be responsible for developing the killer AR app, but they want it on their platform and will happily give away a ton of technology for free in order for that to happen.

As Facebook courts developers, it’s also making itself more attractive to brands. It can now offer them multi-faceted advertising and partnership opportunities: They can run their chatbots on Facebook Messenger, gain targeted insights through computer vision recognition of user-generated videos and develop intelligent brand-sponsored overlays on what users see through the camera. With one F8, Facebook added a whole other category of monetizable real estate for brands! 

Of course, everyone is wondering how this affects Snap, which just announced its own 3D filters. One area could be advertising costs. I see Snapchat as a closed ecosystem and brands currently pay upwards of $800,000 a day for targeted lenses and filters. Facebook will likely bring those costs down by making AR a more open ecosystem.

After these announcements, however, I don’t see Snap as a competitor from an AR perspective. I think Apple could be, however, if they build these capabilities into their own developer platform on iOS. With the rumored depth-sensing, more advanced iPhone 8, it seems like a good possibility. But Facebook is THE social platform, and the social graph insights they can apply to AR experiences will be an advantage over iOS.

Amid all this excitement, it’s important to remember the bigger play is beyond smartphone cameras. Not coincidentally, Facebook revealed it is currently building AR glasses.

VR

Social VR from Facebook's new 'SPACES' product announced at F8

On the VR front, Facebook launched Spaces to make VR more social and interactive. Initially available in beta for Oculus Rift users with Touch controllers, Spaces is a cool hangouts-type app where you and several friends are in the same space and can watch immersive 360-degree videos at the same time. You can also have other friends' avatars join and include live video calls of friends in the room too.

While this isn’t as flashy as the AR news, it does tell us a few interesting things:

Platform-agnostic: It’s interesting that Facebook wants Spaces to be cross platform and not just tied to Oculus. This means you can talk to your Oculus, PSVR and HTC Vive friends in the same hangouts room. It certainly gives social VR companies like Altspace, PlutoVR, High Fidelity and VRChat some competition.

Holograms: I think Spaces has a long way to go, but realistic-looking holograms is the future here. Imagine being able to cohabit spaces with your friends, but in live hologram form, like Princess Leia from Star Wars. Companies like 8i (I’m an investor) are already working on holographic technology.

Samsung Gear: There weren’t really any innovations around 360-degree video, so it appears Facebook is de-emphasizing optimized VR for Samsung Gear, which is powered by Oculus.

With its purchase of Oculus, Facebook made it clear it intended to be the leader in VR. With its announcements this week, it’s clear it also intends to do the same in AR.

Originally published in Fortune Magazine in April 2017. I'm an early stage investor at a fund I helped set up in 2012 named Signia Ventures. I'm passionate about new distribution and content businesses in VR/AR as well as interested more generally in vertical applications of computer vision and in new content discovery archetypes in the emerging Alexa-voice ecosystem.