Mixed Reality Capture: Why it Matters and How to Do It
Gabriele Romagnoli
Showcasing the best of XR & AI for creatives and professionals | Tech Ambassador | Podcast Host | Speaker
In this episode, we sit down with AJ 'Dr Doom' Shewki , the founder of LIV the best mixed reality capture tools, to discuss the importance and challenges of capturing and sharing XR gameplay, Liv’s partnership with Meta, and how they are making MR catpure a breeze thanks to their Creator Kit. For the product Spotlight we will look at a Quest app built by the LIV team that will give you a glimpse into the future of capturing and experiencing immersive gameplay.
Before we dive into the interview I wanted to give a quick shout-out to the XR Awards. If you haven’t secured your ticket yet you should. The XR Awards🔗 will take place in Rotterdam on the 5th of December and bring together a night of red-carpet highlights, the celebration of excellence, and unique access to the world’s most influential names throughout the industry.
Use the code XRAIspotlight15OFF for a 15% discount 😉
Interview with Dr. Doom
Why is it taking so long to develop an easy-to-use solution for recording third-person footage of XR gameplay?
Dr. Doom: It really depends on how you define third-person footage. There’s a difference between using a physical external camera and virtual cameras that are inside the app. When you use an external camera, it adds complexity—like needing a PC to handle the extra processing power. Plus, our solution involves both the Liv app and SDK, which adds some setup. The good news is we’re working on a leaner solution, where you can pull out a virtual camera just like using your phone to take a picture in the real world.
Can Quest apps take advantage of Liv's mixed reality capture, or do they need a PC?
Dr. Doom: For mixed reality capture—the composite of a real person into the virtual scene—you still need a PC. If a developer wants to film high-quality trailers or marketing materials for their app, they’ll need the power of a PC to capture 4K resolution or better. So while the gameplay runs on Quest, the high-end capture still requires external hardware for now.
Is mixed reality capture evolving toward more “augmented reality”?
Dr. Doom: Absolutely. As the industry moves towards AR, recording these experiences becomes crucial. Right now, mixed reality is a bit more challenging because you’re not using a green screen. To segment the human from the background, we rely on post-processing. It’s time-consuming, which is why most developers use this method for high-quality trailers. In the future, as AR headsets become more widespread, capturing those experiences will become more streamlined and necessary for storytelling.
Will we ever see an easy-to-use, no-PC-required option for creators?
Dr. Doom: We’re getting closer. Right now, for high-quality mixed reality, a PC is required. However, we’re launching Quest Creator Kit (QCK) later this year, which will allow users to film gameplay directly on the Quest without a PC. It’s a game-changer for virtual cameras—selfie, third-person, and drone views will be available natively on the headset. This makes content creation more accessible, especially for players who aren’t equipped with high-end setups.
How easy is it for developers to integrate Liv’s Quest Creator Kit into their apps?
Dr. Doom: It’s pretty straightforward! For a competent developer, it should take less than 30 minutes to integrate. You drag and drop the QCK into the project, assign the player avatar, and choose what you don’t want to render in the output—like certain UI elements. It’s as simple as that. If there’s a custom camera system already in place, there might be some back and forth, but we’re always available to help developers sort that out.
What’s the actual value for developers in adding these recording features to their apps?
Dr. Doom: There are two main benefits. First, it gives developers high-quality footage for trailers and marketing, which can boost sales. Mixed reality trailers tend to convert better because they show the gameplay experience in a more relatable way. Second, it empowers content creators in their communities to make and share their own videos. If you give creators easy tools to capture gameplay, they’ll create content that brings more eyes to your game, boosting long-tail engagement.
What are some creative ways content creators are using Liv beyond just gameplay footage?
Dr. Doom: Some creators use Liv to insert their avatars into VR games as a way to represent themselves. We’ve seen VTubers overlay their avatars even outside of VR, like when playing regular games. Competitive gamers also use it to record smooth, high-FOV footage of their highlights. Then there are creators with massive followings, like Lizi and Nacy, who use green screen setups to film high-quality mixed reality content for their TikTok and YouTube channels.
Recommended by LinkedIn
How are non-gaming industries using Liv?
Dr. Doom: About 30% of our users aren’t in gaming at all—they come from industries like education, architecture, police training, and even space agencies. These users are building simulations in VR and need a way to showcase their work to clients or stakeholders. Liv helps them contextualize what they’re building by creating videos that can be shared with people who might not understand VR or have a headset. It’s a powerful way to sell or explain a project in both internal and external environments.
How do you envision immersive gameplay being consumed in the future?
Dr. Doom: The player and viewer experiences are going to converge. Today, you play a VR game fully immersed, but when you share that experience, it’s reduced to a 2D video. In the future, I see immersive video becoming the norm—where the viewer gets a sense of presence just like the player. That’s what we’re working on with Project Spatial, which is aiming to capture and share content volumetrically. It’s early days, but by 2025, we hope to have a public beta that starts bringing that vision to life.
Will volumetric capture allow players to re-experience someone else’s gameplay inside their own world?
Dr. Doom: Exactly! That’s part of the dream. Imagine watching your friend’s gameplay through a volumetric "window" in your own game, or even being able to jump directly into their experience. We’ve experimented with this idea, called Holographic Calls, where you can peek into someone’s gameplay in real-time. Long term, we want to bring players and their fans closer by using the unique sense of presence VR offers. It’s still early, but we’re making strides toward that future.
What role do you see AI playing in improving the capture and gameplay-sharing process?
Dr. Doom: AI is going to help in many ways. We’re already using AI models for background segmentation to speed up the process of cutting out humans from mixed reality footage, which previously required manual rotoscoping. AI will also help with better motion capture and even lower-body puppeteering—there are papers showing it can be done with just a few hours of motion data. Long term, I think AI will improve video encoding and compression, making immersive video more accessible on mobile and standalone devices.
What does your recent partnership with Meta mean for developers and creators?
Dr. Doom: The partnership is a big deal because it unlocks Meta’s Presence Platform for Liv. That means developers using features like hand tracking, scene understanding, and body tracking can now use Liv to capture their content. It required a lot of engineering on our end to integrate with the Oculus runtime, but it means we can accurately capture and render those features in trailers or marketing content. This is a big step toward providing comprehensive capture solutions across all Meta apps and experiences.
How did you navigate the relationship with Meta, given they were developing their own tools?
Dr. Doom: It took us seven years to get to this point. For most of that time, Meta was developing their own solutions. My advice is to stay transparent and keep building. We always showed Meta what we were doing and offered to collaborate, but we respected that they were working on their own tools. Eventually, we knew that if we kept doing great work, they’d come to us. We were focused solely on this, while Meta had many priorities. Our persistence and expertise in this niche paid off when the timing was right.
Product Spotlight: LIV Creator Kit
The LIV Creator Kit (LCK) is a flexible, modular, and lightweight SDK that allows users to record themselves directly within your Unity VR game or app. It includes a plug-and-play camera prefab or a full API to create custom camera prefabs.
I have tried it and it’s awesome! I wish every app had something like this to help the team and users share quick gameplays from other perspectives than the shaky and nauseating first-person view.
Features include:
That’s it for today, and don’t forget to subscribe to the newsletter if you find this interesting
See you next week
CEO @ LIV | Building the leading XR content creation platform
1mothanks Gabriele :) appreciate getting to share on your pod. 🫡