The case for CloudXR - XR Streaming.

Tethered devices

How does a Virtual and Augmented reality (AR/VR) enabled device run an AR/VR app/solution? One way, which is also the way tethered devices (i.e. devices connected by a cord to a computer system like the HTC Vive) do it is to depend on a computer with higher-end graphics and computing power. This allows the XR (Extended Reality (AR/VR/MR)) device to focus on delivering an immersive experience while outsourcing the heavy lifting to the computer system, depending on it for things such as high-quality rendering, real-time lighting and effects, things that require powerful graphic processors and computing power. Hence, for a long time in the VR world one had to choose either a VR headset that provides high-quality graphics & immersive experience but tethered to a location, which usually requires people to move the cord around as one goes through the VR experience, making the experience sometimes tiresome, frustrating collaboration & breaking the “illusion” from time to time. (example below) or a tethered VR headset.

A schematic diagram of the HTC Vive. The participant stands wearing the HMD (Head Mounted display) in an open space of about 2m×2m wide. The HMD is connected to the computer using a long cable. (source: Manivannan Muniyandi)

Definitions(Skip ahead if you must)

OpenXR: is a royalty-free, open standard that provides high-performance access to Augmented Reality (AR) and Virtual Reality (VR)—collectively known as XR—platforms and devices.

Latency: Latency is the time it takes for data to be transferred between its original source and its destination, measured in milliseconds. Internet latency and network latency affect satellite internet connections, cable internet connections, as well as some WiFi

Mobile edge Computing(MEC): MEC helps operators provide low-latency and high-bandwidth 5G applications. MEC uses an architecture dependent on Virtual Network Functions handled by a Virtualization Infrastructure Manager (VIM)

Head Mounted display (HMD): A head-mounted display is a display device, worn on the head or as part of a helmet, that has a small display optic in front of one or each eye. An HMD has many uses including gaming, aviation, engineering, and medicine.        

Mobile/Standalone VR devices

A Mobile (a.k.a Standalone) VR headset on the other hand handles running its apps differently, they are not tethered to computer systems and they have to rely on their own graphics and computing power to run. Hence, XR headset makers have the complicated job of cramming large, complex and powerful computing hardware into small spaces due to the fact that XR devices have to be as small and lightweight as possible, while also making sure it is absolutely safe and doesn’t overheat because they are meant to be on the heads of consumers, and very close to their eyes. Mobile XR devices usually possess limited processing power, low fidelity graphics, insufficient memory for large datasets (or applications) that are useful for enterprise or extremely large or complex solutions. Some XR gadgets like google cardboard even rely solely on the power of mobile phones which are inserted into the headset(s). Although this has held true for a long time, there are now many mobile VR headsets that are doing very well in delivering immersive experiences that can be compared to a tethered device, one of such mobile headsets is the Oculus quest 2 (aka Meta Quest).

Now with the understanding that things like a devices processing power, memory to accommodate large datasets to run complex XR solutions, the fidelity of graphics & ability to move around freely in the virtual world (something mobile XR gadgets excel at over tethered) all contributes in delivering or breaking immersive XR experiences, it has become necessary to bring the processing power and high fidelity graphics that a tethered device has to mobile XR device, this is where NVIDIA’s CloudXR comes in!

Cloud XR - XR Streaming

CloudXR is a solution for streaming virtual reality (VR), Augmented reality (AR), and Mixed Reality (MR) content on a remote server— cloud, data centre, or edge to a mobile XR device. XR apps and contents would be hosted on very powerful servers, some of which are more powerful than most computer systems. The next-generation NVIDIA RTX™ Server is an excellent example, it delivers a giant leap in cloud gaming performance and user scaling. It packs 40 NVIDIA Turing™ GPUs into an 8U blade form factor, though that may sound like gibberish, it simply means it can render and stream even the most demanding games. The server provides cloud gaming performance and user scale that is ideal for Virtual and Augmented reality solutions (and other Mobile Edge Computing (MEC) software). The XR experience can be streamed to various OpenXR enabled mobile XR devices from the server, delivering interactive & immersive experiences that would otherwise be impossible to run on traditional mobile XR devices. To put it simply, the cloudXR approach delivers XR experiences over a network (5G/WiFi), relying on powerful servers promises to eliminate the traditional limitations of mobile XR devices mainly because most of the “hard work” is done outside of the XR device and streamed live, while also receiving inputs from the user in real-time. (see architecture example below)

Augmented and virtual reality are two of the most latency-sensitive applications (i.e. they are dependent on constant communication over a network) for Mobile Edge Computing (MEC), hence they require immense computational power for high frame rates to reduce latency. By deploying NVIDIA RTX Servers at the edge, or other CloudXR architecture, anyone can deliver low-latency AR/VR applications by streaming them over a network to deliver immersive experiences.

Conclusion:

XR gadget makers/manufacturers have done and keep doing some amazing things in advancing the computing strength of mobile XR devices, a key example which I keep going back to is the Quest 2, which currently utilizes the Qualcomm Snapdragon XR2 Chip developed by Qualcomm specifically for delivering “Unrivaled XR experiences” this was indeed a significant leap in mobile VR computing power. Although these chips are considered by some to be ahead of their time, delivering smoother experiences on HMDs and Mobile devices, a lot of advancement still needs to be made to keep up with the growth of the XR industry as adoption increases and demand for better and smoother experiences also increases. They would have to find more ways to integrate such computing power offered by the RTX server into these small mobile XR devices (which they are already trying to do, spending billions on R&D for many years) or they could take the CloudXR route delivering a combination of both high-quality graphics and mobile freedom. It is important to note that whichever mode of execution they choose has its drawbacks, one drawback of CloudXR is that it relies on a sensible internet connection depending on the experience being streamed, writing this article from Nigeria in Africa where the internet connection sometimes get very bad makes this drawback significant for me, although this would be a significant drawback for enterprises use cases because of their access to good internet infrastructure.

There is no doubt that XR streaming is at a very early stage, I look forward to seeing how the space would grow in the coming decade, Until the next article, Keep learning!