BMW achieves notable success with Extended Reality

BMW recognizes the immense possibilities that lie within the incorporation of augmented reality (AR) glasses and virtual reality (VR) devices in cars. The company refers to the convergence of AR, which overlays digital information onto the natural optical field of view through specialized electronic glasses, and VR, which immerses users in entirely synthetic environments through head-mounted displays, as Extended Reality (XR). In collaboration with Meta Reality Labs from California, the Bavarian automobile manufacturer is actively conducting research in this domain.

The collaborative efforts of both companies have yielded the initial outcomes of their research efforts. Presenting XR content within a moving vehicle has proven to be a complex task, requiring the system to effectively distinguish between the user's head movements and the vehicle's pitch, yaw, and roll motions on the road. Maintaining a seamless, uninterrupted XR display despite factors like turns, bumps, or sudden braking poses a significant challenge. While VR headsets employ cameras and motion sensors to accurately determine their position, the dynamics of fast-moving vehicles disrupt the sensors' signals due to the conflicting information from both the device's movement within the vehicle and the vehicle's movement within the world. This discrepancy results in the headset's motion sensors perceiving movement, while the cameras perceive a stationary environment within the car's interior.

The challenge of displaying stable virtual content in a fast-moving vehicle, caused by the discrepancy known as motion in a non-inertial frame of reference, has been addressed through a collaborative effort between BMW and Meta researchers. To overcome this obstacle, the integration of information from the vehicle's sensors into the tracking system of a VR headset was pursued as a solution.

BMW and Meta Reality achieved their initial triumphs through the integration of Meta Reality's tracking system, utilized in the Meta Quest VR headset, with the car's sensor data. This collaborative effort resulted in the realization of stable and captivating gaming, entertainment, productivity, and meditation experiences. The engineers accomplished this by leveraging the tracking system's capability to calculate both the device's position relative to the vehicle and the vehicle's position relative to the external world simultaneously. This accurate anchoring of virtual objects serves as a crucial foundation for delivering immersive VR and MR experiences while driving.

The two collaborators have achieved success in the latter aspect. Through the integration of vehicle-generated motion data and the sensing capabilities of the XR headset, the motion of the vehicle can be effectively subtracted, allowing for more precise tracking of the headset's movement within the vehicle. As a result, it becomes possible to display content in real-time, synchronized with the vehicle's movement, whether inside or outside the vehicle, even at high speeds and irrespective of the road conditions, be it a smooth, straight road or a challenging one with sharp curves.

In collaboration, BMW and Meta Reality have jointly created a proof-of-concept prototype capable of processing motion in six degrees of freedom while accurately determining the position of the headset in relation to the vehicle. Furthermore, the system seamlessly incorporates the vehicle's position relative to the surrounding environment by integrating vehicle signals into the VR system.

Moving forward, the primary objective is to establish an industry standard for the seamless integration of XR devices with vehicles. BMW emphasizes its willingness to collaborate with additional partners in order to achieve this goal.