Exploring the Intersection of Reality: An Introduction to Mixed Reality and the Mechanics of XR Headsets

Mixed Reality (MR) is a technology that merges the real world with virtual objects or environments, creating an immersive and interactive experience. It combines elements of both Augmented Reality (AR) and Virtual Reality (VR) to seamlessly blend digital content with the physical world.

XR headsets, short for Extended Reality headsets, are devices used to experience mixed reality, as well as virtual reality and augmented reality. These headsets typically consist of a display unit that is worn on the head, along with sensors and input devices to track the user's movements and provide a fully immersive experience.

What is Mixed Reality?

In 1994, interface researcher Paul Milgram defined mixed reality as a continuum in which individual immersive technologies such as VR, AR, or video-based AR (passthrough AR) exist.

In his insightful definition, mixed reality is a state characterized by the degree to which real-world and digital content are blended. Mixed reality can theoretically manifest itself in a device as long as it can handle some form of VR and some form of AR.

How are VR, AR and XR headsets different?

Mixed reality glasses are therefore headsets that can handle VR and AR, as well as all forms of mixing digital and physical reality according to the mixed reality continuum defined by Milgram and Kishino.

With a XR headset, you can either hide the physical reality altogether and replace it with a virtual one (Virtual Reality). Or you reconstruct it digitally in the headset (through passthrough technology, see next section) and enhance it with digital elements (Augmented Reality).

In Virtual Reality headsets, the physical environment appears on opaque displays and is entirely digital, in contrast to augmented reality headsets with translucent optics, such as Hololens 2 or Magic Leap 2. Here, the physical environment remains visible through a piece of glass or plastic and is selectively overlaid by digital elements projected directly into the eye.

The industry often equates mixed reality with pass-through technology and makes an unnecessary distinction between mixed reality and Augmented Reality: while the former enables a stronger interaction between digital and physical reality, the latter simply extends and enhances physical reality, it is said.

But mixed reality headsets and augmented reality headsets do the same thing – just based on a different technology.

The passthrough technology of mixed reality headsets has many advantages, but also disadvantages compared to systems with transparent optics. Since sufficiently good displays for AR headsets do not yet exist (see: State of AR technology), the industry is taking a detour via mixed reality headsets, while research and development on AR displays continues.

How do Mixed Reality headsets work?

Examples of existing mixed reality headsets include Meta Quest Pro, Varjo XR-3 and Lynx R-1, while Apple's first headset will also be a mixed reality device, according to numerous reports.

Mixed reality headsets capture the physical environment using cameras built into the headset. Algorithms then reconstruct a stereoscopic view of the world and display it on opaque screens.

This so-called passthrough technology is not new, and has been used in devices like the Meta Quest 2 for quite some time. However, the image quality is usually quite poor. Even newer mixed reality headsets like Meta Quest Pro render the physical environment far from perfect.

Passthrough looks more impressive with the high-priced professional device Varjo XR-3. The following video shows a user threading a yarn using a passthrough view. This example shows how well the technology can work with high-quality sensors and high-resolution displays.

Still, there are big hurdles in advancing passthrough technology, as shown in a recent paper published by Meta.