Apple Glasses could sport ‘inside-out positional, user body and environment tracking’

The rumored “Apple Glasses” could support virtual and mixed reality with inside-out positional, user body and environment tracking. Apple Glasses is the moniker associated with the anticipated head-mounted display (HMD) in the works by the tech giant.

A newly granted patent (number 10,838,206) involves a HMD that uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.

The objective of immersion in a virtual world is to convince a user’s mind to perceive a non-physical world as if it were real. In VR, immersion is achieved by displaying computer generated graphics that simulate a visual experience of a real or imagined world. Apple notes that the quality of immersion is subject to several important factors. For instance, characteristics of the display such as image quality, frame rate, pixel resolution, high dynamic range (HDR), persistence and screen-door effect (i.e., the visible lines between pixels on the screen). 

The quality of the immerse experience decreases when the displayed field of view is too narrow or if the various tracking functions are slow and/or inaccurate (leading to disorientation and nausea; otherwise known as simulation sickness). 

Immersion is also impacted by the camera system performance such as the image quality (noise, dynamic range, resolution, absence of artifacts) and the coherence between the virtual graphics (3D modeling, textures and lighting) and the pass-through images. 

In mixed reality (MR), virtual elements are composited in real-time into the real world environment seen by the user. Physical interaction between the virtual elements and real world surfaces and objects can be simulated and displayed in real-time. 

In the new patent, Apple says that tracking of various elements is essential for achieving a high end VR and MR application experience. Among these elements, positional head tracking, user body tracking and environment tracking play a key role in achieving great immersion. Apple wants its HMD to excel at such functions.

When it comes to Apple Glasses, such a device will arrive next year or 2022, depending on which rumor you believe. It will be a head-mounted display. Or may have a design like “normal” glasses. Or it may be available in both. The Apple Glasses may or may not have to be tethered to an iPhone to work. Other rumors say that Apple Glasses could have a custom-build Apple chip and a dedicated operating system dubbed “rOS” for “reality operating system.”