Apple has been granted a patent (number 10,768,698) for a “method and device for eye tracking with content-embedded glints” that hints at future features of the rumored “Apple Glasses.”
Apple Glasses is an anticipated head-mounted display (HMD) for augmented reality/virtual reality/mixed reality uses. Such a device will arrive next year or 2022, depending on which rumor you believe. It will be a head-mounted display. Or may have a design like “normal” glasses. Or it may be available in both. The Apple Glasses may or may not have to be tethered to an iPhone to work. Other rumors say that Apple Glasses could have a custom-build Apple chip and a dedicated operating system dubbed “rOS” for “reality operating system.”
In the patent filing, Apple notes that virtual reality (VR), augmented reality (AR), and mixed reality (MR) are becoming more popular due to their ability to alter a user’s perception of the world. For example, VR, AR, and MR are used for learning purposes, gaming purposes, content creation purposes, social media and interaction purposes, or the like.
These technologies differ in the user’s perception of his/her presence. VR transposes the user into a virtual space so their VR perception is different from his/her real-world perception. In contrast, AR takes the user’s real-world perception and adds something to it. MR is a combination of VR and AR.
These technologies are becoming more commonplace due to, for example, miniaturization of hardware components, improvements to hardware performance, and improvements to software efficiency. As one example, a user may experience VR content by using a head-mounted device (HMD) that encloses the user’s field-of-view and is tethered to a computer. As another example, a user may experience AR content by wearing an HMD that still allows the user to see his/her surroundings (e.g., glasses with optical see-through).
As one example, the HMD tracks the user’s eyes as an input for interactive AR/VR element. As another example, eye tracking is also used for contextual awareness as to the user’s gaze direction or point of fixation. As yet another example, eye tracking is used to perform foveated rendering in order to reduce the resolution of out of focus content to decrease power consumption.
Apple says that eye tracking is a useful input for the AR/VR experience. However, eye tracking is often accomplished with the use of light-emitting diodes (LEDs) fixed to the HMD. This increases the hardware cost and footprint associated with the HMD. Apple thinks content-embedded glints are a better solution
Here’s the summary of the invention: “In one implementation, a method includes: synthesizing an AR/VR content stream by embedding a plurality of glints provided for eye tracking into one or more content frames of the AR/VR content stream; displaying, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtaining, via the image sensor, light intensity data corresponding to the one or more content frames of the AR/VR content stream that include the plurality of glints, wherein the light intensity data includes a projection of an eye of the user of the HMD having projected thereon the plurality of glints; and determining an orientation of the eye of the user of the HMD based on the light intensity data.
