Apple patent involves ‘monocular gaze estimation’ for ‘Apple Glasses’

Apple has been granted a patent (number 11,443,543) for “electronic devices with monocular gaze estimation capabilities.” It involves the rumored “Apple Glasses,” an augmented reality/virtual reality head-mounted display (HMD).

About the patent

The idea behind the invention is to make sure that there are no impediments to viewing AR/VR images on the device. Per the patent, a left display on an HMD may present a left image through a left lens to a left eye box and a right display may present a right image through a right lens to a right eye box. Right and left gaze tracking sensors may gather left eye gaze information from the left eye box and right eye gaze information from the right eye box, respectively. 

During operation in a first mode in which the right and left gaze tracking sensors aren’t impaired, gaze measurements from the right and left gaze tracking sensors may be used in determining the direction of the user’s gaze and the point-of-gaze of the user in three dimensions. 

This gaze information may be used in presenting foveated and distortion-corrected image content on the displays and/or in otherwise operating the left and right displays. 

During operation in a second mode in which one of the gaze tracking sensors is impaired, gaze information for the impaired sensor may be estimated from gaze information for the unimpaired sensor in combination with an object depth estimate based on depth buffer information, vestibulo-ocular-reflex gain information, and historical eye movement information of either/both eyes. 

Summary of the patent

Here’s Apple’s abstract of the patent: “A head-mounted display may display images in eye boxes. Right and left gaze tracking sensors may gather left eye gaze information from a left eye box and right eye gaze information from a right eye box. During operation in a first mode in which the right and left gaze tracking sensors are not impaired, gaze information from the right and left gaze tracking sensors may be used in presenting foveated and distortion-corrected image content on the displays and/or in otherwise operating the left and right displays. 

“During operation in a second mode in which one of the gaze tracking sensors is impaired, gaze information for the impaired sensor may be estimated from gaze information for the unimpaired sensor in combination with an object depth estimate based on depth buffer information, vestibulo-ocular-reflex gain information, and historical information.”

About Apple Glasses

When it comes to Apple Glasses, the rumors are abundant. Such a device will arrive in mid-to-late 2023. Or maybe 2024. It will be a head-mounted display. Or may have a design like “normal” glasses. Or it may be eventually be available in both. The Apple Glasses may or may not have to be tethered to an iPhone to work. Other rumors say that Apple Glasses could have a custom-build Apple chip and a dedicated operating system dubbed “rOS” for “reality operating system.”