Apple patent involves a ‘Scene Camera’ for the Vision Pro

Apple has been granted a patent (number US 11841510 B1) for a “Scene Camera” for the upcoming Vision Pro.

The US$3,499 (and higher) Spatial Computer is due in early 2024. Apparently, it will only be available in limited quantities at first.

About the patent

Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. 

When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view. Mixed reality (MR) covers a spectrum from augmented reality (AR) systems that combine computer generated information (referred to as virtual content) with views of the real world to augment, or add virtual content to, a user’s view of their real environment, to augmented vitality (AV) systems that combine representations of real world objects with views of a computer generated three-dimensional (3D) virtual world. 

The simulated environments of virtual reality systems and/or the mixed environments of mixed reality systems may be used to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the viewer’s environment, applications that generate 3D virtual worlds, interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the Internet, exploring virtual landscapes or environments, or the like.

Apple’s idea is to equip the Vision Pro with a scene camera for mixed reality (MR) direct retinal projector systems. It would capture images of the real-world scene in front of the user. The images may, for example, be analyzed to locate edges and objects in the scene. In some embodiments, the images may also be analyzed to determine depth information for the scene. 

The information obtained from the analysis could be used to place virtual content in appropriate locations in the mixed view of reality provided by the direct retinal projector system. To achieve a more accurate representation of the perspective of the user, the scene camera is located on the side of the MR headset and facing the inside surface of the lens. 

The lens includes a holographic medium recorded with one or more transmission holograms that diffract a portion of the light from the scene that is directed to the user’s eye to the scene camera. In other words, the scene camera captures images of the environment from substantially the same perspective as the user’s eye.

Summary of the patent

Here’s Apple’s abstract of the patent: “Point to point transmission holograms are used to provide a scene camera for an augmented reality glasses display system. A glass or plastic substrate acts as spectacle style lens. A holographic medium is applied to a surface of the substrate, within which is recorded a series of point to point transmission holograms. 

“The construction points of the holograms are arranged at the eye and at the pupil of a camera placed, ideally, to the temple side of the user’s eye. The recorded transmission holograms act by diffracting a portion of the light from the scene surrounding the user that is heading for the user’s eye towards the scene camera. The hologram efficiency is balanced so that the user is still able to see the surrounding scene. The perspective of the view seen by the scene camera is substantially identical to that seen by the user through the lens.”