Apple has been granted its (by my count) billionth patent for the rumored “Apple Glasses,” an augmented reality/virtual reality/mixed reality head-mounted display (HMD). This patent involves an “AR/VR controller with event camera.”
HMDs are often used in augmented reality (“AR”) and virtual reality (“VR”) systems to present a user with virtual graphical objects that either complement or replace a surrounding real-world environment in the user’s field of view. To enhance that virtualized experience, AR/VR systems may include a secondary device that enables the user to interact with the virtual graphical objects.
For example, the secondary device may correspond to a sword or other implement in the context of the virtualized experience. The AR/VR system may update a depiction of the sword in the virtualized experience to mirror a movement of the secondary device by tracking a correspondence between the secondary device and the HMD or another device.
An existing technique for determining such correspondences involves identifying a position and orientation of the secondary controller and tracking that position and orientation over time using image data obtained from one or more conventional frame-based cameras. The steadily increasing resolution offered by such frame-based cameras (e.g., 10-20 megapixels or more) enables AR/VR systems to identify and track secondary device positions and orientations with increasing precision.
However, Apple says that a tradeoff exist with that increasing precision. Image data from conventional frame-based cameras includes information regarding an absolute light intensity at each pixel sensor of the frame-based camera. When that per-pixel information regarding absolute light intensity is multiplied by the 10-20 million pixels included in a 10-20 megapixel camera, the AR/VR system has a substantial amount of data to process in determining a correspondence between an HMD and a secondary device.
As such, some tradeoffs that come with that increased precision offered by the increasing resolutions of frame-based cameras include the increased latency and increased power budget required to process that substantial amount of data. Apple wants its Apple Glasses to overcome such limitations.
When it comes to Apple Glasses, such a device will arrive next year or 2022, depending on which rumor you believe. It will be a head-mounted display. Or may have a design like “normal” glasses. Or it may be available in both. The Apple Glasses may or may not have to be tethered to an iPhone to work. Other rumors say that Apple Glasses could have a custom-build Apple chip and a dedicated operating system dubbed “rOS” for “reality operating system.”
For those interested in the technical aspects of the patent, it’s for:
A method that “involves obtaining light intensity data from a stream of pixel events output by an event camera of a head-mounted device (“HMD”). Each pixel event is generated in response to a pixel sensor of the event camera detecting a change in light intensity that exceeds a comparator threshold.
A set of optical sources disposed on a secondary device that are visible to the event camera are identified by recognizing defined illumination parameters associated with the optical sources using the light intensity data. Location data is generated for the optical sources in an HMD reference frame using the light intensity data. A correspondence between the secondary device and the HMD is determined by mapping the location data in the HMD reference frame to respective known locations of the optical sources relative to the secondary device reference frame.
