Apple granted patent for interacting with 3D environments

Apple has been granted a patent (number US 11615596 B2) for “devices, methods, and graphical user interfaces for interacting with three-dimensional environments” on Macs, iPads, iPhones, and the company’s rumored AR/VR headset.

About the patent

This patent relates generally to computer systems with a display generation component and one or more input devices that provide computer generated reality (CGR) experiences. It includes, but isn’t limited to, electronic devices that provide virtual reality and mixed reality experiences via a display.

The development of computer systems for augmented reality has increased significantly in recent years. However, Apple says that methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are “cumbersome, inefficient, and limited.”

For example, the tech giant says some such systems offer “insufficient feedback: for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment. Other systems in which manipulation of virtual objects are “complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment.:

In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.

Apple wants to provide improved methods and interfaces for providing computer generated experiences to users that make interaction with the computer systems more efficient and intuitive for a user. 

Summary of the patent

Here’s Apple’s (fairly technical) abstract of the patent: “A computer system, while displaying a view of a computer-generated environment, detects movement of a physical object, and in response: in accordance with a determination that a user is within a threshold distance of a first portion of the physical object and that the physical object meets preset criteria, the computer system changes an appearance of virtual content displayed at a position corresponding to a current location of the physical object’s first portion, without changing an appearance of virtual content displayed at a position corresponding to the physical object’s second portion; and in accordance with a determination that the user is within the threshold distance and that the physical object does not meet the preset criteria, the computer system changes an appearance of virtual content displayed at a position corresponding to a current location of the physical object’s first portion.”