Apple wants users of Macs, iPhones, iPads to be able to easily interact with 3D environments

Apple has been granted a patent (number 11562528 B2) for “devices, methods, and graphical user interfaces for interacting with three-dimensional environments” on Macs, iPhones, and iPads.

About the patent

The invention relates generally to computer systems with a display generation component and one or more input devices that provide computer generated reality (CGR) experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display.

The development of computer systems for augmented reality has increased significantly in recent years. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices, such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch-screen displays for computer systems and other electronic computing devices are used to interact with virtual/augmented reality environments. Example virtual elements include virtual objects include digital images, video, text, icons, and control elements such as buttons and other graphics.

However, Apple says that methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment, and systems in which manipulation of virtual objects are complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment. 

In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Apple wants users to be able to seamlessly interact with 3D environments on its various devices.

Summary of the patent

Here’s Apple’s (long and somewhat technical) abstract of the patent: “A computer system, while displaying a first computer-generated experience with a first level of immersion, receives biometric data corresponding to a first user. In response to receiving the biometric data: in accordance with a determination that the biometric data corresponding to the first user meets first criteria, the computer system displays the first computer-generated experience with a second level of immersion, wherein the first computer-generated experience displayed with the second level of immersion occupies a larger portion of a field of view of the first user than the first computer-generated experience displayed with the first level of immersion; and in accordance with a determination that the biometric data corresponding to the first user does not meet the first criteria, the computer system continues to display the first computer-generated experience with the first level of immersion.”