Apple wants to make interacting with 3D environments seamless on its various devices

Apple has filed for a patent (number US 20230106627 A11) for “devices, methods, and graphical user interfaces for interacting with three-dimensional environments” on Macs, iPads, and, potentially, other devices.

About the patent filing

The patent filing generally to computer systems with a display generation component and one or more input devices that provide computer-generated extended reality (XR) experiences. It includes, but isn’t limited to, electronic devices that provide virtual reality and mixed reality experiences via a display generation component. This means the patent filing could also relate to the rumored “RealityPro” device.

Apple notes that the development of computer systems for virtual reality, augmented reality, and extended reality has increased significantly in recent years. Example augmented reality and extended reality environments include at least some virtual elements that replace or augment the physical world. 

Input devices, such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch-screen displays for computer systems and other electronic computing devices are used to interact with virtual/augmented/extended reality environments. Example virtual elements include virtual objects include digital images, video, text, icons, and control elements such as buttons and other graphics.

However, Apple says that methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, virtual reality environments, and extended reality environments, etc.) are “cumbersome, inefficient, and limited.”

For example, the tech giant says that systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in a virtual/augmented/extended reality environment, and systems in which manipulation of virtual objects are “complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented/extended reality environment.”

For these reasons, Apple says that computer systems with improved methods and interfaces for providing computer-generated experiences to users that make interaction with the computer systems are needed. And such systems should be “more efficient and intuitive for a user.”

Summary of the patent filing

Here’s Apple’s abstract of the patent filing: “A computer system displays an alert at a first position relative to the three-dimensional environment, the alert at least partially overlapping a first object in a first view. The first position has a respective spatial relationship to the user. The computer system detects movement of the user from a first viewpoint to a second viewpoint. 

“At the second viewpoint, the computer system, in accordance with a determination that the alert is a first type of alert, displays the alert at a second position in the three-dimensional environment, the second position having the respective spatial relationship to the user and in accordance with a determination that the alert is a second type of alert, displays the three-dimensional environment from the second viewpoint without displaying the alert with the respective spatial relationship to the user.”