Apple has filed for a patent for “controlling external devices using reality interfaces” with the Vision Pro.
About the patent
In the patent filing, Apple says that techniques for interacting with external devices while using a computer-generated reality system, such as a virtual reality or mixed reality system, are “desirable.” The tech giant thinks the Vision Pro is a good fit for this.
Here’s how it would work: first, image data of a physical environment that includes the external device is obtained with one or more image sensors. A representation of the physical environment according to the first image data is displayed on a display. While displaying the representation of the physical environment, second image data identifying a gesture occurring between the display and the external device in the physical environment is obtained with the one or more image sensors.
The processes determines whether the identified gesture satisfies one or more predetermined criteria associated with the function. In accordance with determining that the identified gesture satisfies one or more predetermined criteria associated with the function, the external device is caused to perform the function.
Summary of the patent
Here’s Apple’s abstract of the patent: “In some exemplary processes for controlling an external device using a computer-generated reality interface, information specifying a function of the external device is received from the external device. First image data of a physical environment that includes the external device is obtained with one or more image sensors. A representation of the physical environment according to the first image data is displayed on a display.
“While displaying the representation of the physical environment, second image data identifying a gesture occurring between the display and the external device in the physical environment is obtained with the one or more image sensors. A determination is made as to whether the identified gesture satisfies one or more predetermined criteria associated with the function. In accordance with determining that the identified gesture satisfies one or more predetermined criteria associated with the function, the external device is caused to perform the function.”
Figure 3D depicts a user interacting with a reality interface of a user device by providing a gesture.
