Apple has filed for a patent (number 20200201444) for “controlling external devices using reality interfaces.” The goal: the ability to interact with external devices such as Macs, televisions, stereos, and more while using a computer-generated reality system, such as a virtual reality or mixed reality system.
If the patent filing ever becomes reality, you’d be able to control such gadgets using a “reality interface” with motions and other controls. A visual representation of the physical (that is, the “real”) environment that includes the aforementioned device with one or more image sensors and displayed on the screen of the reality interface device. Gestures on the secondary device would result in corresponding actions on their real world counterparts.
Here’s the summary of the patent filing: “In some exemplary processes for controlling an external device using a computer-generated reality interface, information specifying a function of the external device is received from the external device. First image data of a physical environment that includes the external device is obtained with one or more image sensors. A representation of the physical environment according to the first image data is displayed on a display.
“While displaying the representation of the physical environment, second image data identifying a gesture occurring between the display and the external device in the physical environment is obtained with the one or more image sensors. A determination is made as to whether the identified gesture satisfies one or more predetermined criteria associated with the function. In accordance with determining that the identified gesture satisfies one or more predetermined criteria associated with the function, the external device is caused to perform the function.”
