Apple wants the iPhone to be able to record computer-generated reality content

Apple has filed for a patent (number 20220207842) for a “media compositor for computer-generated reality.” It involves ways for an iPhone to record, as well as display, computer-generated reality (CGR) content.

About the patent filing

Apple notes that current computing systems and applications don’t adequately facilitate the recording or streaming of CGR content. The patent filing relates to three dimensional (3D) content, and in particular, to systems, methods, and devices for recording or streaming CGR content. 

Apple wants the iPhone to be able to record such content, perhaps in conjunction with the rumored augmented reality headset the company is working on. The tech giant wants the smartphone to be able to store such recordings for playback at a later time or live streamed for live playback on one or more other devices. 

Summary of the patent filing

Here’s Apple’s abstract of the patent filing: “One implementation forms a composited stream of computer-generated reality (CGR) content using multiple data streams related to a CGR experience to facilitate recording or streaming. A media compositor obtains a first data stream of rendered frames and a second data stream of additional data. The rendered frame content (e.g., 3D models) represents real and virtual content rendered during a CGR experience at a plurality of instants in time. 

“The additional data of the second data stream relates to the CGR experience, for example, relating to audio, audio sources, metadata identifying detected attributes of the CGR experience, image data, data from other devices involved in the CGR experience, etc. The media compositor forms a composited stream that aligns the rendered frame content with the additional data for the plurality of instants in time, for example, by forming time-stamped, n-dimensional datasets (e.g., images) corresponding to individual instants in time.”