Apple patent filing involves capturing and playing back 3D content

Apple has filed for a patent (number US 20230281933 A1) for “Spatial Video Capture and Replay”” that involves systems, methods, and devices for capturing three-dimensional (3D) videos or playback of 3D video content. Image and depth sensor data combined with modeling is currently being used to generate 3D representations of an environment. 

About the patent filing

The 3D information may be recorded over time to create 3D video content. However, Apple says that existing technologies for capturing, storing, and playing back such 3D video content have various deficiencies. It wants to overcome such limitations on its devices such as the upcoming Vision Pro, which will be able to record and play back 3D content.

The Vision Pro was previewed at June’s Apple Worldwide Developer Conference. However, the US$3,499 (and up) “Spatial Computer” won’t be available until early 2024 — and then, apparently, only in limited quantities

Summary of the patent filing

Here’s Apple’s abstract of the patent filing: “Various implementations disclosed herein include devices, systems, and methods that create a 3D video that includes determining first adjustments (e.g., first transforms) to video frames (e.g., one or more RGB images and depth images per frame) to align content in a coordinate system to remove the effects of capturing camera motion. 

“Various implementations disclosed herein include devices, systems, and methods that playback a 3D video and includes determining second adjustments (e.g., second transforms) to remove the effects of movement of a viewing electronic device relative to a viewing environment during playback of the 3D video. Some implementations distinguish static content and moving content of the video frames to playback only moving objects or facilitate concurrent playback of multiple spatially related 3D videos. The 3D video may include images, audio, or 3D video of a video-capture-device user.”