iPhones and iPads could be used to help you plan room layouts

iPhones and iPads could soon be used to help you plan room layouts. Apple has been granted a patent (number 11,188,787) for “end-to-end room layout estimation.”

About the patent

In the patent filing, Apple notes that, to understand and generate decisions based on a physical environment’s contexts, computer vision operations often involve having a computing system extract and analyze digital images. Specifically, computer vision operations generally employ an image capturing device, such as a camera or a video recorder, to capture one or more image frames for a physical environment. 

For example, Simultaneous Localization and Mapping (SLAM) technology is able to determine the orientation and/or position of a system relative to a physical environment by utilizing image frames and/or other sensor information, such as inertia-based measurements. SLAM applies the sensor information from a number of sample points to create a scaled geometrical model of the physical environment without requiring pre-knowledge of the physical environment. 

Apple says that, unfortunately, because SLAM typically is limited to a specific number of sample points, the scaled geometrical model can be a sparse representation of the physical environment. 

Still, other computer vision operations are currently being developed to generate a more complete representation of a physical environment. As an example, a room layout estimation operation aims to use an image frame (e.g., a two dimensional 2D image frame) to estimate semantic room geometries, such as the room size and the planar configurations of the physical room. 

The room layout estimation operation then utilizes the semantic room geometries to form a predicted room layout representation. Generating an accurate room layout representation may be applicable in a wide variety of computer vision-based applications that include navigation in an indoor room, scene reconstruction and/or rendering, and augmented reality (AR). Apple says that “continuous improvement in generating a more complete and accurate representation of a physical environment can be beneficial for a wide range of technologies.”

Summary of the patent

Here’s Apple’s abstract of the patent: “Systems, methods, and computer readable media to implementing an end-to-end room layout estimation are described. A room layout estimation engine performs feature extraction on an image frame to generate a first set of coefficients for a first room layout class and a second set of coefficients for a second room layout class. 

“Afterwards, the room layout estimation engine generates a first set of planes according to the first set of coefficients and a second set of planes according to the second set of coefficients. The room layout estimation engine generates a first prediction plane according to the first set of planes and a second prediction plane according to the second set of planes. Afterwards, the room layout estimation engine merges the first prediction plane and the second prediction plane to generate a predicted room layout for the room.”