Apple has been granted a patent (number 9,799,139) for “accurate image alignment to a 3D model.” It involves the company’s ARKit and 3D maps of street-level images.
At this summer’s Worldwide Developer Conference, Apple introduced ARKit. It allows developers to tap into the latest computer vision technologies to build detailed virtual content on top of real-world scenes for interactive gaming, immersive shopping experiences, industrial design and more.
In the patent Apple notes that, typically, 3D maps are formed on the basis of images captured (or acquired) by an airplane scanning a section of terrain to be modeled in 3D. When capturing the images, camera position can be obtained by GPS, and the images further time stamped. The airplane is further equipped with an inertial measurement unit (IMU) such that the angles of rotation of the airplane, known as roll, pitch and yaw, can be recorded. Thus, both the position and attitude of the camera are recorded for each captured image.
Apple says that while 3D modeling using aerial images generally results in high-quality positioning, street-level 3D modeling typically suffers from lower-quality positioning. Factors such as, for instance, GPS signal shadows due to obstacles, signal distortions and the drifting of IMU data in the relatively varied motion of street-level vehicles deteriorate measurements on ground level.
This causes the recorded position of street-level images to be inaccurate. What’s more, mechanical and optical properties of a given real camera differ from those of an assumedly identical camera, resulting in incorrect measurements. Apple says another problem is that alignment of images captured at angels differing greatly is troublesome since it will be difficult to find overlapping image data.
This means that when projecting street-level images onto a 3D model derived from aerial images, there is a significant risk of mismatch since the pose of the ground-level camera used for capturing the street-level images does not comply with the geographic referenced details of the aerial 3D model. Apple wants to improve prior art methods of positioning captured (or acquired) images and, in particular, captured street-level images.
Methods and devices for acquiring, and aligning, an image with a textured three-dimensional (3D) model are described in patent 9,799,139. In one implementation, aligning can include rendering an image in the 3D model at an estimated pose and then aligning the acquired image with the rendered image.
In another implementation, aligning can include projecting the acquired image onto the 3D model at an estimated pose and then aligning the projected acquired image with the textured 3D model. In still another embodiment, aligning can include projecting the acquired image onto the textured 3D model, rendering a first image in the original 3D model and a second image in the 3D model onto which the acquired image has been projected, and then aligning the first and second images.
Of course, Apple files for — and is granted — lots of patents by the U.S. Patent & Trademark Office. Many are for inventions that never see the light of day. However, you never can tell which ones will materialize in a real product.
