Toshiba, the world's first AI technology that can measure real scale from digital camera photos

Toshiba has developed the world's first AI technology that enables real-scale 3D measurement from zoom-up with a zoom lens and image data from several monocular cameras taken with autofocus.

Until now, in images taken with a monocular camera, only relative values ​​can be obtained for depth information, so in order to obtain absolute values ​​such as height, length, and area, it is necessary to add information from the gyro sensor and basic information regarding size. was there. However, by utilizing the AI ​​technology developed this time, it is possible to measure the absolute value by combining the blur information contained in the captured image in addition to the relative depth information obtained from the multi-viewpoint image of the monocular camera. become able to.

In the company's experiment, a combination of a commercially available single-lens reflex camera and a zoom lens with a focal length of 24 to 70 mm was used to take 4 to 5 photos with a shift of about 10 to 20 degrees. By utilizing AI, we were able to measure the size of objects and cracks with high accuracy.

In the future, we will demonstrate with various lenses and cameras such as smartphones, industrial cameras, and drones, aiming for early commercialization.

Nao Mishima, Fellow of Media AI Laboratory, Intelligent Systems Laboratories, Toshiba Research and Development Center, who developed this AI technology, said, "The actual size can be measured by hand other than using a winding scale or crack scale. , There were some that could be measured using a smartphone measurement app or using AI, and some that could be measured using multiple cameras, but each had its own limitations and weaknesses.

For example, in the case of using monocular SfM (structure from motion), the 3D shape is restored based on the difference in the appearance of captured images with different viewpoint positions. It is possible to measure the size by combining it with a gyro sensor, but due to the size restrictions of the device, measurement at a short distance is the main focus.

Also, with AI that uses lens aberrations, it is possible to acquire depth from a single captured image, but it is necessary to fix the lens and pre-learn based on a certain distance. If you were studying at a distance of 3m, you couldn't use it immediately even if you wanted to shoot something closer. When you move the lens, you need to relearn. "

"With just a camera, this AI technology can easily measure deformations such as high places from a distant position, and can be applied to deformations of arbitrary shapes without being bound by two points or a flat surface. In addition, there is an advantage that it is only necessary to take a picture at the site and it is possible to simplify the on-site work. Since it is possible to deal with other than the learned distance, outdoor inspection where the distance changes greatly and drone It can be expected to be used. "

The conventional technology is good for indoor use at short distances or when the shooting distance does not change significantly, but in addition to such applications, this technology is an outdoor infrastructure where the shooting distance changes significantly each time you shoot. It is said that it will be possible to utilize it with an eye on inspection.

In addition, since image data taken with different cameras instead of the same camera can be used in combination, it is possible to measure the size using images taken with multiple drones.

As a result of applying this technology to crack measurement, the company was able to measure the size of cracks 7 m away with high accuracy.

"With the smartphone measurement app, there was not much error at a short distance of 1.5m, but at a long distance of 7m, the error was so large that it could not be used. However, with this technology, even at a long distance of 7m, it is 10 It was within the error range of%, and was at a level that could withstand practical use even in cases such as infrastructure inspections. "

Also, outdoors, I photographed objects 5 to 7 meters away at 11 locations. When those sizes were measured, the size error was 2.5% under ideal conditions with the lens fixed, and the size error could be suppressed to 3.8% even under difficult conditions using a zoom lens.

It is said that this error was confirmed to be at a level where the need for repair can be determined with high accuracy in a numerical simulation based on the concrete crack repair guideline established by the Japan Concrete Engineering Association.

Based on these verification results, Toshiba says that by utilizing this technology, it can be expanded to applications such as infrastructure inspections. It will be possible to easily measure the size of the part to be repaired just by using the picture of. "

It is said that it will be possible to measure cracks on the wall surface at high places, which is difficult to measure, and to measure the rust area at high places.

Toshiba has been developing robot integrated management and inspection image analysis technology so far, and states that this development technology can strengthen size measurement and contribute to inspection planning and efficient operation of robots. ..

Nao Mishima, Fellow of Media AI Laboratory, Intelligent Systems Research Institute, Toshiba Research and Development Center, said, "In Japan, the average age of equipment such as roads, bridges, and tunnels constructed during the high-growth period exceeds 35 years, and it is rapidly aging. On the other hand, there are issues such as aging workers, labor shortages, and cost reductions. At present, most of the infrastructure inspections are visual inspections, which not only complicates the work but also the situation visually. Inspection of high places that are difficult to grasp has become an issue.

In fact, we carry in equipment and trolleys at high places in bridges and tunnels and visually check the deterioration status. " "With the demand for efficient infrastructure maintenance, it is important to determine the degree of deterioration in order to prioritize maintenance work, and in that case, measuring the size of the repaired part is a major point. By being able to measure the size of deformities with high accuracy, we can contribute to the formulation of efficient maintenance plans. "

In addition, Tatsuo Kosakatani, Director of Media AI Laboratory, Intelligent Systems Laboratory, Toshiba Research and Development Center, said, "The on-site work of measuring the size of cracks with a crack scale is quite difficult and requires some familiarity. Utilizing AI technology. By doing so, it is possible for both experienced and beginners to make uniform measurements. It is also a great advantage that it can be used with the simple literacy of shooting. Also, when measuring with a crack scale, be sure to use it as evidence. You are supposed to take a picture. If you forget to measure it at the site, you can measure it using the picture you took. "

The company has indicated that it will utilize this technology to develop maintenance and inspection application systems. Also, at the moment, AI processing is difficult with the CPU of a digital camera, so the captured data will be taken back to the office and processed using the cloud or PC, but in the future, smartphone processors etc. We are also considering processing by utilizing.

The technology will be announced at the international conference on computer vision "The British Machine Vision Conference 2021 (BMVC 2021)" to be held online on November 22nd.