Development of innovative image reconstruction method in low light situations

The research group comprising Professor Takayuki Hamamoto, Professor Yusuke Kameda (currently Assistant Professor of Sophia University Faculty of Science and Technology Department of Information and Communication Sciences), and graduate student Kiyotaka Iwabuchi of Tokyo University of Science Faculty of Engineering Department of Electrical Engineering has succeeded in developing a new image reconstruction method that can control subject blur even in dark environments with few photons by utilizing the high time resolution of the single-photon detection imaging method, which is expected to be a next-generation imaging technology.

The single-photon detection image sensor collects the information in a binary fashion and records the incident photon at each pixel based on whether a photon is detected and records it as a bit-plane image. Research on a method to reconstruct a bit-plane image into a high-quality multi-bit image is currently underway. According to Professor Hamamoto, "The bit-plane image has a high time resolution, but the gradation is binary, so accurate motion cannot be determined using general motion estimation methods. Even in complex scenes where multiple objects overlap each other, it was a difficult task to accurately determine the amount of movement."

The research group proposed a method for directly and accurately estimating motion based on a bit-plane image. The objective of this development is to control subject blur according to the movement of the subject. Specifically, by treating the areas displaying similar movement as a unit of the movement blur removal process, it is possible to obtain an image in which motion blur has been partially removed. In this method, correction is applied based on grouped movement, so it is possible to handle cases where the movement of the subject changes in the middle or cases where multiple moving subjects overlap. There is also no need to specify the area of the subject.

Estimation results of this method showed that the misalignment error was less than 1 pixel, which makes it is more accurate than the conventional method of estimating motion based on multi-bit images; this holds even in a dark environment where the average number of incident photons per frame is 0.1. Furthermore, by performing noise removal processing in addition to motion blur removal, the peak signal-to-noise ratio was improved by 1.2 dB while suppressing subject blur. Professor Hamamoto said, "By using this method, it is possible to obtain a reconstructed image with various characteristics from a bit-plane image; in particular, it is easy to adjust the frame rate of the reconstructed image and the amount of motion blur in each frame. For example, you can add an appropriate amount of motion blur according to the amount of movement and express a moving object clearly without motion blur, and cut out only that part; you can also freely reconstruct it as needed." The proposed method may pave the way for the development of technologies for shooting high-quality images in dark environments with very little light, such as outer space, and Super Slow-Motion images that cannot be captured by current high-speed cameras.