Missing Person Search System utilizing the integration of drones and AI developed through industry-academia collaboration

A research group led by Specially Appointed Associate Professor Yoshito Okada, Specially Appointed Assistant Professor Shotaro Kojima, and Specially Appointed Assistant Professor Plutarco Bezerra Neto Ranulfo at the Tough Cyberphysical AI Research Center, Tohoku University, in collaboration with Tohoku-Drone (headed by Shunsuke Kiriu), announced on May 24 that they have developed 'Missing Person Search System' by integrating autonomous flying drones and person-detection AI. This system fully automates search operations from the air by equipping drones with thermal imaging cameras, AI systems, and position information system. The research group aims to complete developing this system within the current fiscal year. This system is expected to enable searches while ensuring the safety of both the searcher and the missing individual.

According to the Tokyo Metropolitan Police Department (TMPD), the number of mountain accidents has been increasing. In 2012, there were 1988 cases, however, in 2021, the number increased to 2635 cases. When mountain accidents occur, the police and fire departments lead the search and rescue operations, and in certain cases, private rescue organizations also participate, conducting searches with numerous people. These search operations carry inherent risks. Furthermore, survivors of accidents and their families may be billed for the search expenses incurred.

The research group then embarked on developing the 'Missing Person Search System' using drones in 2022. Tohoku-Drone was tasked with developing aircraft and communication systems, while Tohoku University was responsible for creating technologies to detect and locate individuals in distress.

The developed drone autonomously navigates through a pre-set area and automatically detects areas that resemble human figures from the thermal imaging camera's footage. It then sends the coordinates of those positions. The entire process, from powering on the drone to data transmission, has been completely automated.

After taking off, the drone captures infrared footage using an infrared camera. The AI then analyzes the captured footage to identify shapes with higher temperatures (displayed as white in the infrared imagery) compared to the surrounding environment, indicating the presence of an individual. The video is segmented into images, and the AI calculates the positional information related to the recognized objects. After landing or using a device equipped with a SIM card capable of operating in the air, the AI transmits the positional information to a base location. Since only recognized portions are transmitted, it becomes easier to verify the data.

YOLOv7 (an object detection model) is used to detect individuals from the footage. Through custom tuning, objects resembling humans from the infrared image are detected and displayed on the screen as rectangles and text.

To validate the system, the research group participated in the 'Japan Innovation Challenge 2022 (JIC2022),' Alpine Rescue Contest 2022 using robots. This contest's challenges included conducting nighttime searches for simulated disaster victims (mannequins) that emulate human body temperature. Unfortunately, they failed to accomplish the task within the time limit. However, upon reviewing the data from the drones that returned after autonomous navigation, they confirmed that the mannequins were detected, and that the positional coordinates had minimal error.

Furthermore, based on the challenges of JIC2022, an experiment was conducted in January of this year at the Fukushima Robot Test Field, which allowed for the determination of AI evaluation metrics. The experiment achieved a precision rate of 90.5%. However, the recall rate was 50.9%, indicating the need for improvement. The recall rate represents the proportion of correctly detected objects out of the objects that should have been detected and are advantageous when suppressing false negatives or minimizing missed detections. In contrast, the precision rate represents the proportion of correctly detected objects out of the objects detected by the system and is advantageous as an evaluation metric when suppressing false detections and similar issues. Moving forward, the research will continue aiming for further performance improvements while seeking sponsors.

Okada said, "In the future, we aim to improve the recall rate through AI estimation and are planning to perform additional AI training as needed. For example, we intend to add data that was not present in the previous training data, such as cases where the body temperature of an individual in distress has dropped or when parts of the body are obstructed by trees or other objects. This will enable the detection of a wide range of targets similar to the training data."