Software developed by Kumamoto University team provides powerful support for restoring Kumamoto Castle's stone walls

After the Kumamoto earthquake collapsed Kumamoto Castle's ishigaki (stone retaining walls), restoration teams were left with the challenge of identifying where each stone fitted. In response, a research group led by Professor Gou Koutaki of the Faculty of Advanced Science and Technology, Kumamoto University, in collaboration with Associate Professor Keiichiro Shirai of the School of Science and Technology, Shinshu University, and Toppan Inc., developed software that uses image-matching technologies to accomplish this task, which would be far too overwhelming for humans to perform visually. The restoration team adopted the software to design the restoration of the castle's iidamaru turrets. Construction is now scheduled to begin according to the design it produced. The results were published in the International Journal of Computer Vision.

Two massive earthquakes severely damaged Kumamoto Castle in April 2016, and work to restore the damage is expected to take more than 10 years. Due to the Kumamoto Castle's ishigaki being cultural assets, the local municipality must return the collapsed stones to their original correct positions. However, the sheer number of stones makes it extremely difficult to perform this task visually.

With this in mind, the research group developed an ishigaki matching system using image processing technology to assist in returning stones to their original positions. The system utilizes information on the stone's shape and location before and after the collapse to match every position.

The team began by creating a database of contour images and full-scale contours of each stone from photographs of the ishigaki before the earthquake. The team initially hit a dead end because they lacked photos of the ishigaki from before the earthquake. However, Toppan Inc. had taken more than 40,000 high-resolution photos of the walls with SLR cameras around 2011 to create content for a VR Kumamoto Castle. This data was used to create a database of the ishigaki before they collapsed.

Next, workers moved the collapsed stones to a storage yard and measured them in three dimensions using a stereo camera. This provided information on the actual scale in millimeters, which could be converted to a front-side view of the shape.

Finally, corresponding positions were found based on the shape and location of the stone contours before and after the collapse. Since the stones before and after the collapse correspond one to one, like a jigsaw puzzle, the team formulated and solved the puzzle as if it was a 0-1 integer programming problem.

When the developed system was applied to the actual ishigaki of the iidamaru, the team could identify the original locations of 337 (91%) of the 370 collapsed stones. The remaining stones the system could not identify were either small or broken. In addition, using this system to match the stones' positions revealed that prior work by Kumamoto City employees had led to about 10% of the stones being incorrectly positioned, enabling this to be resolved. This result suggests that the hundreds of stones all have similar shapes and that the conventional method of restoration, where photos of stones are compared one by one, would have resulted in a considerable number of human errors. In addition, the system took only about one hour to complete the verification. By comparison, it would have taken several months for the manual visual inspection to be completed.

Journal Information
Publication: International Journal of Computer Vision
Title: SHIGAKI Retrieval System Using 3D Shape Matching and Combinatorial Optimization
DOI: 007/s11263-022-01630-8
Kumamoto Castle English Website