Mental image reconstruction using generative AI — Decoding brain signals into images: New technology developed by QST

A research group including Researcher Kei Majima of the Quantum Life Science Directorate at the National Institute for Quantum Science and Technology (QST), Researcher Naoko Koide-Majima of the Advanced ICT Research Institute at the National Institute of Information and Communications Technology (NICT), and Professor Shinji Nishimoto of the Graduate School of Frontier Biosciences at Osaka University successfully reconstructed an image imagined in the mind (mental imagery). Successful reconstruction was achieved for images observed by the eyes or artificial shapes. This marks the world's first successful reconstruction of mental imagery, regardless of the image type. Majima explained, "We achieved high accuracy by incorporating generative AI assistance." These results were published in Neural Networks.

The use of the functional magnetic resonance imaging (fMRI) in neuroscience research has made rapid strides in recent years. Previous studies have reported success in reconstructing images that individuals actually saw. However, for mental imagery without visual representation, success has been limited to basic elements such as letters or simple geometric patterns, and the reconstruction of complex objects or landscapes has failed.

The research group constructed a brain decoder for brain signals measured by fMRI. They successfully achieved the reconstruction of mental imagery by combining numerical data of features of the image with generative AI.

First, they showed 1200 images, such as objects or landscapes, to human subjects and obtained brain signal data for all of the 1200 images using fMRI. Simultaneously, they input these images into an AI for image recognition, and the feature values of each image (such as the color ratio, strength of 45-degree lines, quantity of stripes and likeness to certain objects—such as bird-likeness in an image of an owl) were converted into approximately 6.13 million columns (inception scores) to create numerical data (matrix) for 1200 images. Based on the brain signal data paired with the inception scores, the brain decoder that translates the brain signals into inception scores was constructed.

Next, the subjects of the fMRI were asked to imagine the images. These images were reconstructed based on the inception scores output by the decoder from the subject's brain signals. During this process, the generative AI repeatedly modified the images automatically based on the inception scores, iterating 500 times (combining Bayesian estimation and Langevin dynamics). This resulted in successful image reconstruction with a high average accuracy rate of 90.7%.

Applying this method facilitates the reconstruction of mental imagery from brain signals. This advancement could lead to the development of a communication device based on mental imagery, commonly known as a brain-machine interface. This offers a communication tool for patients who face difficulties conveying their thoughts to physicians owing to illness or injury. It could also be used as a tool to unravel the brain mechanism during the creation of mental imagery, dreams experienced during sleep and hallucinations created in the brain.

Majima stated: "Humanity has gained the ability to see the universe through telescopes and the microscopic world through microscopes. For the first time, we've stepped into the realm of the human mind. It is a step made possible by the accumulation of extensive research in the past, and I am grateful for that."