When you attempt to move your limbs or think of something, your brain activates correspondingly. In recent years, developments in deep learning, artificial intelligence (AI), and detection devices have dramatically improved brain activity decoding technologies and their accuracy. Professor Takufumi Yanagisawa of the Institute for Advanced Co-Creation Studies, Osaka University, has developed an innovative Brain-Computer Interface (BCI) that combines brain functions with information technology to display text and images. His goal is to produce practical technologies that physicians can use to treat and diagnose various diseases.
Ryohei Fukuma, Specially Appointed Assistant Professor, Graduate School of Medicine, Osaka University
A Diverse Team of Experts Takes on the Challenge of Highly Accurate and Diverse Information Transfer
The brain is the foundation of the mind and is fundamental to human life. Cross-academic research to clarify its structure and functions is being conducted across fields including medicine, psychology, and sociology. One such field of research is BCI, a technology for transmitting electrical signals emitted by neurons to a computer.
BCI has been an active field of medical research since the 2000s. In 2013, researchers at Osaka University conducted a clinical study to evaluate the performance of BCI by implanting electrodes into the brain surface of patients with amyotrophic lateral sclerosis (ALS). The BCI in this study utilized electrocorticography in the motor cortex. Professor Yanagisawa recalled, "It was groundbreaking research, but it did not achieve the accuracy we had anticipated."
Based on the results from this study, Professor Yanagisawa began focusing on technologies to link information from the brain associated with brain activities corresponding to people's imagery and thoughts with a computer (representational space) that displays information in the form of text, images, and video. Through this, he is pursuing BCI which can realize highly accurate and diverse communication of information. The CREST project "Construction of representational Brain-Computer Interaction technology" was initiated to realize symbiotic interactions between the brain and representational spaces and to clarify their impacts and effects.
At the beginning of the project, Professor Yanagisawa produced vast amounts of intracranial EEG big data in a collaboration with three epilepsy centers and proposed a method to realize highly accurate brain information decoding. Specifically, he aims to decode visual information in the brain using informatics to realize a new type of BCI known as representational Brain-Computer Interaction (rBCI) based on interactions between the brain and representational spaces. He also aims to explore the clinical applications of rBC and to elucidate neuroscientific mechanisms (Fig. 1).
The project team comprises researchers with diverse backgrounds in neuroscience, representational spaces, and epilepsy surgery (Fig. 2). Professor Haruhiko Kishima, Department of Neurosurgery, Osaka University, Dr. Hidenori Sugano, a senior associate professor of neurosurgery, Juntendo University, and Dr. Kentaro Tamura, senior lecturer of the Department of Neurosurgery, Nara Medical University, also perform actual surgery at their respective institutions and are collaborating to collect big data on intracranial EEG. There are few opportunities to obtain vision-related intracranial EEG by implanting electrodes in the occipital lobe, so collaboration between multiple facilities will make rBCI development possible.
Yukiyasu Kamitani, a professor of the Graduate School of Informatics, Kyoto University, and the head of the Department of Neuroinformatics, ATR Brain Information Communication Research Laboratory Group, and Professor Shinji Nishimoto of the Graduate School of Frontier Biosciences, Osaka University, specialize in research on decoding and representing information from the brain using fMRI. "My primary role is applying those studies to intracranial EEG, demonstrating rBCI technologies, and realizing clinical applications for rBCI," said Yanagisawa.
BCI for Relieving Phantom Limb Pain
To begin, the teams established identical research environments at each location, and information on brain activity measured at these separate sites was deciphered using AI. From the results obtained, the teams constructed a system that may lead to the diagnosis of diseases and a quantitative information representation model that explains the relationship between cognitive function and brain activity. The teams also began working on producing BCI technologies, in which the brain is directly connected to an external device that controls a PC using a robot operated through information from the brain.
For example, they proposed a BCI-based treatment for phantom limb pain, which can arise after limbs are lost in accidents. Most of those who lose a limb report feeling as if it is still there. In addition, 50 to 80% say they experience intense pain. However, phantom pains cannot be relieved without suppressing the brain activity causing the pain. This peculiarity means common anti-inflammatory, painkillers and medical narcotics are ineffective. No effective treatments have yet been found.
Noticing this gap, Yanagisawa and colleagues developed a use for BCI, where patients move their phantom limbs on a screen using brain signals. This concept was influenced by conventional mirror therapy, in which a healthy limb is reflected using a mirror to strengthen brain activity related to the phantom limb. After training the device on magnetoencephalographic signals - brain information obtained from moving a normal limb - subjects were asked to clench and close a virtual hand projected on the monitor as if they were moving their own lost hand over three days (Fig. 3).
Results showed pain decreases from 45.3 to 30.9 on a visual analog scale (VAS), representing subjective pain intensity based on a 10 cm long line. These effects lasted for five days (Fig. 4).
The researchers found that to decrease pain, it was better to reduce information about the phantom limb in the brain as much as possible. This method has shown greater effectiveness in more patients than mirror therapy. Work is now being performed to make the technique more convenient. If it is established as a new treatment, it will help answer the prayers of many patients suffering from phantom limb pain.
Technology to control imagery using imagination: Hopes for communication with paraplegic patients
The groups then developed an rBCI using an intracranial EEG to present an image of a person's imagined meaning on the screen (Fig. 5). Experiments confirmed that by viewing an image estimated from EEGs and imagining an image with the instructed meaning, an image with the imagined meaning could be presented on the screen (Fig. 6)
.
The researchers also found that imagining an image with a different meaning than the one being viewed brought the EEG closer to that of the imagined image (Fig. 7). This achievement, which uses brain waves to control images without speech or words, is expected to have a variety of medical applications, including communicating with paralyzed patients.
Yanagisawa says there has been active research on technologies for reconstructing and modifying brain functions using AI and robots and applying them to diagnoses in recent years. "The treatment of neurological diseases is an area that is advancing rapidly as brain science develops. Clarifying the pathogenesis of diseases with previously unknown causes and the mechanisms of human brain function is making new treatments and diagnoses possible," he said.
Advanced programming, device development capabilities, and mathematics support these experiments and analyses. Ryohei Fukuma, a specially-appointed assistant professor of the Graduate School of Medicine, Osaka University, is responsible for this project area. The system for providing online feedback of images of deciphered brain information, similar to encryption, was built by Prof. Fukuma through trial and error.
Clinical research must always be done without error so as not to keep patients cooperating with the research waiting. In addition, teams often only have one chance to perform their research. In the BCI experiment with imaginative control of images, the patients could perform the task better than expected from the first trial, perhaps due to the researchers' previous experience. "Our idea worked so well that our expectations rose, and we subsequently considered various other factors to improve the system. However, the more changes we made, the less successful they became, and we continued to grope around for solutions for a year or two. There was a sense of disappointment in the lab," he recalled.
"We then rearranged the conditions we first tried and conducted the experiment again. We were then able to obtain the timing and characteristics of brain activity associated with recalling images in humans." "We were able to do this thanks to Professor Fukuma and his strong engineering background. Based on these experiences, I would like to devote myself not only to research but also to developing human resources in this field," Professor Yanagisawa said, expressing his determination.
Weekly Outpatient Clinics Continue: A Constant Awareness of Clinical Applications
Professor Yanagisawa has a somewhat complex background. As a high school student, he became interested in physics and life sciences, fields that use mathematics to explain all kinds of phenomena. He wanted to acquire a mathematical understanding of the various dynamics that occur in life phenomena, leading him to major in physics at university. In graduate school, he joined a laboratory investigating non-linear non-equilibrium statistical mechanics. His interest was in neuroscience as a dynamic system in which multiple types of cells, including neurons and glial cells, interact non-linearly to process information.
His research used computer simulations to reproduce nervous system behavior. But he felt his lack of brain science knowledge was limiting his research. At the same time, he realized that using mathematics could be useful in medicine, which led him to the Osaka University School of Medicine. Yanagisawa was preparing to study neuroscience abroad, but a newspaper article about the MD-PhD program at the University's School of Medicine, which allows students to earn an MD while enrolled in the program, became a turning point. When he learned about BCI research in neurosurgery, his interest shifted to basic research with an eye toward medical applications. After completing his residency program, he began his current field of research.
Along with his research projects, Professor Yanagisawa continues to provide outpatient epilepsy care once a week at Osaka University Hospital's Department of Neurosurgery. While his research aims to apply the results to clinical applications, Professor Yanagisawa believes the medical treatments he provides keep him aware of his clinical roots. "Patients cooperate with our research to develop BCI using intracranial EEG, even though they experience no direct benefits themselves.
We are always aware that our ultimate goal is clinical applications for treatments and diagnostic techniques helpful to patients," he said.
Experiencing Collaboration Between Medicine and Engineering as a Student
Respecting the Other Party to Produce a Beneficial Collaboration
Currently, the brain information decoding technologies obtained through CREST are being applied to techniques for diagnosing epilepsy and dementia from EEGs. "Our primary focus right now is the clinical application of BCIs using an implantable electroencephalograph," Professor Yanagisawa said.
This technology, in which a device implanted in the skull automatically detects when epileptic seizures occur and then analyzes brain wave characteristics, will lead to treatments. To implement these technologies in society, it is vital to expand the scope of research as well as develop technologies. "I would like to see BCI develop not only in the field of medicine but also as an industry, and I hope that more neurosurgeons in my profession will become interested in and participate in the clinical applications of BCI" he said with hope.
Devices using scalp-based EEGs are beginning to appear on the market, and the base for these technologies appears to be steadily expanding. However, Professor Yanagisawa points out that discussions on how these technologies should be used in society, including safety and efficacy evaluations and rules for development--including ethical aspects--are still far from complete. "We are working with experts in the field to create guidelines to develop the industry into one that is widely accepted," he said. We hope to realize an environment where users can have peace of mind while using safe equipment made according to set standards.
Another important issue is methods for maintaining medical data, which includes a large amount of personal information in a form that developers can use. "About seven years ago, when PRESTO adopted the project, I worked on a system that made big data from intracranial EEG available to many researchers. The CREST project is another theme that emerged from this initiative." he recalls. In the course of the research, the researchers have found that in addition to access to data, talking and collaborating leads to greater results, bringing their knowledge and skills to bear on each other's work.
This realization means that increasing exchange between diverse individuals working in medicine and engineering will require environments that facilitate joint research. Professor Yanagisawa said the key is "Education with an eye toward collaboration between medicine and engineering from the point of university studies." He says he utilizes his experience of studying both engineering and medicine in his research. "When collaborating between disciplines, terminology definitions, prerequisites, and emphasis often differ and pass-by one another. However, sharing goals while respecting the other party's culture should make it possible to conduct beneficial joint research," he said, indicating the significance of collaboration.
Professor Yanagisawa's laboratory has students not only from the School of Medicine but also from the School of Engineering in the form of interns who are pursuing research with the same goals. This is because he believes that having them experience a cross-academic environment as students will lead to future collaboration between these fields. In recent years, medical students tend to actively engage with industry, such as through interning at AI companies. The day may not be far off when rBCI is further perfected and utilized in various fields, including clinical practice.
(Article: Naoko Yoshimoto, Photography: Hideki Ishihara)