Nagoya University develops AI tool for analyzing complex animal behavior

Experiments involving observation of animal behavior require a great deal of patience and time. Accordingly, many AI-based behavior analysis tools have been developed, and the efficiency of animal behavior analysis has been advancing rapidly. However, most of these tools estimate behavior by tracking skeletal features. When animals overlap on screen and parts of their bodies become obscured, accuracy and speed drop significantly.

A research group led by Doctoral Student Hayato M. Yamanouchi, Lecturer Ryoya Tanaka, and Professor Azusa Kamikouchi from the Graduate School of Science at Nagoya University, together with Assistant Professor Ryosuke F. Takeuchi from the Graduate School of Pharmaceutical Sciences of the same university, has developed an AI animal behavior analysis tool called "YORU" that directly analyzes the posture, appearance, and form of animals as they perform specific behaviors. Kamikouchi commented: "Even complex behaviors can be learned and analyzed as long as they are distinguishable to the human eye. We have made YORU freely available as open-source software, and we hope that many researchers will use it." The study was published in Science Advances.

To study the mechanisms underlying animal behavior, it is essential to analyze and quantify each type of behavior individually. YORU applies object detection, a machine learning algorithm, to enable accurate and rapid analysis not only of behaviors involving complex movements-such as copulation and grooming, which were difficult to handle with conventional tools-but also of social behaviors in which multiple individuals interact simultaneously. In addition, by combining a real-time analysis function with a projection optics system, the group succeeded in precisely targeting individual animals displaying a specific behavior and manipulating their neural activity.

Yamanouchi explained: "You simply select the behavior you want to analyze from a video, extract several tens to several hundreds of frames in which it appears, and train the model-that is all it takes to recognize the behavior. If a human can distinguish it by appearance, YORU can classify it with over 90% accuracy, even when animals are overlapping."

In the experiments, the group successfully achieved high-accuracy detection of courtship and copulation behavior in a group of eight Drosophila, trophallaxis (food-transfer behavior) among six ants, and orientation behavior in zebrafish. The researchers also conducted experiments to manipulate neural activity in Drosophila using optogenetics.

Wing extension in Drosophila is a courtship behavior in which a male extends and vibrates one wing, producing a sound known as a courtship song; when females hear this song, their receptivity to copulation increases. The team therefore conducted an experiment in which male wing extension was detected in real time and the activity of the neurons controlling that wing extension was optogenetically suppressed at the moment of detection.

When they used YORU to analyze camera frames in real time and control light delivery in response to detected wing extensions, the researchers found that light was delivered immediately after each male wing extension. This led to a reduction in the frequency of male wing extension accompanying the neuronal suppression, as well as a corresponding decrease in copulation rate, demonstrating that suppression of the courtship song resulted in suppression of copulation.

To enable individual-selective photostimulation in response to the behavior of each animal in a setting with multiple individuals, a module for controlling a projector light source was incorporated into the system. Using this module, the researchers conducted an experiment in which a spotlight-like beam of light was directed at a female at the moment the male's wing extension was detected. Females that had been genetically modified so that their auditory primary neurons would be suppressed by light illumination were used. As a result, selective illumination of the female in response to male wing extension behavior was successfully achieved, and the copulation rate was reduced.

These results demonstrated that YORU can be used to manipulate the neural activity of a specific individual at the moment a target behavior is detected.

Yamanouchi said: "Until two years ago, I was trying to analyze behavior using existing methods, but it wasn't working well, so I changed my approach from skeleton-based to appearance-based. I named it YORU-derived from the Japanese word for 'night'-because I imagined it automatically running analyses overnight. It can reduce the burden of observation, so I hope it will be put to use in research on social behavior and related fields."

Journal Information
Publication: Science Advances
Title: YORU: Animal behavior detection with object-based approach for real-time closed-loop feedback
DOI: 10.1126/sciadv.adw2109