Freely using abilities augmented through machines ― Toward the creation of a society in which anyone can play an active role

As we witness the imminent arrival of a super-smart society that combines physical space and cyberspace, Professor Masahiko Inami of the Research Center for Advanced Science and Technology, The University of Tokyo, is calling for 'freeing bodies' ('JIZAI bodies') that enable people to freely use machine-augmented abilities as if they belonged to their own bodies. He aims to create a society in which anyone can play an active role, through the realization of seamless integration between humans and machines that will enable people to overcome the limits of the physical body with which they were born and gain desirable abilities, and the use of avatars that serve as alternative bodies.

A fragmented cyberspace and movement into the multiverse

The spread of COVID-19 created an opportunity for remote communication such as teleworking and online conferencing to spread throughout society all at once. Now, even the metaverse - 3D space within the computer network - no longer belongs to the world of science fiction. In online conferences, there is no need to share physical space with other participants. Being able to enter the metaverse from any location makes it possible to travel through cyber space using 'bodies' known as avatars.

Thanks to the astounding evolution of this information technology, our bodies have been released from certain physical constraints. On the other hand, it has become easier for us to recognize the value of communication in a physical setting. For example, many people feel that discussions are more active in face-to-face meetings. For those of us who use our physical bodies to communicate with other people, communication without that physicality is not on the same level.

In the future, once the super-smart society arrives, network-based activities will surely increase rapidly. Economic activities in the metaverse are already starting to flourish; cyberspace has a larger presence, and there may also be people whose activities revolve around cyberspace, such as being hired and engaging in work in the metaverse. Devices that will become more convenient and have higher functionality through a network will likely become more commonplace.

"Currently, we think of physical space and cyberspace as a dichotomy, but we can no longer let go of the convenience of cyberspace. There is a possibility that the metaverse will become a fragmented 'multiverse,' with games and communication, like many other countries." This is what Professor Inami sees as the outlook for our society in the future.

If technology advances a little more and a society in which being in cyberspace feels like experiencing reality becomes normalized, the desire to exist simultaneously in different places will manifest, and it is likely that some people will feel that one body is not enough. "When events have overlapped, I have thought that it would be very convenient if I could attend two conferences online at the same time. It would be ideal if I could understand the content of both and make relevant comments."

However much the world of information expands, human bodies do not change, and we are not yet used to being active in multiple locations. To realize a society in which humans seamlessly connect with machines, it is essential to explore human physicality. In light of the recognition of this issue, the ERATO Inami JIZAI Body Project led by Professor Inami is establishing fundamental technologies to create JIZAI bodies that can adapt to a super-smart society (Fig. 1).

Developing senses and awareness together with augmented bodies

Professor Inami's specialist field is information somatics. We understand the body in terms of physiology, motor functions and other aspects, and in the past research has also sought to grasp information from the perspective of brain science. Professor Inami considers the mind to be the information section of the body and is striving to understand it as a broad information system. "We can grasp the physical body in terms of physics and physiology, but our 'body schema' - the form our bodies take and how we learn movement - can be understood as information." Based on this idea, ERATO is researching the editing and modification of body schema, and even carrying out wide-ranging studies of what is possible by gaining an understanding of this.

This editing and modification of the body schema is also known as 'human augmentation.' This includes, for example, exerting a strong force that would be difficult with a real live body, one person moving a second person's body, and multiple people operating a single body. Professor Inami classifies human augmentation into five types: 1) enhancing the senses (super-sensing), 2) enhancing the physical body (superhuman), 3) separating and designing the mind and body (out-of-body experience, transformation), 4) cloning and 5) fusion. He says that when it comes to the body, designing physical parts of the body and mental parts inside the body with a certain degree of independence will enable transformation, cloning, fusion and more.

In engineering terms, bodily 'freedom' (jizai-ka) is broadly divided into two streams: creating human substitutes, and supporting humans. Robotic technology that creates substitutes for the human body, such as cleaning or carrying out dangerous work instead of a human, can be called 'automation.' However, it's not necessarily a good idea to automate everything. If a robot went to an amusement park in place of a human, the human wouldn't enjoy it. In other words, automation is suited for tasks that humans don't want to do and don't have to do. Professor Inami notes, "It's important that machines support you by augmenting your capabilities so that you can do the things you want to do in the way you want to do them, and that you can freely use those capabilities as if you are using your own body. We call this 'freeing' (jizai-ka) the body."

The human augmentation for which Professor Inami aims doesn't just involve demonstrating powerful force from a distance; it means opening the senses and awareness together with an augmented body and making full use of machines attached to the body as if they are actually part of it. Of course, people can't immediately move freely after putting on a machine. However, horse riders feel new emotions, such as exhilaration and joy, from the unity of horse and rider when the two come together. The goal is to create these feelings through the integration of people and machines.

Moreover, the integration of people and machines will be realized thanks to the seamless connection between cyberspace and physical space. It is in fact the body itself that naturally connects the brain and the real world, but we don't even know whether we can truly think of machines and our presence in cyberspace as our own bodies. Thus, the project started with research that explored whether people can be cognizant of an augmented body part as a part of their own body, and whether they can move it as they wish.

From engineering to psychology and brain science: Outcomes created through a fusion of different fields

Transforming the body also has an effect on the mind, in the same way as becoming able to accomplish something that was previously impossible makes people happy. The form of human awareness that accepts machines is the essential key to fully using an augmented body part. As such, the project has also established research themes in fields such as psychology and brain science, in addition to the engineering field. Seven hubs are involved with the project: The University of Tokyo, Waseda University, Keio University, Toyohashi University of Technology, the University of Electro-Communications, the French National Centre for Scientific Research (CNRS) and JIZAIE (Shinjuku-ku, Tokyo). Its researchers are split into five groups organized by research theme and carry out their research across the different hubs.

One of these groups, the JIZAI Body Implementation Group, consists of members who are studying jizai-ka in physical space. They are responsible for research and development to achieve human augmentation that uses technologies such as virtual reality, augmented reality, wearable technologies, robotic technologies, telexistence and machine learning, and its social implementation. Based on cognitive psychology, the Cognitive Psychology and Behavior Analysis Group is working to verify the psychology and behavior modification engendered by JIZAI bodies, and carry out analyses of human cognition and behavior and physiological responses. The System Intelligence and Neural Mechanisms Group is engaged in analysis of the neural mechanisms that support JIZAI bodies, clarification of how edited body schema are expressed in the nervous system, and the improvement of controllability based on optimization associated with the senses and motion.

Moreover, the Virtual Body Implementation Group aims to develop fundamental technologies for JIZAI body manipulation in virtual environments and to enable the manipulation of a shared body in the virtual environment based on actual user body information. They are researching what happens when one person operates multiple avatars, and when multiple people manipulate a single body. Finally, the JIZAI Body Marketing Research Group is working to clarify the social requirements of JIZAI bodies and the possibility of their social implementation through fieldwork, case studies and market research.

It was not easy to gather researchers suited to the project from a wide range of fields, but Professor Inami attaches particular importance to the integration of different fields. "Fusions of different fields with different research cultures and research languages might not be smooth at first. But the young researchers we have trained through this project are already 'multilingual,' with an understanding of multiple research backgrounds. I believe this will lead to the training of talents who will create future innovation."

In fact, in his own career, Professor Inami specialized in bioengineering until his master's course - he researched biosensors - but switched to engineering for his doctorate, and began focusing on VR and other areas that were still new at the time. He is a unique 'bilingual' who has been practically involved in integrating different fields from his youth. Perhaps it was inevitable that he would lead a project that explores the relationship between the human body and awareness, striving to create a new research genre.

An adapted 'sixth finger' and a feeling of something missing when removed

Can people really be aware of an augmented body part as part of themselves? Is it possible for people to think this way about an additional body part with which humans are not naturally equipped, rather than artificial parts that compensate for a missing part, such as prosthetic arms and legs? The project developed an artificial finger, known as the 'sixth finger,' to explore the answer. In the past, researchers developed an artificial finger that the user moved by moving their foot; this is the type of artificial body part that researchers have previously created. In contrast to this, the sixth finger was designed to be attached next to the little finger and operated via movements of the arm muscles that are not normally used, allowing the finger to move independently from other body parts (Fig. 2).

During the experiment, 18 adults wore the sixth finger, and any changes in their senses and actions were observed after they had practiced bending and stretching the finger, plus key tapping, for around an hour. The results showed that all 18 people were able to move the sixth finger intuitively, in the way they wished. Furthermore, the people who really felt that the sixth finger was part of their body found that localization became ambiguous on the side of the hand to which the finger was attached, proving that people can be aware of an artificial body part as part of themselves. When the fully adapted artificial finger was removed, many people felt that something was missing.

So, what level of augmentation or alternative body allows movement with a real sense of integration between human and machine? If there is no sense of freedom, how can support be given so the user feels free? And what is required for users to move a virtual body in a way that feels like moving their own body? Researchers investigated whether, when participants moved their whole bodies while watching randomly arranged images of only hands and feet in cyberspace, the hands/feet and body felt like their own (Fig. 3). This study found that there was a sense of ownership of the hands and feet, but not of the whole body.

In addition, this project has produced a number of very interesting results - for example, it has identified effective timing for a machine to provide support when a human and a machine work cooperatively, and clarified the characteristics of movements when two people share an avatar and carry out joint tasks in cyberspace (Fig. 4 and 5). The former can be applied to the development of a human-machine cooperative system that can smoothly support action without any discrepancy between human will and machine movement, and it is hoped that the latter will lead to suggestions for new ways of carrying out joint work by sharing a body using VR and/or robotics.

The researchers developed a shared task system operated by a two-person avatar in cyberspace, and clarified the characteristics of human movement when using this. From the outcomes of the experiment, the researchers learned that when the avatar was operated, the numerical values of both hand trajectory and jerk were lower (the motion trajectory became shorter and more linear).

Developing both sports and art ― A wish to deliver outcomes to many different people

Professor Inami is also proactively developing sports and media art that combines multiple genres so he can ensure these research outcomes are spread into society. One of these, superhuman sports, refers to new sports that make use of JIZAI bodies, and can be enjoyed by anyone, including people who are bad at sport, people with physical disabilities, and the elderly (Fig. 6). This doesn't only mean enjoying games such as 'just running fast' or 'using augmented sensory functions to score goals without hitting obstacles in the dark.' There are also hackathons to create new and unprecedented sports as participants experience these and discuss them. There are already 31 official sports.

Professor Inami is promoting this outreach so he can deliver these research outcomes to more people. He has loved science since he was a child, but in elementary school he was rarely able to meet any friends who shared his enjoyment. Similarly, even if research institutions produce amazing results, at the moment the number of people who can access that information is limited. "I think it's a good idea to start by having people experience and sympathize with our outcomes so we can deliver them even to people with no interest in science. Fostering new scientific literacy through new outreach methods is also one of the key missions of this project."

He is aiming to make these outcomes into sustainable products, and is also focusing on business creation. In the past, he attempted to connect his outcomes to companies through university technology licensing organizations (TLO), but this was unsuccessful, so he later switched to the method of collaborating with companies from the brainstorming stage. His aim was to create interesting outputs by ensuring discussions between researchers who have the capacity to create strange things and companies looking for new business. Several startups and joint studies with companies have already begun, centered on the JIZAI Body Marketing Research Group.

People overseas are also aware of JIZAI bodies as Japanese research. For example, in the EU, aspects of the augmentation of people with no physical damage are difficult to study as research topics due to social norms. "In contrast to science, which seeks universality, technology is similar to traditional culture and pop culture, in that there are aspects of research rooted in the local culture that lead to originality. I want to establish the new genre of JIZAI bodies as output from Japan."

This research, which may seem like playing around at first glance, has the potential to change social structures and become a foundation that supports a new society. In June 2022, the project succeeded in somatizing the 'third and fourth arms,' also known as the virtual reality version of the MetaLimbs, created in 2019 (Fig. 7). You could say that a future in which avatars with augmented functions are used freely is now another step closer. Maybe the day when people are released from their ties to their bodies and can slip into new, free bodies is not all that far away.

(TEXT: Yuko Sakurai; PHOTO: Hideki Ishihara)