Increasing accessibility and understanding of emotions through 'Cybernetic Being': An interview with Moonshot researchers Yun Suen Pai and Giulia Barbareschi

Japan is well known for its high technological level and deep interest in future technology, with many famous examples across both reality and digital media. A national research program that uses these types of technology and is aiming for innovative outcomes with positive societal impacts is the Moonshot Research and Development Program. Among the various goals and projects being undertaken, one that serves as a great representative of its future oriented outlook is the Cybernetic Being project, part of Moonshot Goal 1, which seeks to enable people to realize their abilities to the fullest and share their diverse skills and experiences with others throughout the world through the use of physical and virtual avatars.

Dr. Yun Suen Pai and Dr. Giulia Barbareschi are two international researchers from diverse backgrounds who are working towards the realization of this project as members of Keio University's Graduate School of Media Design (KMD). We sat down with them at the Cybernetic Being's quirky Tokyo research lab, based out of Tokyo Portcity Takeshiba to chat about their histories, paths to joining the project and the efforts they are putting into achieving its goals.

Contrasting journeys lead to the Cybernetic Being project in Japan

Science Japan (SJ)- What drew you both to science and to your individual research fields?

Pai: I'm from Malaysia, and my interests when I was a HS student were more towards physics and engineering. I majored in this, but became quite interested in AR and VR technology, which was in its infancy at the time, and could be seen in places like the gaming world, I believe Nintendo had a very basic AR game which released in 2011.  I wondered if it would be possible to apply this into some kind of practice in engineering.

I was lucky enough to have a professor at my university who was specifically working in this field, and I worked with him for a while but by the end of my masters I was more interested in the AR/VR side rather than the materials and calculations. I thought that AR/VR was so much cooler. At that point in time, you couldn't find a lot of research labs or companies working on this sort of tech in Malaysa. Japan, however, has always been a place at the forefront of technology, and if you want to do Robotics or XR it is the place to go. I applied for a number of positions there, and after a long period of applications I finally got one at Keio University. I completed my PhD here focusing on applying XR technologies to human behavior.

After completing my PhD, I was searching for the next step, but didn't have full confidence in my abilities, particularly in international labs. My supervisor at the time, Dr. Kai Kunze, shared information about a lab in NZ, called the Empathic Computing Lab  (Now based out of the University of South Australia) directed by Mark Billinghurst, a very famous figure in AI research. In fact, one of the earliest toolkits I used, pre-2010, called the ARToolkit was actually co-developed by Mark. I knew his name and felt that it would be a good fit, and coincidentally he was actually in Japan for a conference at the time, and I met up with him and spoke with him. He thought that what I was doing was a good fit with his lab, as it was a mix between AR and emotions and physiological sensing, which is exactly what I had been doing.

I completed a post-doc during my time at the Lab, between 2019-21. After this, I looked for the next step and found that Keio had received a new budget for the Moonshoot R&D Program, which I thought was really interesting and in-line with my focus at the time. It was an opening as an Assistant Professor and as I had connections here and people that I know, and research that I can pick back up, I thought it was a logical next step. I applied for the position, was selected and I've been back here for a good two and a half years now.

SJ: How about you Giulia, I believe you followed quite a different path, could you share your journey?

Giulia: Sure! I had a very intercontinental experience. I'm always very blown away when I listen to people who have come from more technical backgrounds, like Pai, because my initial fascination with science has always been everything about the body. I got really interested in how people moved and that's how I landed into clinical physiotherapy. After working for a few years in Italy, specifically in rehabilitation after spinal cord injury, which involved a lot of wheelchair skills, I got the idea that I wanted to move abroad for a few years to learn English. I ended up moving to Ireland, hence the accent! (You can hear this on our related podcast!)

While I was there, I worked in sports therapy for a while. The funny thing is that I know that many people adore working in sports physiotherapy, but it wasn't for me! If you have worked in more complex practices before, then it can get repetitive. I decided that I wanted to retrain and there is a big medical device industry in Ireland (Europe's largest per capita employer of medical device professionals and the second-largest exporter of medical device products), so I went and studied medical device science, how to make stents and heart valves. I really liked the technical aspects but was less interested in the products that I was involved with.

My supervisor in Ireland suggested that I should do a PhD to join all of these things I had done together, and I found a lab in the University College London which was doing research into accessibility technology led by Professors Nick Tyler and Catherine Holloway. I applied and ended up moving to London to do my PhD. The very first technology I ended up looking at was exoskeletons, which was great because they were a hot topic, and I was interested in biomechanics and working with people who use wheelchairs for mobility. I did some biomechanical analysis looking at how people walk while wearing exoskeletons. It's not very similar it turns out *laughs*. An important point was that getting into and out of a wheelchair is where a lot of issues arise. A lot of people get hurt at this stage and if you get injured, you can't push, as your arms are both your arms and legs when you are a wheelchair user. I ended up pivoting my research and working on co-designing technologies that help people move in and out of their wheelchair.

After I finished my PhD, I got a very good opportunity to do an internship and the World Health Organization looking at how to get assistive technologies to people, especially in areas where people have very little access and a lot of problems. I worked with them in India, particularly in Bangalore, and then went back to do my post doc in London. It was a project funded by Foreign, Commonwealth and Development Office. This was more related to assistive technologies, but in Africa. I worked in East Africa quite a lot, in Kenya and Uganda. Africa is a hugely fascinating place because they are progressing so fast, and they are skipping steps. They leapfrog from one thing to another. It makes it hugely fascinating.

At a conference of prosthetics and orthotics in Kobe, Japan in 2019, I meet Professor Masa Inakage, the Dean of Keio Graduate School of Media Design, and a lot of the things that they were doing over here when it came to accessibility, was related to engaging with artists and using it to leverage what makes you different. It's a very different approach. I applied for a JSPS fellowship, and I came over in 2021 to work with KMD. During this time, I had an opportunity with Professor Minamizawa (Project Manager of Cybernetic Being) and Professor Kunze, looking at the avatar café that Moonshot was working on. This incredible place bang in the middle of Tokyo, where customer service is delivered by people with disabilities who work from home and control avatar robots to take orders and serve customers. It was a little bit of love at first sight as it's such a wonderful community, it brings together so many different aspects, and one of things that I liked was that it wasn't just about working in the café, but the community that the cafe creates among workers and the customers. When my year of JSPS fellowship finished and I got an offer to remain as an accessibility researcher I was delighted to stay.

Interdisciplinary research into accessibility and deeper understanding of human emotions through XR

SJ: How would you describe "Cybernetic Being" to someone who has never heard the word before?

Pai: Moonshot Goal 1 is about gaining freedom from body, brain, space and time limitations, and envisioning how the society of Japan in 2050 is able to overcome a lot of the limitations that we have today. Considering the goal, the full term should perhaps be Cybernetic Being Avatar. A simple definition for the word Avatar is that it is basically the representation of a person, but the difference between an 'avatar' and a 'cybernetic avatar,' at least how I see it, is that these avatars should be able to assist and augment human abilities. They are not simply representing your face, but they are able to push you forward and you can overcome your limitations using these avatars. A good example of this might be the movie Avatar *laughs*, the main character is in a wheelchair, but by being able to control this alien body avatar he is able to run really fast, jump really high and I think that encompasses the idea quite well, minus the aliens!

SJ: What is the conceptual idea for 2050? Are we going to be walking around with glasses that enable us to see these AR/VR avatars, or will people be interacting with the environment as physical avatars?

Giulia: I think when people think about the future, and the role that technology plays, they tend to focus a lot on the 'thing', for example, the flying cars in 'Back to the Future.' These things are cool, but I think that the key questions are what do we want society to be like? What role does tech play in that? Some of the technology might also be embedded into the environment. So, you can interact with things differently. The fascination for me isn't 'Is it going to be glasses or distributed computed?'

Both of these outcomes are both highly possible. The fascinating aspect is what we are going to do with the technology. It could be something that allows us to see if someone is in a difficult situation and offer help or be embedded in public transport so that people that want to move around are able to do so. Perhaps people that are not able to go to work for all sorts of reasons are going to be able to leverage the technology and feel present in the workplace in the same way without having to negotiate what their restrictions are. I think that is more aligned with what Moonshot is trying to achieve and for me is more interesting.

Pai: It's not so much about walking around with nice glasses but more about trying to make a great social impact. I think one of the reasons that drives the Moonshot program is simply that Japan is one of the first major countries that is facing an aging populace, and how we are going to tackle these very serious issues over the next 20-50 years, and I think that all eyes are on Japan. Moonshot is one of the main initiatives aiming to make this happen.

So, we can imagine that by 2050, when we are living in an aged society, what technology do we have at that point that allows us to contribute to society to still be able to perform to our fullest. It's quite interesting.

SJ: AR and VR are well known in 2023, but could you explain how XR fits in?

Pai: I'll admit that it gets very complicated, and companies like to use extravagant terms because they want to be the one to 'coin' it. Even Apple for example doesn't use the terms VR or Metaverse, they call their technology "Spatial Computing," but they all mean the same thing. XR stands for Extended Reality, which is sort of an umbrella term for AR, VR and Mixed Reality (MR), and the lines between these is quite blurry to be honest. There is a very popular continuum called Milgram's reality-virtuality continuum, which has been referred to quite a lot by AR and VR researchers, on how to define the spectrum from reality to virtuality.

From Reality to Augmented Reality, Augmented Virtuality, and then Virtual Reality. MR encompasses AV and VR and this is how it was defined in the continuum. But the way I like to put it in simple terms is XR covers AR VR and MR, if you want to define AR, it is basically placing objects or overlays to physical space. VR is about placing virtual objects in Virtual space, so then space changes and becomes complete virtualized. MR has a number of papers on its definition, but as a simple explanation it can be referred to where the virtual content can detect and interact with the real environment realistically.  

SJ: When we look at accessibility and disability, I think the quintessential image is of the wheelchair, but what is the idea there when it comes to 2050?

Giulia: I have a sort-of obsession with wheelchairs, and one of my biggest fascinations with them is that when most people who are not familiar with wheelchairs think about them, they automatically think about all of the things these people can't do, right? For me, the wheelchair is one of the great enabling technologies, and when we lose access to it is where a lot of problems start. I think of all of the things that you can do. There is a lot of research about new kinds of wheelchairs that do different things, but if none of them have become hugely popular and people still use basic style wheelchairs, then that should give a hint that this type of technology is not really something that's going change the world. Whether that's because the chair you've built isn't as efficient in other types of environments or ground, and sure you might encounter the occasional stairs, but most of the time you have got other types of terrain.

One of the things that I'm really interested in at the moment is technology that helps in bringing people together and in breaking that separation between who uses a wheelchair to do what, and who does not. A lot of this is talking about the same types of technology that the pilots at the café use. They all have different kinds of disability, and some of them don't have disabilities! These technologies have become flexible enough to allow people with different disabilities to use them in different ways. Think about embedded accessibility features in mobile phones, that allow for a very wide variety of people to take advantage of it, depending on what they can do and how they like to do it.

I am fascinated by distributed systems that are embedded in the environment as well as by flexible and modular toolkits in which you can start to add additional elements. One of my favorite innovations in wheelchairs is 'Smart Drive,' which is produced by a UK based company, it's a power add on for your chair. It is hooked up to the back and it gives extra speed, helps you when you are going uphill, but it does not change the basic form factors of the wheelchair. You hook it up when you need it and remove it when you don't. These sorts of technologies have been proven to be far more effective than trying to rebuild and tweak something from scratch. A lot of things are also about reconceptualizing how we design space, but also renegotiating who has the right to access things.

SJ: How does what you are doing now tie back into the Moonshot goal itself?

Pai: The scale of moonshot, even within one of the goals, is really quite large. It is usually a network of researchers with the head being Professor Minamizawa, but we work directly under the main vision of the cybernetic being avatar. We also have the freedom to propose ideas that we think contribute to the overall goal, so we both have our own topics, and for mine what I am proposing is the "Empathic Metaverse Platform."

In the empathic metaverse I envision how the metaverse and its virtual representations of us can be used for better understanding of each other. I look at it mostly as a tool and how we can actually use it. What I envision is to have a metaverse space where our avatars allow us to understand and convey our emotions with each other better. With the main goal being able to create a more prosocial environment. Typical avatars reflect your appearance and perhaps your motion, but typically don't reflect your emotional state and that's what I'm trying to bring forward. I'm looking into a lot of design factors around it, for example, how do we visualize emotions for neurodivergence? How do we visualize emotions for disabled users as well, and how you know different people have different ways of conveying things, how are we able to predict emotions?

SJ: How do you keep the understanding that a person is still 'themselves' when using avatars?

Pai: That that is one of the main challenges. It's essentially creating a new mental model of empathy. When we look at someone, we have a general idea of whether they are feeling sad or stressed. Humans, however, are very subtle creatures. We generally don't show how stressed or unhappy or happy we are, right? We have an idea, like a subtle turn of the lip, or a twitch in the leg might show that they are stressed. It's something that you can sort of predict and we have this mental model in our minds of how a person potentially feels based on their body language, but that doesn't exist in a virtual space. We have to create this new mapping. We have to have this new understanding and we want it to be as intuitive as possible, but there's going to be some level of training as opposed to understanding.

They will have some form of learning, but we want to keep it as intuitive as possible. We also run workshops where we try to get participants from different target groups. We interview them, we co-design with them on how they imagine that the Avatars will appear across different levels of emotions, emotional valence, emotion, arousal. We have a general idea of how this mapping is, and then we can deploy it. We can test it out and we can refine it from there. It's not like you will immediately understand it like when someone turns red because they're angry. But at some point, you should be able to understand with a certain amount of training, Thats how we look at it.

SJ: How much cross project collaboration is there between what you both do?

Giulia: Quite a lot actually. Within the group there are several people who work on the technological aspects of accessibility. Whereas most of my expertise will be towards the graphical understanding of what people actually do with said technologies and that in the last couple of years has allowed me like a huge amount of freedom to actually cross collaborate between projects that sit under various teams within the within the same goals. I had the opportunity to collaborate with the team that worked more on the use of like parallel systems because one of the fascinating things is that when we normally think about people and avatars, we think of a very linear relationship. You as a person will have one, right?

Well, what if you had two? What if you could actually split yourself into two? Or what if you and I could share the same avatar and then we could do something together. Two of the branches that we have are platforms for parallel abilities. That's actually one of the things we trialed before in the Avatar Cafe. This year we actually had a month-long experiment in the cafe where we are trying three different things.

One was a platform for parallel ability. When you as a customer go to the cafe, there will be a robotic avatar that greets you, will ask you to sanitize your hands, and then we'll send you to one that acts as a table manager which will check your reservation, will tell you where your table is. Once your table is ready, we'll call you. Then you go sit at the table and there is another one. All of those are different people, but the platform that we implemented allows them to hop between one and the other and or to use two at the same time. As you go through this process. The person that you interact with is always the same.

People with disabilities actually learn to do some things particularly well. If you are a person with visual impairment and you learn the skills to use echolocation for navigation, then your ability to process hearing stimuli is significantly better than mine. Blind people have much higher listening rates than people who are not sight impaired and that's as a result of training. One of the things that often observe when we work with people with disabilities who have heavily integrated technology into their life is that they tend to be better at negotiating the multitasking aspects of it, because they're consistently used to switching because there is no universal system that is actually good at doing everything and by doing that you get very good at it. They tend to learn really fast. They also learn to integrate abilities with each other and help each other out so that they can do more and achieve more. That's kind of what is really of interest to me rather than trying to correct what people do, what can we learn from them.

Pai: In the future, I believe all of the senses can be augmented. My interest is this general sense of perception of reality, which includes all sense of feedback and even the 6th sense actually, which is proprioception―the sense of body ownership―and these are these are things that play a very important role in research, like Giulia mentioned right? to be able to pilot two to three robots, to able to collaborate with others and pilot multiple robots or share robot at the same time. The empathic metaverse and is also meant for diverse users, users from minorities to be able to also find a comfortable space within the universe. I also collaborate with Giulia in understanding this and to my understanding as well, the Avatar Cafe also moving towards creating a virtual representation of entire cafe as well, isn't it?

Giulia: I spoke about three experiments during the month-long run. One was the virtual cafe. When you come in as a regular customer. You sit down. And the operator, the pilot that is assigned to you and is controlling your physical avatar. There is also on like a big screen projected on your side there is a virtual reconstruction of the whole cafe and they also, beside their physical robotic avatars, choose a digital representation of themselves, so they switch between the physical robot and the digital screen, and one of the interesting aspects of it that is that allows for different things.

A physical avatar tends to give you a stronger sense of presence, not just because of your experience, but because you are physically present in the space. The people around you react differently. They're less likely to ignore you if you have a robotic avatar compared to a virtual. Virtual offers you more freedom because robots are constrained by their own mobilities. They look a certain way which can be modified to a certain extent, but the virtual space completely breaks those boundaries, The ability to interact with it and like the expertise that people like Pai bring in regarding how you detect when people want to switch. How do you help them sort of like negotiate this kind of like interaction? When we think about technology we often think about users, but what about bystanders? What about the people that interact with the person using the technology? One of the reasons why you need so much cross collaboration is because you need so much expertise and different people to look at all the various scenarios.

Looking towards 2050 and Japan as a place for research

SJ: How would you compare working in a Japanese environment to Malaysia, NZ, Europe or the US?

Pai: The strongest point of working in Japan is how open they are to doing crazy stuff. You find things like Moonshot only in Japan basically. Based on my working history in Malaysia and NZ, if I were to write a proposal to do some of the things that we are doing here most would get rejected probably because, not that it doesn't make sense, it requires them to envision a little bit further than what we normally cover in research, but it's something Japanese culture really treasures. You can also see this in their manga and anime, they really push forward. They really appreciate this vision of the future regardless of age which I find really nice. It is one of the best places to do research. I can here because I wanted to push XR, and Japan was one of the most advanced in the field, and that is still the case today.

Giulia: I would definitely agree on that aspect of trying to have a longer outlook. The UK sometimes is very transactional sometimes when it comes to proposals, where you are exchanging the outcome for the funding, which is good when trying to produce an immediate benefit, but it does hinder things that are 'blue-sky thinking.' One of the things that I find challenging in Japan is that while people tend to be more open to collaboration, making that first contact can be really difficult. In most of Europe I can drop an email to 'Prof. X' that I have never met or spoken with and sometimes it lands, but that just doesn't work in Japan. You have to get a connection, otherwise, it's very difficult to collaborate. I find it a bit stressful. It is getting less challenging now that I have been here for a couple of years as I have built a few contacts. When you network is small, expanding it is difficult, but as it gets bigger it becomes easier.

KMD in itself tends to be very international. In many respects, the difference between the labs I have worked in hasn't been so marked. There are particular things about tech engagement in Japan that I have found really fascinating. Some of the groups that we work with are older adults and introducing technology with them. People tend to be way more open to the use of technologies than I had previously experienced in other countries. I think in part, there is much more of a willingness to experiment in groups where you don't necessarily expect it, which has been very interesting. Japan also tends to have more collaboration with industries, stakeholders, and start-ups, which brings different perspectives and there is more attention in negotiating different paces and interests.

Pai: The way I look at it, besides it being more open, its potentially that its more stressful. In the context of Moonshot for example. It's a very new initiative from the govt, it has a large budget, and so because of that the handling and milestones can be quite stressful. We have to constantly produce reports and outcomes to share with the government. If we compare it with NZ for example, a lot of the research needs to benefit the Māori community, how it can benefit them, it is very heavy on the cultural impacts. Malaysia is also generally heavy on the cultural side, but they are a bit more reserved, you can't go as crazy, mainly because it is a developing country, and we are not quite there yet compared to Japan and NZ. That's how I look at it when doing international research. Japan really allows us to let loose and push as far as we can, and that's something I'm very appreciative of.

SJ: What is something that Japan can do to make it easier for researchers to move to Japan? How important do you think that that type of movement is for a researcher?

Pai: The first thing that comes to mind is language. It's unfair to comment too harshly as we are in Japan, and we should 'do as the Japanese' and pick up the language. At the same time, we have a lot of group meetings with other researchers who try to accommodate us, and they try their best to speak English, but then it switches to Japanese for a while and you are left in the dark, it is a bit difficult.

When we submit proposals, we have to be able to write in Japanese and read and understand it. Which is also hard! This extends towards this MS goal as well and we have meetings with Japanese companies and researchers which can mean there is a barrier. I can't say that there is a clear solution beyond putting extra effort into your Japanese. For me, language is still something that needs to be figured out in some way in the way, whether that's an avatar that can translate in some way, or something like that.

Giulia: I have been here for two years, and my Japanese is improving. I can get by fine in daily life, but it wouldn't be sufficient to work using it, so it is a barrier. It's also a barrier for Japanese for international collaborations and publishing. A lot of the research being done here is very good, but it is published in Japanese, so it gets marginalized. English isn't my first language either, but arguably being Italian, learning English is much easier than for a Japanese person, and I understand this because I am trying to do the reverse and understand the conceptual differences. The simple fact is that you must improve your Japanese and you will likely ask annoying questions at meetings, but most of the time people will always do their best accommodate you.

One thing that I think is necessary, as a woman in tech, is a bit more representation. There are a lot of other departments where I would have been hesitant to work because, you would be a bit more of a marginalized position. Later in your career you have more confidence and gravitas, and it becomes easier to break barriers. Coming here as a post-doc, the fact that KMD was very international and had very good representation was definitely a big boost. If you want to attract more diverse groups, you have to show them it's okay to be there.

SJ: How does the future look for both of you?

Pai: That's the big question, right? Moonshot's current budget continues until 2025, which might be extended based on results but right now we are at the middle-point and just submitted a mid-term report. Personally, in terms of career it's hard to tell, I'm currently a Project Senior Assistant Professor *laughs*. Which is a type of assistant professor, but I will be looking towards a tenured position as a general goal.

Giulia: I have to say I'm not big on definitive future planning, whenever I try it never goes the way I plan, *laughs* so I tend to go with the flow quite a lot. I would like to continue here for the next couple of years and see through all of the things that have been activated.

One of the things that I would like to explore more and see if it can be actioned in the next few years is related to international development, I would love to see more collaboration with countries in the global south. In the Asia and Pacific region, you have a lot of countries on the cusp of big changes, like Indonesia, Malaysia, a lot of the pacific island counties. There could be a lot done to work directly with those countries and I would to be involved. When you have counties on the verge of societal transformation, trying to make sure that that is inclusive of everyone that is around is a very important thing. If the lessons of Moonshot can be passed on, I think that would be tremendously positive.

Pai and Giulia's efforts towards achieving the lofty goals of the Moonshot R&D Program are admirable, and the technology they are advancing has the potential to lead to a truly fascinating future, allowing people both in Japan and around the world to overcome their limitations and engage with society in an even more meaningful way. Science Japan looks forwards to hearing more about their research achievements and seeing the fruits of their work in Japan as Moonshot researchers.

Links
Moonshot Goal 1 - https://www.jst.go.jp/moonshot/en/program/goal1/index.html
Project Cybernetic Being - https://cybernetic-being.org/en/
Science Beyond Limits Podcast: https://www.youtube.com/watch?v=JGziHxMbg6Q
Dr. Yun Suen Pai - https://yunsuenpai.com/
Dr. Giulia Barbareschi - https://giulia-barbareschi.com/