Creating a society that embraces diverse values through Augmented Reality (AR)
― Interview with Tom Kawada
【Feature】Inventors who create the future

This feature looks at inventors trying to change society with unique ideas to create the future, and looks at new innovations. In this first installment, we look at the "eldest" of the "AR Three Brothers," Tom Kawada, who has created unique and humorous augmented reality (AR) works one after another. At the root of Kawada's ideas exists the desire to overcome the gaps in time, space, and experience, and between imagination and reality, where the fun of invention and science lies. He speaks about creating an ideal future society where diverse values are accepted as the norm by learning to see things from multiple perspectives through AR.

Fulfilling a given-up dream

A large screen is placed in a plaza with a four-story-high atrium. In front of it is a conductor's podium. On the stairs surrounding the plaza, people are sitting and relaxing. Once Kawada raises a conductor's baton, an orchestra image appears on the screen and starts playing along with the conductor. The music performance becomes fast if he waves the baton quickly, and the performance slows down if the movement is slow. It looks like Kawada is conducting online, but it is programmed so that the orchestra's performing image and the music are linked to the baton's movement.

This event took place at Hankyu Department Store's Umeda Main Store (Osaka-city). After the store's complete renovation in 2012, Kawada created this AR work as one of the commemorative events for its reopening. People doing shopping gathered there as soon as the performance started, and it ended with cheering and clapping. It was a moment when shoppers turned into spectators.

The people who wanted to could experience conducting by just standing in the waiting line. A woman who was waiting for her turn with her grandchildren called out to Kawada, "It's you who created this, isn't it?" She told him that she had dreamed of becoming a music teacher to conduct, but it did not come true. Kawada says, "When I heard her say, 'I had given up on conducting, but you are going to make my dream come true today. Thanks,' I felt what I did was really worth it."

"Even if you didn't do any professional studies on something, you might sometimes do it easily, or the dream you had given-up on can come true. Shoppers become spectators or conductors ― It's nice to see ordinary people change their mindsets and get inspired to action. I want to entertain people in that way, which should be one of the possibilities of AR," says Kawada.

His origins date back to his days as a developer for a sewing machine manufacturer

AR is short for "augmented reality," referring to the process of adding information to reality (extending the real world) by superimposing digital contents such as videos, images, and 3D data on actual scenery. In some ARs, digital contents appear when you hold a smartphone or tablet over something real. The game "Pokemon Go" is one of them.

Kawada started his work career as a developer for a sewing machine manufacturer. The spark for his work happened when he was traveling Asian countries to explain the patented technology they had developed; A locally hired staff said to him, "I want you to solve our problems. Please make something that would enable us to order replacements of broken parts directly."

"Once a part of a sewing machine breaks, local service staff need to order replacements. In doing so, they have to read a thick dictionary-like parts book because the parts are vast in number. However, back then, some couldn't read," recalls Kawada.

He immediately attached a web camera to his PC and created a prototype system to enable the staff to order on the spot; The system recognizes parts' shapes by outline and identifies the part number.

"It happened to be AR in hindsight." Kawada's AR started with actual use. He says he realized the potential of AR at that time. "You can't grasp detailed information such as part numbers immediately without a certain amount of experience or familiarity with the parts book. But if this technology is applied, you can skip that process and do it right from the beginning. I think that's the brilliant thing about technology," Kawada says with a smile. With technology, you can solve problems, make someone's dreams come true, and even impact society, which may be the best part of the invention.

Making learning fun too through technology

Kawada is also thinking of applying AR to school studies. If you draw a map symbol of a hot spring or a fruit farm and hold your smartphone camera over it, a picture of a hot spring or grapes pops out of it. If you write "1600," the Battle of Sekigahara emerges. Or, if you draw a "museum or art museum" symbol on a plaza, a three-dimensional museum rises into the screen.

This technology makes dry things into something fun using technology and turns "the unrealistic" into "augmented reality". "Isn't this a kind of superpower?" "When I show it to children, they get enthusiastic, reacting like 'wow!' Just memorizing map symbols or era names is boring, but if they memorize the whole program in a fun way, it might motivate them to study, which I think would be great as a gateway to learning," says Kawada.

Filling the gap between ideal and reality

Since he was a child, Kawada has always loved to think about the gap between imagination or ideal and reality. "When I jumped two steps down the stairs with an umbrella, I felt like I was floating, so I thought I could fly from the roof too." He then tried to jump from the balcony on the ninth floor of the apartment building he was living in at that time, but his mother spotted and stopped him, saying, "You're going to die!" He couldn't accept it. He wondered "why she stopped me with that desperate look," so he studied scientific knowledge in many ways. As he did so, he learned about the strength of the human body and that airplanes could fly due to the existence of lifting power, which made him aware, "That sure would kill me." He thought of how to make things look the way he imagined and tried to get closer to the imagination that he would enjoy. While doing so, he was amused, as if science had become a single browser.

His wish to fill the gap between ideal and reality continues even now. For example, when a disaster occurs, he feels a sense of regret, thinking, "If people had recognized their escape route instantly with more AR implemented in society, it could've saved some lives."

What's on Kawada's mind is what Sakyo Komatsu, known as the author of "Japan sinks" and other works, wrote after the Great Hanshin-Awaji Earthquake. "According to that, in the Edo era, the route where people paraded carrying mikoshi (portable shrines) in festivals became an evacuation route. What people were doing as experience with enthusiasm helped in times of emergency. I want to be of such help using AR technology."

For example, scanning the actual city from many angles: Kawada thinks of incorporating AR technology to be synchronized with the world seen there. By playing treasure hunt or hide-and-seek in the town using this technology in peacetime, you can learn where to go in times of need, which will also be helpful in times of disaster. He has actually made the scanned image of the city into shape.

"When implementing this in society, what's necessary is to flag something in space ― create a state where information of some kind is floating in space. We have recently obtained a patent that makes it technically possible. As well as thinking about the cutting edge, I want to ensure the landing spot. Believe me; I'm not just enjoying letting grapes pop out of a fruit farm symbol. Grapes are good, though," Kawada says with a mischievous look.

Pursuing a unique perspective at "Science Agora"

What is the future Kawada wants to realize like?

"It's quite clear. I want to create something that automatically conveys the richness of individual differences ― the fact that age, position, culture, way of thinking, that is, the scale of how we see things that reside in our individuality, is different." People can easily get "connected." However, Kawada sees that they tend to make single-colored values with today's social technology alone. "It's uncomfortable to try to unify our values. People see the same thing differently, and there is beauty and joy in seeing things differently. I want to develop AR technology that can have people understand this instantly. If that happens, our society will be generous enough to accept people with different ideas, positions, and personalities, which will be my goal."

Kawada appeared in four sessions at the Science Agora 2021, held from November 3 to 7. One of them was "Recipe that Scientists Think." Experts in protein and tactile research spoke duing the session. The unique perspective of scientists, different from chefs', is interesting. Kawada is learning a lot from experts' views.

Kawada says that inventions are not only for particular people. "By implementing the ideas you have in your mind, you will find contact between reality and society. In my case, AR or programming applies, but you replace it with something you are good at. You can interact with science from any angle and come up with ideas you haven't had before in abundance." Strangely, we feel like we can do whatever Kawada says easily. We hope you will join us in thinking about the world and the future Kawada envisions through the development results of AR Three Brothers.