Estimating caregivers' states of mind at the NEC Beyond 5G Collaboration Laboratory: Demonstration of probabilistic digital twins commences at Toyonaka facility

The NEC Beyond 5G Collaboration Laboratory, established by Osaka University and NEC, has begun conducting living lab‐based demonstrations at the service‐oriented senior housing facility Shibahara Moca Maison (Toyonaka City, Osaka Prefecture) in March. A living lab is a method that combines the concepts of 'living (living spaces)' and 'lab (experimental venues)', situating research and development closer to people's living spaces to create new services and products by incorporating residents' perspectives. This demonstration is the first in a series of activities to implement the laboratory's research and development achievements produced through living labs in society, thus addressing societal issues.

Although current efforts to improve caregiving environments recognize the importance of compassionate caregiving, the focus is often on improving efficiency by reducing the physical burden on caregivers.

The NEC Beyond 5G Collaboration Laboratory aims to overcome these challenges using a more sophisticated perspective by focusing on enhancing the mental satisfaction of both caregivers and care recipients.

The goal is to materialize an ideal caregiving environment where care recipients feel safe and have ample interaction with caregivers. To achieve these aims, the living lab demonstration uses digital twins to understand and predict emotional states and facilitate appropriate communication opportunities.

Looking towards the Beyond 5G era, the demonstration aims to create an 'emotionally satisfying world' in caregiving environments using digital twins.

Specifically, the demonstration collects data on environmental factors such as temperature and air pressure, care recipients' body temperatures and heart rates, the timing of agitated states, changes in facial expressions and conversation contents, and caregivers' interaction methods and motivations. The collected data is then analyzed to determine relations between environmental changes and comfort levels, as well as relations between the formation of compassion and atmosphere, and applied to the real world.

The demonstration also aims to predict care recipients' emotional changes, identifying the timing of agitated states and adjusting causal factors before these states begin. It also introduces weak robots (robots that coexist with humans, drawing out the human instinct to protect and nurture vulnerable beings) between caregivers and care recipients to create a calm environment. The demonstration aims to reduce psychological burdens on caregivers and care recipients through this fusion of physical and digital technologies.

Furthermore, starting in fiscal year 2024, the laboratory plans to use 'probabilistic digital twins' proposed by the Collaboration Laboratory to further improve object recognition and the accuracy of future predictions. Digital twins recreate real‐world situations in simulation (digital) spaces. Simulations are optimized and verified in simulation spaces, and favorable results obtained are then applied to the real world.

Probabilistic digital twins, however, estimate the real world probabilistically and predict the future, considering uncertain information in collected sensing data, AI recognition errors, and the ever‐changing real‐world environment.

Using this technology, the ultimate goal is to create relationships where caregivers and care recipients can learn from and inspire each other beyond their respective roles.

In this demonstration, Osaka University's School of Information Science and Technology, School of Engineering, and Cybermedia Center) will work on materializing ideal caregiving situations for creating an 'emotionally satisfying world', examining data collection and analysis methods for real‐life situations, and investigating empathetic prediction methods for 'emotional states', including agitation.

Meanwhile, NEC will examine new digital twins for the Beyond 5G era, investigate and verify data collection and analysis methods using digital twins, and study future prediction methods using probabilistic digital twins.