NEDO led project develops and demonstrates remote palpation system enabling doctors in remote locations to share tactile information and videos

As part of the "Research and Development Project of the Enhanced Infrastructure for Post-5G Information and Communication Systems" (FY 2020-2023) commissioned by NEDO (New Energy and Industrial Technology Development Organization), the research group of Hokkaido University, BIPROGY Inc., and TechnoFACE successfully developed a remote palpation system that integrates tactile information and examination videos. The research group demonstrated that the system enables this integrated information to be shared by remote doctors in 3 sites, Hokkaido University Hospital (Sapporo), Obihiro Kosei Hospital (Obihiro), and Hakodate Central General Hospital (Hakodate). This system integrates and synchronizes multimodal signals - which are used to comprehensively analyzes data such as images, sounds, and tactile sensations - on a video frame-by-frame basis and transmits the signals using 5G. The factors that determine communication quality can be reduced to only bandwidth and delay by integrating multimodal signals into existing video-streaming formats.

The research group focused on codecs, a core technology for video and audio distribution, and developed the method to link the performance standards of codecs to slice standards by embedding and integrating multimodal signals into them. As an application of the technique, they built the remote palpation system that shares the tactile information and examination videos among remote doctors. Remote palpation is intended for "DtoD" (doctor-to-doctor transmission), and they aim to, improve the uneven distribution of medical resources and raise treatment efficiency by sharing information with doctors at other locations.

The transmitting doctor converts the palpation information acquired by the sensor group into an elasticity value according to depth as multi-channel stress information and transforms the sequential changes in the information into viscosity values. By embedding such data in accordance with the frames of the visual examination video, the tactile information and the time and space within the video completely synchronize to result in content that includes multimodal information.

Currently the parameters necessary for 5G video transmission will only be those related to the existing codec for video transmission. The parameters are reduced to the video data bandwidth and the delay required by the service, thereby expediting the network slicing setting.

Furthermore, the video content can be used not only for the visual examination but also for the database containing attributes such as palpation position. The receiving doctor can also access the palpation information sequentially and actively reproduce the tactile sensation. At the same time, the content can also be used for longitudinal analysis of the treatment effects - for example, storing palpation history, sharing medical record information, and quantifying palpation - as well as for educational purposes.

In the demonstration experiment, the codec standard was "H.265", and the 30 frames-per-second (fps) video of 4K resolution, captured with a commercially available camera, was transmitted and shared among 2 to 3 core hospitals in Hokkaido connected via 5G. The construction of the 5G network was carried out in cooperation with NTT Communications Corporation.

For palpation of the upper arms, the images were acquired with the camera, integrated with palpation sensor information, and the end-to-end measurements were performed, including remotely reproducing the tactile sensation via 5G transmission. Consequently, it was demonstrated that remote palpation was possible with a delay of 0.8 seconds when connecting 2 locations simultaneously and a delay of 1.7 seconds when connecting 3 locations (Sapporo, Obihiro, and Hakodate) simultaneously, thereby distributing integrated information while storing it in the cloud.

In both cases, the timing of the acquired images and the sensor information were completely synchronized. Even if there was a difference of about 2 seconds between the voice instruction from the remote doctor and the palpation playback image from the on-site doctor in response, there appeared almost no negative effect and a prospect for realizing remote palpation service that would not stress doctors during examinations.

The research group is to improve the scalability and reliability of the system, and they will perform the technology development and the practical application of services that handle multimodal signals. After FY 2024, they are scheduled to conduct full-scale operation and the related technology deployment. They will also work on research and development aimed at non-medical fields.