Automated generation of work improvement advice to resolve shortage of on-site instructors — Technology development using video analysis and generative AI

NEC announced that it has developed a technology utilizing video analysis and generative AI to identify differences between ideal model actions and actual work movements, and then generate advice for work improvement. By utilizing this technology, AI will be able to automatically provide appropriate improvement advice for a wide range of tasks, from delicate hand movements to full-body operations. This enables self-training in various work environments, such as manufacturing, logistics, and construction, allowing workers to improve their skills even without an instructor. The company plans to conduct demonstrations and develop products using this technology in fiscal year 2025, with the aim to launch the service within fiscal year 2026.

The aging of skilled workers has led to a shortage of instructors, making it difficult to pass down technical expertise. Additionally, the increase in high-mix, low-volume production, along with greater workforce diversity and mobility, raises concerns about rising training costs for instructors and a decline in work quality due to insufficient training.

NEC has developed a technology that enables self-training for various tasks by having AI provide guidance in place of human instructors. To achieve this, the company developed a video analysis technology and a generative AI technology. The video analysis technology detects sections where fine movement deviations occur compared to the ideal model actions. The generative AI technology creates, based on the deviations, appropriate advice text to help align movements with the ideal model actions.

The system compares the ideal model action with the actual work and maps corresponding sections where the same movements are performed. In this case, by capturing not only the person's movements but also interactions with the object being worked on, such as "grabbing" or "holding," it becomes possible to accurately map the ideal model actions even if the movement duration differs. This enables the detection of subtle differences in work movements that could not be identified earlier.

Additionally, in the advice text generation technology, the detected deviation section video, along with skeletal information such as hip and knee movements and hand and finger shapes, is fed into a large-scale Vision and Language Model (VLM). By incorporating video and skeletal information, the VLM can accurately identify work postures and movements that must be improved, generating specific advice text.

Furthermore, by presenting the generated advice text alongside the corresponding video segment, the system enables self-training at worksites of various industries, including precise assembly work, packaging, and transportation, without the need for an instructor. This is expected to reduce training costs significantly.