Digital Twin of an Industrial Workstation: A Novel Method of an Auto-Labeled Data Generator using Virtual Reality for Human Action Recognition in the Context of Human Robot Collaboration

The recognition of human actions based on artificial intelligence methods to enable Human-Robot Collaboration (HRC) inside working environments remains a challenge, especially because of the necessary huge training datasets needed. Meanwhile, Digital Twins (DT) of human centered productions are increasingly developed and used in the design and operation phases. As instance, DT are already helping industries to design, visualize, monitor, manage and maintain their assets more effectively. However, few works is dealing with using DT as a dataset generator tool. Therefore, this paper explores the use of the DT of a real industrial workstation involving assembly tasks with a robotic arm interfaced with Virtual Reality (VR) to extract a digital human model. The DT simulates assembly operations performed by humans aiming to generate self-labeled data. Thereby, a Human Action Recognition dataset named InHARD-DT was created to validate a real use case in which we use the acquired auto-labeled DT data of the virtual representation of the InHARD dataset to train a Spatial-Temporal Graph Convolutional Neural Network with skeletal data on one hand. On the other hand, the Physical Twin (PT) data of the InHARD dataset was used for test. Obtained results shows the effectiveness of the proposed method.