Evaluation and Application of Foundation Models for 6D Pose Estimation in Industrial Environments Using Digital Twins H/F

Position offered

0.

Works

Details of the tasks :

This M2 internship is part of the FUSION project and its Work Package 3 (WP3), whose goal is to update the digital twin of an industrial environment based on the robot vision-based perception. Foundation models are increasingly used in the literature across a wide range of applications. Also, this is the case in 6D pose estimation, with Wen et al.’s proposal, titled FoundationPose, which achieves excellent results compared to state-of-the-art methods. The approach has been tested on various datasets.

We aim to evaluate its performance in the context of manufacturing industrial environments through training performed using the digital twin of the production workshop.

The tasks assigned to the intern are as follows:

1. Implement FoundationPose.
2. Generate a synthetic dataset of the production workshop using the digital twin developed in Unity.
3. Train FoundationPose on this data and evaluate the algorithm’s performance.
4. Evaluate the algorithm’s performance on real-world data.
5. If applicable, adjust the initial dataset by incorporating labeled real-world data into the synthetic dataset and assess the performance of the hybrid dataset.

Project Context :

This recruitment is part of the FUSION project (Framework for Universal Software Integration in Open Robotics), which was selected under the “I-Démo – France 2030 Regionalized Normandie” call for projects. The project’s partners are Conscience Robotics (lead), OREKA Ingénierie, and CESI LINEACT.

The main objective of the FUSION project is to democratize the use of robotics by introducing a paradigm shift that places the user at the center of the system through:

1. The introduction of XR for designing robotic missions and teleoperating robots via a digital twin;
2. The reuse and sharing of software modules accessible to everyone;
3. An innovative robotic perception approach using a semantic map to update the digital twin, making robots increasingly autonomous.

The targeted use case focuses on dismantling operations within a nuclear site cell, specifically the cutting of contaminated pipelines. Currently, these operations are carried out by operators remotely controlling the robotic arm using only cameras installed on the intervention site and mounted on the robotic arm. This significantly complicates teleoperation due to the lack of depth perception. Our proposal aims, first, to reduce the complexity of robot teleoperation by replacing environment perception through cameras with immersion in a real-time-generated digital twin of the work area. Secondly, the project seeks to “teach” robots naturally to perform repetitive tasks that require only occasional supervision.

Laboratory Presentation

CESI LINEACT (UR 7527), the Digital Innovation Laboratory for Businesses and Learning in support of Territorial Competitiveness, anticipates and supports technological transformations in sectors and services related to industry and construction. CESI’s historical ties with businesses are a determining factor in its research activities, leading to a focus on applied research in partnership with industry. A human-centered approach coupled with the use of technologies, as well as regional networking and links with education, have enabled cross-disciplinary research that centers on human needs and uses, addressing technological challenges through these contributions.

Its research is organized into two interdisciplinary scientific teams and two application domains:

1. Team 1, “Learning and Innovating,” is primarily focused on Cognitive Sciences, Social Sciences, Management Sciences, Education Science, and Innovation Sciences. The main scientific objectives are understanding the effects of the environment, particularly instrumented situations with technical objects (platforms, prototyping workshops, immersive systems), on learning, creativity, and innovation processes.
2. Team 2, “Engineering and Digital Tools,” is mainly focused on Digital Sciences and Engineering. Its main scientific objectives include modeling, simulation, optimization, and data analysis of cyber-physical systems. Research also covers decision-support tools and studies of human system interactions, especially through digital twins coupled with virtual or augmented environments.

These two teams cross and develop their research in the two application domains of Industry of the Future and City of the Future, supported by research platforms, primarily the Rouen platform dedicated to the Factory of the Future and the Nanterre platform dedicated to the Factory and Building of the Future.