Evaluation of 3D Reconstruction Techniques for the Blending of Real and Virtual Environments

Interaction in immersive virtual reality can be enhanced by merging physical objects with virtual stimuli, thus creating extended reality scenarios. Such approaches could be exploited to implement physical obstacle avoidance and to provide the missing haptic feedback when touching, grasping, and manipulating objects. Effective solutions rely on the correct detection and tracking of the 6DOF pose of the objects and the accurate 3D reconstruction of the physical elements. Here, we focus on the latter aspect, analyzing different 3D reconstruction techniques to create and segment the meshes necessary to merge the physical objects with virtual counterparts. We consider six 3D reconstruction software and six different furnished rooms, analyzing the time necessary to create the 3D mesh from scratch, the reconstruction success rate, and the percentage of volume shift between the real objects and the reconstructed ones. The quantitative results show several off-the-shelf 3D reconstruction techniques that can be used as input for the considered extended reality system. Although fine 3D reconstruction and meshes are not necessary, the considered techniques can catch the sizes of the real objects in the scene and can be used to create virtual environments coherently aligned with the real ones.