Mixed Reality: an Efficient Communication Channel for Human-Robot Collaboration

Collaborative robots represent a technological leap forward, and their adoption could benefit many small and medium-sized enterprises (SMEs). Such robots are cost-effective and allow humans for safe, close-proximity, and highly flexible interactions with the machine. Nonetheless, industrial collaborative robots nowadays lack a key requirement for efficient collaboration, namely the possibility to effectively communicate with human teammates. To tackle this open and challenging aspect in collaborative robotics, the present Ph.D. work has drawn inspiration from social studies on human-human collaboration, where other researchers have demonstrated how efficient interaction is achieved through implicit communication, made up of a series of cues (e.g., gaze, gestures, etc.), which lead individuals to convey their own intentions and infer their teammate’s ones dynamically. Building on this principle, this Ph.D. project’s objective has been attempting to bridge such a communication gap by developing novel interfaces to enable a more intuitive, seamless interaction between humans and robots and to endow the latter with the ability to project their intentions, defined as upcoming planned actions, in a straightforward way. To achieve such a result, various communication alternatives have been evaluated and eventually Mixed Reality has been chosen and thoroughly explored as a suitable channel for building an efficient and intuitive human-robot communication layer. To this extent, a novel robot system architecture has been developed and refined throughout the three years, integrating Mixed Reality with modern and powerful Head-Mounted Display devices. Such architecture brings forth a comprehensive bi-directional, holographic communication interface which can be employed in various collaborative scenarios. On the one hand, robot-to-human communication enables projecting robot’s intentions as holographic, visual cues in a direct way to the human teammate. Specifically, a virtual counterpart of the robot can be superimposed to the real one in the Mixed Reality layer and used to anticipate upcoming robot’s actions via dynamic, holographic animations, potentially offering useful insights and improving human teammate’s awareness throughout the collaborative process. The proposed interface has been tested in multiple user studies under different collaborative contexts, including assembly tasks with fixed robot manipulators and scenarios of mobile collaboration. The results have highlighted that such form of holographic communication ensures a smoother collaboration process, where human and robot are less likely to obstruct and hinder each other, due to the improved awareness of the human, while at the same time increasing the rate of success of joint actions (e.g., handovers). On the other hand, human-to-robot communication can be used to ensure a more direct interaction and to easily control and teach tasks to the robotic teammate. In particular, by interacting with the holographic robot in the Mixed Reality layer through a combination of voice and gestures, the human teammate can intuitively achieve Kinesthetic Teaching and teach both simple motions and complex pick-and-place or handover tasks to the robot. Overall, the result of this Ph.D. work is an open-source, modular architecture which can be employed by other researchers and companies to take advantage of the proposed holographic communication scheme, both in industrial collaborative contexts and in more social scenarios of human-robot interaction.