Achieving dextrous grasping by integrating planning and vision-based sensing

This article deals with the automation of dextrous grasping in a partly known environment using a stereo vision system and a multifingered hand mounted on a robot arm. Effective grasping requires a combination of sensing and planning capabilities: sensing to construct a well-adapted model of the situation and to guide the execution of the task, and planning to determine an appropriate grasping strategy and to generate safe, feasi ble manipulator motions. We propose an integrated approach that combines computer vision, path planning, and manipulator control in three complementary activities: the reconstruction of task-oriented models of the workspace, the determination of ap propriate grasping configurations from computed "preshapes" of the hand, and the automatic generation and execution of hand/arm motions using a hybrid geometric path planner and a hybrid control system. This article outlines the architec ture of our system, discusses the new models and techniques we have developed, and finishes with a brief description of work-in-progress on the implementation and some preliminary experimental results.