Development of techniques for haptic exploration and recognition of objects - a contribution to autonomous robotic hands


Development of techniques for haptic exploration and recognition of objects - a contribution to autonomous robotic hands


Ricardo Martins [ website | social ]


Jorge Dias [ website | social ]

Miguel Castelo-Branco [ website | social ]


During the past few years, a new generation of robotic platforms has begun being integrated in distinct environments (e.g. domestic, healthcare, and entertainment). The robotic platforms must execute autonomously a great variety of tasks (to and in cooperation with humans) in uncertain and dynamic environments. To overcome these challenges, the robotic platforms are equipped with a high diversity of sensory (e.g. monocular and stereo cameras; microphones; and force, torque, and tactile sensing arrays) and actuation apparatus (e.g. dexterous robotic arms and hands, touch screens, humanoid heads, and audio speakers). The research works presented in this thesis are related to the subject of robotic dexterous manipulation and haptic exploration of objects (rigid and soft).

This thesis contributes to the development of robotic platforms with autonomous dexterous manipulation capabilities by studying the human manipulation and haptic exploration skills, presenting several approaches to translate and transfer them to a robotic platform. The study of the human visual and somatosensory systems, the neuronal and functional units supporting the sensory processing pipeline, as well as the behavioural patterns participating in the action-perception loop were used as guidelines and benchmarks throughout the thesis during the formulation and evaluation of three artificial perception applications.

Toward the first application, this thesis presents an approach to model the human strategies executed during a dexterous manipulation task. The human hand is instrumented with a tactile sensing array. The thesis proposes a symbolic description of the tasks using grasping primitives. Each grasping primitive is described by the hand-object contact interaction signature. During the human demonstration of two different dexterous manipulation tasks, the sequence of grasping primitives is recognized by a Bayesian model. The statistical relations emerging from the analysis of the sequence of grasping primitives are used to define the model of the task.

The research works presented in this manuscript contribute to a second application consisting of an artificial perception system to discriminate in-hand explored objects with different hardness properties. The human hand is instrumented with a tactile sensing array and a motion tracking sensor. A Bayesian model integrates features (contact intensity, contact area, and contact indentation) extracted from the sensory data acquired during the press-and-release exploration of the objects. The cutaneous and kinesthetic cues are integrated by a Bayesian model so the system can learn to discriminate between three distinct materials (haptic memory). The learned parameters are used to infer the perceived hardness properties of unknown objects based on the haptic memory of the system.

The final contribution of this thesis is concerning the implementation of a probabilistic approach to perform active haptic exploration of surfaces using dexterous robotic hands (simulation environment). The proposed approach represents the structure of an unknown surface as a probabilistic grid. As long as the haptic exploration of the surface progresses, haptic cues regarding texture and compliance are integrated by a Bayesian model and used to infer the category of material of that region of the workspace. The approach showed an excellent capability to discriminate between ten different types of materials (haptic stimulus). Based on this perceptual representation of the workspace, the robotic system infers the next region of the unknown workspace that should be explored. This decision is made by integrating bottom-up and top-down cues related to the haptic saliency of the stimulus, uncertainty of the current perceptual representation of the workspace, inhibition-of-return mechanisms, objectives of the task, and the current structure of the exploration path. The Bayesian models involved in this approach were tested on a planar surface, during the detection and following haptic discontinuities between three different materials. The following of haptic discontinuity was performed with good structural accuracy. The tactile attention mechanisms of the system demonstrated a high specificity, following the discontinuities of interest and ignoring the others. The role and impact of the different cues (haptic saliency, inhibition-of-return, uncertainty, and structure of exploration path) was also studied by removing each of these components from the Bayesian models.

Supporting materials: code and data

Additional details in the PhD thesis full-text manuscript. Check below.

[Open Data: Human dexterous manipulation multimodal database]

[Open Code: importDatasetTB MATLAB toolbox: integration of multi-modal data from human manipulation demonstrations]

[Open Code: Human body and hand kinematic structure analysis in a 3D virtual environment]

[Open Code: Reconstruction of octrees (volumetric grid) from 3D point-cloud]

[Open Code: Instrumented Rubik cube touch data visualization too]

Alternative full-text PDF:

Chapter 0: Cover, Acknowledgment, Abstract - [download pdf - web version]

Chapter 1: Introduction - [download pdf - web version]

Chapter 2: Fundamentals - [download pdf - web version]

Chapter 3: Dexterous manipulation and exploration: from Humans to robots - [download pdf - web version]

Chapter 4: Recording Human manipulation and exploration movements - [download pdf - web version]

Chapter 5: Recognition of grasping primitives using tactile sensory data - [download pdf - web version]

Chapter 6: Categorization of soft objects during haptic exploration tasks - [download pdf - web version]

Chapter 7: Active haptic exploration of surfaces using robotic hands - [read and cite Neurocomputing paper] | [read and cite IROS 2014 paper]


University of Coimbra

Date Published:


Publisher website:




artificial perception, robotics, Bayesian modelling



Development of techniques for haptic exploration and recognition of objects - a contribution to autonomous robotic hands