GelSLAM: A Real-Time, High-Fidelity, and Robust 3D Tactile SLAM System

TL;DR: Extremely high-fidelity, real-time 3D reconstruction using only tactile input. Also supports accurate long-horizon tactile-based object pose tracking for manipulation and dexterous manipulation tasks.

Accurately perceiving an object's pose and shape is essential for precise grasping and manipulation. Compared to common vision-based methods, tactile sensing offers advantages in precision and immunity to occlusion when tracking and reconstructing objects in contact. This makes it particularly valuable for in-hand and other high-precision manipulation tasks.

In this work, we present GelSLAM, a real-time 3D SLAM system that relies solely on tactile sensing to estimate object pose over long periods and reconstruct object shapes with high fidelity. Unlike traditional point cloud–based approaches, GelSLAM uses tactile-derived surface normals and curvatures for robust tracking and loop closure. It can track object motion in real time with low error and minimal drift, and reconstruct shapes with submillimeter accuracy, even for low-texture objects such as wooden tools.

GelSLAM extends tactile sensing beyond local contact to enable global, long-horizon spatial perception, and we believe it will serve as a foundation for many precise manipulation tasks involving interaction with objects in hand.

Reconstructing a peanut and a rock with the GelSight Mini sensor, and a tree trunk with the GelBelt sensor.

GelSLAM reconstruction results for the 15 objects, ordered roughly from largest to smallest. (Video coming soon)

Comparison between GelSLAM-reconstructed shapes and ground truth CAD models of 3D-printed objects. (Video coming soon)