GelSLAM: A Real-Time, High-Fidelity, and Robust 3D Tactile SLAM System

Tactile sensing offers precision and immunity to occlusion that vision-based methods lack when tracking and reconstructing objects in contact, making it especially valuable for in-hand and other high-precision manipulation tasks.

We present GelSLAM, a real-time 3D SLAM system that uses only tactile sensing to estimate object pose over long periods and reconstruct object shapes with high fidelity. Instead of matching tactile-derived point clouds of the object's surface, GelSLAM registers its surface normal and curvature maps for robust tracking and loop closure. It tracks object motion in real time with minimal drift and reconstructs shapes with submillimeter accuracy, even for low-texture objects such as wooden tools. GelSLAM extends tactile sensing beyond local contact to enable global, long-horizon spatial perception, and we believe it will serve as a foundation for many precise manipulation tasks involving interaction with objects in hand.

Reconstructing a peanut and a rock with the GelSight Mini sensor, and a tree trunk with the GelBelt sensor.

GelSLAM tracks the sensor’s motion relative to the object from what the gel feels: instead of matching tactile-derived point clouds of the object's surface, it registers the object's surface normal and curvature maps sensed at each contact, which stay informative even on smooth, low-texture objects. Loop closures are detected in the object's curvature maps and optimized in a global pose graph, keeping drift minimal over tens of thousands of frames and hundreds of contact breaks. The tracked poses then fuse every touch into a single high-resolution signed-distance surface, producing submillimeter-accurate meshes in real time.

We show interactive viewers for the smaller of the 15 reconstructed objects; the full-resolution meshes of the rest are too large to display on the web.

These objects were 3D printed from CAD models and scanned with GelSLAM. Each reconstruction is shown beside its CAD model in the same orientation. We call the CAD model a reference rather than ground truth because printing can slightly alter fine surface details.

Tactile keyframes densely cover the object's surface, so GelSLAM closes orders of magnitude more loops than typical indoor SLAM. Below are the pose graphs behind four reconstructions, from the smallest object to the largest.

On a 140-episode benchmark with motion-capture ground truth, GelSLAM tracks object pose in real time with only about 1 mm and 4 degrees of error, even as each episode accumulates hundreds of degrees of rotation. Loop closure cuts rotation error nearly in half compared to tracking alone, and the system reliably relocalizes whenever contact breaks and resumes.