RevoSim: Scalable Multimodal Tactile Simulation for Dexterous Manipulation

Project Website · English · 中文

RevoSim integrates visual appearance, joint kinematics, dynamics, and multimodal tactile sensing in a two-hand model for real-to-sim-to-real workflows.

Each hand has 21 revolute joints, 247 piezoresistive channels, and five fingertip visuotactile regions. Control objects through the Python API or keyboard and read RGB, marker displacement, depth, six-axis contact wrench, and piezoresistive responses.

Environment

The target environment is Linux, Python 3.11, Isaac Sim 5.1.0, Isaac Lab v2.3.2, and CUDA PyTorch 2.7.0. An NVIDIA RTX GPU and driver supported by Isaac Sim are required. Runtime validation used Ubuntu 24.04 and an RTX 5090 with 32 GB VRAM.

Create an environment and install dependencies:

conda create -n revosim python=3.11 -y
conda activate revosim
python -m pip install --upgrade pip
python -m pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install 'isaacsim[all,extscache]==5.1.0' --extra-index-url https://pypi.nvidia.com

git clone --branch v2.3.2 --depth 1 https://github.com/isaac-sim/IsaacLab.git ../IsaacLab
python -m pip install -e ../IsaacLab/source/isaaclab

# Run from the repository root.
python -m pip install -e .
python tools/doctor.py

If you already have a compatible environment, run python -m pip install -e . from the repository root.

Installation reference: Isaac Sim 5.1 · Isaac Lab v2.3.2.

Run the demos

# Read sensor data.
python examples/read_sensors.py --headless --device cuda:0

# Three-object tactile demo.
python examples/parallel_demo.py --headless --device cuda:0 --fps 60 --seed 42 --output outputs/parallel

# Longer sequential-object demo, including ballistic transfers across the palms.
python examples/slide_demo.py --headless --device cuda:0 --shape all --fps 30 --output outputs/demo

# Interactive visualization: open Isaac Sim and the ten-finger tactile dashboard.
python examples/interactive.py --device cuda:0 --shape sphere --output outputs/interactive

# Record the finger-dance video.
python examples/finger_dance.py --headless --device cuda:0 --output outputs/finger-dance

Keyboard controls: W/S forward/backward, A/D left/right, Q/E up/down, Space to release, R to reset, and Esc to exit.

Dashboard and data

The output directory contains:

File	Contents
dashboard.mp4	Synchronized views of both hands and all ten fingertip sensor panels
observations.h5	Raw floating-point RGB, depth, markers, pressure, wrench, poses, and timestamps, with lossless gzip compression
calibration.npz	Original marker order, validity masks, camera coordinates, and registration resources
sensor_layout.json	Runtime tactile sampling positions, normals, owning links, joint names, and angle limits
metadata.json	Units, coordinate systems, left/right and finger ordering, and data ranges
frames.jsonl	Per-frame time and demonstration phase
physics.json	Collision geometry added at runtime and records of missing-mass regularization
report.json	Coverage, per-channel peaks, and release state
telemetry.npz	Per-frame wrench, pressure, marker and depth peaks, contact flags, object poses, and optical target indices for the parallel demo

The first array dimension follows left, right. The second dimension, where it indexes fingers, follows thumb, index, middle, ring, little:

Field	Shape per frame	Meaning
rgb	[2,5,3,240,320]	Raw simulated visuotactile RGB, float32, range 0–1
depth	[2,5,240,320]	Contact indentation depth in meters
marker	[2,5,100,3]	Displacement in the fingertip soft-pad link's local coordinates, in meters; use with the validity mask
wrench	[2,5,6]	Resultant contact force and torque between the fingertip soft pad and target object: Fx,Fy,Fz,Tx,Ty,Tz, in N and N·m
contact	[2,5]	Whether the fingertip soft pad and target have normal contact
pressure	[2,247]	Nonnegative normalized response after spatial diffusion; may exceed 1 and is not measured in Pa, N, or resistance
pad_pose	[2,5,7]	Fingertip soft-pad world position and wxyz quaternion
joint_pos	[2,21]	Measured joint angles in radians
object_pose	[7]	World pose of the first target object: position and wxyz quaternion
object_poses	[number of objects,7]	World poses of all target objects
optical_object_index	[2,5]	Per-fingertip optical contact target index in multi-object mode; -1 when there is no indentation

The six-axis wrench includes only normal and frictional forces between the DIP_rubber_link soft pad and the target object. It does not aggregate contacts on proximal/middle phalanges or rigid shells. Torque is referenced to the DIP_rubber_link origin, and the wrench is expressed in that link's local frame. The dashboard converts N·m to N·mm for readability; raw data always uses N·m.

Piezoresistive channel order: middle MCP 23 / PIP 4, index 23 / 4, ring 23 / 4, little 23 / 4, thumb 18 / 4, and palm 117. Use revosim.resources.pressure_regions() to retrieve the slices.

Python API

Start the Isaac application before creating a World:

import argparse
from isaaclab.app import AppLauncher
from revosim.launch import configure

parser = argparse.ArgumentParser()
AppLauncher.add_app_launcher_args(parser)
app = AppLauncher(configure(parser.parse_args()), multi_gpu=False).app
try:
    from revosim.world import World
    world = World(shape="sphere", device="cuda:0")
    point, normal = world.surface_point("left", "index", "tip")
    world.reset_object(point + normal * 0.025)
    world.drive_object(point + normal * 0.005)
    for _ in range(120):
        world.step()               # Four physics steps at 240 Hz by default.
        data = world.observe()
        print(data["wrench"][0, 1])
finally:
    app.close()

world.sensor_layout() returns sensor sampling positions, coordinate systems, and joint ordering in a directly JSON-serializable structure.

world.set_joints(side, {joint_name: radians}) sets joint targets, and world.release_object() releases the object. World(shape=("sphere", "cube", "polyhedron"), hand_spacing=0.20) creates a three-object scene; drive_object(position, object_name="cube") controls a specific object. Each fingertip optical sensor supports one contact object at a time.

Project structure

src/revosim/       World API, scenes, dashboard, recording, and minimal launcher
  tactile/        Piezoresistive, depth, HydroShear, and Taxim backends
  assets/         Left/right hand USDs, PBR textures, tactile resources
examples/         Sensor access, three-object demos, and keyboard interaction
tools/            Environment checks, data validation, and diagnostics
tests/            Channel contracts, coordinate registration, wrench, and independence tests

See THIRD_PARTY_NOTICES.md and the individual asset directories for third-party licenses and file notices.

Citation

To cite this project, use the following BibTeX entry:

@misc{zhang2026revosim,
  title = {{RevoSim}: Scalable Multimodal Tactile Simulation for Dexterous Manipulation},
  author = {Zhang, Ke and Xu, Lixin and Wang, Ziyi and Dong, Xiyue and Tan, Jie and Li, Chuanyu and Xu, Renjing},
  year = {2026},
  url = {https://davidlxu.github.io/RevoSim-web/}
}