Adds visual-based tactile sensor with shape sensing example #3420

JuanaDd · 2025-09-10T06:18:34Z

Description

This is an implementation of TacSL integrated with Isaac Lab, which demonstrates how to properly configure and use tactile sensors to obtain realistic sensor outputs including tactile RGB images, force fields, and other relevant tactile measurements.

Fixes # (issue)

Type of change

New feature (non-breaking change which adds functionality)

Screenshots

The screenshots of added documentation and simulation outputs.

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

clean up license and configs; use full import path; add doc for visuo_tactile_sensor

JuanaDd · 2025-09-10T06:22:21Z

docs/source/overview/core-concepts/sensors/visuo_tactile_sensor.rst

+
+References
+~~~~~~~~~~
+


@iakinola23 Could you help view this documentation here? Thanks!

Hi @iakinola23 updated the documentation with your edits.

JuanaDd · 2025-09-10T06:35:59Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_r15_data/polycalib.npz

@kellyguo11 Hi Kelly, I will upload the npz binaries, and the USD files to nucleus once the licensing concerns are resolved. Thanks!

kellyguo11

just a few high-level comments, haven't gone through the code in detail yet

kellyguo11 · 2025-09-23T19:48:23Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+    taxim_gelsight = gelsightRender("gelsight_r15")
+    import ipdb
+
+    ipdb.set_trace()


why is there a breakpoint here?

deleted in the newest commit.

kellyguo11 · 2025-09-23T19:50:47Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+        self._camera_sensor = TiledCamera(self.cfg.camera_cfg)
+
+        # Initialize camera if not already done
+        # TODO: Juana: this is a hack to initialize the camera sensor. Should camera_sensor be managed by TacSL sensor or InteractiveScene?


maybe we can do something similar as the Raycaster camera?

Hi Kelly, I checked the RayCasterCamera implementation, and it differs slightly from the tacsl sensor logic. The tacsl_sensor includes a camera sensor that can be enabled or disabled via the cfg, while RayCasterCamera only uses CameraData without having a camera or other sensor as a member. I’ve kept the current implementation and cleaned up the comments. Please let me know if you have any further concerns or suggestions.

kellyguo11 · 2025-09-23T19:51:17Z

docs/source/_static/overview/sensors/tacsl_demo.png

images should be .jpg

all changed to jpg format and documentation is updated accordingly.

kellyguo11 · 2025-09-23T19:52:14Z

docs/source/overview/core-concepts/sensors/visuo_tactile_sensor.rst

+.. code-block:: bash
+
+    conda activate env_isaaclab
+    pip install opencv-python==4.11.0 trimesh==4.5.1 imageio==2.37.0


do we need to add these to setup.py?

Added opencv-python to setup.py; trimesh is already included, and imageio's dependency is not required in the newest commit.

kellyguo11 · 2025-09-23T19:54:35Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+----------------
+Tensorized implementation of RGB rendering of gelsight-style visuo-tactile sensors


can you also add any licenses required for gelsight under docs/licenses?

Updated the license header to be consistent with other files.

kellyguo11 · 2025-09-23T21:22:11Z

please also make sure to run ./isaaclab.sh -f for the linter checks

…ency and readability; Clean comments

iakinola23 · 2025-09-25T01:40:17Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+    return dzdx, dzdy
+
+
+def generate_normals(height_map):


generate_normals not needed and deprecated. only the tensorized version is used.

iakinola23 · 2025-09-25T01:41:21Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+        self.background_tensor = torch.tensor(self.background, device=self.device)
+        print("Gelsight initialization done!")
+
+    def render(self, heightMap):


Delete render as it is not used. Only the tensorized version (render_tensorized ) is used.

deleted render , generate_normals, and compute_image_gradient , padding as well.

zxhuang97 · 2025-09-30T19:31:02Z

Thank you for making TacSL available on IsaacLab! I have three quick questions:

Have you evaluated the physical plausibility of the GelSight finger? For example, would it allow stable grasps of objects like pegs or bolts, similar to what TacSL demonstrated in Isaac Gym?
I noticed that only the USD for gelsight_r15 is currently available. Do you plan to migrate gelsight_mini as well? Alternatively, could you share guidance on converting the URDF to USD while preserving realistic physics?
Is there a way to change the resolution of the tactile image? Currently it's hard-coded in configuration to be 320 x 240

Thanks!

kellyguo11 · 2025-10-17T21:40:16Z

@ooctipus @Mayankm96 could you please help do a review of this one?

JuanaDd · 2025-10-18T04:03:00Z

@zxhuang97 Hi, thanks for your questions and for checking out TacSL on IsaacLab.

Yes. A full task environment (like object grasping) will be supported later after this PR. This PR mainly serves as a reference for how to implement a custom visual-based sensor in the IsaacLab framework.
The GelSight Mini will likely be supported later, as long as there aren’t any license issues. In the meantime, you can try using the URDF importer tool in IsaacSim or the convert_urdf.py script in IsaacLab to convert it yourself. You can tweak the physical properties — such as compliance_stiffness and compliant_damping — in the tacsl sensor config if needed.
The tactile image resolution can be modified through the camera config. Right now, it’s using the values defined in the camera prim included in the asset.

Hope this helps!

Mayankm96 · 2025-10-21T09:00:21Z

docs/source/_static/overview/sensors/tacsl_diagram.jpg

Only this diagram seems to be getting used. Please remove any unused images.

Mayankm96 · 2025-10-21T09:03:24Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+conf_r15 = {
+    "data_dir": "gelsight_r15_data",
+    "background_path": "bg.jpg",
+    "calib_path": "polycalib.npz",
+    "real_bg": "real_bg.npy",
+    "h": 320,
+    "w": 240,
+    "numBins": 120,
+    "pixmm": 0.0877,
+}
+conf_gs_mini = {
+    "data_dir": "gs_mini_data",
+    "background_path": "bg.jpg",
+    "calib_path": "polycalib.npz",
+    "real_bg": "real_bg.npy",
+    "h": 240,
+    "w": 320,
+    "numBins": 120,
+    "pixmm": 0.065,
+}


Following all other sensors, please make these into configclass objects with the correct doc-strings. We usually don't want to keep sensor-specific settings in the core codebase.

Sensors configuration live here: https://github.com/isaac-sim/IsaacLab/tree/main/source/isaaclab_assets/isaaclab_assets/sensors

Please choose variable names that are intuitive to read and understand. "h" and "w" are insufficient.

Thanks for the suggestion. A Config class has been added here and gelsight configs are put under isaaclab_asets/sensors

Mayankm96 · 2025-12-16T10:44:04Z

scripts/demos/sensors/tacsl_sensor.py

+    )
+
+
+def mkdir_helper(dir_path):


Please add type annotations correctly along with docstrings to the code here.

Mayankm96 · 2025-12-16T10:44:33Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

+    """Visualize the tactile shear field.
+
+    Args:
+        tactile_normal_force: Array of tactile normal forces.


Missing shape information.

Mayankm96 · 2025-12-16T10:45:02Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

+    """Visualize the penetration depth.
+
+    Args:
+        penetration_depth_img: Image of penetration depth.


Missing shape information. How are the defaults chosen? Would be good to have some reference.

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

Mayankm96 · 2025-12-16T10:47:15Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

+        """Generate the gradient magnitude and direction of the height map.
+
+        Args:
+            img: Input height map tensor.


Missing shape information.

Mayankm96 · 2025-12-16T10:47:26Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

+            kernel_sz: Size of the kernel. Defaults to 5.
+
+        Returns:
+            Filtering kernel.


Missing shape information.

Mayankm96 · 2025-12-16T10:47:35Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py

+        """Apply Gaussian filtering to the input image tensor.
+
+        Args:
+            img: Input image tensor.


Missing shape information.

Mayankm96 · 2025-12-16T10:49:20Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+        if self.cfg.debug_vis:
+            self._initialize_visualization()
+
+    def get_initial_render(self) -> dict | None:


Is this value used anywhere downstream? How does it handle nominal value when environments are getting reseted? I was wondering if this call should happen inside the sensor reset

Actually, the sensor uses this stored "no-contact" image to calculate deformation (diff = nominal - current) every step. Capturing it once should be enough and safe because a reset might happen while the robot is holding an object.

I've also added the warning to the docstring. It now explicitly states that the user must ensure the sensor is in a "no contact" state when calling this method to avoid incorrect baselines. Thanks for pointing this out—your feedback helped clarify this important detail.

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

Mayankm96 · 2025-12-16T10:51:39Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+
+import isaacsim.core.utils.torch as torch_utils
+import omni.log
+from isaacsim.core.prims import SdfShapePrim


As we are working towards getting rid of isaacsim dependencies, would it be possible to use PhysX views directly and not rely on Isaac Sim classes for SDF? I don't think they are doing much besides creating the PhysX views internally.

Changed to call create_sdf_shape_view from physx view directly.

Mayankm96 · 2025-12-16T10:51:58Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+from scipy.spatial.transform import Rotation as R
+from typing import TYPE_CHECKING, Any
+
+import isaacsim.core.utils.torch as torch_utils


Is it possible to use the math utils provided with Isaac Lab: isaaclab.utils.math?

Mayankm96 · 2025-12-16T10:59:19Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+    # Timing statistics
+    #########################################################################################
+
+    def get_timing_summary(self) -> dict:


To stay consistent with other sensor classes, this shouldn't belong to the sensor class itself. All benchmarking should be done outside in user code.

Removed benchmarking code.

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

Mayankm96 · 2025-12-16T11:00:46Z

source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cfg.py

+    compliant_contact_stiffness: float | None = None
+    compliant_contact_damping: float | None = None


Missing doc strings.

Mayankm96 · 2025-12-16T11:02:41Z

source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cfg.py

+    func: Callable = from_files.spawn_from_usd_with_physics_material_on_prim
+    compliant_contact_stiffness: float | None = None
+    compliant_contact_damping: float | None = None
+    apply_physics_material_prim_path: str | None = None


apply sounds more of a function name which makes it confusing. My expectation would be that this is a bool variable which says whether to apply or not. But it is a string. Is there a better name we can use here instead that is less ambiguou?

Updated to "physics_material_prim_path", hope it's less confusing :)

source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cfg.py

jtigue-bdai · 2025-12-16T16:24:14Z

source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cfg.py

+    func: Callable = from_files.spawn_from_usd_with_physics_material_on_prim
+    compliant_contact_stiffness: float | None = None
+    compliant_contact_damping: float | None = None
+    apply_physics_material_prim_path: str | None = None


does this only work for one rigidbody in the usd?

Added support for multiple rigidbodies here, thanks.

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py

jtigue-bdai · 2025-12-16T16:38:07Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+            contact_object_velocities = self._contact_object_body_view.get_velocities()
+            contact_object_linvel_w = contact_object_velocities[env_ids, :3]
+            contact_object_angvel_w = contact_object_velocities[env_ids, 3:]
+
+            elastomer_velocities = self._elastomer_body_view.get_velocities()
+            elastomer_linvel_w = elastomer_velocities[env_ids, :3]
+            elastomer_angvel_w = elastomer_velocities[env_ids, 3:]


One thing to be careful with here is that the velocities returned from the body view are the velocity of the COM of that body. If your *_body_view.get_coms() has non zero transforms your velocity values will be incorrect. I would suggest getting the getting the COMs of the contact body and elastomer body at init and then doing the linear velocity transforms (see IMU update call)

Thanks for pointing that out and for the detailed explanation. I’ll update the implementation here to use the COMs of the contact body and elastomer body for the velocity transforms as you suggested.

…der.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

…sor.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

…sor_cfg.py Co-authored-by: James Tigue <[email protected]> Signed-off-by: Juana <[email protected]>

…g.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

…aming and doc

visual-based tactile sensor impl. and shape sensing example

6808a09

clean up license and configs; use full import path; add doc for visuo_tactile_sensor

JuanaDd requested review from Mayankm96, jtigue-bdai, kellyguo11 and ooctipus as code owners September 10, 2025 06:18

JuanaDd commented Sep 10, 2025

View reviewed changes

JuanaDd mentioned this pull request Sep 10, 2025

visual-based tactile sensor impl. and shape sensing example #3341

Closed

6 tasks

Mayankm96 changed the title ~~visual-based tactile sensor impl. and shape sensing example~~ Adds visual-based tactile sensor with shape sensing example Sep 10, 2025

Mayankm96 added the enhancement New feature or request label Sep 10, 2025

Merge branch 'main' into tacsl_demo

5f83dd2

github-actions bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Sep 11, 2025

kellyguo11 reviewed Sep 23, 2025

View reviewed changes

Merge branch 'main' into tacsl_demo

8eecdc0

JuanaDd added 4 commits September 24, 2025 13:28

load assets from nucleus

7c90c47

cherrypick document update

ec93980

update documentation and setup.py, formatting

9115636

Optimized tensor operations in VisuoTactileSensor to improve effici…

e86b370

…ency and readability; Clean comments

iakinola23 reviewed Sep 25, 2025

View reviewed changes

Delete unused functions

a8e1c98

Mayankm96 added this to Isaac Lab Oct 10, 2025

Mayankm96 moved this to In review in Isaac Lab Oct 10, 2025

Mayankm96 reviewed Oct 21, 2025

View reviewed changes

Mayankm96 reviewed Dec 16, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py Outdated Show resolved Hide resolved

Mayankm96 reviewed Dec 16, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_render.py Show resolved Hide resolved

Mayankm96 reviewed Dec 16, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py Show resolved Hide resolved

Mayankm96 reviewed Dec 16, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py Outdated Show resolved Hide resolved

Mayankm96 reviewed Dec 16, 2025

View reviewed changes

source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cfg.py Outdated Show resolved Hide resolved

jtigue-bdai reviewed Dec 16, 2025

View reviewed changes

JuanaDd and others added 12 commits December 24, 2025 18:34

Merge branch 'main' into tacsl_demo

f657072

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_ren…

cf70346

…der.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_ren…

648b93c

…der.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

09844e2

…sor.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

b47e826

…sor.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

e959e4c

…sor_cfg.py Co-authored-by: James Tigue <[email protected]> Signed-off-by: Juana <[email protected]>

Update source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cf…

811ce7c

…g.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>

Improve docstrings

18ac547

Removed the benchmarking code from the sensor

e3e5cd1

Support applying physics materials to multiple rigid bodies, better n…

3b085da

…aming and doc

velocity correction

a93c29e

Call create_sdf_shape_view from physx view directly

49e1d65

		compliant_contact_stiffness: float \| None = None
		compliant_contact_damping: float \| None = None


		References
		~~~~~~~~~~

Adds visual-based tactile sensor with shape sensing example #3420

Are you sure you want to change the base?

Adds visual-based tactile sensor with shape sensing example #3420

Conversation

JuanaDd commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Screenshots

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JuanaDd Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kellyguo11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kellyguo11 commented Sep 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zxhuang97 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kellyguo11 commented Oct 17, 2025

Uh oh!

JuanaDd commented Oct 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

JuanaDd commented Sep 10, 2025 •

edited

Loading

JuanaDd Sep 10, 2025 •

edited

Loading

zxhuang97 commented Sep 30, 2025 •

edited

Loading