Course 4 - Developing With Software-in-the-Loop (SIL) In Isaac Sim

Course 4 - Developing With Software-in-the-Loop (SIL) In Isaac Sim

Image Credits

Image from NVIDIA - Tutorial for DNN Image Segmentation with Isaac Sim

This course covers the concept of software-in-the-loop (SIL) simulation, demonstrating its application through image segmentation on a simulated robot in Isaac Sim.

❗

Unfortunately, I was unable to complete the last module (Image Segmentation with SIL) is incomplete. See below for more information.

Module 1: Intelligent Robots

No additional comments for this module.

Module 2: Software-in-the-Loop (SIL)

No additional comments for this module.

Module 3: Using Omni-Graph to Drive a Robot in Isaac Sim

This module walks through how to control a robot in Isaac Sim through keyboard control.

‣
What does /cmd_vel stand for? What is it?
‣
What is the “behind-the-scenes” of the action graph?

Module 4: Foundations of Isaac Sim Scripting With Python

To create a ground plane in Isaac Sim, we usually go to Create > Physics > Ground Plane in the GUI. But in this module, we use code to accomplish the same thing.

‣
What are USD APIs and Isaac Sim Core APIs, and what is the difference between them?

In each of the two activities below, we set up a scene that obeys physics with a cube and a ground plane - one time with USD APIs, and another with Isaac Sim Core APIs.

Activity: Working with USD APIs

Unfortunately, the USD API script provided gave me errors, such as Cannot import name UsdPhysics from pxr.

Below is a working version (based on Omni Physics 107.3).

‣
Full code

Activity: Isaac Sim Core APIs

The code provided for this section also gave me an error: AttributeError: ‘NoneType’ object has no attribute ‘material’.

‣
Full code
‣
Programming in VS Code

Resources/ Additional Info:

Quickstart with Isaac Sim — Isaac Sim DocumentationQuickstart with Isaac Sim — Isaac Sim Documentation

Isaac Sim Core TutorialsIsaac Sim Core Tutorials

Omni PhysX Commands — Omni PhysicsOmni PhysX Commands — Omni Physics

Simulation Control — Omni PhysicsSimulation Control — Omni Physics

Module 5: Foundations of Image Segmentation, ROS 2, and Isaac ROS

No additional comments for this module.

Module 6: Image Segmentation Without Simulation

This module shows you how to run an image segmentation model that segments people in an image and displaying them as colorized masks. I faced a couple of errors during the setup process, so here is what I did:

The rest of the instructions (running the launch file and visualizing results) had no issues for me.

‣
What is Docker and why are we using it?

Module 7: Image Segmentation with SIL

In the last course, we ran image segmentation on a static image. This module aims to perform the image segmentation on the Nova Carter’s simulated camera feed.

❗

I faced errors when I reached “Run the Image Segmentation Package” step. The steps below go up till then.

  1. Assuming you have already installed and setup the ROS2 bridge, you can open Isaac Sim Selector and ensure that “ROS Bridge Extension” is populated.
  2. To open the sample scene in Isaac Sim, open the window IsaacSim Assets [Beta] in the bottom left. Then select ROS2 → Scenario → carter_warehouse_apriltags_worker
    1. Loading this sample scene is quite intensive.
  3. In Ubuntu via WSL2, launch the Isaac ROS development container.
    1. Follow “Build isaac_ros_unet” from the previous module
    2. Follow “Prepare PeopleSemSegnet Model” from the previous module
    3. Source your ROS 2 workspace:
      4. echo "export ISAAC_ROS_WS=${HOME}/workspaces/isaac_ros-dev/" >> ~/.bashrc
source ~/.bashrc

      Resource/Additional Info: Developer Environment Setup — isaac_ros_docs documentationDeveloper Environment Setup — isaac_ros_docs documentation

  4. While the simulation in Isaac Sim is running, run ros2 topic list and confirm that /front_stereo_camera/left/image_rect_color appears
  5. Launch the image segmentation package
    1. During this step, I faced the errors
      2. gxf/core/runtime.cpp@791 could not deinitialize entity 'FZDLULTAVL_utils' E113: GXF_Failure gxf/std/program.cpp@582 deactivation failed

      This last part of the module is incomplete as this error still persists.