2025-11-06: Issac Sim - Setup jupyter notebook with WebRTC Client

While following Nvidia’s Issac Sim Tutorial 9: Pick and Place Example, I ran into an issue running the included python script examples for the Gipper Control Example.

My setup consists of running Issac Sim in docker and interfacing via the IsaacSim WebRTC Streaming Client. The problem here is when following the Gipper Control Example and invoking python.sh, that does not latch on to the, already existing, running IsaacSim instance and instead runs an entirely new instance in the background (see IsaacSim: SimulationApp).

[!IMPORTANT] References to “Standalone Isaac Sim” implies it running an instance of Isaac Sim. Even though examples show headless: True, a new instance will be started. headless being in the backround (no UI) launched.

Issac Sim docker setup

To get started with running IsaacSim in docker, we need to make sure Nvidia's Container Toolkit is installed, which for my case is under ArchLinux’s nvidia-container-toolkit package. This allows the docker runtime to fully utilize the host’s GPU.

After installing the package, we need to apply it:

$ sudo nvidia-ctk runtime configure --runtime=docker
INFO[0000] Config file does not exist; using empty config
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.

and verify it works:

% docker run --rm -it --runtime=nvidia --gpus all ubuntu:22.04 nvidia-smi
Sun Oct 26 22:03:31 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02             Driver Version: 570.153.02     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5070        Off |   00000000:01:00.0  On |                  N/A |
|  0%   36C    P8              7W /  250W |    1070MiB /  12227MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Now running the IsaacSim image, nvcr.io/nvidia/isaac-sim:5.1.0, I made 3 scripts to help setting up and run the environments I want.

setup_docker.sh

This is a base script that ensures the directories, permissions, and docker image is ready. This includes jupyter notebook direcotires which will be used in getting scripts to work/interact with the running headless IsaacSim process:

#!/usr/bin/env bash
set -ex

IMAGE="nvcr.io/nvidia/isaac-sim:5.1.0"

docker pull ${IMAGE}

# Set up cached directories
CACHE_DIRS=( \
  cache/main/ov \
  cache/main/warp \
  cache/computecache \
  config \
  data/documents \
  data/Kit \
  logs \
  pkg \
  jupyter_notebook_docs \
  jupyter_config \
)

for p in ${CACHE_DIRS[@]}; do
  if [ ! -e "$p"  ]; then
    mkdir -p "$p"
    sudo chown -R 1234:1234 "$p"
  fi
done

run_docker-headless.sh

This is the main script that I’m using to run IsaacSim in headless mode for which to attach the WebRTC client to later on:

#!/usr/bin/env bash
set -ex
source ./setup_docker.sh

docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
  -e "PRIVACY_CONSENT=Y" \
  -v ./cache/main:/isaac-sim/.cache:rw \
  -v ./cache/computecache:/isaac-sim/.nv/ComputeCache:rw \
  -v ./logs:/isaac-sim/.nvidia-omniverse/logs:rw \
  -v ./config:/isaac-sim/.nvidia-omniverse/config:rw \
  -v ./data:/isaac-sim/.local/share/ov/data:rw \
  -v ./pkg:/isaac-sim/.local/share/ov/pkg:rw \
  -v ./jupyter_notebook_docs:/isaac-sim/exts/isaacsim.code_editor.jupyter/data/notebooks \
  -v ./jupyter_config:/isaac-sim/.local/share/jupyter \
  -u 1234:1234 \
  ${IMAGE} ./runheadless.sh -v -–/app/window/dpiScaleOverride=1.0 --/app/livestream/allowResize=true

run_docker-bash.sh

The final script which I occasionally use for troubleshooting. It throws the user into an interactive shell within the conatiner, for which I use it to discover the existing scripts and assets and whatnot.

#!/usr/bin/env bash
set -ex
source ./setup_docker.sh

docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
  -e "PRIVACY_CONSENT=Y" \
  -v ./cache/main:/isaac-sim/.cache:rw \
  -v ./cache/computecache:/isaac-sim/.nv/ComputeCache:rw \
  -v ./logs:/isaac-sim/.nvidia-omniverse/logs:rw \
  -v ./config:/isaac-sim/.nvidia-omniverse/config:rw \
  -v ./data:/isaac-sim/.local/share/ov/data:rw \
  -v ./pkg:/isaac-sim/.local/share/ov/pkg:rw \
  -v ./jupyter_notebook_docs:/isaac-sim/exts/isaacsim.code_editor.jupyter/data/notebooks \
  -v ./jupyter_config:/isaac-sim/.local/share/jupyter \
  -u 1234:1234 \
  ${IMAGE}

WebRTC Client

Now run the run_docker-headless.sh script the the background and lets get the client setup.

Within Nvidia’s IsaacSim WebRTC Streaming Client docs, there’s a link to downloading the prebuilt client, which links to Installation: Latest Release page. Download the Isaac Sim WebRTC Streaming Client for your appropriate platform and run it. Then connect to 127.0.0.1, which is where the headless script’s server will be bound to.

Jupyter Notebook

Now that we have docker and the client setup, we need to set up the Jupyter Notebook extension within IsaacSim. All you have to do is Enable and Autoload the extension:

To begin, enable this extension using the Extension Manager by searching for isaacsim.code_editor.jupyter.

Once that’s done, open a browser on the host and enter http://127.0.0.1:8228/lab.

Translating example scripts

Before we get started, create a new Omniverse python kernel file in the jupyter notebook browser tab.

Following Nvidia’s Issac Sim Tutorial 9: Pick and Place Example, the gripper control example references the script at standalone_examples/api/isaacsim.robot.manipulators/ur10e/gripper_control.py, which contains:

# SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": True})

import argparse

import numpy as np
from isaacsim.core.api import World
from isaacsim.core.utils.stage import add_reference_to_stage
from isaacsim.core.utils.types import ArticulationAction
from isaacsim.robot.manipulators import SingleManipulator
from isaacsim.robot.manipulators.grippers import ParallelGripper
from isaacsim.storage.native import get_assets_root_path

parser = argparse.ArgumentParser()
parser.add_argument("--test", default=False, action="store_true", help="Run in test mode")
args, unknown = parser.parse_known_args()

my_world = World(stage_units_in_meters=1.0)
assets_root_path = get_assets_root_path()
if assets_root_path is None:
    raise Exception("Could not find Isaac Sim assets folder")

asset_path = assets_root_path + "/Isaac/Samples/Rigging/Manipulator/configure_manipulator/ur10e/ur/ur_gripper.usd"
add_reference_to_stage(usd_path=asset_path, prim_path="/ur")
# define the gripper
gripper = ParallelGripper(
    # We chose the following values while inspecting the articulation
    end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
    joint_prim_names=["finger_joint"],
    joint_opened_positions=np.array([0]),
    joint_closed_positions=np.array([40]),
    action_deltas=np.array([-40]),
    use_mimic_joints=True,
)
# define the manipulator
my_ur10 = my_world.scene.add(
    SingleManipulator(
        prim_path="/ur",
        name="ur10_robot",
        end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
        gripper=gripper,
    )
)

my_world.scene.add_default_ground_plane()
my_world.reset()

i = 0
reset_needed = False
while simulation_app.is_running():
    my_world.step(render=True)
    if my_world.is_stopped() and not reset_needed:
        reset_needed = True
    if my_world.is_playing():
        if reset_needed:
            my_world.reset()
            reset_needed = False
        i += 1
        gripper_positions = my_ur10.gripper.get_joint_positions()
        if i < 400:
            # close the gripper slowly
            my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] + 0.1]))
        if i > 400:
            # open the gripper slowly
            my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] - 0.1]))
        if i == 800:
            i = 0
    if args.test is True:
        break

simulation_app.close()

Translating: Imports

See Isaac Sim: Python API documentation for more info about what the libraries do.

Importing/instantiating a SimulationApp class will not work as the running environment is not set up for running Standalone Isaac Sim. So delete that.

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": True})

We don’t care about the argparse logic but we do care about most of the libraries so we’re left with:

import numpy as np
from isaacsim.core.api import World
from isaacsim.core.utils.types import (
    ArticulationAction,
    Usd,
)
from isaacsim.robot.manipulators import SingleManipulator
from isaacsim.robot.manipulators.grippers import ParallelGripper
from isaacsim.storage.native import get_assets_root_path

We don’t care about this function as the expectation in Tutorial 9 is to have gotten to a point where the assets are already in the scene AND open in the Isaac Sim WebRTC client:

from isaacsim.core.utils.stage import add_reference_to_stage
from isaacsim.storage.native import get_assets_root_path

Translating: World class

Instantiating our World class, which is inherits from SimulationContext and handles time-related events such as rendering and physics.

The example includes a reference to the example gripper asset but we already have it in our environment and is open:

my_world = World(stage_units_in_meters=1.0)
assets_root_path = get_assets_root_path()
if assets_root_path is None:
    raise Exception("Could not find Isaac Sim assets folder")

asset_path = assets_root_path + "/Isaac/Samples/Rigging/Manipulator/configure_manipulator/ur10e/ur/ur_gripper.usd"
add_reference_to_stage(usd_path=asset_path, prim_path="/ur")

so instead of adding the gripper to the world, we bind to the existing one like so:

# Attach to current world and grab the gripper XPrim object.
my_world = World(
    stage_units_in_meters=1.0,
    physics_prim_path="/ur/physicsScene",
)

ee_link_prim: Usd.Prim = my_world.stage.GetPrimAtPath("/ur/ee_link")
print( "ee_link children:", ee_link_prim.GetChildrenNames() )

robotiq_base_link: Usd.Prim = ee_link_prim.GetObjectAtPath("/ur/ee_link/robotiq_arg2f_base_link")

The reason I’ve included the physics_prim_path, is due to the defaulting to /physicsScene and creating one if it doesn’t exist (World class doc). But we have one in the scene so I’m referencing mine.

Translating: Gripper/Manipulator

The gripper instance can remain the same. It references the right prim:

gripper = ParallelGripper(
    # We chose the following values while inspecting the articulation
    end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
    joint_prim_names=["finger_joint"],
    joint_opened_positions=np.array([0]),
    joint_closed_positions=np.array([40]),
    action_deltas=np.array([-40]),
    use_mimic_joints=True,
)

Same with the manipulator instance, however do note that this block can only run ONCE as there’s a unique name attached to it and thus only one instance can exist in the scene:

my_ur10 = my_world.scene.add(
    SingleManipulator(
        prim_path="/ur",
        name="ur10_robot",
        end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
        gripper=gripper,
    )
)

Translating: Running the simulation

For sanity’s sake, we can verify that we’re connected to the right application and it’s running:

# Access running simulation via the world
print(my_world.app.get_app_name())
assert my_world.app.is_running()

expecting an ouput of: Isaac-Sim Streaming

For the following lines:

my_world.scene.add_default_ground_plane()
my_world.reset()

we can include a default ground plane, which is very nice actually, however, don’t reset the world just yet. Our World instance is not yet initialized. We need to initialize the simulation context, which will also handle setting up the physics context so we can step through our physics sim:

my_world.scene.add_default_ground_plane()

# This method is intended to be used in the Isaac Sim’s Extensions workflow
# where the Kit application has the control over timing of physics
# and rendering steps
await my_world.initialize_simulation_context_async()
assert my_world.get_physics_context()

# Let er rip!
if my_world.is_stopped():
    my_world.reset()

Now since we didn’t instantiate a SimulationApp class, but we do have a running sim instance, instead of relying on the entire instance running:

i = 0
reset_needed = False
while simulation_app.is_running():
    my_world.step(render=True)
    if my_world.is_stopped() and not reset_needed:
        reset_needed = True
    if my_world.is_playing():
        if reset_needed:
            my_world.reset()
            reset_needed = False
        i += 1
        gripper_positions = my_ur10.gripper.get_joint_positions()
        if i < 400:
            # close the gripper slowly
            my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] + 0.1]))
        if i > 400:
            # open the gripper slowly
            my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] - 0.1]))
        if i == 800:
            i = 0

we can instead just check that our simulation is in Play mode via my_world.is_playing :shrug: like so:

i = 0
while my_world.is_playing():
    my_world.step(render=True)
    i += 1

    gripper_positions = my_ur10.gripper.get_joint_positions()
    if i < 400:
        # close the gripper slowly
        my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] + 0.1]))
    if i > 400:
        # open the gripper slowly
        my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] - 0.1]))
    if i == 800:
        i = 0

my_world.stop()
print("Simulation ended")

and that’s it. to stop it, just click the stop button in the WebRTC client to stop the simulation and the block will exit.

Translating: Outcome

Here’s the fully translated script, split by jupyter notebook blocks as comments:

# 1. Library imports
import numpy as np
from isaacsim.core.api import World
from isaacsim.core.utils.types import (
    ArticulationAction,
    Usd,
)
from isaacsim.robot.manipulators import SingleManipulator
from isaacsim.robot.manipulators.grippers import ParallelGripper
from isaacsim.storage.native import get_assets_root_path

# 2. Attach to current world and grab the gripper XPrim object.
my_world = World(
    stage_units_in_meters=1.0,
    physics_prim_path="/ur/physicsScene",
)

ee_link_prim: Usd.Prim = my_world.stage.GetPrimAtPath("/ur/ee_link")
print( "ee_link children:", ee_link_prim.GetChildrenNames() )

robotiq_base_link: Usd.Prim = ee_link_prim.GetObjectAtPath("/ur/ee_link/robotiq_arg2f_base_link")

# 3. Instantiating gripper instance
gripper = ParallelGripper(
    # We chose the following values while inspecting the articulation
    end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
    joint_prim_names=["finger_joint"],
    joint_opened_positions=np.array([0]),
    joint_closed_positions=np.array([40]),
    action_deltas=np.array([-40]),
    use_mimic_joints=True,
)

# 4. Instantiating manipulator instance
my_ur10 = my_world.scene.add(
    SingleManipulator(
        prim_path="/ur",
        name="ur10_robot",
        end_effector_prim_path="/ur/ee_link/robotiq_arg2f_base_link",
        gripper=gripper,
    )
)

# 5. Access running simulation via the world
print(my_world.app.get_app_name())
assert my_world.app.is_running()


# 6. Run simulation
my_world.scene.add_default_ground_plane()

# This method is intended to be used in the Isaac Sim’s Extensions workflow
# where the Kit application has the control over timing of physics
# and rendering steps
await my_world.initialize_simulation_context_async()
assert my_world.get_physics_context()

# Let er rip!
if my_world.is_stopped():
    my_world.reset()

i = 0
while my_world.is_playing():
    my_world.step(render=True)
    i += 1

    gripper_positions = my_ur10.gripper.get_joint_positions()
    if i < 400:
        # close the gripper slowly
        my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] + 0.1]))
    if i > 400:
        # open the gripper slowly
        my_ur10.gripper.apply_action(ArticulationAction(joint_positions=[gripper_positions[0] - 0.1]))
    if i == 800:
        i = 0

my_world.stop()
print("Simulation ended")

2025-11-06: Issac Sim - Setup jupyter notebook with WebRTC Client#

Issac Sim docker setup#

setup_docker.sh#

run_docker-headless.sh#

run_docker-bash.sh#

WebRTC Client#

Jupyter Notebook#

Translating example scripts#

Translating: Imports#

Translating: World class#

Translating: Gripper/Manipulator#

Translating: Running the simulation#

Translating: Outcome#

References#