Skip to content

python_env_setup.md

Latest commit

 

History

History
401 lines (308 loc) · 18.2 KB

python_env_setup.md

File metadata and controls

401 lines (308 loc) · 18.2 KB
Raw

Headless Python environment for the data-collection scripts

How to set up a Python environment on a headless server that can run:

  • scripts/data_collection/replay_demos.py
  • scripts/data_collection/rollout_hdf5_demos.py
  • scripts/data_collection/run_multiple_data_collection_configs.py (which spawns scripts/data_collection/collect_synthetic_data.py as a subprocess)

It follows the Isaac Lab pip installation method: a conda env with Python 3.11, PyTorch and Isaac Sim from pip, and Isaac Lab installed from a source clone.

Quick start

bash scripts/setup/install_env.sh     # ~20-40 min, ~20 GB
bash scripts/setup/restore_assets.sh  # recover gitignored USD/USDZ assets
source scripts/setup/activate_env.sh  # in every new shell

Then verify:

bash scripts/setup/check_env.sh

No sudo is required as long as cmake is present (it is on this machine); isaaclab.sh --install only shells out to apt-get when cmake is missing.

Why Isaac Sim 5.0 / Isaac Lab 2.x, and not 3.0

The Isaac Lab 3.0 pip page pins Isaac Sim 6.0, Python 3.12 and torch 2.10. Do not use it for this repo. Isaac Lab 3.0 is a breaking rewrite and this codebase targets the 2.x API throughout:

Isaac Lab 3.0 change Where this repo breaks
.data.* returns wp.array (Warp) instead of torch.Tensor The entire MDP/mimic layer indexes and does arithmetic on these as torch tensors
Default quaternion convention flipped wxyz → xyzw Every pose computation, e.g. _convert_action_to_env_action in rollout_hdf5_demos.py
write_*_to_sim(data, env_ids) split into _index / _mask variants 29 call sites under source/, e.g. tasks/world_modeling/mdp/events.py:64, tasks/world_modeling/orca_synthetic_data_gen_mimic_env.py:210

Porting to 3.0 is a substantial project in its own right, not an environment setup. The pinned stack below is the one this code was written against.

Pinned versions

Component Version
Python 3.11
PyTorch 2.7.0 + torchvision 0.22.0, CUDA 12.8 wheels (cu128)
Isaac Sim isaacsim[all,extscache]==5.0.0 from pypi.nvidia.com
warp-lang 1.9.0 (isaaclab leaves it unpinned; a fresh resolve grabs 1.17.0)
Isaac Lab source checkout at commit 21bcb476b27ceedccccd63afef6bbd822adc2b2b

Why Isaac Lab is pinned to a commit and not a tag

The Isaac Lab release tags do not line up with what this repo needs. The pinned commit (2025-09-29, "Randomizes viscous and dynamic joint friction based on IsaacSim 5.0") is the unique revision producing:

isaaclab 0.46.3 · isaaclab_tasks 0.11.1 · isaaclab_mimic 1.0.14
isaaclab_assets 0.2.2 · isaaclab_rl 0.4.1

which is exactly the stack this codebase is developed against. The next commit touching that file bumps isaaclab to 0.46.4.

The v2.2.1 tag does not work - this was tried and it fails concretely. The tag is older (isaaclab 0.45.9 / isaaclab_tasks 0.10.47) and does not have isaaclab.utils.assets.check_usd_path_with_timeout, which faive_lab/sim/spawners/from_files/from_files.py:30 imports at module scope. The result is an ImportError deep inside the faive_lab.tasks registration chain, so orca-synthetic-mimic-ik-abs-v0 is never registered:

ImportError: cannot import name 'check_usd_path_with_timeout'
             from 'isaaclab.utils.assets'

There is no v2.2.2 tag and PyPI publishes no 2.2.x past 2.2.0, so a commit pin is the only way to reproduce this stack.

To move the pin later, find the commit whose five config/extension.toml versions match your target and set ISAACLAB_REF:

ISAACLAB_REF=<sha> bash scripts/setup/install_env.sh

The installer fetches a single commit (git fetch --depth 1 origin <ref>), so ISAACLAB_REF accepts either a SHA or a tag, and it will move an existing checkout onto the requested ref.

The torch pin matters: isaaclab.sh's ensure_cuda_torch() looks for exactly 2.7.0+cu128 and will uninstall and replace anything else. Installing torch first, at that version, makes the step a no-op.

All pins live in scripts/setup/config.sh and can be overridden from the environment:

ENV_PREFIX=/some/other/path bash scripts/setup/install_env.sh

Where things are installed

Path What
${ENV_PREFIX} (default $HOME/envs/faive_lab) The conda env (prefixed, ~20 GB)
${ISAACLAB_DIR} (default $HOME/IsaacLab) Isaac Lab source checkout at the pinned commit

The env is a prefixed env rather than one inside the conda installation, because Isaac Sim plus its shader/asset caches need ~20 GB. Override ENV_PREFIX (and ISAACLAB_DIR) to point at wherever you have space; Omniverse's own caches (~/.cache/ov, ~/.nv, ~/.local/share/ov) follow $HOME and need no separate redirection.

What the installer does, and why

  1. Preflight — NVIDIA driver, the Vulkan ICD at /usr/share/vulkan/icd.d/nvidia_icd.json (without it Isaac Sim starts but every camera renders black), cmake, ≥40 GB free.

  2. Conda env at $ENV_PREFIX with Python 3.11.

  3. PyTorch 2.7.0 + cu128, before Isaac Sim, for the ensure_cuda_torch() reason above.

  4. Isaac Sim isaacsim[all,extscache]==5.0.0. This is the ~19 GB step.

  5. setuptools pinned below 81, and flatdict pre-built. Isaac Lab pins flatdict==4.0.1, an sdist whose setup.py does import pkg_resources. setuptools 81 removed pkg_resources, so on a fresh env (which gets setuptools 83) the build fails and takes the whole isaaclab core install down with it — leaving isaaclab_tasks/isaaclab_mimic installed but import isaaclab broken. The validated reference environment runs setuptools 78.1.1. The installer pins setuptools<81 and builds flatdict once with --no-build-isolation so it uses that pin rather than a fresh overlay.

  6. Isaac Lab single-commit fetch at the pinned SHA, then ./isaaclab.sh --install none.

    none means "no RL frameworks". None of the three data-collection scripts import rsl-rl, rl-games, skrl or stable-baselines3, and installing them drags in a torch that conflicts with the pinned one (stable-baselines3 wants torch>=2.8). Set ISAACLAB_RL_FRAMEWORKS=all if you also need to train, and expect a pip dependency-resolver warning about it.

    The installer then re-asserts the torch pin and reinstalls the cu128 build if anything moved it, so this stays correct whichever ref is checked out.

  7. Both source packages — source/faive_lab and source/faive_data_generation. The README only mentions the first, but collect_synthetic_data.py:78-81 imports faive_data_generation.datagen.*, so both are required.

  8. pytorch_kinematics, the one external dependency of the vendored hand-tracking retargeter (see below).

The hand-tracking retargeter is vendored in-repo

tasks/world_modeling/orca_synthetic_data_gen_mimic_env_cfg.py imports FaiveRetargeterCfg, and devices/faive_retargeter.py imports the retargeting algorithm from faive_lab.retargeter.retargeter (under source/faive_lab/faive_lab/retargeter/) - a copy of the lab's own retargeter code, vendored so the gym environment doesn't depend on an external, separately-installed package. Isaac Lab's _walk_packages re-raises on import failure, so if this import ever breaks, the gym environment fails to register and gym.make(...) fails with a confusing "environment not found" - check_env.sh checks the retargeter's dependency and, in its end-to-end step, the registration itself for that reason.

The retargeter's only external dependency is pytorch_kinematics, installed separately by the installer.

Running the three scripts headless

source scripts/setup/activate_env.sh

activate_env.sh also sets OMNI_KIT_ACCEPT_EULA=YES (so nothing blocks on the EULA prompt) and raises the file descriptor limit to 65536, for the same reason as scripts/data_collection/launch_record.sh.

Add PYTHONUNBUFFERED=1 whenever you redirect output to a log - see the gotchas below.

# 1. Replay demonstrations
#    --headless must be passed explicitly: the script's own
#    `args_cli.headless = True` is commented out (replay_demos.py:46).
python scripts/data_collection/replay_demos.py --headless \
  --task orca-synthetic-mimic-ik-abs-v0 \
  --dataset_file datasets/<your_demos>.hdf5 \
  --num_envs 1

# 2. Roll out an HDF5 dataset and re-record it
#    --enable_segmentation_cameras implies --enable_cameras (set at line 123).
python scripts/data_collection/rollout_hdf5_demos.py --headless \
  --input_file  datasets/torus_demos_50.hdf5 \
  --output_file datasets/torus_demos_50_rollout.hdf5 \
  --object_type torus \
  --enable_segmentation_cameras

# 3. Batch data collection
#    Already forces headless + enable_cameras internally (lines 563-565), so no
#    --headless flag. But you MUST pass --output_file (see Known gaps).
python scripts/data_collection/run_multiple_data_collection_configs.py \
  --output_file /path/to/datasets/SIM_DATA

On a multi-GPU machine, pin one with export CUDA_VISIBLE_DEVICES=0.

Two headless gotchas worth knowing

1. Always set PYTHONUNBUFFERED=1 when redirecting output to a file. Isaac Sim's simulation_app.close() terminates the process before Python flushes block-buffered stdout, so redirecting a run to a log silently loses most of its output - including tracebacks and result summaries. Interactively (a TTY) stdout is line-buffered and you see everything, which makes this very confusing.

PYTHONUNBUFFERED=1 python scripts/data_collection/replay_demos.py --headless ... > run.log 2>&1

scripts/list_envs.py is the clearest victim: its table never appears in a redirected run. That is why check_env.sh uses scripts/setup/verify_task_registration.py instead, which flushes explicitly.

2. Recording needs ORCA_ENABLE_SEGMENTATION_CAMERAS=true, or you get no dataset - silently. OrcaSyntheticDataGenMimicEnvCfg.__post_init__ only assigns a recorder config with actual terms when segmentation (or RGB+depth) is enabled. Otherwise recorders stays a bare RecorderManagerBaseCfg with zero terms, and Isaac Lab's RecorderManager.__init__ returns early ("Do nothing if no active recorder terms are provided") before creating the HDF5 file handler. The run then completes, reports demos as generated, and writes nothing at all.

run_multiple_data_collection_configs.py and rollout_hdf5_demos.py --enable_segmentation_cameras set this for you. If you invoke collect_synthetic_data.py directly, set it yourself:

ORCA_ENABLE_SEGMENTATION_CAMERAS=true python scripts/data_collection/collect_synthetic_data.py \
  --headless --enable_cameras ...

Se3Keyboard and --headless

replay_demos.py used to construct an Se3Keyboard unconditionally, which crashes headless - omni.appwindow is not loaded, giving AttributeError: module 'omni' has no attribute 'appwindow'. The script now only creates the keyboard when not headless; pause/resume (B/N) has no meaning without a window. This is the one source change made outside scripts/setup/.

Verification

bash scripts/setup/check_env.sh            # full, launches Isaac Sim
bash scripts/setup/check_env.sh --skip-sim # fast, no simulator

It checks the interpreter and CUDA, every pinned package, the vendored faive_lab.retargeter import, each USD/USDZ asset on the default code path (including detecting unsmudged Git LFS pointers), the dataset paths the scripts default to, and finally runs scripts/list_envs.py and greps for orca-synthetic-mimic-ik-abs-v0.

That last step is the real test: it launches Isaac Sim headless and exercises the whole faive_lab.tasks registration chain, so it proves torch/CUDA, Isaac Sim startup, the retargeter, both source packages, and gym registration at once.

What has been verified end to end

All four scripts were run on this machine against the pinned environment:

Script Result
collect_synthetic_data.py (random motion) Generated a 2-demo, 3.7 MB HDF5 (actions (480, 24), initial_state, states, obs, segmentation_camera)
replay_demos.py --headless Replayed both episodes: "Finished replaying 2 episodes."
rollout_hdf5_demos.py --headless --enable_segmentation_cameras Re-recorded both episodes to a new 3.6 MB HDF5
run_multiple_data_collection_configs.py Validated its input, built the right command, spawned the child, and the child reached env creation

The reproducible smoke test, from nothing:

source scripts/setup/activate_env.sh
mkdir -p /tmp/faive_smoke

# 1. generate a small dataset (no source demos needed in random-motion mode)
ORCA_ENABLE_SEGMENTATION_CAMERAS=true PYTHONUNBUFFERED=1 \
python scripts/data_collection/collect_synthetic_data.py \
  --headless --enable_cameras --num_envs 1 --total_num_episodes 2 \
  --output_file /tmp/faive_smoke/gen.hdf5 --dataset_export_mode all
# writes /tmp/faive_smoke/gen_rank_0.hdf5

# 2. replay it
PYTHONUNBUFFERED=1 python scripts/data_collection/replay_demos.py --headless \
  --dataset_file /tmp/faive_smoke/gen_rank_0.hdf5 --num_envs 1

# 3. roll it out into a new dataset
PYTHONUNBUFFERED=1 python scripts/data_collection/rollout_hdf5_demos.py --headless \
  --input_file /tmp/faive_smoke/gen_rank_0.hdf5 \
  --output_file /tmp/faive_smoke/rollout.hdf5 \
  --object_type banana --enable_segmentation_cameras --overwrite

Throughput observed: ~10 env steps/s with segmentation cameras on a single RTX 3090, ~18 steps/s without.

run_multiple_data_collection_configs.py was only exercised as far as spawning its child, because its 15 configs are hardcoded to 1000 episodes each (~14 h per config at this throughput) and its MimicGen configs need annotated source demos.

Known gaps you must supply

These are outside the Python environment but will block an actual run.

1. Robot USD

assets/data/orca_v1b/orcav1b_franka_vmnt_v4_only_hand_semantics.usd is tracked with Git LFS; restore_assets.sh pulls it and check_env.sh verifies it.

Do not substitute orcav1b_franka_vmnt_v4.usd for it. That was tried: the robot loads, but the scene expects rigid bodies it does not have, failing with found a prim '.../right_pinky_pip_jointbody' which is not a rigid body.

2. Object and scene assets

misc/red_cube_SAM3D.usdz (used by object_type=dice), misc/color_arena_SAM3D_fixed.usdz and the cylinder and torus meshes are tracked with Git LFS and restored the same way. misc/duck_unit.usd, used by object_type=duck, is not part of the repository.

3. There is no datasets/ directory

Every default dataset path is dead:

  • replay_demos.py defaults to datasets/dataset_cube_big_annotated.hdf5
  • run_multiple_data_collection_configs.py references ./datasets/{banana,cup,cylinder,torus,dice}_demos*.hdf5

Supply the HDF5 files, or pass explicit paths.

Note that the MimicGen configs need annotated source demos (subtask signals written by scripts/data_collection/annotate_demos.py), not raw generator output. Feeding an unannotated dataset fails inside the datagen info pool with IndexError: index 0 is out of bounds for dimension 0 with size 0 (datagen_info_pool.py:193), because no subtask termination signal is ever non-zero.

If you have no source demos at all, you can bootstrap one: random-motion mode does not read --input_file (it is only loaded when use_mimicgen=True), so collect_synthetic_data.py can generate a dataset from nothing. See the verification section below.

4. Batch output path

run_multiple_data_collection_configs.py writes to datasets/generated unless --output_file is given. Pass --output_file to put a batch on another disk.

Troubleshooting

gym.error.NameNotFound for orca-synthetic-mimic-ik-abs-v0 The faive_lab.tasks import chain died. Run bash scripts/setup/check_env.sh --skip-sim; the usual cause is a broken faive_lab.retargeter import (see above). To see the real traceback: python -c "import faive_lab.tasks" will not work on its own (Isaac Lab modules need the app launched first), so use python scripts/list_envs.py and read the error.

Cameras render black, or Isaac Sim fails to create a Vulkan device Check /usr/share/vulkan/icd.d/nvidia_icd.json exists and that vulkaninfo runs. Headless rendering uses EGL/Vulkan directly and needs no X server or DISPLAY.

An asset check reports "LFS pointer, not content" The file is an unsmudged Git LFS pointer. Run bash scripts/setup/restore_assets.sh.

isaaclab.sh --install wants to run sudo apt-get cmake is missing. Install it, or accept the sudo prompt.

ERROR: Failed to build 'flatdict' / ModuleNotFoundError: No module named 'pkg_resources' setuptools >= 81 is in the build environment. The installer pins setuptools<81 before touching Isaac Lab; if you installed by hand, do the same and then pip install --no-build-isolation flatdict==4.0.1. Symptom to watch for: the check reports isaaclab-tasks and isaaclab-mimic as installed but isaaclab core not importable.

check_env.sh reports torch as 2.13.0+cu130 (or anything but 2.7.0+cu128) Something re-resolved torch from the default PyPI index - almost always the Isaac Lab RL-framework install step, which is why ISAACLAB_RL_FRAMEWORKS defaults to none. Older isaaclab.sh revisions have no ensure_cuda_torch() guard. Re-run bash scripts/setup/install_env.sh; it detects this and restores the pinned build.

ImportError: cannot import name 'check_usd_path_with_timeout' The Isaac Lab checkout is older than the pinned commit. Check with git -C "$ISAACLAB_DIR" rev-parse HEAD and re-run the installer, which will move it onto ISAACLAB_REF.

Warp CUDA error: Failed to get driver entry point 'cuDeviceGetUuid' at startup Benign - ignore it. These come from the warp build bundled inside Isaac Sim (note the /builds/omniverse/warp/... path in the message), not from the pip warp-lang package, so changing the pip version does not affect them. Verified: the messages are byte-identical under warp-lang 1.9.0 and 1.17.0, and the validated reference environment on this machine emits them too while working fine. Simulation, cameras and task registration all work regardless.

Out of disk during install The env needs ~20 GB. Override the location: ENV_PREFIX=/big/disk/env bash scripts/setup/install_env.sh.