Skip to content

Simulated RealSense D455 in MuJoCo for RTAB-Map SLAM - #275

Open
zabsy wants to merge 4 commits into
mainfrom
slam-sim-mujoco
Open

zabsy wants to merge 4 commits into
mainfrom
slam-sim-mujoco

Conversation

@zabsy

@zabsy zabsy commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Description

Adds src/perception/perception/slam/slam_sim_mj/, a simulated RealSense D455 that
walks a corridor loop in MuJoCo and feeds RTAB-Map, producing a 2D floor plan.

This is a simulation testbed for the real-camera procedure in slam-quickstart.md,
so the SLAM side can be developed and regression-tested without hardware: the room is
known exactly, the trajectory is repeatable, and a full run takes ~3 minutes on a
laptop with no GPU.

Design

Two halves that share no code and talk only over ROS topics:

  • mj_slam_publisher.py renders the camera and publishes what a real driver
    would — /camera/camera/color/image_raw, .../depth/image_rect_raw,
    .../color/camera_info, .../imu.
  • launch/slam_sim_mj.launch.py runs madgwick → static TFs → rtabmap.

Topic names, encodings and frame ids match realsense2_camera, so the SLAM half
cannot tell simulation from hardware, and the launch file runs the same
imu_filter_madgwick node with the same parameters as the real-hardware procedure.
Swapping the publisher for a real camera leaves the SLAM half untouched.

Intrinsics (fx=383.682, fy=383.165 at 640×480) are taken from a real D455 recording
rather than invented, since RTAB-Map derives scale and geometry from them.

The IMU is synthesised by differentiating the known trajectory. RTAB-Map uses an IMU
as a gravity prior in RGB-D mode; without one, odometry drifts in z on a path that is
flat by construction, and since Grid/MaxGroundHeight is 0.15 m that drift destroys
ground/obstacle classification. A physics-based IMU sensor cannot be used here — the
camera is kinematically posed, so such a sensor reads zero.

Running it

python3 -m venv --system-site-packages .venv     # repo root, once
.venv/bin/pip install mujoco imageio imageio-ffmpeg
src/perception/perception/slam/slam_sim_mj/run.sh

Outputs go to outputs/slam_mj/ (gitignored, matching outputs/rl/ and
outputs/train/). The README covers the two-terminal RViz flow and how to explore
the room interactively.

Notes for review

- The scene is generated, not committed. scene/build_room.py builds room.xml
  and its 59 textures on first run, deterministically from a fixed seed, so every
  checkout gets an identical room without carrying binaries in the repo.
- Not a ROS package. No package.xml; nothing here is built or installed by
  colcon, and the launch file is launched by path. It lives under perception/slam/
  because it is perception work, next to the quickstart it mirrors.
- 149 tests, ~0.3 s, no GPU required. The trajectory and IMU maths are pure numpy
  and tested independently — rotation and accelerometer sign errors are invisible at
  runtime and surface much later as a map that is silently tilted or mirrored.
- MuJoCo is the only new dependency, installed in a venv rather than added to any
  package manifest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant