DEEPS: Depth Estimation Enhanced Pseudo s

Pseudo Label Generation for 3D Object Detection

A comprehensive framework for generating pseudo labels for 3D object detection for autonomous driving using multiple complementary approaches.

🎯 About This Repository

This repository implements several (state-of-the-art) pseudo-labeling approaches for 3D object detection in autonomous driving scenarios. The framework combines zero-shot depth estimation and object segmentation, spatial clustering, appearance-based tracking, bounding box heuristics and several other techniques to generate high-quality pseudo labels for LiDAR-based 3D object detection.

Key Features:

• Depth estimation using MapAnything
• Object segmentation and tracking using Grounded SAM2
• UNION based Lidar segmentation
• Support for nuScenes dataset with different class mapping schemes (class agnostic, 3 classes or 8 classes)
• Evaluation pipeline with nuScenes metrics
• Rerun-based visualization tools for debugging and analysis

System Requirements

• Ubuntu 22.04.3 LTS (or compatible Linux distribution)
• NVIDIA GPU with CUDA 12.8+ support
• Docker and Docker Compose

🔧 How It Works

Pipeline Overview

The framework operates in a multi staged pipeline, with an initial setup step:

⚙️ Configuration & Dataset Setup (Preliminary)

• Config Files: Define pseudo-labeling parameters in the configs/<stage>/ directory, see examples
• Dataset Preparation: Ensure proper formatting and placement of raw data (e.g., nuScenes), location is defined in the config file

---

1. Pseudo-Label Generation

The approach is implemented in the main_pseudo_*.py scripts:

Stages:

• Depth Estimation (main_depth_estimation.py): Estimates depth images from color images
• Image Segmentation (main_segmentation.py): Segments color images and labels detected objects and tracks them
• Image Box Estimation (main_seg_depth_merge.py): Merges segmentation with depth information to obtain image based pseudo boxes
• LiDAR detection (./vespa/main_pseudo_union.py): LiDAR clustering
• LiDAR Image merge (main_merge_cam_lidar.py): Merges LiDAR based boxes with Image based boxes

🚀 Quick Start

1. Clone Repository

# Clone the repository
git clone <repository-url>
cd <repository-name>

2. Estimate LiDAR Boxes

This project uses LiDAR Box estimation based on the UNION implementation from VESPA (for details see the VESPA README).

Environment Setup

# Build and run LiDAR based pipeline with Docker Compose
docker-compose up vespa -d
docker-compose exec vespa bash

Generate pseudo boxes

# UNION approach (LiDAR clustering)
python main_pseudo_lidar.py 
    --config configs/union_lidar/base_mini.yaml \ 
    --workers 10 # Optional

Generated boxes will be saved to outputs/union/<config_name>/lidar_boxes.json.

3. Estimate Image Boxes

Environment Setup

# Build and run image based pipeline and merge using Makefile
make build-image
make run
docker exec -it deeps-deeps-1 bash

Generate Pseudo Boxes

# Depth Estimation. Only config argument is required
python main_depth_estimation.py \ 
    --config configs/depth/scene-0061.yaml \ 
    --single_source \ 
    --num_worker 2

# Image Segmentation and tracking. Only config argument is required
python main_segmentation.py \ 
    --config configs/segmentation/scene-0061.yaml

# Image Box estimation
python main_seg_depth_merge.py \ 
    --config configs/merge/scene-0061.yaml \ 
    --num_worker 10 # Optional

4. Merge Boxes

# LiDAR image box fusion
python main_merge_camera_lidar.py \ 
    --config configs/merge_cam_lidar/scene-0061.yaml \ 
    --num_worker 10 # Optional. Only recommended for large datasets

Generated labels will be saved to outputs/<stage>/<dataset>.

📊 Evaluation

The framework provides comprehensive evaluation using nuScenes detection metrics:

• mAP (mean Average Precision): Primary detection metric
• mATE (mean Average Translation Error): Localization accuracy
• mASE (mean Average Scale Error): Size estimation accuracy
• mAOE (mean Average Orientation Error): Heading accuracy
• mAVE (mean Average Velocity Error): Velocity estimation accuracy
• NDS (nuScenes Detection Score): Overall weighted score

To run evaluation start the deeps container as explained above and execute

python main_eval.py \ 
    submission_file.json

🐛 Debugging & Visualization

Rerun Visualization

You can visualize a scene with the following command.

python main_visualization.py \ 
    --data_root /datasets/nuscenes/ \ 
    --scene scene-0061 \ 
    --results_file /deeps/outputs/matched/nuscenes/nusc_pseudo_labels_8class.json \ 
    --seg_dir outputs/gsam2/nuscenes/scene-0061 \           # Optional
    --depth_dir outputs/map_anything/nuscenes/scene-0061    # Optional

Furthermore, using --cam_box_path and --lidar_box_path with the corresponding .json file allows the visualization of intermediate results. --seg_dir requires visualization flag to be set when running segmentation.

You may have to run xhost si:localuser:root on your host for GUI rendering. This is temporary and can be undone using xhost -si:localuser:root

Visualization Features:

• LiDAR point clouds with instance color-coded clusters
• Image colorized depth estimation point clouds
• 3D bounding boxes
• Camera images and segmentation (if visualized)

🎓 Research Background

This research work was conducted as the Master Thesis of Patrick Goroncy, under the supervision of Esteban Rivera at the Department of Mobility Systems Engineering of the TUM School of Engineering and Design at the Technical University of Munich.

📚 Acknowledgments

This work builds upon several open-source projects and research contributions:

• nuScenes: Large-scale autonomous driving dataset and evaluation metrics
• Grounded SAM2: Zero-shot object detection, segmentation and tracking
• Rerun: Real-time visualization and debugging
• VESPA/UNION: State-of-the-art approach for pseudo labeling of autonomous driving scenes