Authors: Keshav Gupta, Tejas S. Stanley, Pranjal Paul, Arun K. Singh, K. Madava Krishna
- 🚀 Quick Start
- 🐍 Environment Setup
- 📊 Dataset Preparation
- 🔮 Inference
- 📈 Evaluation
- 🏋️ Training
- 🤝 Contributing
This repository implements Diffusion-FS, a novel approach for multimodal free-space prediction in autonomous driving using diffusion models. From a dataset of raw driving logs containing image and ego trajectory pairs, our self supervised method processes such an unannotated dataset to generate freespace segments essential for autonomous driving. At inference, our model denoises a fixed number of noise samples into freespace segments. We showcase predictions across various weather conditions, times of day, road topologies, and obstacle layouts. We show the results of our method on both simulated (CARLA) and real-world (NuScenes) datasets.
- Python 3.10.16
- CUDA-compatible GPU (recommended)
- Conda package manager
- Create and activate conda environment:
conda create -n diff_fs_env python=3.10.16 -y
conda activate diff_fs_env
- Install PyTorch with CUDA support:
pip install torch==2.3.1+cu118 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Note: Replace
cu118
with your CUDA version (e.g.,cu121
for CUDA 12.1)
- Install additional dependencies:
pip install -r requirements.txt
We perform experiments on both CARLA (simulated) and NuScenes (real-world) datasets.
- Download from HuggingFace Dataset Repository
- Contains cached training/evaluation data and scenario-wise situation classes (no lane, single lane, multilane, intersection)
- Merge the splits:
mv carla/carla_train/split_*/*.npz carla/carla_train/
rm -r carla/carla_train/split_*
- Download data from LAV repository
- Cache manually using
datasets_all/fs_dataset.py
- Update
data_dir_train
anddata_dir_test
indatasets_all/fs_cached.yaml
- Download from HuggingFace Dataset Repository
- Merge the splits:
mv nuscenes/FS/split_*/*.npz nuscenes/FS/
mv nuscenes/fs_meta/split_*/*.npz nuscenes/fs_meta/
rm -r nuscenes/FS/split_*
rm -r nuscenes/fs_meta/split_*
Use the script: dataset_all/nusc_save.py
Download checkpoints from Google Drive:
- CARLA Base:
pretrained_ckpts/carla_base.ckpt
- CARLA Command Conditioning:
pretrained_ckpts/carla_cls.ckpt
- NuScenes Base:
pretrained_ckpts/nuscenes_base.ckpt
python3 infer.py
Configuration Options (modify at the beginning of infer.py
):
- Config file path
- Checkpoint path
- Output visualization path
- Noise template usage (optional)
- Obstacle guidance during inference (optional)
Output: 6 inferred freespace segments overlayed on each input image.
python3 evaluate.py
python3 dd_calc.py
Run all experiments (base, obs, cls, noise, obs+cls) sequentially:
python3 eval_all_dd.py
Parse and display results:
python3 calc.py
Configuration Options (modify at the beginning of each script):
- Checkpoint path
- Config file path
- Noise template usage (optional)
- Obstacle guidance during inference (optional)
- Situation classes folder path (for
eval_all_dd.py
)
Precomputed Results: Available in results/
folder
python3 evaluate.py
Use the same optional arguments as CARLA evaluation.
Modify hyperparameters in:
configs/carla.yaml
(for CARLA dataset)configs/nuscenes.yaml
(for NuScenes dataset)
Available Parameters:
- Model hyperparameters
- Optimizer settings
- Checkpointing paths
- Diffusion denoising parameters
python3 train.py
We welcome contributions! Please feel free to submit issues and pull requests.
📧 Contact: For questions or issues, please open an issue on GitHub or contact the authors.
🌟 Star this repo if you find it helpful!