SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery

NeurIPS 2024 Spotlight

Jian Song^1,2, Hongruixuan Chen¹, Weihao Xuan^1,2, Junshi Xia², Naoto Yokoya^1,2

¹ The University of Tokyo, ² RIKEN AIP

🔭 Overview

SynRS3D: Largest Remote Sensing Synthetic Dataset

SynRS3D is a high-quality, high-resolution synthetic remote sensing (RS) dataset designed to enhance global 3D semantic understanding. It contains 69,667 images with ground sampling distances (GSD) ranging from 0.05 to 1 meter, and provides annotations for height estimation, land cover mapping, and building change detection across a wide variety of scenes.

RS3DAda: UDA & DG Multi-Task Baseline Method in RS

RS3DAda is a novel unsupervised domain adaptation (UDA) and domain generalization (DG) multi-task baseline, designed for transferring knowledge from synthetic to real remote sensing data.

🛎️Updates

Oct 10th, 2024: Codes and data are ready! You are welcome to use it.
Sep 27th, 2024: Codes and data are coming soon! Please stay tuned!!
Sep 26th, 2024: SynRS3D has been accepted at NeurIPS D&B Track 2024 as a Spotlight!!

✅ To-Do List

Release the SynRS3D generation code
Release the building change detection code
Release the SynRS3D dataset
Release the RS3DAda code

⚙️ Installation

Note that the code in this repo runs under Linux system. We have not tested whether it works under other OS.

Clone this repository:

git clone https://github.com/JTRNEO/SynRS3D.git
cd SynRS3D

Create and activate the conda environment:

conda create -n synrs3d python=3.8
conda activate synrs3d
conda install pytorch=2.2.1 torchvision=0.17.1 torchaudio=2.2.1 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install gdal
pip install albumentations tqdm ever-beta==0.2.3 huggingface_hub rasterio

💾 Datasets Preparation

Download the SynRS3D dataset:

SynRS3D:

Place all zip files under ./SynRS3D/data and extract them.

Real-World Elevation Datasets

Here is the table of real-world elevation datasets we used:

Types	Datasets	Data Source	License/Conditions of Use
Target Domain 1	DFC18	2018 IEEE GRSS Data Fusion Challenge	Creative Commons Attribution
	DFC19_JAX	Data Fusion Contest 2019	Creative Commons Attribution
	DFC19_OMA	Data Fusion Contest 2019	Creative Commons Attribution
	geonrw_urban	GeoNRW	Creative Commons Attribution
	geonrw_rural	GeoNRW	Creative Commons Attribution
	Potsdam	ISPRS	Redistribution is not allowed
Target Domain 2	OGC_ATL	Overhead Geopose Challenge	Creative Commons Attribution
	OGC_ARG	Overhead Geopose Challenge	Creative Commons Attribution
	Vaihingen	ISPRS	Redistribution is not allowed
	Nagoya	NTT DATA Corporation and Inc. DigitalGlobe	End User License Agreement (Commercial)
	Tokyo	NTT DATA Corporation and Inc. DigitalGlobe	End User License Agreement (Commercial)

❗❗❗ We provide those redistributable datasets that have been preprocessed by ourselves. For datasets like ISPRS, which do not allow redistribution, we provide a script to process them in ./prepare_ISPRS_datasets. For commercial datasets like Nagoya and Tokyo, we cannot provide them directly.

(Optional) Land Cover Mapping Real-World Dataset

For land cover mapping, we used the OpenEarthMap dataset. Please download it from the following link and reorganize it with the same structure as SynRS3D:

OpenEarthMap

Redistributable Datasets Download Links

The redistributable datasets can be downloaded from the links provided below:

DFC18 (Houston dataset): Google Drive

Place all zip files under ./SynRS3D/data and extract them.

DFC19 (JAX & OMA): Google Drive

DFC19 (JAX & OMA): After downloading, place the files in ./SynRS3D/data and extract them. In the folders ./SynRS3D/data/DFC19_JAX and ./SynRS3D/data/DFC19_OMA, we have prepared training and testing lists for JAX and OMA, respectively. Please use the following soft link commands to set up the DFC19 dataset:

ln -s /your/project/path/SynRS3D/data/DFC19/opt /your/project/path/SynRS3D/data/DFC19_JAX/opt

ln -s /your/project/path/SynRS3D/data/DFC19/gt_ss_mask /your/project/path/SynRS3D/data/DFC19_JAX/gt_ss_mask

ln -s /your/project/path/SynRS3D/data/DFC19/gt_nDSM /your/project/path/SynRS3D/data/DFC19_JAX/gt_nDSM

ln -s /your/project/path/SynRS3D/data/DFC19/opt /your/project/path/SynRS3D/data/DFC19_OMA/opt

ln -s /your/project/path/SynRS3D/data/DFC19/gt_ss_mask /your/project/path/SynRS3D/data/DFC19_OMA/gt_ss_mask

ln -s /your/project/path/SynRS3D/data/DFC19/gt_nDSM /your/project/path/SynRS3D/data/DFC19_OMA/gt_nDSM

This will create the necessary symbolic links for the DFC19 dataset setup.

GeoNRW (Rural & Urban): Google Drive
- GeoNRW (Rural & Urban): The original GeoNRW dataset contains more than 120,000 512x512 images. In order to avoid data imbalance, we only selected some cities in our original paper for experimentation and divided them into rural and urban. All DSMs were preprocessed and converted to nDSMs. After downloading, place the files in ./SynRS3D/data and extract them. In the folders ./SynRS3D/data/geonrw_rural and ./SynRS3D/data/geonrw_urban, we have prepared training and testing lists for rural cities and urban cities, respectively. Please use the following soft link commands to set up the GeoNRW dataset:
```
ln -s /your/project/path/SynRS3D/data/GeoNRW/opt /your/project/path/SynRS3D/data/geonrw_rural/opt

ln -s /your/project/path/SynRS3D/data/GeoNRW/gt_nDSM /your/project/path/SynRS3D/data/geonrw_rural/gt_nDSM
```
```
ln -s /your/project/path/SynRS3D/data/GeoNRW/opt /your/project/path/SynRS3D/data/geonrw_urban/opt

ln -s /your/project/path/SynRS3D/data/GeoNRW/gt_nDSM /your/project/path/SynRS3D/data/geonrw_urban/gt_nDSM
```
  This will create the necessary symbolic links for the GeoNRW dataset setup.
OGC_ATL: Google Drive

Place all zip files under ./SynRS3D/data and extract them.
OGC_ARG: Google Drive

Place all zip files under ./SynRS3D/data and extract them.

Data Structure:

For real-world datasets, the data is expected in the following structure:

${DATASET_ROOT} # Dataset root directory, e.g., /home/username/project/SynRS3D/data/DFC19_JAX
├── opt              # RGB images saved as .tif
├── gt_nDSM          # Normalized Digital Surface Model images saved as .tif
├── gt_ss_mask       # Land cover mapping labels saved as .tif (some datasets like DFC18 do not include land cover mapping labels)
├── train.txt        # List of training data names without suffix
└── test.txt         # List of testing data names without suffix

For SynRS3D, it contains 17 folders. Download and extract all of them, ensuring each folder follows this structure:

${DATASET_ROOT} # Dataset root directory, e.g., /home/username/project/SynRS3D/data/grid_g05_mid_v1
├── opt             # RGB images saved as .tif, also post-event images in building change detection
├── pre_opt         # RGB images saved as .tif, also pre-event images in building change detection
├── gt_nDSM         # Normalized Digital Surface Model images saved as .tif
├── gt_ss_mask      # Land cover mapping labels saved as .tif
├── gt_cd_mask      # Building change detection mask saved as .tif (0 = no change, 255 = change area)
└── train.txt       # List of training data names

Class Mapping for `gt_ss_mask` in the SynRS3D dataset:

Bareland: 1
Rangeland: 2
Developed Space: 3
Road: 4
Trees: 5
Water: 6
Agriculture land: 7
Buildings: 8

🤖 Training

Due to restrictions on open-sourcing certain real-world datasets, it may not be possible to reproduce all the exact results from the original paper. However, to assist with reproducibility, we provide results from experiments conducted on redistributable real-world datasets. Below are the key findings.

Height Estimation Results

Best results based on the average of DFC18, DFC19_JAX, DFC19_OMA, geonrw_rural, geonrw_urban, OGC_ARG, and OGC_ATL datasets.

Model	MAE ↓ (Whole)	MAE ↓ (High)	RMSE ↓ (Whole)	RMSE ↓ (High)	F1^HE ↑ (δ < 1.25)	F1^HE ↑ (δ < 1.25^2)	F1^HE ↑ (δ < 1.25^3)
RS3DAda	2.534	4.961	4.122	6.242	0.467	0.605	0.672
Source-Only	3.103	6.087	5.040	7.552	0.408	0.550	0.628

Land Cover Mapping Results

Best results obtained on the OEM dataset.

Class	Bareland	Rangeland	Developed	Road	Tree	Water	Agriculture	Buildings	mIoU
RS3DAda	7.97%	46.34%	16.91%	44.23%	62.98%	70.37%	70.95%	63.02%	47.85%
Source-Only	8.69%	37.95%	22.54%	49.05%	60.16%	46.64%	35.40%	65.19%	40.70%

The figure below shows the evaluation results over training iterations.

Training Scripts

To train RS3DAda, you can use the script:
```
. RS3DAda.sh
```
For Source-Only training with SynRS3D, use:
```
. sourceonly.sh
```

You should expect results similar to those shown above.

Notes:

To evaluate land cover mapping with the OEM dataset, download the dataset from this link.
Organize its file structure similarly to SynRS3D and enable the --eval_oem flag in the script for evaluation.

💽 Inference Using Our Best Model

You can find pretrained RS3DAda checkpoints here. At the moment we provide the following model versions:

Model	Size
RS3DAda_vitl_DPT_height	366.54M Params
RS3DAda_vitl_DPT_segmentation	366.54M Params

Please download and place them in the folder ./SynRS3d/pretrain.

Infer Height Maps

To infer height maps using trained RS3DAda on the SynRS3D dataset:

python infer_height.py \
--data_dir path/to/your/image \
--restore_from ./pretrain/RS3DAda_vitl_DPT_height.pth \
--output_path path/to/your/output \
--use_tta

Infer Land Cover Maps

To infer land cover maps using trained RS3DAda on the SynRS3D dataset:

python infer_segmentation.py \
--data_dir path/to/your/image \
--restore_from ./pretrain/RS3DAda_vitl_DPT_segmentation.pth \
--output_path path/to/your/output \
--use_tta

📜Reference

If this code or dataset contributes to your research, please kindly consider citing our paper and give this repo ⭐️ :)

@article{song2024synrs3d,
  title={SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery},
  author={Song, Jian and Chen, Hongruixuan and Xuan, Weihao and Xia, Junshi and Yokoya, Naoto},
  journal={arXiv preprint arXiv:2406.18151},
  year={2024}
}

@inproceedings{song2024syntheworld,
  title={SyntheWorld: A Large-Scale Synthetic Dataset for Land Cover Mapping and Building Change Detection},
  author={Song, Jian and Chen, Hongruixuan and Yokoya, Naoto},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={8287--8296},
  year={2024}
}

🤝Acknowledgments

This project is based on DepthAnything (paper, code), DAFormer (paper, code), dsm2dtm repository (code). Thanks for their excellent works!!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery

🔭 Overview

SynRS3D: Largest Remote Sensing Synthetic Dataset

RS3DAda: UDA & DG Multi-Task Baseline Method in RS

🛎️Updates

✅ To-Do List

⚙️ Installation

💾 Datasets Preparation

Download the SynRS3D dataset:

Real-World Elevation Datasets

(Optional) Land Cover Mapping Real-World Dataset

Redistributable Datasets Download Links

Data Structure:

Class Mapping for `gt_ss_mask` in the SynRS3D dataset:

🤖 Training

Height Estimation Results

Land Cover Mapping Results

Training Scripts

Notes:

💽 Inference Using Our Best Model

Infer Height Maps

Infer Land Cover Maps

📜Reference

🤝Acknowledgments

Files

README.md

Latest commit

History

README.md

File metadata and controls

SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery

🔭 Overview

SynRS3D: Largest Remote Sensing Synthetic Dataset

RS3DAda: UDA & DG Multi-Task Baseline Method in RS

🛎️Updates

✅ To-Do List

⚙️ Installation

💾 Datasets Preparation

Download the SynRS3D dataset:

Real-World Elevation Datasets

(Optional) Land Cover Mapping Real-World Dataset

Redistributable Datasets Download Links

Data Structure:

Class Mapping for gt_ss_mask in the SynRS3D dataset:

🤖 Training

Height Estimation Results

Land Cover Mapping Results

Training Scripts

Notes:

💽 Inference Using Our Best Model

Infer Height Maps

Infer Land Cover Maps

📜Reference

🤝Acknowledgments

Class Mapping for `gt_ss_mask` in the SynRS3D dataset: