Spatial-IQ Analyses

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. Licensed under the BSD-3-Clause License.

Analysis code for the paper Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests.

🌐 Project Page →  |  📊 Dataset (HuggingFace) →

The analysis code is under analyses/; its main entry point is analyses/spatial-IQ_analyses.py. It reads model inference-result CSVs and generates the paper-aligned analysis tables, statistics, and figures used for the manuscript and appendix. The benchmark task prompts are provided under inference/.

What This Script Does

The analysis pipeline has three stages:

1. Discover raw model-result CSVs in analyses/inference_results/.
2. Export analysis-ready CSV tables with paper figure/table identifiers.
3. Render paper figures from those exported CSV tables.

This table-first design is meant for appendix review: each generated figure has a corresponding CSV with counts, uncertainty estimates, and statistical annotations where applicable.

Repository Layout

Spatial-IQ/
├── analyses/                       # Paper-aligned analysis and figure code
│   ├── spatial-IQ_analyses.py      # Main entry point
│   ├── inference_results/          # Raw eval_raw_*.csv inputs
│   ├── analyses_results/           # Generated paper tables, figures, and manifest
│   └── requirements.txt            # Analysis dependencies
├── inference/                      # Benchmark task prompts (paper Appendix A)
│   └── prompt_*_Type1.json         # Text / MCQ / image prompt specifications
├── docs/                           # Sphinx source for the project page (GitHub Pages)
│   ├── conf.py
│   ├── index.md                    # Project landing page
│   ├── _static/                    # Custom CSS and figures for the page
│   ├── make_demo.py                # Regenerates _static/figures/demo.gif
│   ├── demo_gt/                    # Ground-truth scene inputs to make_demo.py
│   └── demo_gemini/                # Gemini prediction inputs to make_demo.py
├── pyproject.toml                  # uv-managed docs build dependencies
├── uv.lock
├── .github/workflows/pages.yml     # GitHub Pages auto-deploy workflow
├── LICENSE                         # BSD-3-Clause license
├── THIRD_PARTY_LICENSES.md         # Third-party dependency notices
└── README.md

Project Page

The Spatial-IQ project page is live at nvidia.github.io/Spatial-IQ.

It is built with Sphinx and MyST-Markdown from the docs/ folder, and deployed automatically to GitHub Pages via .github/workflows/pages.yml on every push to main.

To build the page locally:

# Install docs build dependencies (uv-managed)
uv sync

# Build to docs/_build/html/
uv run sphinx-build -b html docs docs/_build/html

# Serve locally
uv run python -m http.server -d docs/_build/html 8765
# then visit http://127.0.0.1:8765/

The page copy lives in docs/index.md; shared figures under docs/_static/figures/; custom CSS in docs/_static/spatial_iq.css. To regenerate the animated demo GIF, run uv run python docs/make_demo.py.

Dataset

The Spatial-IQ benchmark dataset is available on Hugging Face:

https://huggingface.co/datasets/patrickqrim/spatial-iq

Quick Start

All analysis code lives under analyses/. Install its dependencies and run from that directory:

cd analyses
pip install -r requirements.txt        # pandas, numpy, scipy, matplotlib
python spatial-IQ_analyses.py --stage all

Generated files are written to analyses/analyses_results/ by default:

analyses_results/
├── paper_output_manifest.csv
├── paper_output_manifest.json
├── tables/
└── figures/

Using New Model Outputs

This repository does not run model inference by itself. It assumes that inference has already been run on the Spatial-IQ benchmark examples and that the results have been saved in the CSV structure described below.

To evaluate a new model, use the Spatial-IQ dataset at https://huggingface.co/datasets/patrickqrim/spatial-iq, ask the model the paper prompts for each task, score the model responses against the provided ground truth, and save the scored results as eval_raw_*.csv files in analyses/inference_results/.

The paper prompt specification is in Appendix A, “Prompts, Definitions, and Task Specifications.” The appendix describes the shared prompt structure, concept definitions, modality-specific wording, and wrong-answer taxonomy. The machine-readable prompts are provided under inference/ (prompt_text_Type1.json, prompt_mcq_Type1.json, prompt_mcq3options_Type1.json, prompt_mcq4options_Type1.json, prompt_image_Type1.json). The task-specific query text is also summarized in Table 5 of the paper and exported by this script as analyses/analyses_results/tables/table_05_task_queries.csv.

Once a model's inference results are converted to the expected CSV format, the same analysis command can be used for any model:

python spatial-IQ_analyses.py --stage all

CLI Options

python spatial-IQ_analyses.py \
  --input-dir inference_results \
  --output-dir analyses_results \
  --stage all \
  --groups text mcq5 mcq345 image trained_text

--stage can be all, analysis, figures, or manifest.

--groups controls which modalities are processed. Supported groups are text, mcq5, mcq345, image, and trained_text.

Input CSV Discovery

Model lists are not hardcoded. The script discovers inputs by filename pattern:

Group	Expected inputs
text	eval_raw_*_text.csv and eval_raw_human_frq.csv, excluding trained checkpoints
mcq5	eval_raw_*_mcq.csv, excluding 3CQ and 4CQ variants
mcq345	all eval_raw_*_mcq.csv files
image	eval_raw_image_*.csv files
trained_text	trained checkpoint text CSVs, including names such as dapo*, sft-*, and Qwen2.5-VL*

Adding a new model usually only requires placing the matching CSV in analyses/inference_results/ and rerunning the pipeline.

Input CSV Structure

Each CSV should contain one row per evaluated scene/request. Task correctness columns should be numeric binary values, where 1 means correct and 0 means incorrect.

Core task columns:

Column	Meaning
task1 ... task11	Binary correctness for each Spatial-IQ sub-task/target column used by the raw evaluation code.
main	Binary correctness for Object Counting. Image-output CSVs may use task_main; the script renames it to main.

Recommended metadata columns for reproducing all paper analyses:

Column	Meaning
view	Scene/view identifier. Camera metadata and within-scene analyses are derived from this field when present.
object_type	Object appearance label, such as cube, factory_box, soup_can, or spam.
total_blocks	Total number of objects in the scene.
num_hidden_blocks	Number of hidden objects.
num_columns	Number of object columns.
num_layers	Number of vertical layers.

Optional modality-specific columns:

Columns	Used for
task*_pred, task*_gt, main_pred, main_gt	Text/trained-text numerical composition diagnostics, including task5_pred + task8_pred versus main_pred.
task*_choice_type, main_choice_type	MCQ wrong-answer preference tables and heatmaps. Use correct for correct answers and a stable label for each wrong-answer reason.
task*_verdict, task_main_verdict	Optional image-output scoring labels; binary task* columns are still the values used by the analysis script.

Example text CSV header:

view,object_type,total_blocks,num_hidden_blocks,num_columns,num_layers,task1,task1_pred,task1_gt,...,task11,task11_pred,task11_gt,main,main_pred,main_gt

Example MCQ CSV header:

view,object_type,total_blocks,num_hidden_blocks,num_columns,num_layers,task1,task1_choice_type,...,task11,task11_choice_type,main,main_choice_type

Output Conventions

The manifest files list every paper figure and table, including static figures that are not generated by this script.

Generated tables are saved in analyses_results/tables/. These CSVs are the statistical source for the generated figures and include fields such as raw counts, accuracy, binomial standard error, Wilson confidence intervals, chance-adjusted metrics for MCQ settings, p-values, and figure-display flags where relevant.

Generated figures are saved in analyses_results/figures/. Figure filenames use the paper numbering, for example figure_04_text_task_accuracy.png.

Task Numbering

Raw CSV columns keep their original development names, such as task2 or task9. Generated tables also include paper-facing IDs such as Task2 or Target1.

When both are present:

Column	Meaning
source_task	Original raw CSV task column
task	Paper-facing task or target ID
task_name	Manuscript task name

This avoids breaking compatibility with existing raw outputs while keeping tables aligned with the paper.

Code Organization

spatial-IQ_analyses.py is the recommended reproducibility script for the paper figures and tables.

The script uses a non-interactive Matplotlib backend, so it can run in local terminals, CI jobs, and server environments without opening windows.

License

This project is released under the BSD-3-Clause license. See LICENSE for the full license text.

Each source file in this repository carries an SPDX header (SPDX-License-Identifier: BSD-3-Clause) identifying it as NVIDIA-authored code licensed under BSD-3-Clause.

Third-Party Notices

This project depends on third-party open-source packages (pandas, numpy, scipy, matplotlib) installed by the end user via requirements.txt. The corresponding upstream license notices are reproduced in THIRD_PARTY_LICENSES.md for attribution. No third-party source code is redistributed inside this repository.

Contributing

This repository is a reproducibility artifact for the paper and does not accept external contributions. Please fork the repository and cite the paper for any derivative work.

Citation

If you use this code or the Spatial-IQ benchmark in your research, please cite the paper:

@article{spatialiq2026,
  title  = {Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests},
  author = {<author list>},
  year   = {2026},
  note   = {<venue / arXiv id>}
}