Reasoning in Space via Grounding in the World

Sota on VSI-Bench 🚀
Yiming Chen1,2,3, Zekun Qi4, Wenyao Zhang5,6, Xin Jin6, Li Zhang2,7*, Peidong Liu1*,
1Westlake University 2Shanghai Innovation Institute 3Zhejiang University 4Tsinghua University 5Shanghai Jiao Tong University 6Eastern Institute of Technology 7Fudan University
arXiv Code Huggingface
0:00 / 0:00
The video is captured in the wild without sensory 3D inputs, highlighting the strong generalization capability of our model.

Abstract

In this paper, we claim that 3D visual grounding is the cornerstone of spatial reasoning and introduce the Grounded-Spatial Reasoner (GS-Reasoner) to explore the effective spatial representations that bridge the gap between them. Existing 3D LLMs suffer from the absence of a unified 3D representation capable of jointly capturing semantic and geometric information. This deficiency is manifested either in poor performance on grounding or in an excessive reliance on external modules, ultimately hindering the seamless integration of grounding and spatial reasoning. To address this, we propose a simple yet effective dual-path pooling mechanism that tightly aligns geometric features with both semantic and positional cues, constructing a unified image patch-based 3D representation that encapsulates all essential information without increasing the number of input tokens. Leveraging this holistic representation, GS-Reasoner is the first 3D LLM that achieves autoregressive grounding entirely without external modules while delivering performance comparable to state-of-the-art models, establishing a unified and self-contained framework for 3D spatial reasoning. To further bridge grounding and spatial reasoning, we introduce the Grounded Chain-of-Thought (GCoT) dataset. This dataset is meticulously curated to include both 3D bounding box annotations for objects referenced in reasoning questions and step-by-step reasoning paths that integrate grounding as a core component of the problem-solving process. Extensive experiments demonstrate that GS-Reasoner achieves impressive results on 3D visual grounding, which in turn significantly enhances its spatial reasoning capabilities, leading to state-of-the-art performance.

Method Overview

Method Pipeline
Overview of GS-Reasoner framework. Our method builds a semantic-geometric hybrid 3D scene representation, enabling 3D LLM to perform 3D visual grounding autoregressively, which allows grounding to be integrated as a chain-of-thought within the spatial reasoning process.

Dataset Overview

Dataset Pipeline
Overview of Grounded Chain-of-Thought (GCoT) Dataset. We first construct spatial QA pairs without CoT, and then prompt GPT-4o to generate CoT paths based on the bird's-eye view, object information, and QA pairs.

Performance

Spatial Reasoning (VSI-Bench)

Spatial Reasoning Table
Evaluation on VSI-Bench. GS-Reasoner achieves state-of-the-art performance on most tasks, with further gains using more accurate (ground-truth) depth.

3D Visual Grounding

3D Visual Grounding Table
Evaluation on 3D Visual Grounding. GS-Reasoner achieves performance comparable to 3D LLMs using mesh proposals or external grounding, without any external components.

General 3D Tasks

General 3D Tasks Table
Evaluation on General 3D Tasks. GS-Reasoner outperforms state-of-the-art 3D LLMs on Scan2Cap and achieves comparable results on ScanQA and SQA3D.

BibTeX

@misc{chen2025reasoningspacegroundingworld
      title={Reasoning in Space via Grounding in the World}, 
      author={Yiming Chen and Zekun Qi and Wenyao Zhang and Xin Jin and Li Zhang and Peidong Liu},
      year={2025},
      eprint={2510.13800},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.13800}, 
}