Abstract

Despite the remarkable progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, their application to complex 3D scene manipulation remains underexplored. In this paper, we bridge this critical gap by tackling three key challenges in 3D object arrangement task using MLLMs. First, to address the weak visual grounding of MLLMs, which struggle to link programmatic edits with precise 3D outcomes, we introduce an MCP-based API. This shifts the interaction from brittle raw code manipulation to more robust, function-level updates. Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools to analyze scene state, gather spatial information, and validate action outcomes. This perceptual feedback loop is critical for closing the gap between language-based updates and precise 3D-aware manipulation. Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework with designated roles for planning, execution, and verification. This decomposition allows the system to robustly handle multi-step instructions and recover from intermediate errors. We demonstrate the effectiveness of our approach on a diverse set of 25 complex object arrangement tasks, where it significantly outperforms existing baselines.

Interactive 3D Object Arrangement

Given an input scene and text prompt, our system iteratively transforms instructions into precise 3D actions. A team of Collaborative Agents leverages specialized 3D Scene Tools to manipulate objects step-by-step, producing a final arrangement that satisfies the user's request.

Method Overview

Our method solves a long-horizon task through an iterative multi-agent process. The Planner agent examines the global context (the user instruction and all previous rendered states) to formulate a concrete plan for the current movement. The Executor implements this single-step plan in the 3D scene using API tools and solvers. A set of Evaluators and an automatic floating check assess the execution quality. The entire Plan-Execute-Evaluate loop repeats until the Planner validates that the final arrangement fulfills the original user instruction.

Results

Our model performs complex 3D editing using either explicit per-step instructions (Top) or high-level general instructions (Bottom). For general inputs, the system automatically decomposes the main goal into the sequence of sub-tasks displayed below the corresponding images.

Per-Step Instruction Results

General Instruction Results

 

Bibtex

                        @misc{kuang2025vulcantoolaugmentedmultiagents,
                              title={VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement}, 
                              author={Zhengfei Kuang and Rui Lin and Long Zhao and Gordon Wetzstein and Saining Xie and Sanghyun Woo},
                              year={2025},
                              eprint={2512.22351},
                              archivePrefix={arXiv},
                              primaryClass={cs.CV},
                              url={https://arxiv.org/abs/2512.22351}, 
                        }