REVIEW 1 major objections 1 minor 38 references
A benchmark of 4,331 instances shows vision-language models remain far below human performance on spatial tasks for low-altitude UAVs.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-07-01 06:48 UTC pith:KZVSRESM
load-bearing objection SpatialUAV releases a new benchmark covering multi-view UAV collaboration and motion tasks that prior datasets skipped, with model evaluations showing clear gaps. the 1 major comments →
SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes SpatialUAV, a real low-altitude UAV benchmark with 4,331 curated instances across 14 fine-grained task types supporting multiple input configurations and answer formats. Through systematic evaluation, it finds that current vision-language models are far from human-level performance with clear weaknesses in cross-view association, structured grounding, geometric reasoning, and temporal viewpoint understanding.
What carries the argument
The SpatialUAV benchmark with its unified visual-input-question-answer schema, supported by a construction pipeline that uses detector-assisted regions, depth supervision, metadata rules, and multi-turn human validation to produce reliable heterogeneous outputs.
Load-bearing premise
The 14 task formulations and 4,331 instances accurately capture the spatial intelligence needed for real low-altitude UAV perception and collaboration without selection bias or annotation artifacts.
What would settle it
A controlled evaluation in which any vision-language model reaches or exceeds human-level scores across all 14 tasks on the SpatialUAV benchmark would falsify the reported performance gaps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SpatialUAV, a real low-altitude UAV benchmark with 4,331 curated instances across 14 task types covering semantic discrimination, spatial relations, aerial-aerial and aerial-ground collaboration, and motion understanding. It uses a unified visual-input-question-answer schema supporting seven input configurations and nine answer formats, constructed via detector-assisted regions, depth supervision, metadata rules, manual annotation, blind filtering, and multi-turn human validation. Evaluations of representative vision-language models show they remain far from human-level performance, with bottlenecks in cross-view association, structured grounding, geometric reasoning, and temporal viewpoint understanding.
Significance. If the benchmark construction is robust, the work supplies a valuable, publicly released resource that exposes concrete gaps in current models for UAV spatial tasks and supplies empirical guidance for future model development in low-altitude perception and collaboration.
major comments (1)
- [Data Construction Pipeline] Data Construction Pipeline (abstract and §3): the pipeline is described at high level (detector-assisted regions, depth supervision, blind filtering, multi-turn validation). To substantiate that the 4,331 instances faithfully measure the intended spatial intelligence without selection bias or annotation artifacts, the manuscript must supply quantitative inter-annotator agreement statistics, explicit exclusion criteria, and precise definitions of the task-specific metrics used for the nine heterogeneous answer formats.
minor comments (1)
- [Abstract] Abstract: the phrase 'extensive manual annotation' is used without indicating the number of annotators or the exact validation protocol; this detail belongs in the main text even if summarized in the abstract.
Simulated Author's Rebuttal
We thank the referee for the positive evaluation and recommendation of minor revision. The single major comment concerns the level of detail in the data construction pipeline; we address it directly below and will incorporate the requested information in the revised manuscript.
read point-by-point responses
-
Referee: [Data Construction Pipeline] Data Construction Pipeline (abstract and §3): the pipeline is described at high level (detector-assisted regions, depth supervision, blind filtering, multi-turn validation). To substantiate that the 4,331 instances faithfully measure the intended spatial intelligence without selection bias or annotation artifacts, the manuscript must supply quantitative inter-annotator agreement statistics, explicit exclusion criteria, and precise definitions of the task-specific metrics used for the nine heterogeneous answer formats.
Authors: We agree that the current description in §3 is high-level and that quantitative evidence of annotation quality would strengthen the claim that the benchmark faithfully measures spatial intelligence. In the revision we will expand §3 with: (1) inter-annotator agreement statistics (Cohen’s κ and raw agreement percentages) computed on the multi-turn human validation subset; (2) the explicit exclusion criteria applied during blind filtering (e.g., minimum region size, depth consistency thresholds, and semantic ambiguity rules); and (3) precise, task-by-task definitions of the nine answer-format metrics, including how region-identifier, geometric-value, cross-view correspondence, and free-form motion-description outputs are scored. These additions will be accompanied by a short table summarizing the statistics and criteria. revision: yes
Circularity Check
No significant circularity; empirical benchmark with no derivations or fitted predictions
full rationale
The paper introduces SpatialUAV as a benchmark dataset and evaluates existing models on it. No equations, first-principles derivations, parameter fitting, or 'predictions' appear in the abstract or described pipeline. Construction uses detector-assisted regions, depth supervision, metadata rules, manual annotation, blind filtering, and multi-turn validation, but these are procedural safeguards rather than self-referential definitions or fitted inputs renamed as outputs. No self-citation chains, uniqueness theorems, or ansatzes are invoked. The central claim (models lag human performance on spatial tasks) rests on external model evaluations against the released data, not on any internal reduction. This matches the expected non-finding for benchmark releases.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Human annotators following the described multi-turn validation protocol produce consistent and accurate labels for spatial relations and motion descriptions.
- domain assumption The selected 14 task types and input configurations collectively capture the spatial intelligence requirements of low-altitude UAV perception and collaboration.
read the original abstract
Spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation. However, existing UAV benchmarks often emphasize image-level recognition, single-view understanding, or narrow answer formats, leaving 3D spatial inference, multi-view collaboration, scene dynamics, and diverse task formulations insufficiently evaluated. To address these gaps, we introduce SpatialUAV, a real low-altitude UAV benchmark comprising 4,331 curated instances across 14 fine-grained task types, covering semantic discrimination, spatial relation, aerial--aerial collaboration, aerial--ground collaboration, and motion understanding. SpatialUAV organizes all samples into a unified visual-input--question--answer schema, while supporting seven input configurations and nine answer formats, including option labels, region identifiers, geometric values, cross-view correspondences, and free-form motion descriptions. To ensure reliable and grounded evaluation, our data construction pipeline integrates detector-assisted regions, depth supervision, metadata-derived rules, extensive manual annotation, blind filtering, and multi-turn human validation, together with task-specific metrics for heterogeneous outputs. Evaluating representative vision-language models across three categories, we show that current models remain far from human-level performance, with pronounced bottlenecks in cross-view association, structured grounding, geometric reasoning, and temporal viewpoint understanding. These results offer empirical guidance for advancing low-altitude UAV spatial intelligence. Code and data are available at https://github.com/Hyu-Zhang/SpatialUAV.
Figures
Reference graph
Works this paper leans on
-
[1]
Long-short match for lost control in uav multi-object tracking,
Z. Zou, M. Ye, L. Ji, L. Zhou, S. Tang, Y . Gan, and S. Li, “Long-short match for lost control in uav multi-object tracking,”IEEE Transactions on Multimedia, vol. 28, pp. 786–800, 2026
2026
-
[2]
Fre-stformer: A frequency-based spatio-temporal transformer for uav human action recognition,
T. Xiang, X. Xia, J. Yuan, and Z. Tu, “Fre-stformer: A frequency-based spatio-temporal transformer for uav human action recognition,”IEEE Transactions on Multimedia, pp. 1–13, 2026
2026
-
[3]
Mode-track: Robust multi-object tracking with motion decoupling in uav videos,
Z. Song, Y . Li, S. Zhou, W. Tang, and L. Wang, “Mode-track: Robust multi-object tracking with motion decoupling in uav videos,”IEEE Transactions on Multimedia, pp. 1–11, 2026
2026
-
[4]
A survey of robotic language grounding: tradeoffs between symbols and embeddings,
V . Cohen, J. X. Liu, R. Mooney, S. Tellex, and D. Watkins, “A survey of robotic language grounding: tradeoffs between symbols and embeddings,” inProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024, pp. 7999–8009
2024
-
[5]
Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai,
T. Wang, X. Mao, C. Zhu, R. Xu, R. Lyu, P. Li, X. Chen, W. Zhang, K. Chen, T. Xueet al., “Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19 757–19 767
2024
-
[6]
Egothink: Evaluating first-person perspective thinking capability of vision-language models,
S. Cheng, Z. Guo, J. Wu, K. Fang, P. Li, H. Liu, and Y . Liu, “Egothink: Evaluating first-person perspective thinking capability of vision-language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 291–14 302
2024
-
[7]
Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,
B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 455–14 465
2024
-
[8]
Thinking in space: How multimodal large language models see, remember, and recall spaces,
J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in space: How multimodal large language models see, remember, and recall spaces,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 10 632–10 643
2025
-
[9]
Urbanvideo-bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces,
B. Zhao, J. Fang, Z. Dai, Z. Wang, J. Zha, W. Zhang, C. Gao, Y . Wang, J. Cui, X. Chenet al., “Urbanvideo-bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 32 400– 32 423
2025
-
[10]
Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space,
Y . Zhao, K. Xu, Z. Zhu, Y . Hu, Z. Zheng, Y . Chen, Y . Ji, C. Gao, Y . Li, and J. Huang, “Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 12 465–12 480
2025
-
[11]
Anti-uav: A large-scale benchmark for vision-based uav tracking,
N. Jiang, K. Wang, X. Peng, X. Yu, Q. Wang, J. Xing, G. Li, G. Guo, Q. Ye, J. Jiao, J. Zhao, and Z. Han, “Anti-uav: A large-scale benchmark for vision-based uav tracking,”IEEE Transactions on Multimedia, vol. 25, pp. 486–500, 2023
2023
-
[12]
Open3d-vqa: A benchmark for embodied spatial concept reasoning with multimodal large language model in open space,
W. Zhang, Z. Zhou, X. Zeng, L. Xuchen, J. Fang, C. Gao, J. Cui, Y . Li, X. Chen, and X.-P. Zhang, “Open3d-vqa: A benchmark for embodied spatial concept reasoning with multimodal large language model in open space,” inProceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 12 784–12 791
2025
-
[13]
L. Zhang, Y . Zhang, H. Li, H. Fu, Y . Tang, H. Ye, L. Chen, X. Liang, X. Hao, and W. Ding, “Is your vlm sky-ready? a comprehensive spatial intelligence benchmark for uav navigation,”arXiv preprint arXiv:2511.13269, 2025
-
[14]
Scanqa: 3d question answering for spatial scene understanding,
D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe, “Scanqa: 3d question answering for spatial scene understanding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 129–19 139
2022
-
[15]
Sqa3d: Situated question answering in 3d scenes,
X. Ma, S. Yong, Z. Zheng, Q. Li, Y . Liang, S.-C. Zhu, and S. Huang, “Sqa3d: Situated question answering in 3d scenes,” inInternational Conference on Learning Representations, 2023
2023
-
[16]
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
Y . Hong, J. Liu, H. Yin, M. Li, L. Guibas, F.-F. Li, J. Wu, and Y . Choi, “ESI-Bench: Towards embodied spatial intelligence that closes the perception–action loop,”arXiv preprint arXiv:2605.18746, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[17]
Aircopbench: A benchmark for multi-drone collaborative embodied perception and reasoning,
J. Zha, Y . Fan, T. Zhang, G. Chen, Y . Chen, C. Gao, and X. Chen, “Aircopbench: A benchmark for multi-drone collaborative embodied perception and reasoning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 2, 2026, pp. 1507–1515
2026
-
[18]
S. Dai, Z. Ma, Z. Luo, X. Yang, Y . Huang, W. Zhang, C. Chen, Z. Guo, W. Xu, Y . Sun, and M. Sun, “MM-UA VBench: How well do multimodal large language models see, think, and plan in low-altitude uav scenarios?” arXiv preprint arXiv:2512.23219, 2025
-
[19]
Y . Zhan and Y . Yuan, “Uavbench and uavit-1m: Benchmarking and enhancing mllms for low-altitude uav vision-language understanding,” arXiv preprint arXiv:2603.14336, 2026
-
[20]
D. Liu, J. Feng, D. Li, Y . Zheng, G. Li, W. Dong, and G. Shi, “Are vlms lost between sky and space? LinkS 2Bench for uav-satellite dynamic cross-view spatial intelligence,”arXiv preprint arXiv:2604.02020, 2026
-
[21]
Cambrian-s: Towards spatial supersensing in video,
S. Yang, J. Yang, P. Huang, E. L. Brown II, Z. Yang, Y . Yu, S. Tong, Z. Zheng, Y . Xu, M. Wanget al., “Cambrian-s: Towards spatial supersensing in video,” inThe Fourteenth International Conference on Learning Representations, 2026
2026
-
[22]
arXiv preprint arXiv:2512.10863 , year=
J. Lin, R. Xu, S. Zhu, S. Yang, P. Cao, Y . Ran, M. Hu, C. Zhu, Y . Xie, Y . Long, W. Hu, D. Lin, T. Wang, and J. Pang, “Mmsi-video-bench: A holistic benchmark for video-based spatial intelligence,”arXiv preprint arXiv:2512.10863, 2025
-
[23]
C. Gao, B. Zhao, W. Zhang, J. Mao, J. Zhang, Z. Zheng, F. Man, J. Fang, Z. Zhou, J. Cui, X. Chen, and Y . Li, “Embodiedcity: A benchmark platform for embodied agent in real-world city environment,”arXiv preprint arXiv:2410.09604, 2024
-
[24]
Real-time and accurate uav pedestrian detection for social distancing monitoring in covid-19 pandemic,
Z. Shao, G. Cheng, J. Ma, Z. Wang, J. Wang, and D. Li, “Real-time and accurate uav pedestrian detection for social distancing monitoring in covid-19 pandemic,”IEEE Transactions on Multimedia, vol. 24, pp. 2069–2083, 2022
2069
-
[25]
M. A. Ferrag, A. Lakas, and M. Debbah, “UA VBench: An open benchmark dataset for autonomous and agentic ai uav systems via llm- generated flight scenarios,”arXiv preprint arXiv:2511.11252, 2025
-
[26]
Bedi: A comprehensive benchmark for evaluating embodied agents on uavs,
M. Guo, M. Wu, J. He, S. Li, H. Li, and C. Tao, “Bedi: A comprehensive benchmark for evaluating embodied agents on uavs,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 232, pp. 910–936, 2026
2026
-
[27]
Multiview aerial visual recognition (mavrec): Can multi-view improve aerial visual perception?
A. Dutta, S. Das, J. Nielsen, R. Chakraborty, and M. Shah, “Multiview aerial visual recognition (mavrec): Can multi-view improve aerial visual perception?”arXiv preprint arXiv:2312.04548, 2023
-
[28]
Airscape: An aerial generative world model with motion controllability,
B. Zhao, R. Tang, M. Jia, Z. Wang, F. Man, X. Zhang, Y . Shang, W. Zhang, W. Wu, C. Gao, X. Chen, and Y . Li, “Airscape: An aerial generative world model with motion controllability,”arXiv preprint arXiv:2507.08885, 2025
-
[29]
University-1652: A multi-view multi- source benchmark for drone-based geo-localization,
Z. Zheng, Y . Wei, and Y . Yang, “University-1652: A multi-view multi- source benchmark for drone-based geo-localization,” inProceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 1395–1403
2020
-
[30]
Metric3d: Towards zero-shot metric 3d prediction from a single image,
W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen, and C. Shen, “Metric3d: Towards zero-shot metric 3d prediction from a single image,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 9043–9053
2023
-
[31]
S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Geet al., “Qwen3-vl technical report,”arXiv preprint arXiv:2511.21631, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[32]
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shaoet al., “Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,”arXiv preprint arXiv:2508.18265, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[33]
Scaling spatial intelligence with multimodal foundation models,
Z. Cai, R. Wang, C. Gu, F. Pu, J. Xu, Y . Wang, W. Yin, Z. Yang, C. Wei, T. Zhouet al., “Scaling spatial intelligence with multimodal foundation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 7879–7890
2026
-
[34]
Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence,
D. Wu, F. Liu, Y .-H. Hung, and Y . Duan, “Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence,”Advances in neural information processing systems, vol. 38, pp. 13 569–13 597, 2026
2026
-
[35]
Visual spatial tuning.arXiv preprint arXiv:2511.05491,
R. Yang, Z. Zhu, Y . Li, J. Huang, S. Yan, S. Zhou, Z. Liu, X. Li, S. Li, W. Wanget al., “Visual spatial tuning,”arXiv preprint arXiv:2511.05491, 2025
-
[36]
Spatial understanding from videos: Structured prompts meet simulation data,
H. Zhang, M. Liu, Z. Li, H. Wen, W. Guan, Y . Wang, and L. Nie, “Spatial understanding from videos: Structured prompts meet simulation data,”Advances in Neural Information Processing Systems, vol. 38, pp. 103 202–103 229, 2026
2026
-
[37]
Spaceera++: A unified framework towards 3d spatial reasoning in video,
W. Guan, H. Zhang, M. Liu, Q. Xiang, Y . Wang, and L. Nie, “Spaceera++: A unified framework towards 3d spatial reasoning in video,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026
2026
-
[38]
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.