SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation

Aman Chadha; Amitava Das; Niyati Rawal; Saksham Jain; Shah Alam Abir; Suranjana Trivedy; Sushant Ravva; Vinija Jain

Current vision-language models produce somewhat coherent trajectories in 3D scenes but fail systematically under occlusion, interaction constraints, and multi-step instructions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 22:48 UTC pith:GYYL47OI

load-bearing objection SleepWalk introduces a tiered single-scene benchmark for VLM trajectory prediction that shows expected difficulty scaling, but the pointwise judge protocol needs closer inspection for global coherence. the 1 major comments →

arxiv 2605.10376 v2 pith:GYYL47OI submitted 2026-05-11 cs.CV

SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation

Niyati Rawal , Sushant Ravva , Shah Alam Abir , Saksham Jain , Aman Chadha , Vinija Jain , Suranjana Trivedy , Amitava Das This is my paper

classification cs.CV

keywords vision-language navigationVLMs3D scene understandingspatial reasoningembodied planningbenchmark evaluationtrajectory prediction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SleepWalk as a benchmark that generates navigable 3D scenes from text and organizes navigation tasks into three tiers of increasing spatial and temporal difficulty. Models receive rendered views plus a natural-language instruction and must output a trajectory that respects geometry, avoids collisions, and ends at an action-compatible spot. Evaluation uses a standardized pointwise judge protocol across 2,472 scenes and nine instructions each. Results show clear performance drops as difficulty rises, exposing limits in grounded spatial reasoning while noting that some trajectories still meet basic coherence, executability, and alignment criteria. The work targets localized interaction-centric reasoning rather than long-range room-to-room exploration.

Core claim

SleepWalk shows that frontier VLMs can generate trajectories that are simultaneously spatially coherent, plausibly executable, and aligned with intended actions in single-scene 3D worlds, yet they exhibit systematic failures in grounded spatial reasoning that grow worse under occlusion, interaction constraints, and multi-step instructions.

What carries the argument

Three-tier benchmark of localized, interaction-centric tasks in text-generated navigable 3D scenes, scored by pointwise judge-based evaluation of trajectory coherence and action compatibility.

Load-bearing premise

The pointwise judge protocol and navigability filter on text-generated scenes give reliable, unbiased measures of whether a predicted path respects geometry and action constraints.

What would settle it

A controlled test in which models maintain high success rates on the hardest tier across occluded and multi-step instructions without any architectural change to spatial reasoning modules would falsify the reported systematic failures.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Performance on SleepWalk declines with each added layer of compositional complexity, indicating that current models lack robust mechanisms for maintaining spatial consistency across multiple reasoning steps.
Failures concentrate around occlusion and interaction constraints, suggesting that models trained primarily on 2D image-text pairs transfer poorly to 3D geometry-aware planning.
The benchmark isolates localized embodied reasoning from long-range exploration, allowing targeted diagnosis of where language grounding breaks in action sequences.
By providing a scalable, controlled testbed, the work supplies a measurable signal for iterative improvement of vision-language-action pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the observed drop with difficulty holds across larger models, training regimes may need explicit 3D geometry supervision or simulation-based reinforcement rather than scale alone.
The single-scene focus could be extended to test whether failures persist when models must chain decisions across changing viewpoints or partial observability.
Success on easier tiers but failure on harder ones implies that current VLMs handle local visual grounding better than compositional planning over time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

SleepWalk introduces a tiered single-scene benchmark for VLM trajectory prediction that shows expected difficulty scaling, but the pointwise judge protocol needs closer inspection for global coherence.

read the letter

The core takeaway is that this paper adds SleepWalk, a benchmark with three difficulty tiers for testing how VLMs map natural language instructions to trajectories inside single generated 3D scenes. It reports performance drops on three frontier models as occlusion, interaction constraints, and multi-step instructions increase.

What is new is the deliberate focus on localized, interaction-centric tasks rather than the usual long-range multi-room exploration. Generating scenes from text descriptions, applying a navigability filter, and running nine instructions per scene across 2472 environments gives a controlled way to measure grounding. The tier structure supports finer analysis of where spatial reasoning breaks.

The work is straightforward in its setup and the reported patterns align with known VLM weaknesses in embodied settings. That makes the failure modes under higher tiers a useful data point for the field.

The soft spot is the evaluation protocol. A pointwise judge that scores waypoints independently can accept locally plausible points while missing whether the full interpolated path stays collision-free or respects occlusion and action constraints across the sequence. The navigability filter on text-generated scenes also risks making the tier progression partly reflect curation choices instead of pure model limits. Details on judge implementation and any global checks would help here.

This paper is for researchers building or testing VLMs for grounded navigation and embodied planning. Anyone working on benchmark design or spatial reasoning evaluation will get concrete failure examples from the tier breakdowns.

It deserves peer review. The benchmark framing is clear enough that referees can usefully pressure-test the judge and scene generation steps.

Referee Report

1 major / 0 minor

Summary. The paper introduces SleepWalk, a three-tier benchmark for stress-testing instruction-guided vision-language navigation in single-scene 3D environments generated from textual descriptions and filtered for navigability. Tasks require VLMs to predict trajectories from rendered observations and natural-language instructions that respect geometry, avoid collisions, and end at action-compatible locations. The benchmark evaluates three frontier VLMs across 2,472 scenes and nine instructions per scene using a pointwise judge-based protocol, reporting systematic performance drops with increasing difficulty (especially under occlusion, interaction constraints, and multi-step instructions) while concluding that current VLMs can somewhat produce spatially coherent and plausibly executable trajectories.

Significance. If the pointwise evaluation protocol reliably captures global trajectory properties, SleepWalk would supply a scalable, controlled testbed for localized embodied reasoning that complements existing long-range navigation benchmarks. The tiered design and focus on interaction-centric failures could usefully guide progress in grounded multimodal planning and action-capable agents.

major comments (1)

[Abstract] Abstract (evaluation protocol paragraph): The central claim that VLMs produce 'spatially coherent, plausibly executable' trajectories rests on the standardized pointwise judge-based evaluation protocol. If individual waypoints are scored independently without an integrated verification step (e.g., continuous collision detection along the interpolated path or explicit checks that the full sequence respects occlusion and interaction constraints), locally acceptable scores can still correspond to globally invalid trajectories. This is load-bearing for the reported performance drops across tiers and the overall assessment of model capabilities.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful analysis of our evaluation protocol and its implications for the central claims. We address the major comment below and commit to revisions that strengthen the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract (evaluation protocol paragraph): The central claim that VLMs produce 'spatially coherent, plausibly executable' trajectories rests on the standardized pointwise judge-based evaluation protocol. If individual waypoints are scored independently without an integrated verification step (e.g., continuous collision detection along the interpolated path or explicit checks that the full sequence respects occlusion and interaction constraints), locally acceptable scores can still correspond to globally invalid trajectories. This is load-bearing for the reported performance drops across tiers and the overall assessment of model capabilities.

Authors: We agree that the distinction between pointwise and global validity is important and that our current description does not fully address it. The protocol prompts the judge with the complete trajectory, instruction, and scene to evaluate waypoints in sequence context, but it does not include automated continuous collision detection or explicit global constraint checks. This is a genuine limitation of the reported results. In revision we will (1) qualify the abstract claim to specify that coherence and executability are assessed via the pointwise judge protocol, (2) add a dedicated paragraph in the methods section describing the judge prompt and its scope, and (3) include a small-scale human validation experiment measuring agreement between pointwise scores and global trajectory validity. These changes will make the performance-drop findings more precisely supported. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark paper with no derivation chain or fitted predictions

full rationale

The paper introduces a new benchmark (SleepWalk) for evaluating VLMs on instruction-guided trajectory prediction in 3D scenes. It organizes tasks into three tiers, generates scenes from text, applies navigability filtering, and uses a pointwise judge protocol to score model outputs on 2,472 environments. No mathematical derivations, first-principles predictions, parameter fitting, or self-referential definitions appear in the abstract or described structure. Central claims rest on external VLM evaluations rather than internal reductions. No self-citation load-bearing steps, ansatz smuggling, or renaming of known results are present. This matches the default case of an empirical benchmark whose observations are independent of its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are introduced; the contribution is an empirical benchmark and evaluation rather than a theoretical derivation.

pith-pipeline@v0.9.1-grok · 5828 in / 1158 out tokens · 19647 ms · 2026-06-30T22:48:21.962660+00:00 · methodology

0 comments

read the original abstract

Vision-Language Models (VLMs) have advanced rapidly in multimodal perception and language understanding, yet it remains unclear whether they can reliably ground language into spatially coherent, plausibly executable actions in 3D digital environments. We introduce SleepWalk, a benchmark for evaluating instruction-grounded trajectory prediction in single-scene 3D worlds generated from textual scene descriptions and filtered for navigability. Unlike prior navigation benchmarks centered on long-range exploration across rooms, SleepWalk targets localized, interaction-centric embodied reasoning: given rendered visual observations and a natural-language instruction, a model must predict a trajectory that respects scene geometry, avoids collisions, and terminates at an action-compatible location. The benchmark covers diverse indoor and outdoor environments and organizes tasks into three tiers of spatial and temporal difficulty, enabling fine-grained analysis of grounding under increasing compositional complexity. Using a standardized pointwise judge-based evaluation protocol, we evaluate three frontier VLMs on 2,472 curated 3D environments with nine instructions per scene. Results reveal systematic failures in grounded spatial reasoning, especially under occlusion, interaction constraints, and multi-step instructions: performance drops as the difficulty level of the tasks increase. In general, current VLMs can somewhat produce trajectories that are simultaneously spatially coherent, plausibly executable, and aligned with intended actions. By exposing failures in a controlled yet scalable setting, SleepWalk provides a critical benchmark for advancing grounded multimodal reasoning, embodied planning, vision-language navigation, and action-capable agents in 3D environments.

Figures

Figures reproduced from arXiv: 2605.10376 by Aman Chadha, Amitava Das, Niyati Rawal, Saksham Jain, Shah Alam Abir, Suranjana Trivedy, Sushant Ravva, Vinija Jain.

**Figure 1.** Figure 1: Overview of SleepWalk. A language instruction is converted into a single-scene 3D environment using Hunyuan3D-3.0. For each scene, given language instructions, different VLMs predict trajectories, which are visualized using top-down views. A fixed judge model scores trajectories in a pointwise manner, and rankings are aggregated across environments. clusion, respect environmental constraints, and generate … view at source ↗

**Figure 2.** Figure 2: SleepWalk pipeline. Starting from a natural-language scene description, we reconstruct a single-scene 3D environment with Hunyuan3D-3.0, render top-down and oblique observations, and use Qwen3-8B-VL to generate tiered navigation instructions (easy, medium, hard). Given the scene views and an instruction, a VLM predicts a continuous action trajectory, which is then evaluated by a judge model (GPT-5-mini) us… view at source ↗

**Figure 3.** Figure 3: Qualitative trajectory comparison across difficulty levels. Column (a) shows the reconstructed scene view, while columns (b), (c), and (d) show trajectory predictions from Qwen3-VL, Gemini Robotics ER-1.5, and GPT-5-mini, respectively. Across levels, the figure highlights differences in startlocation grounding, goal accuracy, obstacle avoidance, and robustness under compositional instructions. fails on b… view at source ↗

**Figure 4.** Figure 4: Factor-wise performance across difficulty tiers. Comparison of Qwen3-VL, Gemini Robotics ER-1.5, and GPT-5-mini on four evaluation factors—start location, goal location, obstacle avoidance, and trajectory efficiency—across easy, medium, and hard tiers. 3.2 Overall Model Ranking [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: From predicted path to humanoid execution. We animate the trajectory generated by GPT-5-mini for the instruction “Walk from the bookshelf to the wall-mounted lamp” and “Approach the yellow spherical object and then move to the northern tree” using TLControl (Wan et al., 2024) and MotionGPT (Jiang et al., 2024). This provides a qualitative check of whether a path that appears correct in top-down space rema… view at source ↗

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

14 extracted references

[1]

a single-scene, interaction-centric benchmark setting for grounded trajectory reasoning,
[2]

a scalable construction pipeline that reconstructs 3D scenes, generates tiered instructions, and evaluates predicted paths, and
[3]

Why this matters

an empirical diagnosis showing that current frontier VLMs degrade substantially on composi- tional, interaction-heavy, and multi-step instructions. Why this matters. The paper should not be read as claiming that any one component—for exam- ple, the judge or the scene generator—is the sole novelty. Rather, the novelty lies in defining and instantiating a b...
[4]

Start Location Consistency,
[5]

Obstacle Avoidance, and
[6]

Trajectory Efficiency. This means the evaluation is designed to reflect not only where a model ends up, but whether it gets there through a path that is spatially plausible and aligned with the intended action. What the paper does and does not claim. The manuscript does not claim that judge-based eval- uation is perfect. It makes the narrower methodologic...
[7]

the benchmark differentiates strong models,
[8]

the revealed failures are systematic rather than trivial, and
[9]

current VLMs are bad at navi- gation

performance degrades under higher compositional and interaction demands. What the results already show. Among the evaluated models, GPT-5-mini performs best across all four reported factors, but even the strongest system degrades as the tasks become harder. This supports the benchmark’s central diagnostic claim. Takeaway. Broader model coverage would cert...
[10]

Image 1: Oblique View (Perspective)
[11]

Image 2: Top-Down View (Map) ### Critical Pre-Condition Analyze the Top-Down View first. If the Top-Down view is incomplete, obstructed, or not clearly visible, you must ignore all other instructions and output ONLY this exact phrase:,→ ”The view is not clear to generate instructions” --- ### Phase 1: Environment Analysis If the views are clear, analyze b...
[12]

Object Grounding: - Every START and END location must refer to a specific, visible object found in the images.,→ - Correct: ”Start: Near the red chair” - Incorrect: ”Start: Near the wall” or ”Start: At the start point”
[13]

- BANNED WORDS: left, right, front, back, center, centre, middle, top, bottom, upper, lower.,→ - Tasks must be valid regardless of the agent 's facing direction

Forbidden Terminology (Spatial Hallucination): - NEVER use viewpoint-dependent or relative directional terms. - BANNED WORDS: left, right, front, back, center, centre, middle, top, bottom, upper, lower.,→ - Tasks must be valid regardless of the agent 's facing direction
[14]

Only reference items clearly visible in the provided images

Object Consistency: - Do not hallucinate objects. Only reference items clearly visible in the provided images. --- ### Output Format Step 1: Provide an *ENVIRONMENT SUMMARY* (2-3 sentences describing the room type and listing 10-15 key visible objects).,→ Step 2: Output the task list following this exact schema: LEVEL_1 | TASK: <instruction> | START: Near...

[1] [1]

a single-scene, interaction-centric benchmark setting for grounded trajectory reasoning,

[2] [2]

a scalable construction pipeline that reconstructs 3D scenes, generates tiered instructions, and evaluates predicted paths, and

[3] [3]

Why this matters

an empirical diagnosis showing that current frontier VLMs degrade substantially on composi- tional, interaction-heavy, and multi-step instructions. Why this matters. The paper should not be read as claiming that any one component—for exam- ple, the judge or the scene generator—is the sole novelty. Rather, the novelty lies in defining and instantiating a b...

[4] [4]

Start Location Consistency,

[5] [5]

Obstacle Avoidance, and

[6] [6]

Trajectory Efficiency. This means the evaluation is designed to reflect not only where a model ends up, but whether it gets there through a path that is spatially plausible and aligned with the intended action. What the paper does and does not claim. The manuscript does not claim that judge-based eval- uation is perfect. It makes the narrower methodologic...

[7] [7]

the benchmark differentiates strong models,

[8] [8]

the revealed failures are systematic rather than trivial, and

[9] [9]

current VLMs are bad at navi- gation

performance degrades under higher compositional and interaction demands. What the results already show. Among the evaluated models, GPT-5-mini performs best across all four reported factors, but even the strongest system degrades as the tasks become harder. This supports the benchmark’s central diagnostic claim. Takeaway. Broader model coverage would cert...

[10] [10]

Image 1: Oblique View (Perspective)

[11] [11]

Image 2: Top-Down View (Map) ### Critical Pre-Condition Analyze the Top-Down View first. If the Top-Down view is incomplete, obstructed, or not clearly visible, you must ignore all other instructions and output ONLY this exact phrase:,→ ”The view is not clear to generate instructions” --- ### Phase 1: Environment Analysis If the views are clear, analyze b...

[12] [12]

Object Grounding: - Every START and END location must refer to a specific, visible object found in the images.,→ - Correct: ”Start: Near the red chair” - Incorrect: ”Start: Near the wall” or ”Start: At the start point”

[13] [13]

- BANNED WORDS: left, right, front, back, center, centre, middle, top, bottom, upper, lower.,→ - Tasks must be valid regardless of the agent 's facing direction

Forbidden Terminology (Spatial Hallucination): - NEVER use viewpoint-dependent or relative directional terms. - BANNED WORDS: left, right, front, back, center, centre, middle, top, bottom, upper, lower.,→ - Tasks must be valid regardless of the agent 's facing direction

[14] [14]

Only reference items clearly visible in the provided images

Object Consistency: - Do not hallucinate objects. Only reference items clearly visible in the provided images. --- ### Output Format Step 1: Provide an *ENVIRONMENT SUMMARY* (2-3 sentences describing the room type and listing 10-15 key visible objects).,→ Step 2: Output the task list following this exact schema: LEVEL_1 | TASK: <instruction> | START: Near...