LCVN
SR
↑ Higher is betterSuccess rate
Fraction of routes that reach the goal.
Language-conditioned visual navigation
NeurIPS 2026 Oral
LCVN
Language-Conditioned World Modeling
for Visual Navigation
1University of Washington2National University of Singapore3Drexel University4Clemson University5Microsoft Research
Recorded scenes from the LCVN dataset
Overview
LCVN studies action-sequence prediction from one initial egocentric image and a language instruction, without a goal image or intermediate environmental feedback. The proposed agents also imagine future observations.
How the 39,016 trajectories are split
Share of all trajectories · counts reported in the paper
From routes to language
HUMAN-VERIFIED ANNOTATIONS
39,016trajectories117,048instructions
THREE STYLES PER TRAJECTORY
Concise · Intricate · Landmark-based
Train and both validation splits are publicly available. The test split is held out.
Browse dataset files ↗One trajectory, three instruction styles
Walk straight down the hallway, turn left at the corner, and stop near the end of the corridor.
Original annotation · validation seen · same indoor episode as below
ATE
↓ Lower is betterAbsolute trajectory error
Error over the predicted action trajectory.
RPE
↓ Lower is betterRelative pose error
Error in relative motion along the route.
SSIM
↑ Higher is betterStructural similarity
Structural fidelity to recorded observations.
PSNR
↑ Higher is betterPeak signal-to-noise ratio
Pixel reconstruction quality, measured in dB.
LPIPS
↓ Lower is betterPerceptual distance
Learned perceptual distance to the recorded image.
DreamSim
↓ Lower is betterDreamSim distance
Perceptual distance between predicted and recorded views.
How the instructions are builtFrom recorded routes to verified language
- Prepare trajectories
Normalize motion and segment routes from five source datasets.
- Generate three styles
Human seed examples guide Qwen-3.5-9B using frames, a 2D path and metadata.
- Verify and revise
Human review checks route alignment and the three instruction styles.
Frames and maps support annotation; navigation inference receives only the initial image and instruction.
Appendix A · Figure A1 ↗Explore a navigation episode
Start with one image and a language instruction. Reveal the recorded future observations and position samples to explore a dataset trajectory.
Initial observation
Drag to compare
The only camera view available to the agentRecorded data
Paper Figure 1
Language-conditioned visual navigation
Provided to the agent
Initial observation os and language instruction I. The goal is specified through language.
Predicted over the route
An action sequence of planar displacements and yaw, ending with a null action. The proposed agents additionally predict future observations.
Two modeling paradigms
Learn to act inside an imagined world, or predict actions and observations with one shared model.
LCVN-WM + LCVN-AC
Learn a navigation policy through future states imagined by a world model.
Figure 2 · Original figure from the paper · Click to enlarge
Algorithm & inference procedure
LCVN-WM predicts future visual latents conditioned on actions and language. LCVN-AC learns a navigation policy inside the frozen world model, using expert-plan alignment and intrinsic rewards. At inference, the policy selects actions from imagined states and the instruction.
Read the full algorithm and pseudocode in the paper ↗LCVN-Uni
Predict the next action and observation together in one shared sequence.
Figure 3 · Original figure from the paper · Click to enlarge
Algorithm & inference procedure
Actions, instructions and observations are tokenized into one shared sequence. Starting with the initial observation and instruction, LCVN-Uni predicts the next action and observation autoregressively, feeding predicted observations into subsequent steps until the null action.
Read the full algorithm and pseudocode in the paper ↗Leaderboard
Five planning metrics alongside visual prediction quality. Compare models across seen, unseen and held-out environments.
FirstSecondThird
Ranked per metric within this split; ties share a rank. — = not reported.
Metrics & evaluation protocol
Planning follows the five metrics defined by WorldNav. Full-route fidelity and heading and turn fidelity are not reported in the paper. The challenge’s composite Score requires per-episode results and cannot be reconstructed from these aggregates.
Visual prediction uses ground-truth actions. Single-frame and eighth-step rollout scores are separate settings, reported for validation seen only. Navigation and imagination are separate evaluations; pairing them here does not imply a joint rollout measurement.
WorldNav Challenge
RoboWorld 2026 · Track 1
We organize WorldNav as Track 1 of RoboWorld 2026, built on the LCVN benchmark and associated with the RoboPAD Workshop at NeurIPS 2026. The workshop takes place in Paris, France, on December 12, 2026.
Given one initial egocentric RGB image and a language instruction, participants build agents that generate a complete navigation trajectory without a goal image or intermediate environmental feedback. World-model, vision-language-action (VLA), and policy-only approaches are welcome.
Submit predictions through CodaBench for evaluation on held-out test trajectories. Top five teams receive official certificates, and selected top-performing teams will be invited to present their methods at the workshop. Awards are scheduled to be announced on December 12.
Scenes from the public LCVN dataset
Imagined futures
Qualitative comparisons of predicted observations on validation seen.
Citation
@inproceedings{dong2026language, title={Language-Conditioned World Modeling for Visual Navigation}, author={Dong, Yifei and Wu, Fengyi and Dai, Yilong and Kong, Lingdong and Chen, Guangyu and Sha, Yetong and Hu, Qiyu and Liu, Feng and Huang, Siyu and Dai, Qi and Cheng, Zhi-Qi}, booktitle={Advances in Neural Information Processing Systems}, year={2026}, url={https://arxiv.org/abs/2603.26741} }
Acknowledgement
LCVN builds on trajectories from five public navigation datasets: Go Stanford, ReCon, SCAND, HuRoN and TartanDrive. We thank their authors for collecting and releasing this data.