Transferring the Intelligence of VLMs to Robotic Control
RoboDawn gives a frozen vision-language model a small set of robot commands — move, rotate, grip — and a few demonstrations. The model then controls the robot in a closed loop, without task-specific robot training.
Humans adapt to both physical and digital worlds, suggesting that intelligence itself may transfer across the digital-to-real gap. Can the intelligence of vision-language models (VLMs) similarly generalize to robotic control? We study this with RoboDawn, a human-intuitive interface that exposes robot control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. The VLM observes the current visual state, reasons about the next action, executes it, and adapts to the result. An in-context learning scheme uses a few demonstrations to ground the model in both interface usage and task-solving strategy.
Without task-specific robot training, zero-shot RoboDawn already outperforms several strong policies trained on benchmark-specific robot data, and a single in-context demonstration yields state-of-the-art results. On RoboTwin 2.0 C2R, success rises from 53.2% zero-shot to 73.6% one-shot, above π0.5 (46.0%). On RoboDojo, it rises from 35.67% to 47.17%. The same framework transfers to real robots, performing block-in-basket and block stacking on Franka.
Figure 1. RoboDawn transfers general-purpose VLM intelligence to physical robot control through a human-intuitive action interface and in-context demonstrations.
Recorded episodes
Each replay pairs the recorded video with the model’s decisions and the commands it sent. Select an episode, then press play or jump to a chapter. Every evaluation episode and in-context demonstration is in the results browser.
GPT-6 Astra · RoboDojo · simulation recording
Align the blocks
Head camera
Use the ruler to align the three blocks.
Awaiting cameras
Left wristRight wrist
Right GIP — / — / — cmSteps 0 / 500
R
Agent traceGPT-6 Astra · recorded session
Loading recorded session…
– / –0:00 / —
step completedstep with a failed or rejected commandwidth = time the step takes in the replay
Guided replay: pauses are added for reading and recorded motion is slowed. Decision notes and tool calls are aligned to control steps; this is a replay, not a live model connection.
The VLM’s weights stay frozen. At every round it receives camera views, the measured robot state, feedback from the last commands, and its own memory, then returns a short plan and a batch of commands.
Figure 2. Overview of RoboDawn. The VLM is conditioned on (a) a semantic action interface and (b) in-context demonstrations. Each round it observes (c), reasons (d), emits commands (e) that are planned and executed (f), and receives grounded feedback and memory updates (g).
A human-intuitive interface
Instead of joint angles or high-frequency end-effector targets, the model issues game-like commands. All motion is defined at the gripper interaction point (GIP), the midpoint between the fingertips, which is also used for visual annotations and state reporting.
<arm> move <x|y|z><d><arm> rotate <roll|pitch|yaw><θ><arm> point <down|forward|down45><arm> gripper <open|close|0–1><arm> home · wait · done
<arm> is left or right; axes are in the world frame. Steps are capped at 20 cm and 90° per command. Each command becomes a planned motion to a target GIP pose and runs until the robot is stationary.
In-context demonstrations
Written rules leave ambiguity about step size, rotation effects and conventions. The context therefore combines a shared command primer, which shows the effect of every primitive, with task demonstrations showing how primitives compose into a solution.
Expert trajectories are reduced to waypoints and rewritten as commands the online model could have issued. A VLM then adds a short rationale for each round. Demonstrations come from scenes disjoint from evaluation — in simulation, the same clean trajectories used to train the baselines.
0, 1 and N task demonstrations give the zero-, one- and few-shot settings.
Closed-loop control
Observe. Annotated multi-view images, robot state, and grid-based localization.
Reason. Estimate progress, update the plan, write to a scratchpad.
Act. Send a batch of semantic commands.
Adapt. Execution feedback (reached, partial, failed) and the new state update memory for the next round.
No privileged object poses are used online.
Simulation results
Baselines are post-trained on each benchmark’s full training set. RoboDawn uses no parameter updates and at most one task demonstration in context.
50 bimanual tasks, 10 evaluation runs per task. Baselines are trained on 50 clean demonstrations per task and evaluated in domain-randomized scenes; RoboDawn’s one demonstration also comes from the clean scenes. Success rate (%).
RoboDawn (ours)Trained on the full setAgentic VLA, trained VLA backbone
Ablations on 2.0
Success rate (%) for the number of demonstrations, the backbone VLM, and harness components.
(a) Number of demonstrationsGemini 3.8 Flash
Gains flatten after one example; eight examples slightly hurt, consistent with long-context difficulty.
(b) Backbone modelone-shot
Performance scales with the underlying VLM.
(c) Harness componentsGemini 3.8 Flash, zero-shot
Grid localization and explicit reasoning matter most.
42 simulation tasks on a bimanual platform, spanning generalization, memory, long-horizon, precision and open-instruction tasks. RoboDawn uses GPT-6 Astra; results average 5 runs.
Liber-0 Preview and Liber-0 Lite (LiberAI) are taken from the official RoboDojo sim leaderboard (updated 2026-09-17); all other baselines are from Table 4 of the report. Excluding the 8 open-instruction tasks, for which other methods may have no training data (34 tasks), RoboDawn reaches 33.96% success zero-shot and 43.33% one-shot (progress scores 39.02 and 51.29).
Figure 3. RoboDojo success rate under varying per-episode command budgets.
Test-time scaling
Success keeps rising as the command budget grows. One-shot success increases from 31.2% with 60 commands to 47.2% with 240; zero-shot from 23.7% to 35.7%.
The report attributes this to failure recovery and memory: RoboDawn adapts its next actions to earlier observations and execution outcomes, so extra interaction turns into task success.
Inference and execution time
Per decision, averaged over the 50 RoboTwin 2.0 tasks and measured with Seed-2.1-Pro for RoboDawn. The inference-to-motion ratio indicates how much thinking time surrounds each unit of robot motion; below 0.5, inference could overlap with motion for streaming execution.
Model
Inference
Actions
Motion time
Inference / motion
π0.5
101 ms
45.0 steps
2.70 s
0.037
StarVLA
70 ms
16.0 steps
0.96 s
0.073
X-VLA
143 ms
28.6 steps
1.71 s
0.084
FastWAM
523 ms
27.3 steps
1.64 s
0.32
Motus
1.93 s
16.0 steps
0.96 s
2.01
LingBot-VA
8.89 s
22.2 steps
1.33 s
6.67
RoboDawn · Seed-2.1-Pro
9.74 s
3.4 commands
2.09 s
4.65
Real-world deployment
The same harness runs on physical robots zero-shot with Gemini 3.8 Flash: no task-specific training and no task demonstration in context. Stacking needs more precise placement than dropping a block into a basket. Cloth folding, which requires large end-effector rotations, remains hard: several attempts came close, but the folds were not neat enough to count as successes.
Failure analysis
Three representative failure modes on RoboDojo.
1 Insufficient precision
The intent is right, but the last centimeter fails — for example, inserting a coin into a narrow slot. Discrete commands are coarse near contact.
2 Collision during IK execution
A sensible command sequence produces a motion that sweeps into nearby objects and disturbs the scene.
3 Premature success judgment
The model pours liquid into the cup but stops before enough has been transferred — its notion of “done” does not match the benchmark’s success criterion.
Limitations
Speed. Iterative VLM reasoning is slower than end-to-end VLA and WAM policies, so high-frequency control is out of scope.
Rotations. VLMs handle translation more reliably than precise rotations and orientation changes; demonstrations narrow but do not close this gap.
Fine contact. The discrete interface can be too coarse for subtle corrections during grasping and placement.
Safety. Erroneous commands can cause collisions or damage, which matters more as systems become general-purpose.
Citation
@article{guo2026robodawn,
title = {Transferring the Intelligence of VLMs to Robotic Control},
author = {Guo, Meng-Hao and Mo, Zhe-Han and Wang, Jia-Jun and
Zhang, Yi and Wang, Kejin and Deng, Yi-Xuan and
Zhang, Jia-Peng and Rao, Yongming and Hu, Shi-Min},
journal = {arXiv preprint arXiv:2609.22966},
year = {2026}
}
