Building generalizable robot agents for diverse applications remains a fundamental challenge. While imitation learning-based policies can perform well in familiar training environments, they often struggle to generalize to novel scenes, layouts, and task compositions. To this end, we present World Action Planner, an agentic robot planning system in which the agent searches for and composes executable action plans through imagination with an action-conditioned world model. The search proceeds in a coarse-to-fine manner. First, the agent performs global action optimization by reasoning over imagined world-model rollouts to identify potential failures and refine the proposed action plan. It then performs local action search, comparing the imagined future outcomes of neighboring candidates to select the best action for execution. Across compositional long-horizon tasks, novel object layouts, and real-robot planning on novel tasks without expert demonstrations, World Action Planner consistently outperforms state-of-the-art end-to-end generalist policy models and VLM planners, demonstrating the effectiveness of world-model-based action search for generalizable robot decision making.
We introduce World Action Planner, a planning system in which the agent reasons about and refines its action plans via world model imagination, generalizing to novel scenarios and solving new tasks at test time.
Real Robot Demonstration
Real-world execution on a Franka arm for the task “Put the green can into the white basket.” The VLM agent proposes grasp and placement waypoints, the action-conditioned world model imagines the resulting rollouts to check and adjust the plan before execution, and the refined plan is then executed on the physical robot.
Algorithm
Generalizable Decision-Making
Compositional Generalization We evaluate compositional generalization on LIBERO-Long tasks, each composed of two LIBERO-90 tasks seen during training. After completing the first subtask, the VLM proposes transition actions toward the next object, and the action-conditioned world model imagines the resulting rollout to verify that the policy can subsequently resume and complete the next subtask.
New Layout Generalization We evaluate our method on LIBERO-Object tasks with modified object layouts. The VLM agent identifies the relocated target object and navigates the gripper to its proximity, where a weak policy can reliably complete the grasp. The action-conditioned world model then imagines placement trajectory, allowing the planner to detect and avoid potential collision risks before execution.
New Task Generalization We evaluate on Robosuite StackCube and PickPlaceCan without expert demonstrations or a learned robot policy. Using only hard-coded grasp/release primitives, the planner leverages world model imagination to iteratively refine fine-grained actions, such as precise cube alignment for successful stacking.
Robot World Model
Pose-Image Conditioned World Model We compute robot joint poses via forward dynamics and render them as joint skeleton images from the corresponding viewpoints, which are then used as conditioning inputs for action-conditioned world modeling.
