WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions

Key Takeaways WorldOracle is a benchmark for action-conditioned world models (ACWMs) that scores each model’s predicted rollout against the paired real-world execution, with both starting from the same initial observations and following the same action sequence. Seven representative models were evaluated; arm motion was predicted relatively well, but object positions and movements often deviated from reality, and performance dropped over complete operation sequences. The platform opened to developers on October 16 and accepts both open-weight and closed-source models. In one WorldOracle case, the robot’s grasp missed, yet the world model predicted a successful pick. Credit: [WorldOracle team / AgiBot] In one case showcased by WorldOracle, a robot’s grasp missed, yet the world model predicted that the robot had successfully picked up the fruit. Given the initial frame and action sequence of a real manipulation episode, the model generated a rollout that looked plausible on screen but differed from the actual execution. The success it depicted never happened in reality. Researchers from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA and AgiBot have jointly introduced WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) by comparing their predictions with real-world executions that start from the same initial observations and follow the same action sequences. The team evaluated seven representative models — among them DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0 — and identified two prominent problems. First, while the motion paths of robot arms are predicted relatively well, the positions and movements of manipulated objects can still deviate from reality. Second, models that perform well over short horizons can degrade significantly over complete operation sequences. On October 16, WorldOracle, an evaluation platform for action-conditioned world models, officially opened to developers. The research team has collected real manipulation episodes in advance, recording both video and actions. Developers use the initial frames and action sequences provided in the evaluation set to generate predicted videos and submit them to the platform, which scores each submission against the corresponding real execution record. The results show whether a model has accurately predicted the consequences of the actions, helping developers pinpoint weaknesses and verify improvements. Project page: https://worldoracle-benchmark.github.io/ Paper: https://arxiv.org/abs/2610.18826 How Do Seven World Models Perform? Arm motion is right, but objects don’t always follow The predicted motion of a robot arm reaching toward a piece of fruit and closing its gripper may closely match its actual motion, yet the model can still incorrectly predict whether the fruit is lifted and where it moves next. To assess the two separately, the research team compared the position and motion of the robot arm and of the manipulated object in each predicted video against the real-world recording, and scored them individually. Spatial alignment scores for the robot arm and the manipulated object. Credit: [WorldOracle team / AgiBot] The results show that all five models evaluated on the human-operated AgiBot G2 data scored higher on the arm than on the object. In other words, even when the arm in the generated video follows the given actions well, the position and movement of the object may still depart from reality. IRASim scored 78.8 on the robot arm but only 29.1 on the manipulated object, while Cosmos 3 scored 89.4 and 45.4, respectively. The remaining models showed gaps of varying size. Accuracy declines as the rollout horizon grows Time presents a second challenge. Completing a continuous task often requires multiple grasps, placements and adjustments, and the object state left by one interaction becomes the starting point for the next. Whether a model stays close to reality must therefore be tested over the full process. WorldOracle evaluates both the first 30 seconds of each trajectory and the complete long sequence. WorldOracle-SV scores of the same models under the two evaluation horizons. Credit: [WorldOracle team / AgiBot] On the same set of AgiBot G2 teleoperation data, the WorldOracle-SV scores of all five models fell. DreamDojo dropped from 50.56 to 27.08, and Cosmos 3 from 53.99 to 26.60, nearly halving in both cases. GE-Sim 2.0 fell from 73.86 to 63.33; it posted the highest score of the five models at both horizons and the smallest decline over the full sequence. The relative ranking also shifted. IRASim scored below Ctrl-World, DreamDojo and Cosmos 3 over the first 30 seconds, but above all three over the full sequence. What Can WorldOracle Measure? WorldOracle evaluation framework: real actions are recorded, the model predicts what follows, and the result is scored against the real execution. Credit: [WorldOracle team / AgiBot] Accurately predicting a failure earns a high score WorldOracle starts from pre-collected records of real manipulation. A model is given the initial frame and the action sequence and generates a predicted video, which is compared with the real execution of the same actions. What models must predict includes missed grasps, drops and spills. If an operation fails in reality, accurately predicting that failure counts as a correct prediction. Failed executions therefore serve as an important class of test samples: they reveal whether a model follows the given actions and their consequences, or defaults to a successful version of the task. The benchmark covers two robot embodiments, AgiBot G2 and YAM, as well as first-person human-hand manipulation. Four tasks (fruit picking, towel hanging, water pouring and drawer pulling) involve interaction with rigid, deformable, fluid and articulated objects, respectively. Task coverage varies by embodiment, as defined in the evaluation set. The human-hand setting is intended for world models that can accept hand-motion input, testing whether a model can predict subsequent object changes and interaction outcomes from a recorded sequence of hand poses. Models with egocentric world-modeling capabilities are evaluated zero-shot in this setting, without any additional hand-adaptation data provided by the platform. Real execution footage for the four task types. Credit: [WorldOracle team / AgiBot] The data come from two sources. The first is human-operated data collection, including robot teleoperation trajectories and human hand-manipulation trajectories. The second is policy rollouts, in which an autonomous policy controls a real robot. When a robot executes autonomously, it can produce pauses, jitter and motion corrections that differ from human operation. Including such data helps test whether a world model remains accurate when faced with actions from different sources. The platform offers three tracks: WorldOracle-Core: evaluates predictions over the first 30 seconds of human-operated trajectories. WorldOracle-Policy: evaluates predictions over complete episodes of autonomous policy rollouts on AgiBot G2. WorldOracle-Long: evaluates sustained prediction over complete, human-operated long sequences. The WorldOracle study compares several embodied world-model benchmarks. Of these, WorldOracle is the only one that simultaneously covers action-sequence input, real-world execution data, failure cases, long-horizon interaction, multiple embodiments, multi-view evaluation and autonomous policy rollouts. Breaking down prediction error: outcomes and process The predicted and real robot arm positions may closely match even when the model gets the fruit’s grasp state wrong; two final states may look alike while the paths that led to them differ. A single score can hardly capture such differences. WorldOracle checks predictions with four complementary metrics: Physical State Correctness: checks whether object positions, grasp states and liquid transf