When I started my PhD in 2021, embodied AI looked roughly like this. We had cool simulation demos, we had robot policies that could learn impressive behaviors in simulation, and we had very little that worked on an actual robot.
Transfer from sim to real hardware was sparse. You could get it to work if you constrained the action space enough, or learned on top of some pre-existing robot skills, and even then success rates were low. Plenty of serious people doubted whether simulation policies would even transfer at all and argued that real-world data was the only relevant data.
Robot Locomotion and the “Randomize & Learn” Button
The breakthrough came from robot locomotion, specifically quadrupeds [1]. Researchers trained quadrupeds to walk in simulation, but randomized simulation rollouts such that each quadruped was trained in a slightly different world, with different terrains (slopes, stairs, rough ground), ground friction, random shoves and noisy sensors. The complexity of these changes in environment was also increased over time as the trained policy improved.
ANYmal robots learning to walk in large-scale physics simulations [1].
This was domain randomization [2], applied to physics and dynamics at scale. Let’s call it large-scale physics randomization. The big picture was that now they could transfer and run the same policy on the real robot with no further training. This recipe blew sim2real for locomotion wide open.
The recipe transferred to several locomotion problems and meant that, as a researcher, you could train a quadruped policy in simulation, press a “randomize & learn” button, see a large number of simulations roll out with slightly different physics, terrain settings, and then see a robust policy learned and working on your real robot the next day! Essentially, once real-world physics was inside the training distribution of randomized physics, real-world performance followed.
It might seem obvious now, but this was a big breakthrough back then, and went a long way towards getting the classical robotics community to believe in learned robot policies.
But Manipulation Needs Visual Input and Realistic Environments
Since then, researchers have been trying to bring the same capability to robot manipulation. But there hasn’t been the same amount of success.
Locomotion could get away with geometric perception, such as depth cameras or lidar. Manipulation usually can’t. It needs fine-grained visual understanding of the scene, because many tasks depend on what things look like, not just their shape (for example, folding laundry and sorting it by color). Some tasks, like grasping, do work from depth alone [3], and realistic depth is fairly easy to simulate. But the north star for manipulation is visual policies that map pixels directly to robot actions, since that is the most likely route to solving general-purpose manipulation.
Vision is also much harder to randomize [2]. For a walking robot, randomizing terrain and physics was enough, since a quadruped’s dynamics live in a smaller space. A camera in somebody’s kitchen can see almost anything. So a visual policy needs whole environments that look realistic, with rooms, furniture, clutter, materials and lighting, and enough variety to cover the homes the robot will actually work in. Building and randomizing that many realistic scenes is a much harder task.
As a result, simulation for manipulation was largely left behind, widely seen as something that does not really work or does not pay dividends, and mainly useful for benchmarking.

Procedurally generated households from MolmoSpaces [4], the kind of realistic, varied environments a visual policy needs.
Pressing the Button for Manipulation
During my year-long research scientist internship at the Allen Institute for AI, I wanted to work on exactly this but for mobile manipulation. Mobile manipulation raises the bar further. A mobile manipulator can move through the entire scene, so the whole home has to be built and randomized, not just a tabletop. Ai2 had already seen visual sim2real work for household navigation, and the question was whether the recipe extended to manipulation and mobile manipulation.
MolmoB0T [5] is our attempt at answering that. It builds on MolmoSpaces [4], which has over 230K indoor environments, from handcrafted homes to procedurally generated multi-room houses, and 48K manipulable objects built from Objaverse assets and human-authored doors and articulated objects.
On top of that, we use fairly standard data generators that rely on ground-truth simulation information such as object poses, collision bodies and bounding boxes. They solve pick-and-place, door opening and articulated-object opening with IK controllers, collision-aware IK and cuRobo [6] motion planning.
None of these ingredients is new. What we changed is the scale and the amount of variation. We generated data across textures, lighting, object placement and dynamics, with the aim of making real-world observations look like just another sample from the training distribution.
MolmoBot data generation, with the same tasks generated across thousands of randomized households [5].
For a sense of scale, MolmoBot-Data is 1.7 million trajectories across more than 94,000 environments and 11,000 objects, roughly 5,700 robot-hours of experience. Every episode is randomized on top of that. We vary where objects are placed and how they are oriented, the number, position, color and intensity of the lights, the surface textures (both procedural and real-world texture maps), the camera poses, and physical parameters such as friction, object mass and joint damping.
The New Workflow for Sim2Real Manipulation
The policies run on real robots without any real-world fine-tuning. In one of the demos, the desk the robot is mounted on is raised and lowered mid-rollout, and the policy keeps going. The training data had randomized robot heights, so to the policy this is just another variation it has already seen.
The bigger change is in how you build a manipulation policy. Even a policy prototyped in simulation usually had to be redone on the real robot with teleoperated demonstrations, so the simulator ended up as a testbed rather than a source of training data.
With enough randomization, the workflow starts to look like the locomotion one:
- Write a data generator for the task in simulation. The simulator knows every object’s pose and geometry, so a scripted planner can solve the task without any learning.
- Press “randomize” and generate data at scale. Run the generator across thousands of households, randomizing the objects, textures, lighting, camera poses and robot setup in every episode. This is the step that large-scale households and randomization make possible, and it is the manipulation version of locomotion’s “randomize physics” button.
- Train a visual policy via imitation learning on the generated demonstrations.
- Deploy it zero-shot on a real arm or mobile manipulator.
To further improve policy performance on the real robot, you can often simprove the simulation, for example by randomizing something you missed, and generate again, instead of collecting real-world tele-op demonstrations. A new task means writing a new generator rather than recording tele-op demonstrations from scratch.
This helps most for whole-body mobile manipulation tasks like opening doors and cupboards, which are tedious to teleoperate. Instead, a cuRobo-based whole-body motion planner can be tweaked to solve the task in simulation, and the same recipe then transfers the resulting policy zero-shot to a real mobile manipulator.
"Put the blue tape in the wooden box" while the desk moves up and down [5].
"Put the banana in the brown bowl." When the brown bowl is out of view the policy heads for the black one, then changes course once it sees the right bowl [5].
So Is Sim2Real Solved?
Not so fast!
For rigid-body manipulation and mobile manipulation, partly. In our pick-and-place tests the policy reaches about 80% success zero-shot, which is a useful starting point for fine-tuning with real data.
Zero-shot pick-and-place on a Franka FR3, over 120 real-world trials with no real-world fine-tuning. MolmoBot-Pi0 keeps the π0 [8] architecture and swaps in simulation data, so much of the gain comes from the data. The baseline is π0.5 [9] trained on DROID [10]. Results from [5].
For deformable and soft objects, and for more complex objects in general, it remains to be seen. Simultion fidelity can also be a bottleneck for tasks that require very precise physics. But the current results are promising!
The open question for me is how far this scales and how many more variations we can add. There is still a lot left to vary.
Promising Recent Progress
MolmoBot is not the only recent result pointing this way. AnyTask [11] from RAI automates task and data generation for sim2real policy learning, and SimVLA [12] trains a mobile-manipulation policy entirely in procedurally generated houses and runs it zero-shot in real homes.
A complementary direction is large-scale reinforcement learning in simulation, distilled into a policy that sees only RGB. Zhang et al. [13] scale RL to over a million parallel environments by sampling simulator resets at the edge of what the policy can currently do, and transfer zero-shot to real assembly tasks such as gear insertion. DoorMan [14] from NVIDIA uses a similar recipe to teach a humanoid to open real doors.

Gear insertion in simulation (left) and zero-shot on a real UR5e (right) [13].

DoorMan in simulation with randomized doors, walls and lighting (left three) and opening a real door zero-shot (right) [14].
Taken together, these results point the same way. With enough scale and randomization, simulation is becoming a real source of training data for manipulation. The policy can imitate generated demonstrations, be trained with parallel RL and distilled into vision policies, or combine the two, with imitation pretraining followed by RL fine-tuning.
There is still plenty to improve around this workflow. We need more simulated household environments, and more realistic ones. We need better data generators, and agentic pipelines are already proving quite capable here. And we need simulators and physics engines that come closer to reality.
It is an exciting time for sim2real in manipulation. Stay tuned for more on this!
MolmoBot was accepted as an Oral (top 3% of papers) at the Conference on Robot Learning (CoRL) 2026! You can read the paper here, and see more on the project page.
References
- N. Rudin, D. Hoeller, P. Reist, M. Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Conference on Robot Learning (CoRL) 2021.
- J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, P. Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017.
- S. Jauhri, I. Lunawat, G. Chalvatzaki. NeuGraspNet: Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering. RSS 2024.
- Y. Kim, W. Pumacay, O. Rayyan, et al. MolmoSpaces: Large-Scale Open Ecosystem for Robot Manipulation and Navigation. RSS 2026.
- S. Jauhri*, A. Deshpande*, M. Guru*, R. Hendrix*, et al. MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation. Conference on Robot Learning (CoRL) 2026.
- B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, et al. cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation. arXiv 2023.
- C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.
- K. Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv 2024.
- Physical Intelligence. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv 2025.
- A. Khazatsky et al. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. RSS 2024.
- R. Gong, X. Zhang, J. Shang, et al. AnyTask: An Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning. arXiv 2025.
- K. Baik, Y. Lee. SimVLA: Zero-Shot Sim-to-Real VLA Learning for Mobile Manipulation. arXiv 2026.
- O. Zhang, M. Guaman Castro, P. Yin, I. Dagnino, A. Gupta, R. Scalise, B. Boots. A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control. Conference on Robot Learning (CoRL) 2026.
- H. Xue, T. He, Z. Wang, Q. Ben, W. Xiao, et al. Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer. arXiv 2025.
