Sim-to-real transfer
Also known as: sim2real, sim-to-real, simulation-to-reality transfer
Sim-to-real transfer is the process of training a robot's skills in a computer simulation and then making them work on a real robot.
Simulation lets robots practise millions of times quickly and safely, without wearing out motors or breaking things. While sim-to-real has produced impressive walking and grasping skills, the 'reality gap,' meaning differences between the simulation and the messy real world, means skills often need adjustment once they hit real hardware.
Sim-to-real transfer, explained at three levels
At home
It's like a pilot training in a flight simulator before flying a real plane. Robots practise in a virtual world first, then try what they learned in real life.
At work
Simulation training cuts the cost and risk of developing robot skills, because virtual robots can fall, crash, and retry without damaging equipment.
At home
Robots in simulation can practise thousands of times faster than real time. The challenge is that virtual floors, objects, and lighting never perfectly match the real world.
At work
Techniques include domain randomization, system identification (measuring real robot properties to tune the simulator), and fine-tuning on a small amount of real-world data. Reinforcement learning is commonly used to train policies in simulation.
At home
Things like soft fabrics, liquids, and squishy objects are very hard to simulate. In my opinion, that's why robots learned to walk long before they learned to fold laundry well.
At work
Key considerations include simulator fidelity versus speed, contact physics accuracy, sensor noise modelling, photorealistic rendering for vision, and validation protocols. Digital twins of real facilities are increasingly used to test robot behaviour before deployment.
Real-world examples
At home
Sophie watched a video of a four-legged robot learning to walk in a simulation, falling thousands of times in minutes. Later, the real robot walked across a snowy hill on its first try.
At work
Marcus leads robotics at a Montreal logistics startup. His team trains a robot arm to pick odd-shaped packages in simulation, then fine-tunes it with a few hours of real-world practice.
How it compares
| feature | Sim-to-real transfer | Real-world training |
|---|---|---|
| Speed of practice | Thousands of times faster | Real time only |
| Hardware damage | None during training | Wear and breakage |
| Realism | Approximate | Exact |
| Cost to scale | Compute | Robots and staff |
Simulation scales cheaply. Real-world data keeps it honest.
For analysts
Simulation can shorten robot projects a lot. I'd still budget real-world testing time, because the reality gap tends to show up in exactly the edge cases that matter.
History & origin
Roboticists used simulation for decades, but skills learned there often failed on real machines. In 2017, researchers at OpenAI and UC Berkeley, including Josh Tobin, popularized 'domain randomization,' which varies colours, lighting, friction, and other settings in simulation so the robot learns to cope with variety. OpenAI used heavy randomization to train a robot hand that solved a Rubik's Cube in 2019. ETH Zurich trained the ANYmal four-legged robot in simulation to walk over rough terrain, publishing influential results in 2019 and 2020. Physics simulators running on GPUs, such as NVIDIA Isaac Sim and Isaac Lab and Google DeepMind's MuJoCo, made massive parallel training practical. In my view, sim-to-real is the main reason legged robots suddenly got so good at walking in the 2020s.