Sim-to-real transfer
Also known as: sim2real, sim-to-real, simulation-to-reality transfer
Sim-to-real transfer is the process of training a robot's skills in a computer simulation and then making them work on a real robot.
Simulation lets robots practise millions of times quickly and safely, without wearing out motors or breaking things. While sim-to-real has produced impressive walking and grasping skills, the 'reality gap,' meaning differences between the simulation and the messy real world, means skills often need adjustment once they hit real hardware.
Sim-to-real transfer, explained at three levels
At home
Think of a pilot training in a flight simulator. The simulator copies the physics of flying closely enough that skills carry over to a real cockpit, and crashes cost nothing. Sim-to-real transfer does the same for robots: a virtual copy of the robot practises a task millions of times in a physics simulation, and the learned controller is then loaded onto the real machine.
At work
Sim-to-real transfer is the practice of developing a robot's skills in a physics simulation and then deploying them on physical hardware. Simulation runs many robots in parallel, faster than real time, with no damage or safety risk. For businesses, that shortens development, cuts hardware wear, and lets teams test rare or dangerous situations before a robot reaches a real floor.
At home
Most sim-to-real training uses reinforcement learning. Thousands of virtual robots try a task at once, earn a reward for progress, such as walking forward without falling, and gradually improve their control policy. The trouble is the reality gap: real motors have delays, real floors have different friction, and real cameras see glare the simulator never modelled. A policy that exploits a simulator quirk can fail instantly on real hardware.
At work
Teams close the reality gap three ways. Domain randomization varies mass, friction, lighting, and sensor noise in simulation, so the policy learns to cope with any value in a range. System identification measures the real robot's motors and masses and tunes the simulator to match. Fine-tuning adapts the policy with a small amount of real data. GPU-based simulators such as NVIDIA Isaac Lab and MuJoCo's MJX run thousands of environments in parallel.
At home
Rigid bodies like legs and floors are relatively easy to simulate, but contact, friction, and soft or deformable objects are not. Cloth, liquids, and cables behave in ways simulators approximate poorly. While legged robots now learn to walk almost entirely in simulation, in my opinion that gap explains why robots mastered rough terrain years before they could fold laundry reliably.
At work
Key engineering choices are simulator fidelity versus speed, contact models, actuator models that capture motor delays and limits, sensor noise, and photorealistic rendering for camera-based policies. Validation needs staged real-world tests, since success rates in simulation overstate real ones. Digital twins built in tools such as NVIDIA Omniverse let teams test robot fleets against a facility's actual layout. I'd budget real-world testing time, because the reality gap shows up in edge cases.
Real-world examples
At home
Sophie watched a video of a four-legged robot learning to walk in a simulation, falling thousands of times in minutes. Later, the real robot walked across a snowy hill on its first try.
At work
Marcus leads robotics at a Montreal logistics startup. His team trains a robot arm to pick odd-shaped packages in simulation, then fine-tunes it with a few hours of real-world practice.
How it compares
| feature | Sim-to-real transfer | Real-world training |
|---|---|---|
| Speed of practice | Thousands of times faster | Real time only |
| Hardware damage | None during training | Wear and breakage |
| Realism | Approximate | Exact |
| Cost to scale | Compute | Robots and staff |
Simulation scales cheaply. Real-world data keeps it honest.
For analysts
Simulation can shorten robot projects a lot. I'd still budget real-world testing time, because the reality gap tends to show up in exactly the edge cases that matter.
History & origin
Roboticists used simulation for decades, but skills learned there often failed on real machines. In 2017, researchers at OpenAI and UC Berkeley, including Josh Tobin, popularized 'domain randomization,' which varies colours, lighting, friction, and other settings in simulation so the robot learns to cope with variety. OpenAI used heavy randomization to train a robot hand that solved a Rubik's Cube in 2019. ETH Zurich trained the ANYmal four-legged robot in simulation to walk over rough terrain, publishing influential results in 2019 and 2020. Physics simulators running on GPUs, such as NVIDIA Isaac Sim and Isaac Lab and Google DeepMind's MuJoCo, made massive parallel training practical. In my view, sim-to-real is the main reason legged robots suddenly got so good at walking in the 2020s.