Reinforcement Learning for Robots

Phase	What happens	Main condition	Planning use
Define the task	The goal is turned into a reward function and the policy is set up to maximise it.	Dense reward design is labour-intensive and needs domain expertise.	Budget expert time for the reward before budgeting compute.
Train in simulation	GPU-supported simulators such as Isaac Gym and Isaac Sim run the bulk of the trial and error.	A simulator is a simplification of reality.	Treat simulated success as a candidate, not a result.
Check the transfer	The trained policy is compared across simulators or moved to hardware.	The simulator choice changed joint error even with an identical reward and training length.	Validate the policy outside the simulator it was trained in.
Train or adapt on hardware	The robot keeps learning in the real world with a supervisor present.	One reported method needed around 1 to 2 hours of real-world RL, while sparse rewards did not succeed.	Plan supervised robot time and a safe working area.

The table summarises findings from the GCR paper on arXiv and the TIAGo simulator comparison on arXiv.

Reinforcement learning for robots means training a policy by trial and error to earn long-term reward. It works best when the reward is well specified and trials are cheap, which usually means simulation first.

This guide follows the loop from reward to simulator to deployment. It is one part of artificial intelligence in robotics, and it ends with the cases where a different method is the safer choice.

The learning loop on a physical robot

Reinforcement learning seeks the best actions for a given environment, a mapping called a policy, so that long-term reward is maximised. A review of Sutton and Barto’s textbook describes the goal as achieving the maximum long-term reward, and notes that, unlike other machine learning methods, RL does not require you to specify subgoals.

On a real machine, the loop repeats in the same order each time:

1. Observe the state. The policy sees only what the robot’s sensors report, so the quality of state estimation limits what the policy can learn. Sensor fusion robotics covers how those readings are combined.
2. Act. The current policy chooses a command, and the robot executes it.
3. Receive reward. A reward function scores what happened.
4. Update the policy. Learning shifts the policy toward behaviour that earned more reward.

Exploration is the costly part. According to the GCR paper, a robot with only a sparse reward can spend hours randomly exploring its environment before chancing upon a goal state. That is why the reward and the training setting decide whether a project is practical.

Reward function design for robots

A reward function is hard to specify because a dense, informative one is labour-intensive to design and requires domain expertise, as the GCR paper states. The alternative, a sparse reward that only marks success, avoids hand design but pushes the cost into exploration time.

Learned rewards bring a different problem. The same paper notes that it is possible, and often easy, for the agent to hack a learned value function, reaching states the function rates highly that do not actually satisfy the goal. A reward that looks right on paper can therefore be satisfied in the wrong way.

For that reason, a short review routine before a long training run pays off:

1. State the goal in plain terms, then write the reward to match it.
2. Look for states that would score highly without achieving the goal.
3. Watch early rollouts for behaviour that exploits the score instead of the task.
4. Revise the reward before committing hardware time.

The GCR paper reports that its learned dense reward let model-free RL solve about twice as many tasks as the baseline reward learning methods it was compared against. That supports investing in reward quality, though it is one paper’s result and does not remove the exploitation risk.

When no reward can be written with confidence, learning from demonstrations is a different route, covered in imitation learning robotics.

Training in simulation

Simulation is where most trial and error happens, because it avoids wear and supervised hardware time. A comparison of simulators on the TIAGo robot notes that simulators with GPU support appeared, among them NVIDIA’s Isaac Gym and Isaac Sim, and that several RL libraries were developed, such as OpenAI’s Gymnasium. Stable-baselines is cited in the same context. OpenAI Gym is the older, familiar name for this style of environment interface.

The same paper describes how Isaac Gym handles joint control: a PD controller, using stiffness and damping values, tracks each new reference. A learned policy therefore often sits on top of a classical low-level controller instead of replacing it.

A sensible simulation phase runs in this order:

1. Choose a simulator that supports the robot model and GPU training.
2. Connect it to an RL library through a standard environment interface.
3. Fix the reward and training length before comparing any results.
4. Keep a second simulator or hardware test ready to check the outcome.

Sim-to-real transfer and the gap that matters

Sim-to-real transfer means taking a policy trained in simulation and running it on a physical robot. The gap matters because, as the TIAGo paper puts it, what happens inside a simulator is nothing more than a simplification of reality, so models need adaptation before real use.

The same paper shows that the choice of simulator alone changes outcomes. On TIAGo, both simulators used the same reward, the negative absolute error of each degree of freedom, and the same 100K training epochs. After steady state, the accumulated joint error was 114.3006 rad in Isaac Gym and 152.0785 rad in Isaac Sim. Identical reward and training length still produced different results.

For planning, that supports a cautious habit. Judge a policy by how it behaves outside the simulator that produced it, and expect some adaptation step before deployment.

Real-robot examples and training times

The clearest sourced example here is the GCR paper (arXiv version dated 14 May 2025). It tested in simulation on RoboMimic and MimicGen tasks and in the real world on a Franka arm and a Spot quadruped. It reports that its method enabled real-world RL in around 1 to 2 hours, whereas learning with sparse rewards did not succeed.

The TIAGo study is a second example. It compared how a robot trained in two different GPU simulators performed on joint tracking, with the reward and training length held fixed.

• Manipulation: the Franka arm shows reward-driven learning on a real arm.
• Legged locomotion: the Spot quadruped is the second hardware platform in the GCR experiments.
• Joint control: TIAGo shows how simulator choice shifts accumulated error.

Real-robot training also has a supervision cost. The TIAGo paper notes that performing RL on real robots still requires supervision to ensure that no unexpected scenario is met.

When classical control is the better choice

Reinforcement learning is a worse choice when the costs described above outweigh its benefit. The sourced evidence points to a few clear conditions, and each one favours a conventional controller or a hybrid.

• You cannot write a trustworthy reward. Dense rewards demand domain expertise, and learned ones can be exploited. If the task has a well-defined reference to track, a PD controller already does that job.
• Hardware time is scarce. Sparse rewards can mean hours of random exploration, and real-robot RL still needs supervision. Without a safe, supervised space, keep learning in simulation or use another method.
• Speeding up means buying more robots. Running several real robots at once is an economically expensive way to collect data faster.
• The simulator is a poor match. If the simulation simplifies the task too far, the adaptation burden can erase the savings.

A practical middle path is common. Keep the classical low-level controller, and use learning only for the part of the task that is hard to write down by hand.

Frequently asked questions

Does reinforcement learning need subgoals?

No. The review of Sutton and Barto says RL needs only the ultimate goal. Subgoals are not required, though shaping the reward can still help learning.

What is a TD error?

It is the difference between a predicted state value and the actual one. The same review says the learner updates only when a prediction is wrong, so it learns only from its mistakes.

Can several real robots speed up training?

Yes, but at a price. The TIAGo paper calls acquiring multiple robots an economically expensive way to make data collection faster.

Where is Stable-Baselines3 documented?

The TIAGo paper cites Stable-baselines3 in the Journal of Machine Learning Research 22.1 (2021). It appears alongside Gymnasium among the RL libraries used for robot training.