JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Learning by Watching: Behavioral Cloning

When you have demonstrations but no reward function, the simplest move is to treat control as supervised learning. Meet behavioral cloning.

Why imitate instead of explore?

Classic reinforcement learning learns by trial and error, guided by a reward signal. But in many real problems the reward is hard to write down (what exactly is the reward for driving 'well'?), and pure exploration is slow or dangerous. Imitation learning sidesteps both issues: instead of a reward, you give the agent demonstrations — recorded examples of an expert doing the task — and ask it to behave like the expert.

The learning paradigms side by side: behavioral cloning takes the supervised route instead of the reward-driven reinforcement-learning one.

Diagram contrasting supervised, unsupervised, and reinforcement learning.

This reframing is powerful. Humans are far better at showing a skill than at specifying a reward for it. A surgeon, a forklift driver, or a video-game champion can produce excellent trajectories on demand, even though none of them could hand you a clean reward function.

Behavioral cloning: control as supervised learning

Behavioral cloning (BC) is the most direct form of imitation. You take every demonstrated state and its action, and train a policy π(a | s) to predict the expert's action from the state — exactly like a classifier predicts a label from an input. The reinforcement-learning problem dissolves into ordinary supervised learning.

Concretely: the inputs are states, the labels are the expert's actions, and the loss is whatever fits your action type — cross-entropy for discrete actions, mean-squared error for continuous ones. You never need a simulator, a reward, or a single environment step during training.

Behavioral cloning recasts control as the supervised-learning loop: states are the inputs and the expert's actions are the labels.

The supervised-learning loop: data feeds a model, predictions are compared to labels by a loss, and the model is updated.

# Behavioral cloning: pure supervised learning on (state, action) pairs
dataset = [(s, a) for trajectory in expert_demos
                  for (s, a) in trajectory]

for epoch in range(num_epochs):
    for (states, actions) in batches(dataset):
        pred = policy(states)              # predicted action distribution
        loss = action_loss(pred, actions)  # cross-entropy or MSE
        loss.backward()
        optimizer.step()
Behavioral cloning is just classification/regression — the trajectories are flattened into one big labeled dataset.

A minimal recipe

  1. Collect demonstrations. Record an expert acting in the environment; store each (state, action) pair. Broad coverage of states matters more than many repeats of the same state.
  2. Choose a policy class. Pick a network mapping states to an action distribution, matching the action space (softmax over discrete actions, or a Gaussian for continuous control).
  3. Fit by maximum likelihood. Minimize the supervised loss between predicted and demonstrated actions. Use standard tricks: shuffling, a validation split, early stopping.
  4. Roll out and inspect. Deploy the policy in the environment and watch where it drifts away from expert-like states — that failure pattern is the subject of the next guide.
\theta^{\star} = \arg\max_{\theta} \sum_{(s,a)\in\mathcal{D}} \log \pi_{\theta}(a \mid s)

Behavioral cloning fits the policy by maximum likelihood — maximizing the log-probability of each expert action under the policy.

Where cloning shines — and its blind spot

Behavioral cloning is a fantastic baseline: it is fast, stable, needs no environment interaction, and often gives a strong starting policy you can fine-tune with RL later. That is exactly why offline-RL practitioners keep a BC baseline around for comparison. When demonstrations are plentiful and the deployment states look like the training states, BC is hard to beat.

Its blind spot is subtle but fatal: the policy is trained only on states the expert visited. The moment the learner makes a small mistake, it lands in a state the expert never demonstrated, has no idea what to do, makes a bigger mistake — and the errors snowball. We unpack this compounding error problem next.