JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Frontiers: Observation-Only, One-Shot, and Preference-Based Imitation

What if you only see the expert's states, not their actions? Or one demo of a brand-new task? Or no demos at all, just preferences? The modern edge of learning from people.

Learning from observation: no action labels

Everything so far assumed you can read the expert's actions. But the richest source of demonstrations — video of humans and animals — gives you only states: you see the basketball go in, not the precise muscle torques. Learning from observation (LfO) tackles imitation when actions are unobserved.

Two ideas dominate. One learns an inverse dynamics model that infers 'what action takes you from this state to the next?', then labels the observation-only demos with guessed actions and falls back to behavioral cloning. The other matches the distribution of state transitions the expert produces — an observation-only cousin of GAIL whose discriminator compares state pairs (s, s′) rather than state–action pairs.

a_t = I_\phi(s_t,\, s_{t+1})

The inverse dynamics model infers the action that carried the agent from one observed state to the next, relabeling action-free demonstrations.

One-shot imitation

One-shot imitation learning asks for the ultimate data efficiency: watch a single demonstration of a never-before-seen task and perform it immediately. The trick is to not learn one task at all, but to meta-learn across a huge family of related tasks, so the model acquires a general skill — 'condition on a demo, then act' — rather than a single policy.

At deployment the demonstration is fed in as context: the policy attends to it and generalizes to the new instance, much as a language model follows an in-context example. One-shot imitation is where imitation learning meets few-shot, prompt-driven generalization — and a direct ancestor of today's generalist robot policies that take a demo or instruction as input.

One-shot imitation treats the single demonstration as context the policy attends to, much like in-context learning in a language model.

Interactive self-attention: click a word to see which other words it attends to.

Demonstration-guided RL: imitation plus reward

Pure imitation caps you at the expert; pure RL from scratch can be hopelessly slow when rewards are sparse. Demonstration-guided RL blends them: use demonstrations to bootstrap exploration and shape early learning, then let a real reward push the agent past the expert. Demonstrations answer 'where do I even start?'; the reward answers 'how do I get better than my teacher?'.

\mathcal{L}(\theta)\;=\;\mathcal{L}_{\text{imitation}}(\theta)\;+\;\lambda\,\mathcal{L}_{\text{RL}}(\theta)

Demonstration-guided RL optimizes a blended objective: an imitation term keeps the policy near the demos while a weighted RL term lifts the expert's ceiling.

  1. Warm-start. Pre-train the policy with behavioral cloning so the agent begins near competent behavior instead of random flailing.
  2. Seed the buffer. Inject demonstrations into the replay buffer so the agent revisits expert transitions during off-policy RL updates.
  3. Blend the objective. Combine an imitation loss with the RL loss, then anneal the imitation weight down so the reward gradually takes over and the agent can surpass a suboptimal expert.

Preference-based reward learning

Sometimes you cannot demonstrate at all — you cannot show a robot how to do a backflip — but you can judge which of two attempts looks better. Preference-based reward learning exploits exactly this. Show a human pairs of trajectory segments, collect 'A is better than B' comparisons, and fit a reward model whose scores agree with those preferences. Then optimize that learned reward with ordinary RL.

Preference-based reward learning is the engine behind RLHF: human comparisons train a reward model that then tunes the policy.

Diagram: a human preference between two outputs feeds a reward model that tunes the policy.

Comparative feedback is often easier and more reliable than either demonstrating or hand-writing a reward — judging is cheaper than doing, and a noisy comparison still carries signal. This pipeline is the conceptual core of preference-based RL and, scaled up to language models, of RL from human feedback (RLHF), which aligns modern AI assistants by turning human preferences into a reward.