JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Learning to Learn, and to Keep Learning

Instead of training one policy for one task, train agents that adapt fast, carry knowledge forward without forgetting, and even grow skills with no reward at all.

From 'a policy' to 'a learning algorithm'

Meta-RL flips the target of optimization. Ordinary RL produces a policy good at one task. Meta-RL trains across a distribution of tasks so that the output is a fast learner — an agent that, dropped into a new but related task, figures it out in a handful of episodes. This is the RL face of meta-learning, 'learning to learn.' Two recipes dominate: optimization-based meta-RL learns an initialization that fine-tunes quickly, and context-based (recurrent) meta-RL lets a memory absorb the new task on the fly, treating adaptation itself as something the network runs rather than something you retrain.

Q-learning masters a single gridworld; meta-RL trains across a whole distribution of such tasks to produce a fast learner instead of one fixed policy.

An interactive gridworld where an agent learns a policy by trial and error.

Transfer — don't start from scratch

Transfer in RL reuses something learned in a source task to speed up a target task — a feature extractor, a value function, a set of skills, or a whole pretrained policy you then fine-tune. The promise is enormous (it directly attacks both the generalization and sample-efficiency gaps), but so are the failure modes. Negative transfer happens when source knowledge actively misleads the target; representations that were perfect for one reward can be exactly wrong for another. Transfer is reliable only when source and target genuinely share structure — dynamics, goals, or causal mechanism.

A practical cousin is closing the reality gap from simulation to hardware: domain randomization trains across many randomized simulators so the real world looks like just one more variation. That is transfer plus generalization, deployed.

Continual and lifelong learning

Transfer is usually one hop, source to target. Continual RL and lifelong RL demand a stream: a non-stationary sequence of tasks, learned one after another, with no clean boundary and no return trips. The central enemy is catastrophic forgetting — gradient steps for the new task overwrite the weights that solved the old one. The deep tension is the stability–plasticity dilemma: too stable and you cannot learn anything new; too plastic and you erase everything you knew.

  1. Rehearsal: keep or generate samples from old tasks and mix them into new training so gradients don't drift too far.
  2. Regularization: penalize changes to weights that mattered for past tasks (protect what was hard-won).
  3. Architecture: give new tasks fresh capacity (added modules or masks) so they don't compete for the same parameters.

Unsupervised RL — practice before the test

What if an agent could get better with no reward at all? Unsupervised RL does exactly that: in a long reward-free phase the agent pursues intrinsic objectives — curiosity, novelty, empowerment, or diversity — to build a repertoire of behaviors and a useful world representation. Then, when a real reward finally arrives, learning is dramatically faster. The flagship idea is skill discovery: learn a set of distinguishable skills with no task in mind, so that solving a downstream task becomes mostly a matter of choosing among skills you already have.

Unsupervised RL is RL's analogue of self-supervised pretraining — a reward-free phase that builds broad competence before any task reward arrives.

Diagram contrasting the three learning paradigms: supervised, unsupervised, and reinforcement learning.

Offline meta-RL — fast adaptation from logs

Stack two frontier ideas and you get offline meta-RL: meta-learn a fast adapter using only logged datasets from many tasks, with no fresh interaction during meta-training. It is wildly attractive for the high-stakes domains where new trials are costly, but it inherits the hardest parts of both parents — the distributional shift of offline RL (the data only covers what some old behavior policy did) and meta-RL's demand to generalize to genuinely new tasks. Getting both right at once is an active research front.

\theta^\star=\arg\max_{\theta}\;\mathbb{E}_{\mathcal{T}\sim p(\mathcal{T})}\Big[\,\mathbb{E}_{\tau\sim\pi_{\phi_{\mathcal{T}}}}\big[R(\tau)\big]\,\Big],\qquad \phi_{\mathcal{T}}=U\!\big(\theta,\,\mathcal{D}^{\mathrm{tr}}_{\mathcal{T}}\big)

The meta-RL objective; offline meta-RL optimizes this same goal using only logged datasets, with no fresh interaction.