A3C (asynchronous advantage actor-critic)

A3C was an influential early way to make deep policy gradients work on ordinary CPUs. Spawn many actor-learners, each with its own copy of the environment, all exploring different parts of the world at once. Every worker computes gradients on its own little batch and applies them to one shared set of parameters whenever it is ready — no waiting for the others. The diversity of simultaneous explorers is what stabilises learning.

The asynchrony is the defining feature: workers read the shared parameters, run a short rollout, compute an advantage actor-critic gradient, and push the update back without locking. This decorrelates the data in time (a substitute for experience replay) and was fast on multi-core machines of its era. The cost is staleness — a worker's gradient may be computed against slightly out-of-date parameters.

A3C mattered historically because it showed parallelism, not a replay buffer, could stabilise on-policy deep RL. But its non-determinism makes it hard to reproduce and tune, and the later finding that a synchronous version (A2C) works as well or better, while being simpler and GPU-friendly, has made A2C and PPO the more common choices today.