A2C (synchronous advantage actor-critic)

A2C is the tidy, synchronous version of advantage actor-critic. You run many copies of the environment in parallel, let each take a fixed number of steps, then stop and gather every transition into one big batch before doing a single update. Think of a row of game machines all paused at once so you can read their screens together — the simultaneity is the whole point.

Batching across parallel actors gives you decorrelated, diverse experience without a replay buffer, which both stabilises the gradient and uses hardware efficiently — one large matrix update instead of many tiny ones. Each update computes advantages (often via GAE), takes one actor step and one critic step, then all workers resume from where they paused. It is deterministic and easy to reproduce, unlike its asynchronous cousin.

Historically A2C was introduced as the observation that the asynchrony in A3C was not actually necessary — a synchronous version matched or beat it while being simpler to implement and debug. It remains a clean, strong baseline and the conceptual stepping stone to PPO, which adds a clipped objective on top of essentially this loop.