stochastic computation graphs

Ordinary backpropagation differentiates a chain of deterministic operations. But an RL objective has random sampling baked into the middle — you sample an action, the environment samples a next state — and you cannot backpropagate through a coin flip in the naive way. Stochastic computation graphs are a unifying framework for differentiating objectives that contain such sampling nodes, with a clear recipe for what to do at each one.

There are two ways past a sampling node, and the framework makes the choice explicit. The score-function (likelihood-ratio) route works for any sampler: weight the downstream value by ∇_θ log p_θ(x); it is general but high-variance. The pathwise (reparameterisation) route rewrites the sample as a deterministic function of θ and fixed noise, x = g_θ(ε), so ordinary gradients flow through; it is low-variance but needs the sampler to be differentiable and continuous. Most discrete-action policies must use the first; many continuous ones can use the second.

Seeing RL through this lens is clarifying. REINFORCE is just the score-function estimator on the action node; the deterministic and soft actor-critic methods are pathwise gradients through the policy; and the framework tells you exactly where bias and variance enter. It is the bridge connecting policy gradients to the broader world of differentiable programming and variational inference.