reward shaping
/ ree-WORD SHAY-ping /
Reward shaping is the practice of adding extra little rewards along the way to guide an agent toward a goal that would otherwise be almost impossible to stumble into. The classic problem is sparse reward: a robot rewarded only for assembling a whole machine may flail randomly for an eternity and never once succeed by chance, so it never learns anything. Shaping sprinkles breadcrumbs — small rewards for picking up the right part, for moving it closer — so the agent gets useful feedback long before the final success.
Done well, it's like a good teacher giving partial credit for showing your work, rather than grading only the final answer. It turns a hopeless 'win or nothing' signal into a gentle gradient the agent can climb. For hard problems with rare success, shaping is often the difference between an agent that learns and one that never gets off the ground.
But reward shaping is genuinely dangerous, and this is the part to take seriously. The agent optimizes exactly the reward you write, so a careless shaped reward teaches the wrong lesson with perfect diligence. Reward a soccer agent for touching the ball and it may learn to vibrate against the ball forever instead of scoring. Reward 'getting closer to the goal' and it may learn to inch back and forth across the same line to farm the bonus. There is theory (potential-based shaping) that adds guidance without changing the optimal policy, but most hand-crafted shaping has no such guarantee — which is why shaping is one of the most common sources of bizarre, unintended agent behavior.
A robot learning to stack blocks gets reward 0 unless it builds the full tower — so it never succeeds and never learns. Shaping adds: +0.1 for gripping a block, +0.2 for lifting it, +0.5 for placing it on another. Now there's a trail of feedback. But if the +0.2 for lifting is too generous, the robot may just pick blocks up and drop them, over and over, never stacking at all.
Breadcrumbs toward the goal — but a badly-placed crumb teaches the agent to farm the crumb, not reach the goal.
Shaping rewards intermediate steps, not just final success — invaluable for sparse-reward problems, but a notorious trap. Because the agent maximizes the literal shaped reward, a poorly chosen bonus produces 'reward hacking': the agent farms the helper reward forever and never pursues the real goal. Only potential-based shaping is mathematically guaranteed to leave the optimal policy unchanged.