successor features

Successor features separate where a policy will take you from what those future places are worth. They summarize the expected discounted future occupancy of features under a policy, independently of any particular reward. Once you have them, plugging in a new reward weighting yields that reward's value almost instantly, which is what makes them powerful for transfer across tasks.

Assume the reward is linear in a feature vector, the dot product of features and a weight vector. The successor features of a policy are the expected discounted sum of those features along its trajectory, so the action-value for any task is simply the successor features dotted with that task's weight vector. Generalized policy improvement then takes a library of policies' successor features and, for a new task, acts greedily with respect to their maximum, immediately producing a good policy without relearning from scratch.

Successor features generalize Dayan's successor representation from raw states to learned features. The catch is that the transfer is exact only insofar as the reward really is linear in the chosen features.