decision transformer (DT)

Decision Transformer throws out value functions and Bellman equations and asks a different question: what if RL were just sequence prediction? Feed a transformer a trajectory written as a stream of desired-return, state, action, return, state, action, and train it to predict the next action. To act, you tell it the return you want and let it autocomplete.

Each timestep is tokenized as a triple of return-to-go, state, and action. The model is trained with a plain supervised next-token loss to predict actions given the history and the remaining desired return. There is no temporal-difference learning at all — credit assignment is handled implicitly by attention over the sequence.

It is elegant and stable, inheriting the strengths of large sequence models. But conditioning on a return you have never achieved can fail, and pure return-conditioning struggles to stitch together sub-optimal trajectories the way value-based methods can.