Offline RL as Sequence Modeling

A completely different framing

Everything so far fought distributional shift inside the Bellman machinery. The sequence-modeling view sidesteps it: forget value iteration, treat a trajectory as a sequence of tokens, and train a Transformer to predict the next action — exactly like a language model predicts the next word. There is no `max`, so there is no over-estimation to fear. Offline RL becomes plain supervised learning over logged sequences.

Decision Transformer

The headline method is the Decision Transformer. Its key move is to feed the model not just states and actions but the return-to-go — the total future reward from each step onward. The model learns the association "if this much reward still lies ahead from this state, the logged action was this." At test time you simply set the return-to-go to a high value you want, hand it the current state, and read off the action. You are commanding outcomes rather than optimizing for them.

\hat{R}_t=\sum_{t'=t}^{T} r_{t'}

Return-to-go: the total future reward from step t that the Decision Transformer is conditioned on.

# Decision Transformer: trajectory as a token sequence, in time order
#   R_1, s_1, a_1, R_2, s_2, a_2, ...   where R_t = return-to-go at step t
#
# Train like a language model: predict each action token from everything before it.
# (cross-entropy for discrete actions, MSE for continuous ones)
#
# At deployment, you CONDITION on the return you want:
#   set R_1 = target_return       # "I want this much reward"
#   feed (R_1, s_1) -> model     -> a_1
#   act, observe r_1, s_2; set R_2 = R_1 - r_1; repeat

Return-conditioned prediction: ask for an outcome, the model proposes the matching action.

Trajectory Transformer and return conditioning

A sibling, the Trajectory Transformer, goes further and models whole trajectories — states, actions, and rewards all discretized into tokens — then plans by beam-searching over high-return continuations, blending sequence modeling with autoregressive planning. Both belong to the broader family of return-conditioned policies: instead of learning which action is best, they learn which action leads to which return and let you dial the return at run time.

Honest limits

The framing is elegant but not magic. Because it is fundamentally conditioned imitation — closer to weighted behavior cloning than to dynamic programming — it struggles to stitch: if no single trajectory ever reached a high return, asking for one can produce incoherent actions. Value-based methods like IQL/CQL can in principle combine good fragments from many mediocre trajectories; pure return conditioning often cannot.