Plans and Memory: Decomposing Goals, Remembering the Past

Why long-horizon tasks need scaffolding

Ask an agent to 'refactor this codebase' or 'plan a week-long trip' and the reasoning–acting loop alone is not enough: the task spans far more steps than fit comfortably in one context, and a single slip early on compounds. Two kinds of structure rescue it. Planning gives the agent a skeleton of the future so it does not wander. Memory gives it continuity with the past so it does not forget what it already learned. Vol I sketched both under planning and memory in agents; here we make each precise.

LLM planning: plan-then-act vs. interleave

LLM planning is using the language model to turn a goal into an ordered structure of steps before (or while) executing them. The two poles are plan-then-execute — write the whole plan up front, then run it step by step — and interleaved planning, where the agent re-plans after each observation (the reasoning–acting loop is the extreme of this). Up-front plans are legible and easy to inspect but brittle when the world surprises you; interleaved planning adapts but can lose the thread of the overall goal. Most strong systems do both: a coarse standing plan, re-examined and revised as observations arrive.

Subgoal decomposition and hierarchy

The engine of planning is subgoal decomposition: split a hard goal into smaller subgoals, each of which is either directly solvable or decomposed further, forming a hierarchy. A high-level controller manages subgoals; lower levels execute them, often with their own private reasoning–acting loops. This recursion is what lets a fixed-context model tackle tasks far larger than its window — each subgoal is solved in a fresh, focused context, and only its result bubbles up.

There is a deep parallel to hierarchical reinforcement learning. An option is a temporally extended action — a sub-policy with its own initiation and termination — and a subgoal is essentially an option named in natural language. The same questions arise: how to decompose, how to know a subgoal succeeded, and how to compose subgoal results without their errors accumulating.

o = \langle \mathcal{I},\, \pi,\, \beta \rangle, \qquad \mathcal{I} \subseteq \mathcal{S}, \;\; \pi : \mathcal{S} \to \Delta(\mathcal{A}), \;\; \beta : \mathcal{S} \to [0,1]

An option in hierarchical RL — an initiation set, a sub-policy, and a termination condition — the formal analogue of a subgoal in the decomposition hierarchy.

def solve(goal, depth=0):
    if is_atomic(goal) or depth > MAX:
        return run_react_loop(goal)        # base case: just do it
    subgoals = model.plan(goal)            # decompose
    results = [solve(g, depth+1) for g in subgoals]
    return model.synthesize(goal, results) # compose results up

Recursive decomposition: keep splitting until a subgoal is atomic, solve those with the agent loop, then synthesize results back up the hierarchy.

Memory architectures

Agent memory architectures give the agent state that outlives a single context window. The standard split mirrors human memory. Short-term (working) memory is the live context: the running thought–action–observation trace, naturally bounded by the context length. Long-term memory is external storage the agent reads from and writes to: episodic memory of past events (often a vector database queried by embedding similarity), semantic memory of distilled facts, and procedural memory of learned skills or routines.

The hard part is not storage but management: what to write, when to retrieve, and how to forget. Naively dumping everything into memory drowns retrieval in noise; aggressive summarization loses the detail you will later need. A practical architecture writes compact, timestamped summaries, retrieves the top-k most relevant on each step, and periodically consolidates old episodes into semantic facts — the same compression-and-recall trade-off a human notebook faces.

Reflexion: learning from verbal feedback

Memory plus reflection unlocks learning without touching the weights. Reflexion runs an agent on a task, and when it fails, asks the model to write a short verbal self-critique — 'I should have checked the edge case before submitting' — which is stored in episodic memory. On the next attempt that reflection is prepended to the context, so the agent literally reads its own past lesson. The feedback signal is natural language rather than a scalar gradient, which is why it is framed as verbal reinforcement learning: the policy improves across episodes, but the update lives in text, not parameters.