Thinking Wider: Trees, Graphs, and Self-Consistency

Why one chain is fragile

Chain-of-thought prompting dramatically improves reasoning by letting the model think step by step before answering. But a single greedy chain commits to an early step and can never take it back — one wrong turn near the start dooms everything after it. Worse, the model has no way to recognize that a different first step would have been easier. The fix is structural: instead of one line of reasoning, generate many, and combine them.

This is the heart of test-time compute: spend more inference effort on hard problems, the way a person spends longer on a hard puzzle. The three techniques below sit on a ladder of increasing structure — a flat set of independent samples, a branching tree, and finally a general graph.

Self-consistency: vote over independent samples

Self-consistency is the simplest and often the strongest idea here. Sample several reasoning chains at non-zero temperature, then marginalize over the reasoning by taking a majority vote on the final answers. The insight is that there are many correct ways to reach a right answer but idiosyncratically many wrong ways to reach a wrong one, so correct answers cluster while errors scatter. You keep the answer, throw away the reasoning, and let the agreement be your confidence signal.

Self-consistency marginalizes over reasoning: sample k chains and return the majority answer.

answers = [extract_answer(model.sample(prompt, temp=0.7))
           for _ in range(k)]
final = most_common(answers)
confidence = answers.count(final) / k

Tree of Thoughts: search with lookahead

Tree of Thoughts (ToT) turns reasoning into deliberate search. Each node is a partial solution — a few reasoning steps so far. From a node the model proposes several next thoughts (the branching factor), a value function (often the model itself, asked to rate promise) scores each child, and a classic search strategy — breadth-first, depth-first, or best-first — decides which frontier nodes to expand. Crucially, a dead end can be pruned and abandoned, and the search backtracks to a more promising branch. This is the deliberate, slow 'System 2' counterpart to a single fast chain.

If this feels like Monte Carlo Tree Search with a language model in the roles of both the policy (proposing moves) and the value network (scoring states), that is exactly the right analogy. ToT is search over the space of reasoning, and like all search its cost grows quickly with depth and branching — so you tune both to the problem's difficulty.

Graph of Thoughts: merge and refine

A tree forbids two branches from ever recombining, but real problem-solving often aggregates partial results — solve two sub-parts separately, then merge. Graph of Thoughts (GoT) generalizes the tree to an arbitrary directed graph of thoughts, where a node can have multiple parents. New operations become possible: aggregation (combine several thoughts into one), refinement (a thought loops back to improve itself), and generation (branch out as in a tree). Tasks with a divide-and-conquer shape — sorting, merging documents, set operations — benefit most.

Reflection inside the reasoning

All three methods need a way to evaluate a thought. The most flexible evaluator is the model itself, used as a critic — the core idea behind self-reflection agents. After producing an answer or a partial chain, a second pass asks the model to find flaws, check constraints, and propose a revision. Used as the value function in ToT, reflection scores branches; used after a final answer, it catches mistakes before they ship. The catch is calibration: models are not reliable judges of their own correctness, and a confident wrong critique can steer the search away from the right answer.

All of this composes back onto the reasoning–acting loop of the previous guide: each branch can itself take tool actions, so the value of a thought may depend on observations from the world, not just the model's self-assessment. That grounding — verifying a branch by running code or querying a source rather than trusting the model's hunch — is the single most reliable way to make test-time search actually pay off.