What Alignment Means — and Why Pretraining Isn't Enough

The gap a base model leaves open

After pretraining, a model is astonishingly good at one thing: guessing the next token of human-written text. Ask a raw base model a question and it does not answer you so much as continue the document. Type "List three tips for sleeping better" and it might helpfully comply — or it might write a fourth tip, then invent a comment section, then drift into an unrelated blog post. It is doing exactly what it was trained to do; it just was never trained to treat your message as a request to be fulfilled.

This is the central insight behind alignment: capability and cooperation are different things. Pretraining buys raw capability — knowledge, fluency, reasoning latent in the weights. Alignment is the separate effort of shaping behavior so the model reliably uses that capability the way people want: following instructions, staying on task, declining clearly when it should, and admitting when it doesn't know.

The three things we ask of an assistant

Most alignment programs organize their goals around three words: helpful, harmless, and honest (often abbreviated HHH). Helpful means the model actually addresses what you asked, in a usable form. Harmless means it declines to assist with serious harms and avoids gratuitously toxic or dangerous output. Honest means it conveys what it actually believes given its training — flagging uncertainty rather than bluffing, and not fabricating sources.

These three pull against each other, and that tension is the whole game. A model tuned to be maximally helpful will try to answer everything, including dangerous requests. A model tuned to be maximally harmless slides into refusing benign questions — the over-cautious assistant that won't help you write a villain for your novel. Good alignment is not maximizing any one axis; it is finding the trade-off surface where the model is as helpful as it can be without crossing harm lines, and honest about both.

Where alignment sits in the training stack

Modern chat models are built in layers. First comes pretraining on a vast corpus — that is where capability is born. Then comes a post-training phase that does the aligning, and it usually has two moves. The first is supervised fine-tuning (SFT), also called instruction tuning: show the model thousands of example conversations where an assistant responds the way we want, and have it imitate them. The second is preference-based learning — RLHF and its relatives — which is the subject of the rest of this track.

Pretraining — learn language and world knowledge by predicting next tokens on web-scale text. Produces a base model.
Supervised fine-tuning (SFT) — imitate curated example conversations to learn the assistant format and basic helpfulness.
Preference optimization (RLHF / DPO / AI feedback) — learn from comparisons of which answer is better, to refine judgment, tone, and safety.

SFT teaches the model what a good answer looks like; preference learning teaches it how to choose between two plausible answers. The second is subtler and harder — there is no single gold response, only the messy human sense that this reply is a bit better than that one. Capturing that sense at scale is the engineering problem the next four guides unpack.

Why this is a hard problem, not a settled one

It is tempting to think alignment is just "add some politeness training and ship." It is not. The deeper difficulty, shared with the broader field of AI alignment, is that we are trying to optimize a system toward goals we can only specify indirectly — through examples and comparisons, never through a clean rulebook the model can read. Whenever you optimize hard against an imperfect signal, the system finds the gaps in your specification. Much of this track is about those gaps and how practitioners fight them.

\max_{\pi}\;\mathbb{E}_{x,\;y\sim\pi(\cdot\mid x)}\!\big[\,r_\phi(x,y)\,\big]\;-\;\beta\,\mathbb{D}_{\mathrm{KL}}\!\big(\pi(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big)

RLHF as optimization against a proxy: maximize the reward-model score while a KL penalty keeps the policy near its reference.