weak-to-strong generalization

Can a weak teacher train a student who ends up smarter than the teacher's own flawed lessons? Picture a bright pupil taught by a mediocre instructor: if the pupil truly grasps the subject, they can get the gist, see past the teacher's occasional mistakes, and end up understanding the material better than the teacher ever explained it. Weak-to-strong generalization asks whether AI models can do the same, learning to perform better than the imperfect labels they were trained on.

Precisely, in a 2023 OpenAI study researchers used a weak model's labels as a stand-in for the limited, error-prone supervision a human could give a future superhuman system, and fine-tuned a much stronger pretrained model on those weak labels. The question was whether the strong student would merely copy the weak teacher's mistakes, or would instead recover its own latent capability and exceed the teacher. They found that strong students often do recover much of their underlying ability from weak supervision, but not all of it, leaving a measurable performance gap.

It matters as an empirical stand-in for the superalignment problem: if humans will be weak supervisors of superhuman AI, can weak human feedback still elicit the strong system's full, aligned capability? The honest caveats run deep. It is an analogy, not the real situation, weak models make different kinds of mistakes than humans do. And recovering capability is not the same as recovering aligned values: a strong model generalizing past weak labels could just as easily generalize toward the wrong goal.