in-context learning

In-context learning is the surprising knack a large model has for picking up a brand-new task from a few examples sitting in its prompt, without any training step. Show it a pattern, like converting dates from one format to another, and it just does the next one. Nothing inside the model changes; the lesson lives entirely in the words you fed it, and it is forgotten the moment that prompt is gone.

This is strange because we normally think learning means updating weights through gradient descent. Here the weights are frozen, yet the model behaves as if it learned. The leading explanation is that, during pretraining, predicting the next token across countless documents forced the model to get good at spotting and continuing patterns. A few examples in the prompt are just another pattern to continue. In-context learning is what makes prompting so powerful: you adapt the model to your task at the moment you ask, not in a training lab.

Its limits are worth remembering. The lesson must fit in the context window, it does not accumulate across separate conversations, and the model can latch onto a superficial pattern rather than the one you intended. For anything you need permanently and reliably, fine-tuning still has its place.