semantic vector space

Once every token or word is a vector, you can think of them all as points scattered through a high-dimensional space. The remarkable thing learned from training is that this space has geometry: points for similar meanings end up close together, and directions can carry meaning. "King" and "queen" land near each other, and the step from "king" to "queen" can resemble the step from "man" to "woman".

Closeness here is usually measured by cosine similarity, the angle between two vectors, rather than plain straight-line distance. This geometry is never designed; it emerges because the model is rewarded for predicting words from their neighbours, so words that share contexts get pulled together. It is the reason embeddings can power search, clustering, and analogy. But the structure is approximate and full of quirks — a useful map of meaning, not a perfect one.