contextual vs. static embeddings

The word "bank" means one thing in "river bank" and another in "savings bank", yet older methods such as word2vec and GloVe gave it a single fixed vector no matter the sentence. Those are static embeddings: one vector per word, looked up the same way every time. Contextual embeddings, produced by transformer models, instead give each token a vector that depends on the whole sentence around it, so "bank" carries different numbers in each context.

Inside an LLM the token embedding is only the starting point. As that vector passes up through the layers, attention mixes in information from the surrounding tokens, so by the upper layers each position's representation reflects its context. This is why modern models can tell apart word senses that static vectors hopelessly blur. The trade-off is cost: static vectors are cheap and can be precomputed once, while contextual ones must be recomputed for every new sentence.