JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Meaning as Geometry: Static vs Contextual Vectors

Once words are vectors, meaning becomes distance and direction. We will see the famous king − man + woman trick, why a single fixed vector per word is not enough, and how attention upgrades static embeddings into context-aware ones.

A space where closeness means similarity

When every token is a list of, say, 4096 numbers, you can picture each one as a point in a 4096-dimensional space — the semantic vector space. We cannot visualize 4096 dimensions, but the intuition from 2D holds: points that are close together are similar, points far apart are unrelated. Because training pushes tokens used in similar ways toward similar vectors, this space organizes itself by meaning. ` cat`, ` dog`, and ` hamster` form a little neighbourhood of pets; ` Paris`, ` Tokyo`, ` Berlin` cluster as capitals, far from the pets.

\cos\theta=\frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}

Cosine similarity: closeness is the angle between two vectors, not their raw length.

The king − man + woman trick

The idea that words could be useful vectors was made famous by word2vec back in 2013. Its showstopper: not only are similar words close, but directions carry meaning too. Take the vector for king, subtract man, add woman — and the nearest token to the result is queen. The "male → female" change and the "singular → plural" change each correspond to a consistent direction you can add or subtract. Meaning had become arithmetic.

vec("king") - vec("man") + vec("woman")  ~=  vec("queen")
vec("Paris") - vec("France") + vec("Japan")  ~=  vec("Tokyo")
Word analogies fall out of vector arithmetic — meaning lives in directions, not just positions.
Word vectors put meaning in directions — king minus man plus woman lands near queen.

Word2vec analogy diagram showing king minus man plus woman approximately equals queen.

This is the same kind of embedding idea that lives in the embedding matrix from guide 3 — just demonstrated on its own. It is genuinely how the input layer of a language model starts to encode meaning.

The fatal flaw of one vector per word

But there is a problem hiding in plain sight. The embedding table gives each token exactly one fixed vector. These are static embeddings: bank always maps to the same point, whether you mean the bank of a river or a bank that holds your money. The poor vector has to be a blurry average of both senses. Apple the fruit and Apple the company collapse into one. For real language, where words shift meaning with context, a single frozen vector per word is simply not enough.

Contextual embeddings: let the neighbours vote

Modern LLMs fix this with contextual embeddings. The static vector from the table is only the starting point. As it passes up through the transformer, each token's vector is repeatedly updated by looking at the other tokens in the sentence — a process called context mixing, powered by self-attention. By the upper layers, the vector for bank in "sat on the river bank" and the vector for bank in "deposit at the bank" have moved to two completely different points in space. Same starting token, two different meanings — finally.

Click a word to watch attention let the neighbours vote, turning a static vector into a contextual one.

Interactive self-attention widget highlighting which other words a clicked word attends to.

So the embedding layer you met in guide 3 is not the final word on meaning — it is the seed. Static embeddings get a token onto the map; contextual embeddings let the rest of the sentence drag it to exactly the right spot. That hand-off is the bridge from this track into how transformers actually work.