embeddings as low-rank geometry

An embedding turns discrete things, words, users, graph nodes, into points in a continuous space so that geometry encodes meaning: nearby points are similar, and directions can carry relationships. Underneath, this is linear algebra, taking a huge sparse relationship table and squeezing it into a few dense dimensions.

The recurring recipe is low-rank factorization. Build a matrix M of co-occurrence or similarity statistics (which words appear near which, which users touched which items), then factor M approx W C^T with thin factors. Each row of W is an object's embedding vector, and the inner product <w_i, c_j> reconstructs the statistic, so similarity in the data becomes an inner product in the embedding space.

Many famous embeddings are exactly this in disguise. Word2vec's skip-gram with negative sampling provably factorizes a shifted pointwise-mutual-information matrix; GloVe factorizes log co-occurrence counts; classic latent semantic analysis is just a truncated SVD of a term-document matrix. The chosen rank is the embedding dimension, and the SVD again gives the best low-rank fit.

Why it matters: embeddings are the input layer of modern NLP, recommendation, and graph learning, and their geometry supports analogies and nearest-neighbor search. The caveats: a linear inner-product model captures only second-order, pairwise structure (modern deep models add nonlinearity and context), and embeddings faithfully absorb whatever biases live in the co-occurrence data.