tied input-output embeddings

A language model needs to do two mirror-image jobs. At the input it turns each token into a vector, and at the very end it turns its final vector back into a score for every token in the vocabulary so it can predict the next one. Both jobs involve a table whose shape is vocabulary by dimension. Weight tying simply reuses the same matrix for both, instead of learning two separate ones.

This is natural because both tasks really concern the relationship between tokens and the same vector space. Sharing the weights cuts a large number of parameters — for big vocabularies the embedding table is enormous — and in practice often improves quality by keeping the two halves consistent. The output step uses the transpose of the input embedding matrix to produce a score, called a logit, for each token, which softmax then turns into probabilities. Not every model ties its embeddings, but many do.