weight tying and scaling

Weight tying is a thrifty trick at the model's two ends. The input embedding turns each token id into a vector; the output layer turns a vector back into scores over the vocabulary. These two are mirror images, so many models share one matrix for both, halving a large chunk of parameters and often improving quality because the same word lives at the same point going in and coming out. For very large vocabularies, though, some recent models leave them untied.

Scaling is the broader question of how to spend a fixed parameter budget: more layers (depth) or wider layers (width), and how those should grow with model size. Scaling laws give empirical guidance, showing loss falling predictably as parameters, data, and compute grow together, and warning that a bigger model starved of data is wasteful. The shape of the network, its width-to-depth ratio and head count, is tuned alongside these laws so the model trains stably and uses its budget well.