maximum likelihood estimation

Among all the values a parameter could possibly take, which one makes the data you actually observed the most probable? That single question is the whole idea of maximum likelihood. It formalizes the intuition that the best explanation of what you saw is the one under which what you saw was the least surprising, and it turns fitting a model into a well-defined optimization.

Given data and a model with parameters theta that assigns a probability (or probability density) to any possible dataset, the likelihood L(theta) is that probability read as a function of theta with the observed data held fixed. Maximum likelihood picks the theta_hat that maximizes L, which in practice means maximizing the log-likelihood ln L, a sum instead of a product and far friendlier numerically, by solving d(ln L)/d theta = 0. The sharpness of the peak sets the uncertainty: the variance is approximately the inverse of -d^2(ln L)/d theta^2 (the Fisher information), so a tightly peaked likelihood means a precisely determined parameter and a flat one means a poorly constrained one.

This is the most general and widely used estimation principle in physics, and least-squares fitting is exactly its special case for independent Gaussian errors, since maximizing exp(-chi^2 / 2) is the same as minimizing chi^2. It underlies particle-physics fits, neutrino-oscillation analyses, and cosmological parameter extraction. The honest caveat: the likelihood is the probability of the data given the parameter, not the probability of the parameter given the data; converting it into a probability statement about theta requires Bayes' theorem and a prior. And MLE's nice properties, such as being unbiased, hold only asymptotically, so it can be biased or badly behaved for small samples or near a physical boundary.