maximum likelihood estimation

You have data and a family of candidate models indexed by an unknown parameter, and you want to pick the best one. Maximum likelihood estimation makes the most natural choice: pick the parameter value under which the data you actually observed would have been most probable. It is the principle of 'the explanation that makes the evidence least surprising', and it is the dominant estimation method in frequentist statistics and a backbone of machine learning.

Here is the mechanism. Write the likelihood L(theta) = p(data given theta) — the probability of your observed data viewed as a function of the parameter theta. The maximum likelihood estimate is the theta that maximizes L. In practice you almost always maximize the log-likelihood ln L(theta) instead, because the data are usually independent so the likelihood is a product and the log turns it into a sum (sum from i=1 to n of ln p(x_i given theta)), which is far easier to differentiate. You then set the derivative to zero and solve. For example, with n independent Bernoulli trials giving s successes, maximizing the log-likelihood gives the wonderfully intuitive answer theta-hat = s/n, the sample proportion; for normal data the MLE of the mean is just the sample average.

MLE earns its dominance through good large-sample behavior: under mild conditions it is consistent (it homes in on the true value as n grows), asymptotically normal, and asymptotically efficient (no competitor has smaller variance in the limit). It also connects cleanly to the Bayesian world — the MLE is the posterior mode under a flat prior. But be honest about its limits: in small samples MLE can be biased (the MLE of a normal variance divides by n, not n - 1, and is too small on average), it can overfit when the model is flexible, and its optimism rests entirely on the model being correctly specified — maximize the likelihood of the wrong model and you get the best wrong answer.