the best mean-square predictor

Suppose you must predict X from information Y, and you will be scored by the average squared size of your error — big misses hurt much more than small ones. Among all conceivable prediction rules, which one minimizes that expected squared error? The clean and complete answer is: predict with the conditional expectation. E[X given Y] is the single best mean-square predictor of X based on Y, full stop.

Precisely, over all functions g, the choice g(Y) = E[X given Y] minimizes E[ (X - g(Y))^2 ]. The reason is the geometry of the L^2 projection: E[X given Y] is the orthogonal projection of X onto the space of functions of Y, hence the closest such function in mean-square distance. Any other guess h(Y) pays a strictly larger error, and by Pythagoras you can see exactly how much more: E[ (X - h(Y))^2 ] = E[ (X - E[X given Y])^2 ] + E[ (E[X given Y] - h(Y))^2 ]. The first term is the irreducible error you can never beat (its size is E[ Var(X given Y) ] by the law of total variance); the second term is the avoidable penalty for choosing the wrong rule, and it vanishes only when h equals the conditional expectation.

This single fact is why conditional expectation is THE notion of 'optimal estimate' throughout statistics, signal processing, filtering, and machine learning — a regression model is literally an attempt to approximate E[X given Y]. Two honest caveats. First, 'best' here means under squared-error loss; change the loss and the optimum changes — the best predictor under absolute-error loss is the conditional median, not the conditional mean. Second, the famous linear least-squares predictor (the regression line) is only the best AMONG LINEAR rules; the true best predictor E[X given Y] can be curved, and they coincide only in special cases such as the bivariate normal.