total least squares
Ordinary least squares quietly assumes one thing: that your inputs (the x's, the columns of A) are exact and only the outputs (b) carry noise. It measures error as the VERTICAL gap from each point to the line. But what if the x's are noisy too — both coordinates of every measured point are uncertain? Then minimizing only vertical distance is biased. Total least squares fixes this by minimizing the PERPENDICULAR distance from each point to the fitted line, treating errors in the inputs and outputs on an equal footing.
Where ordinary least squares solves A x = b allowing only b to be adjusted, total least squares allows BOTH A and b to be perturbed, seeking the smallest total correction (in Frobenius norm) to the data matrix [A | b] that makes the system exactly consistent. The beautiful result is that this is solved by the SVD: form the augmented matrix [A | b], take its SVD, and the solution comes from the singular vector belonging to the SMALLEST singular value — the direction of least variance, which is the noise direction you collapse. Geometrically, for a line fit, total least squares is exactly orthogonal regression: it minimizes the sum of squared perpendicular distances rather than vertical ones.
When does it matter? Whenever the predictors are themselves measured with error — calibration where both instruments are imperfect, fitting a line to a noisy point cloud, errors-in-variables models in econometrics and geometry (like fitting a plane to scanned 3D points). The honest caveats: total least squares assumes the noise in all variables is comparable in scale (otherwise you must rescale first), it amplifies noise more than ordinary least squares and so can be less stable, and if you genuinely know the inputs are exact, ordinary least squares is the correct, lower-variance choice. Use total least squares precisely when the data MATRIX, not just the right-hand side, is noisy.
Fitting a line through a tilted noisy cloud of points where both x and y are measured imperfectly: ordinary least squares (vertical distances) tilts the line toward flatter, biased by the x-noise; total least squares (perpendicular distances) follows the cloud's true axis — the direction of its largest singular vector, with the line's normal being the smallest singular vector of [A | b].
Perpendicular distances, not vertical ones — the right fit when both coordinates are noisy.
Total least squares is the right model only when the inputs are genuinely noisy; if the columns of A are known exactly, it adds variance for no benefit and ordinary least squares is better. It also assumes comparable noise scales across all variables — otherwise you must weight or rescale first.