Two variables, one scatter plot
Imagine you are a teacher with a simple question: does studying more actually go together with higher exam scores? You quietly record, for five students, how many hours each spent studying and the score each earned. That is the shape of almost every regression problem — you have pairs of numbers, two measurements on the same thing, and you want to know how they move together.
Each of the two columns is a variable — something that changes from one student to the next. Here the two variables are hours studied and exam score. By convention we call the variable we use to explain or predict the explanatory variable (or predictor) and write it x; the variable we are trying to understand or predict we call the response (or outcome) and write it y. Choosing which is x and which is y is a decision about the question you are asking — here it is natural to ask how score (y) depends on hours (x).
hours = [2, 4, 6, 8, 10] # hours studied before the exam score = [55, 65, 70, 80, 85] # exam score (out of 100) # five students, five (x, y) pairs
The first thing to do — always — is to draw the data, not summarize it. The natural picture for two numerical variables is a scatter plot: a flat space whose horizontal axis is x (hours) and whose vertical axis is y (score). Each student becomes a single dot, placed at their (hours, score). Five students, five dots. Stand back, and the dots usually form a cloud whose shape is the whole story.
An interactive scatter plot of five points, hours studied on the x-axis and exam score on the y-axis, rising from lower-left to upper-right, with a straight line you can drag to fit through them.
Covariance and the correlation coefficient
Our eyes say the cloud tilts upward, but eyes disagree with one another and cannot be written into a report. We want a single number that captures how two variables move together. The starting idea is co-movement: when a student is above average in hours, are they also above average in score?
First we need each variable’s center. The mean (the ordinary average — add the values, divide by how many there are) is the usual choice. For our five students the mean of hours is (2 + 4 + 6 + 8 + 10) / 5 = 6, written x̄ (read ‘x-bar’), and the mean of scores is (55 + 65 + 70 + 80 + 85) / 5 = 71, written ȳ.
Now measure each student as a deviation from the mean — how far above or below average they are. The student with 2 hours is 2 − 6 = −4 hours below average; their score 55 is 55 − 71 = −16 points below average. Multiply those two deviations: (−4) × (−16) = +64. The sign is the magic. If a student is below average in both, two negatives multiply to a positive; if above in both, two positives also give a positive; only when a student is above in one and below in the other do you get a negative product. So the product is positive whenever the two variables agree in direction.
Covariance, in words: the average — over all data points — of (x’s deviation) times (y’s deviation).
For our data the five products are 64, 12, 0, 18 and 56; their sum is 150, so the covariance is 150 / 5 = 30. It is positive, which confirms in a number what our eyes saw: the upward tilt.
The fix is to strip out the units. We divide the covariance by the standard deviation of x and by the standard deviation of y — each standard deviation being the typical distance of a variable from its own mean, carrying that variable’s units. Dividing by both cancels the units entirely. The result is the correlation coefficient, written r: a pure, unit-free number that, as we will see, is trapped between −1 and +1.
The correlation coefficient r is just the covariance, standardized by both standard deviations so the units cancel.
Running the numbers for our students gives r ≈ 0.99 — an extremely tight upward relationship.
Correlation’s range — and what it misses
The correlation coefficient always lands between −1 and +1, and that range is its whole vocabulary. A value near +1 means the points sit almost perfectly on an upward-sloping line; near −1, almost perfectly on a downward-sloping line; near 0, no straight-line pattern at all. The sign is the direction; the distance from zero is the strength.
As a rough reading: an |r| around 0.1 is a barely-there whisper of a trend, around 0.3 a modest one, around 0.5 a moderate one, and above 0.8 a strong one. But these are rules of thumb, not laws — what counts as ‘strong’ depends entirely on your field. In physics 0.9 might be disappointing; in social science 0.4 might be a celebrated finding.
Here is the trap that catches everyone. The correlation coefficient measures only the strength of a straight-line relationship. A relationship can be powerful, obvious and perfectly real, yet have r ≈ 0. Picture how alertness depends on coffee: a cup or two lifts you, but ten cups leave you jittery and useless. Plotted, that is an upside-down U — a strong relationship — and yet its rising and falling halves cancel, leaving r near zero. So ‘r = 0’ means ‘no linear relationship,’ never ‘no relationship.’
Correlation is also fragile against outliers. A single bizarre point — one student who studied 20 hours but panicked and scored 10 — can drag r dramatically up or down, making a weak relationship look strong or hiding a strong one. The number cannot tell you a rogue point is lurking; only a picture can. The most famous demonstration of all this is Anscombe’s quartet, a set of four datasets built to share the same summary numbers while looking utterly different.
Four small scatter plots in a grid; all share the same means, the same correlation and the same best-fit line, yet one is a tidy rising line, one a smooth curve, one a line dragged off course by a single high outlier, and one a vertical stack of points with one far-off point.
What makes a line ‘best’?
Correlation gives us a verdict on strength and direction, but it does not give us a tool. We usually want more: a single line that summarizes the cloud, lets us read off a typical y for any x, and lets us predict. Through a cloud of points you could draw infinitely many lines. Which one deserves to be called best?
To compare lines we need a way to score them. Pick any candidate line. For each student, the line offers a prediction — the height of the line directly above that student’s hours, written ŷ (read ‘y-hat’). The student’s actual score is y. The gap between them, the residual = actual − predicted, is what the line got wrong for that student.
The residual for student i: their actual score minus what the line (with intercept a and slope b) predicted.
We measure that gap vertically, because the line’s job is to predict y from x, so a vertical gap is exactly the prediction error in y. A good line makes these residuals small. But we cannot simply add them up: a line can sit dead-center with huge errors that happen to cancel — large positives above, large negatives below, summing to zero. We need a measure of total error that never lets positives and negatives cancel.
Two honest fixes exist: add the absolute values of the residuals, or add their squares. Squaring wins for three reasons. It makes every error positive, so nothing cancels. It punishes big misses far more than small ones — a residual of 4 contributes 16, while a residual of 2 contributes only 4 — so the line is pulled hard toward avoiding gross errors. And squares are smooth, which (with a little calculus) yields one clean, exact formula for the best line instead of a messy search. Adding absolute values is a genuine alternative, but squares are the classic choice, and the method built on them is called least squares.
A cloud of scattered points with a single straight line drawn through the middle so the points are balanced above and below it.
Least squares: minimizing squared residuals
Now we can state the goal in one sentence. Add up the squares of all the residuals to get the sum of squared residuals (often written SSR). Every candidate line produces its own SSR. The line of best fit is the one line, out of all possible lines, that makes the SSR as small as it can possibly be. Fitting a line this way is called ordinary least squares, or OLS.
The sum of squared residuals. Least squares searches over all intercepts a and slopes b for the pair that makes this total smallest.
The wonderful thing is that you do not have to try millions of lines. Because squares are smooth, calculus hands us an exact answer — and you have already computed its ingredients. The best slope b is the very numerator and denominator we met for covariance: the sum of the deviation-products on top, divided by the sum of squared x-deviations on the bottom. The best intercept a is then whatever makes the line pass through the cloud’s center of mass, the point (x̄, ȳ).
The least-squares slope and intercept. Notice the slope’s top is the same sum (150) we used for covariance.
Recall our sums: the deviation-products added to 150, and the squared x-deviations (16 + 4 + 0 + 4 + 16) add to 40. So the slope is b = 150 / 40 = 3.75, and the intercept is a = ȳ − b·x̄ = 71 − 3.75 × 6 = 48.5. The least-squares line for our five students is therefore:
Predicted score = 48.5 + 3.75 × (hours studied).
The slope is the correlation rescaled by how spread out y is relative to x.
- Compute the two means, x̄ and ȳ.
- For each point, find its deviations (xᵢ − x̄) and (yᵢ − ȳ).
- Multiply the two deviations at each point and add the products — that is the slope’s numerator.
- Square each x-deviation and add them — that is the slope’s denominator.
- Divide to get the slope b; then set a = ȳ − b·x̄ to get the intercept.
- Write the line ŷ = a + b·x, and sanity-check that it passes through (x̄, ȳ).
import numpy as np hours = np.array([2, 4, 6, 8, 10]) score = np.array([55, 65, 70, 80, 85]) # slope (b) and intercept (a) of the least-squares line b, a = np.polyfit(hours, score, deg=1) print(round(b, 2), round(a, 2)) # -> 3.75 48.5 # the correlation coefficient r r = np.corrcoef(hours, score)[0, 1] print(round(r, 3)) # -> 0.993
A scatter of points with a fitted line through them; short vertical segments connect each point to the line, marking the residuals — the gaps the line failed to capture.
Reading the slope and intercept
A fitted line is only useful if you can read it. The slope b = 3.75 is the engine of interpretation: it says that comparing students who differ by one hour of study, the predicted score differs by about 3.75 points. A slope is always a rate — a change in y per one-unit change in x — and it always carries units, here points per hour. Double the x-step and you double the predicted change: two extra hours buy about 7.5 predicted points. This number, the slope, is what people mean when they speak of a regression coefficient.
The intercept a = 48.5 is where the line crosses the vertical axis — the predicted score for x = 0, a student who studied zero hours. Sometimes that is meaningful, sometimes nonsense. Read it only if x = 0 is realistic and sits inside the range of your data. Our smallest observed x is 2 hours, so x = 0 is outside what we measured; here 48.5 is best treated as a number that positions the line, not as a trustworthy prediction for a student who never opened a book.
Prediction is just plugging in. For a student who studied 7 hours, the line predicts 48.5 + 3.75 × 7 = 74.75 points. Trust such a prediction only within the range you actually observed — here 2 to 10 hours. Stretching the line far beyond the data, called extrapolation, is where confident predictions go to die: the straight-line pattern that held from 2 to 10 hours need not continue at 30 hours, where exhaustion might flatten the curve or even bend it back down.
Everything here is the skeleton of linear regression, the workhorse of the next guide. There we will measure how much of the variation the line actually explains (a quantity called R-squared), attach honest uncertainty to the slope, and check the assumptions a line quietly makes about your data.
Correlation is not causation (a preview)
We end on the single most important sentence in this whole track: a correlation, however strong, and a slope, however steep, do not by themselves prove that x causes y. Our data shows score rising with hours, but the numbers alone cannot tell us whether studying lifts scores, or whether already-strong students simply choose to study more.
The classic illustration: across a summer, ice-cream sales and drowning deaths rise and fall together, a strong positive correlation. Ice cream does not drown anyone. A third variable — hot weather — pushes both up at once. A variable that secretly drives both x and y, and so manufactures a correlation between them, is called a confounder, and it is the reason that correlation is not causation.
Whenever you see x and y correlated, at least four stories could explain it: x really causes y; y actually causes x (reverse causation); some hidden third thing causes both (confounding); or it is pure coincidence in a small or cherry-picked sample. Regression, by itself, cannot tell these apart — it faithfully measures association and stays silent on cause.
You now own the foundation of regression: a scatter plot to see the relationship, a correlation coefficient to measure its strength and direction, residuals to define what ‘wrong’ means, and least squares to pin down the one best line — plus the honesty to read its slope and intercept without overclaiming. Next we build the full machinery of simple linear regression on top of exactly these ideas.