Simple linear regression

The model: y = a + b·x + error

In the last guide we stared at a scatter plot — a cloud of dots — and met the idea of a best-fit line drawn through it. This guide gives that line a job. Once we can write the line down as a formula, we can read a relationship straight off it, predict a new value, and judge how much to trust the whole thing. That formula is called a simple linear regression model: one predictor, one outcome, one straight line.

Here is a concrete scene. You are apartment-hunting and you jot down five listings, each with its size (in 坪, the Taiwanese unit of floor area, about 3.3 square metres) and its monthly rent. Bigger places clearly cost more — but how much more per 坪? A regression turns that vague feeling into one number you can actually use.

In words, the model says: the outcome equals a starting value, plus a constant amount for every unit of the predictor, plus an error term that absorbs everything the straight line cannot explain. Written as a formula:

The simple linear regression model, in symbols.

Read it piece by piece. The y is the outcome (or response) — the thing we want to explain or predict, here the rent. The x is the predictor (or explanatory variable) — here the size. β₀ (beta-nought) is the intercept: where the line crosses the vertical axis. β₁ (beta-one) is the slope: how steeply the line rises. And ε (epsilon) is the error — the honest admission that real points never sit exactly on any line.

The fitted line we estimate from the sample; the hats mark estimates.

That little ‘+ ε’ is the most honest symbol in the whole equation. It says out loud that the model is a simplification — rent also depends on location, building age, floor, light, and luck. A regression never claims to capture all of that. It claims only to capture the straight-line part of the story, and to be upfront about the rest.

Fitting and interpreting the slope (with units!)

How do we pick the two numbers — the slope and the intercept — that define our line? We let the data vote. For any candidate line, each data point sits some vertical distance above or below it; that gap is the point's residual. The method of ordinary least squares (OLS) chooses the single line that makes the total of the squared gaps as small as possible. Squaring keeps positive and negative gaps from cancelling, and punishes big misses far more than small ones.

The least-squares slope: it weighs how x and y move together against how much x spreads.

Run that recipe on our five apartments and the slope comes out to 1.5, in units of thousands of NT$ per 坪. Now interpret it — and never quote a slope without its units. It reads: each additional 坪 of floor space is associated with about NT$1,500 more rent per month, on average. A slope is a rate: it carries the units of y per one unit of x.

Notice the careful words ‘associated with’ and ‘on average’. The slope describes how rent and size move together in this data; it does not prove that bolting an extra 坪 onto an apartment would cause its rent to rise by NT$1,500. Bigger apartments differ from smaller ones in many ways at once. This is the famous warning that correlation is not causation — establishing real causation takes far more than a tilted line, and we return to it throughout this track.

One more honesty point. Our slope of 1.5 came from just five apartments; a different five would have given a slightly different number. The slope is an estimate, and every estimate carries uncertainty. Good regression output reports a standard error and a confidence interval for the slope, and often a p-value testing whether the slope could plausibly be zero (that is, no relationship at all). Treat the single number as the centre of a range, not a fact carved in stone.

The intercept and when it's meaningful

The intercept, β₀, is simply the predicted outcome when the predictor is zero — the height at which the line meets the vertical axis. In our fitted line the intercept is 0.7, i.e. about NT$700. Taken literally, that is the predicted rent of a zero-坪 apartment.

Which is, of course, nonsense — there is no zero-坪 apartment, and zero is nowhere near our data, which runs from 10 to 20 坪. This is the trap of extrapolation: a regression line is only trustworthy across the range of x you actually observed. Push it far outside that range and it will happily hand back confident, meaningless numbers.

The intercept is set so the line passes through the point of averages (x̄, ȳ).

So is the intercept ever meaningful? Yes — whenever x = 0 is real and sits inside your data. If x were daily advertising spend, then ‘zero spend’ is a genuine, interpretable case, and the intercept is the baseline sales with no ads at all. A common trick when zero is unnatural is to center the predictor — subtract its mean from every x — so the new intercept becomes the predicted y at the average x (here, the rent of a typical 15-坪 flat, about NT$23,200), which is far easier to read.

Residuals: what the line missed

A residual is the part of reality the line missed: the actual value minus the predicted value. It is the single most informative quantity in all of regression, because every later judgement — how good the fit is, whether the assumptions hold — is really just the residuals examined under different lights.

A residual: actual minus predicted, for each data point i.

Take the 12-坪 flat. Our line predicts 0.7 + 1.5 × 12 = 18.7, i.e. NT$18,700. It actually rents for 20 (NT$20,000). Its residual is +1.3: it costs about NT$1,300 more than the line expected — perhaps it is newly renovated, or steps from an MRT station. A positive residual sits above the line, a negative one below. Here are all five:

Every apartment's prediction and residual (rent in thousands of NT$); the residuals sum to zero.

Two facts make residuals special. First, by the very way least squares is built, the residuals always sum to (essentially) zero — the overpredictions and underpredictions balance out. Second, least squares is precisely the rule that makes the sum of their squares as small as possible; no other straight line beats it on this data by that measure. Keep the residual column in view — the next two sections do nothing but interrogate it.

R-squared: how much variation is explained

How good is the line? Here is the trick that defines R². Imagine you had no line at all and someone asked you to guess any apartment's rent. Your best single guess would be the overall mean rent, ȳ (here 23.2, about NT$23,200). The points still scatter around that flat mean line, and that total scatter is the variation you are trying to explain.

The regression line explains part of that scatter; what remains as residual scatter is the part it could not. The coefficient of determination (R², ‘R-squared’) is simply the fraction of the total variation that the line explains:

R-squared: one minus the leftover scatter divided by the total scatter.

Plugging in our numbers — total scatter SS_tot = 158.8 and leftover scatter SS_res = 5.8 — gives R² ≈ 0.96. In plain words: size alone explains about 96% of the variation in rent across these apartments. That is a strong fit. (R² always lands between 0 and 1; for a simple one-predictor regression it is exactly the square of the correlation coefficient.)

1. A high R² does not mean the model is correct. A badly curved relationship can still post a high R² while the straight line is plainly the wrong shape — always look at the residual plot (next section).
2. R² says nothing about causation. Size explaining 96% of rent's variation does not prove that size sets the rent.
3. A low R² is not automatically bad. Human behaviour is noisy; a real and useful effect can live happily inside an R² of 0.1.
4. R² never goes down when you add another predictor, so it is the wrong thing to chase in multiple regression — that road leads straight to overfitting.

Checking assumptions with residual plots

Every number above — the slope, the intercept, R², the standard errors — rests on a small set of assumptions. When they hold, the line and its uncertainty are trustworthy; when they fail, the very same numbers can quietly mislead. For simple linear regression the big assumptions, in plain words, are these:

1. Linearity — the true relationship really is roughly a straight line, not a curve in disguise.
2. Independence — the observations don't lean on one another (one apartment's rent isn't dictated by its neighbour's).
3. Constant variance, or homoscedasticity — the scatter around the line is about the same size everywhere, not tight on the left and fanning out on the right.
4. Roughly normal residuals — the leftover errors aren't wildly skewed or riddled with outliers. This one matters for the standard errors, confidence intervals and p-values, not for the slope estimate itself; the relevant bell shape is the normal distribution.

You check these mostly with one humble picture: the residual plot — residuals on the vertical axis against the fitted values on the horizontal. If the assumptions hold, you should see a shapeless, patternless cloud hovering around zero. The classic warning signs are a curve or smile (the relationship is non-linear), a funnel that widens to one side (non-constant variance), and a lone point far from the rest (an outlier or unusually influential observation).

This is the deepest reason the field insists you must plot your data. Anscombe's quartet — four little datasets with nearly identical means, variances, correlation and regression line, yet four utterly different shapes — is the cautionary tale every analyst learns. The summary numbers agreed; only the picture told the truth. Treat assumption-checking as part of exploratory data analysis, not a box-ticking afterthought.

A full worked example

Let's run the whole pipeline once, start to finish, on the five apartments — so you can do it yourself by hand. The data: sizes 10, 12, 15, 18, 20 (坪) and rents 15, 20, 22, 29, 30 (in thousands of NT$). Here is the procedure to follow every time:

1. Plot first. Make a scatter of rent against size and confirm the cloud looks roughly straight before you fit anything.
2. Find the centres. Compute the mean size x̄ and the mean rent ȳ.
3. Compute the slope from the deviations, using the OLS formula.
4. Compute the intercept so the line passes through the point of means (x̄, ȳ).
5. Write the equation and predict — but only within the observed range of x.
6. Form the residuals and look at their plot for curves, funnels, or outliers.
7. Report R² and, just as importantly, the slope's uncertainty (its standard error or confidence interval).
8. Interpret in plain language, with units, and with the correlation-is-not-causation caveat firmly attached.

Step 2: the centres of x and y.

Steps 3–4: slope first, then intercept.

The finished model for our five apartments.

Now predict the rent of a new 16-坪 flat: 0.7 + 1.5 × 16 = 24.7, i.e. about NT$24,700. Because 16 sits comfortably between our smallest (10) and largest (20) apartments, this is honest interpolation. Asking the same line about a 40-坪 luxury flat would be extrapolation — outside the data we ever saw, and not to be trusted.

import pandas as pd
import statsmodels.formula.api as smf

df = pd.DataFrame({
    "size_ping": [10, 12, 15, 18, 20],
    "rent_k":    [15, 20, 22, 29, 30],
})

fit = smf.ols("rent_k ~ size_ping", data=df).fit()
print(fit.params)      # Intercept 0.7,  size_ping 1.5
print(fit.rsquared)    # 0.963...
print(fit.conf_int())  # a confidence interval for each coefficient

From here the path forks in two natural directions. Real outcomes usually depend on more than one thing at once — that is multiple regression, where we also meet multicollinearity and the subtle phrase ‘holding everything else constant’. And once a model has many knobs to turn, it can start memorising noise instead of learning the pattern — the problem of overfitting, kept in check by cross-validation. Both build directly on the single line you just learned to read and to trust.