Multiple regression & interpreting coefficients

Adding predictors: why and how

In the last two guides you fit a line through a single predictor: one input, like a house’s floor area, and one outcome, its price. That single line is powerful, but the real world is rarely so tidy. Two houses of exactly the same size can sell for very different prices because one is brand new and the other is fifty years old, or because one sits next to a metro station and the other is a long walk away. Floor area alone simply cannot see those differences.

Multiple regression is the natural fix: instead of one predictor, we let several predictors explain the outcome at the same time. The recipe is exactly the one you already know — least squares still chooses the numbers that make the prediction errors as small as possible — only now there is one number (a coefficient) for each predictor.

Written out, the model says the outcome y is a baseline value plus a separate contribution from each predictor, plus an error term for everything the model doesn’t capture:

The multiple regression model: a baseline β₀ plus one coefficient β times each predictor x, plus an error term ε for the rest.

Here each x is a predictor, β₀ (read “beta-zero”) is the intercept or baseline, and each β is the coefficient for one predictor. For a concrete feel, suppose we fit a model for apartment prices (in thousands of dollars) from floor area (in square meters), age (in years), and whether the unit is near a metro station (a 0/1 flag). Least squares might return:

A worked apartment model we will use throughout this guide. The little hat on “price” marks it as a predicted value.

We will live inside this one example for the rest of the guide. Notice we now get richer predictions and — just as importantly — each coefficient becomes a cleaner statement about one factor, precisely because the other factors are in the model too. That second benefit is the whole point of the next section. The deep-dive term for all of this is multiple regression.

“Holding other variables constant”

Here is the single most important sentence in this entire guide: in a multiple regression, each coefficient tells you the effect of its predictor while all the other predictors are held fixed. Statisticians call this the ceteris-paribus interpretation — Latin for “all else equal.”

Take the age coefficient, −1.5. It does not say “older apartments are cheaper” in some vague overall way. It says something sharp: among apartments of the same floor area and the same station status, each additional year of age is associated with a price about 1.5 thousand dollars lower. The model has, in effect, lined up apartments that match on size and location, and asked what age alone does within that matched set.

More formally, holding the other predictors fixed, a one-unit increase in a predictor x_j changes the predicted outcome by exactly its coefficient β_j:

Each coefficient is the change in the prediction for a one-unit change in its own predictor, with the others held still.

Why does this matter so much? Because a coefficient from a simple one-predictor regression is a blend. If newer apartments also tend to be bigger, a size-only model would quietly credit some of size’s effect to newness, and vice versa. By putting both in the model, multiple regression statistically “removes” the part of each predictor that the others can explain, and reports only the unique, leftover contribution. This is what people mean when they say a result is “controlling for” age, or “adjusted for” location.

Categorical predictors: dummy variables

So far every predictor has been a number you can multiply by a coefficient. But many of the most useful predictors are categories, not numbers: neighborhood, apartment type, brand, yes/no flags. You cannot literally multiply a coefficient by the word “downtown.” You first have to turn the category into numbers.

The standard trick is a dummy variable (also called an indicator variable): a column that is 1 when a condition holds and 0 otherwise. Our near_station predictor is already a dummy — 1 for near, 0 for far. Its coefficient, +8, reads as: holding area and age fixed, an apartment near a station is predicted to cost about 8 thousand dollars more than an otherwise-identical one that isn’t.

What about a category with more than two levels, such as neighborhood A, B, or C? You make one dummy per level but deliberately leave one out as the reference (baseline) category. With A as the reference, you create two columns, “is B?” and “is C?”. A coefficient of +12 on the B dummy then means: neighborhood B is predicted to be 12 thousand dollars more expensive than neighborhood A, holding everything else fixed. Every category is read as a comparison to the one you dropped.

1. List the categories of the variable (for example: A, B, C).
2. Pick one as the reference category — often the most common level or a natural baseline.
3. Create a 0/1 dummy column for each of the remaining categories.
4. Read each dummy’s coefficient as the difference from the reference, holding all other predictors fixed.

import pandas as pd

# turn the 'neighborhood' column (A, B, C) into 0/1 dummy columns,
# dropping the first level (A) as the reference category
dummies = pd.get_dummies(df['neighborhood'], prefix='nb', drop_first=True)

# -> new columns: nb_B, nb_C   (neighborhood A is the baseline)

Interaction effects

In the model so far, every coefficient is a fixed number: each extra square meter is worth +2.5, full stop, whether the apartment is downtown or in the suburbs. But that assumption is often wrong. Space near a busy station may command a higher premium per square meter than space far away. When the effect of one predictor depends on the level of another, we say the two predictors interact.

We capture an interaction by adding a new predictor that is the product of the two originals — here, area multiplied by the near_station flag:

An interaction term is just the product of two predictors, given its own coefficient β₃.

Now the slope on area is no longer a single number. When near_station = 0, the product term vanishes and each square meter is worth β₁. When near_station = 1, the product term switches on and each square meter is worth β₁ + β₃. So the effect of area is itself a small formula:

With an interaction, the “effect of area” depends on the station flag — it is two different slopes, not one.

With real numbers, suppose β₁ = 2.0 and β₃ = 1.0. Then far from a station each square meter adds 2.0 thousand dollars; near a station it adds 2.0 + 1.0 = 3.0. The interaction coefficient β₃ is exactly that gap — how much more (or less) the area effect becomes when you switch the station flag on.

Multicollinearity: when predictors overlap

Multiple regression shines when each predictor brings its own information. It struggles when two or more predictors carry nearly the same information — when they are highly correlated with each other. This condition is called multicollinearity, and it is one of the quietest ways a regression can mislead you.

Picture the extreme case. Suppose you accidentally include floor area measured in square meters and the very same area measured in ping (a Taiwanese unit, about 3.3 square meters). The two columns are perfectly redundant — one is just the other times a constant. The model literally cannot decide how to split the credit between them: it could put all the weight on one, all on the other, or any mix in between, and fit equally well. Software either refuses outright or returns wild, meaningless coefficients. Everyday versions are subtler: number of bedrooms and floor area tend to rise together, so they partly step on each other.

The tell-tale symptoms: standard errors balloon, so confidence intervals become very wide; coefficients show up with surprising or flipped signs; and the estimates swing dramatically when you add or drop a single variable. All of these are the model honestly admitting, “I can’t tell which of these overlapping predictors deserves the credit.” A common diagnostic is the variance inflation factor (VIF), a number computed per predictor that grows large when that predictor is well explained by the others.

Reading a regression table like a pro

When you fit a model, your software prints a table — and to a beginner it looks like a wall of numbers. Let’s fit our apartment model and then decode every column, one at a time.

import statsmodels.formula.api as smf

model = smf.ols('price ~ area + age + near_station', data=df).fit()
print(model.summary())

term            coef    std err      t     p>|t|         95% CI
intercept      30.20     8.10     3.73    0.000    [ 14.3,  46.1]
area            2.48     0.21    11.81    0.000    [ 2.07,   2.89]
age            -1.52     0.34    -4.47    0.000    [-2.19,  -0.85]
near_station    7.90     2.60     3.04    0.003    [ 2.80,  13.00]

R-squared = 0.78     Adjusted R-squared = 0.77     n = 240

Column by column: coef is the estimated coefficient — the partial effect we have been interpreting all along. std err (the standard error) measures how much that estimate would wobble from sample to sample; smaller means more certain. The t column is simply coef divided by its standard error — how many standard errors the estimate sits away from zero.

The p>|t| column is the p-value for the hypothesis that the true coefficient is exactly zero (that the predictor has no effect once the others are accounted for). A small p-value (here, well under 0.05) says the data would be surprising if that coefficient were really zero. Be careful, though: a p-value is not the probability that the coefficient is zero, and it says nothing about how big or important the effect is — only how compatible the data are with a zero effect. The 95% CI is a confidence interval: a range of plausible values for the true coefficient. The honest reading is about the procedure — if we repeated the whole study many times, about 95% of the intervals built this way would contain the true value — not “a 95% chance the truth is in this particular range.”

Finally, the bottom line. R-squared (the R²) is the fraction of the variation in price the model explains — 0.78 means 78%. But R² never goes down when you add a predictor, even a useless random one, so it quietly rewards bloated models. Adjusted R-squared fixes this by penalizing each extra predictor; if adjusted R² barely moves (or falls) when you add a variable, that variable is probably not earning its place. And n is just the number of observations the model was fit on.

Interpretation traps to avoid

You can now read a multiple regression. The last, most valuable skill is knowing how it can fool you. Here are the traps that catch even experienced analysts.

Closely related is omitted-variable bias: leaving out a variable that affects the outcome and is correlated with a predictor you kept will distort that predictor’s coefficient. This is the flip side of “controlling for” — control for too little, and your numbers are biased.

Do not read a bigger coefficient as a more important predictor. Coefficients carry the units of their predictors: a coefficient of 2.5 per square meter and a coefficient of 8 per station-flag live on completely different scales, so their raw sizes are not comparable. To compare importance fairly, put predictors on a common footing first — for example by standardizing them to the same scale before fitting.

Two more, quickly. Statistical significance is not practical importance: with a large enough sample, a trivially small effect can have a tiny p-value, while a big, business-relevant effect in a small sample may not reach significance — always look at the size of the coefficient and its confidence interval, not just the asterisks. And beware extrapolation: a model fit on apartments of 20–120 square meters tells you little about a 500-square-meter penthouse. Trust predictions inside the range of your data, and be very cautious outside it.

1. State the question first: are you predicting, or trying to interpret one coefficient? Your goal changes what matters.
2. Interpret each coefficient as a partial effect — “holding the other predictors fixed.”
3. Check for multicollinearity before trusting any individual coefficient.
4. Plot the residuals; confirm the model’s shape and assumptions look reasonable.
5. Report coefficients with their confidence intervals, not p-values alone.
6. Resist causal language unless your design — not just your control variables — earns it.

That is multiple regression: many predictors at once, each coefficient a careful “all else equal” statement, with dummies for categories, interactions for effects that change, and a healthy suspicion of overlap and overreach. Next in this track, logistic regression adapts these same ideas to yes/no outcomes, and the guide on overfitting and regularization shows how to keep a model with many predictors honest. The interpretive discipline you built here carries into all of it.