Logistic regression & classification

Why a straight line fails for yes/no

So far in this track, regression has predicted a number: a price, a temperature, a test score. The fitted line answered the question “how much?” But a huge share of real questions are not “how much?” at all — they are “yes or no?” Will this customer cancel? Is this email spam? Does this patient have the disease? Did the student pass? The answer is one of just two labels, and our job is to predict that label from the data.

Let's make it concrete. Imagine 200 students. For each one we record how many hours they studied and whether they passed the final exam — we write 1 for pass and 0 for fail. If we plot hours on the horizontal axis and the pass/fail flag on the vertical axis, every point sits at one of just two heights: a row of dots along the bottom at 0 (failed) and a row along the top at 1 (passed). There is no such thing as “1.7 passes.”

Your first instinct, fresh from linear regression, might be to draw the ordinary least squares straight line through this cloud. It sort of works in the middle — students who studied more did pass more often — but two things break badly. First, the line never stops at the edges: extend it far enough to the right and it predicts a “probability of passing” of 1.3; far enough to the left and it predicts −0.2. Neither is a real probability. Second, the line insists that each extra hour adds the same amount of pass-probability everywhere, when common sense says the jump from 1 to 2 hours should matter far more than the jump from 20 to 21.

What we actually want to predict is not the bare label but the probability of the label — a number between 0 and 1, such as “this student has a 0.73 chance of passing.” A probability can never leave the 0-to-1 band, and near the extremes it should flatten out: once you are already almost certain, a little more studying barely moves the needle. So we need a curve that is S-shaped — gently flat near 0, steep in the middle, gently flat again near 1 — not a straight line. That curve is the whole idea behind logistic regression.

The logistic (sigmoid) curve

The S-shaped curve we need has a name: the logistic function, also called the sigmoid (from the Greek for “S-shaped”). Think of it as a little machine that takes any number at all — from minus a million to plus a million — and gently squashes it into the open interval between 0 and 1, so the output can always be read as a probability.

Here's how it behaves in words. Feed it a very negative number and it returns something just above 0; feed it exactly 0 and it returns exactly 0.5; feed it a very positive number and it returns something just below 1. In between, it rises smoothly in that telltale S shape. Written as a formula, with the input called z:

The logistic / sigmoid function. Here e is the mathematical constant about 2.718; whatever z you put in, the output sits strictly between 0 and 1.

Now we connect it to predictors. Inside, z is just the familiar linear formula from ordinary regression — an intercept plus a coefficient times each predictor. With one predictor (hours studied), z = β₀ + β₁·x. Logistic regression feeds that straight-line score z through the sigmoid to bend it into a probability:

The logistic regression model with one predictor: build the linear score, then squash it through the sigmoid into a predicted probability p̂.

Concrete numbers make it click. Say the fitted coefficients are β₀ = −4 and β₁ = 0.5 (per hour). A student who studied 4 hours has z = −4 + 0.5·4 = −2, and σ(−2) ≈ 0.12 — about a 12% chance of passing. A student who studied 10 hours has z = −4 + 0.5·10 = 1, so σ(1) ≈ 0.73 — a 73% chance. A student at exactly 8 hours has z = 0, giving exactly 0.5. The predictor value where the curve crosses 50% (here, 8 hours) is the natural tipping point of the model.

How are β₀ and β₁ chosen? Not by least squares — squared error misbehaves badly for 0/1 data. Instead logistic regression uses maximum likelihood estimation: the computer searches for the coefficients that make the passes and fails we actually observed as probable as possible under the model. You will almost never do this by hand — one line of software does it — but it helps to know the line is fitted to make the observed data look likely, not to minimize a vertical distance.

Log-odds and odds ratios

The sigmoid is elegant, but it makes the coefficient β₁ awkward to read directly: because of the curve's bend, one extra hour adds a different amount of probability depending on where you start. To recover a clean, constant interpretation we switch from probabilities to odds — the same currency bookmakers use.

Odds are simple. If the probability of passing is p, the odds of passing are p divided by the probability of not passing, 1 − p. A probability of 0.8 means odds of 0.8 / 0.2 = 4 — “4 to 1 on,” four times as likely to happen as not. A probability of 0.5 is odds of 1 (“even odds”). Probability and odds carry exactly the same information, just dressed differently.

Odds turn a probability into a ratio of “happens” to “doesn't happen.” p = 0.8 gives odds of 4; p = 0.5 gives odds of 1.

Now the magic. Take the natural logarithm of the odds — the log-odds, also called the logit — and the messy S-curve straightens out completely: the log-odds turn out to equal exactly the linear score z. So while the probability is an S-shaped function of the predictors, the log-odds are a plain straight line of them. This is the defining equation of logistic regression:

Logistic regression is an ordinary linear model — but for the log-odds, not for the probability itself. That straight line is what we are really fitting.

On the log-odds scale, β₁ is an ordinary slope: each extra hour simply adds β₁ to the log-odds. But nobody thinks in log-odds. Exponentiate the coefficient and you get something you can actually feel: e^{β₁} is the odds ratio — the factor by which the odds get multiplied for each one-unit increase in the predictor.

The odds ratio: exponentiate a coefficient to read it as a multiplier on the odds, per one-unit change in that predictor.

Numbers again. If β₁ = 0.5, then e^{0.5} ≈ 1.65, so each extra hour of study multiplies a student's odds of passing by about 1.65 — a 65% increase in the odds. If a coefficient were 0.7, then e^{0.7} ≈ 2.0 — each unit doubles the odds. An odds ratio of 1 means the predictor does nothing (its coefficient is 0); above 1 means it raises the odds; below 1 means it lowers them.

Choosing a threshold

Logistic regression hands you a probability — 0.73, 0.08, 0.51. But a decision is usually yes/no: move the email to the spam folder or not; call the patient back for a biopsy or not. To turn a probability into a label you pick a threshold (a cutoff): if the predicted probability is at least the threshold, predict “yes”; otherwise “no.” The default everyone reaches for is 0.5 — but 0.5 is a convention, not a law of nature.

The threshold is a business and ethics decision, not a statistical one. Lower it (say to 0.2) and you flag many more cases as positive: you catch almost every true case, but you also raise a lot of false alarms. Raise it (to 0.8) and you only flag the cases you are very sure about: few false alarms, but you miss many real ones. Where you set it depends entirely on which kind of mistake hurts more.

A concrete contrast. For a cancer screening test, missing a real case (telling a sick person they are fine) is far worse than a false alarm (a frightening call-back that turns out benign). So you deliberately set a low threshold — you accept many false alarms in order to miss as few real cases as possible. For an automatic spam filter the calculus flips: dumping one real, important email into the spam folder may annoy the user far more than letting the occasional junk message slip through, so you set a conservative, higher threshold. Same model, same probabilities — different cutoff, because the costs differ.

To talk about “false alarms” and “misses” precisely — and to compare thresholds fairly — we need a scoreboard that counts every kind of right and wrong answer. That scoreboard is the confusion matrix.

The confusion matrix

Once a threshold turns probabilities into yes/no predictions, every prediction lands in one of four boxes, depending on what we predicted and what was actually true. By convention we call positive the outcome we are trying to detect (has the disease, is spam, will churn). The four boxes are:

1. True positive (TP): we predicted yes, and it really was yes — a real case correctly caught.
2. True negative (TN): we predicted no, and it really was no — a healthy person correctly cleared.
3. False positive (FP): we predicted yes, but it was actually no — a false alarm (also called a type I error).
4. False negative (FN): we predicted no, but it was actually yes — a miss (a type II error).

Arrange these four counts in a 2×2 table — one axis for what is actually true, the other for what we predicted — and you have the confusion matrix, the single most useful summary of a classifier. Let's fill it in with real numbers.

Suppose we screen 1,000 people for a disease that 100 of them actually have (so 900 are healthy). Our model, at its chosen threshold, flags 120 people as “likely sick.” Of those 120, 80 truly are sick (TP) and 40 are healthy false alarms (FP). Among the 880 it cleared, 60 actually were sick but slipped through (FN) and 820 were genuinely healthy (TN). Quick check: 80 + 40 + 60 + 820 = 1,000, and the truly sick add up to 80 + 60 = 100. It balances.

The most obvious score is accuracy: the fraction of all predictions that were correct, (TP + TN) divided by everyone. Here that's (80 + 820) / 1,000 = 0.90 — a tidy 90%. Sounds great, until you notice that a lazy model which simply predicts “healthy” for everyone also scores 900 / 1,000 = 90%, while catching exactly zero sick people. When one class is rare (here only 10% are sick), accuracy becomes a vanity metric: it rewards ignoring the very cases you most care about. This class imbalance problem is exactly why we need sharper measures.

Accuracy: the share of all predictions that are correct. Useful when classes are balanced, dangerously misleading when they are not.

Precision, recall, and the ROC curve

Two questions cut straight through the imbalance trap, and they ask genuinely different things. Precision: when the model says “yes,” how often is it right? Recall (also called sensitivity, or the true-positive rate): of all the cases that really are yes, how many did the model actually catch? One is about trusting the alarms; the other is about not missing the real thing.

Precision: of everyone we flagged as positive, the share who truly were positive.

Recall: of everyone who truly was positive, the share we managed to catch.

Back to the screening numbers. Precision = 80 / (80 + 40) = 0.67 — when we flag someone, we are right two-thirds of the time. Recall = 80 / (80 + 60) = 0.57 — we caught 57% of the people who were actually sick. And notice: the lazy “everyone is healthy” model now scores a recall of 0 — exactly the catastrophic failure that 90% accuracy quietly hid.

Precision and recall pull against each other through the threshold. Lower the cutoff and you flag more people: recall climbs (you miss fewer real cases) but precision usually falls (more of your flags are false alarms). Raise the cutoff and the opposite happens. You usually cannot maximize both at once — you choose the balance the problem demands. When you truly need a single number, the F1 score — the harmonic mean of precision and recall — is a common compromise, though it hides which of the two you actually favored.

To see a classifier's quality across all thresholds at once, draw the ROC curve (short for receiver operating characteristic). For every possible cutoff, plot the true-positive rate (recall) on the vertical axis against the false-positive rate — the share of truly-negative cases that get wrongly flagged — on the horizontal axis. A useless coin-flip classifier just traces the diagonal line; a good one bows up toward the top-left corner, catching many true positives while raising few false alarms.

The area under that curve — the AUC — squashes the whole picture into one number between 0.5 (no better than chance) and 1.0 (perfect). AUC even has a lovely plain-English meaning: it is the probability that the model gives a randomly chosen true-positive case a higher score than a randomly chosen true-negative one.

A worked classification example

Let's run the whole pipeline once, end to end, on a fresh problem: predicting which customers will churn — that is, cancel their subscription next month. The positive class is “churned.” Say we have a year of data on 5,000 customers: their monthly spend, how many days since they last logged in, and whether they ended up churning.

We fit a logistic regression with churn as the outcome and the two predictors as inputs. In practice that's only a few lines:

from sklearn.linear_model import LogisticRegression

# X: columns [monthly_spend, days_since_login]; y: 1 = churned, 0 = stayed
model = LogisticRegression()
model.fit(X_train, y_train)

# predicted PROBABILITIES, not hard labels
probs = model.predict_proba(X_test)[:, 1]

# turn probabilities into yes/no with a chosen threshold
predictions = (probs >= 0.30)

Suppose the coefficient on “days since last login” comes out as 0.04 per day. Exponentiate it: e^{0.04} ≈ 1.04, so each extra idle day multiplies a customer's odds of churning by about 1.04 — roughly a 4% rise in the odds per day. And it compounds: 30 idle days multiply the odds by 1.04³⁰ ≈ 3.2. That one number tells the retention team exactly where to aim — at customers who have gone quiet.

Because winning back a wavering customer is cheap (an email, a small discount) but losing one is expensive, we set a low threshold of 0.30 — high recall, accept some false alarms. On a held-out test set of 1,000 customers (120 of whom actually churned), the model flags 200 people: 90 of them really churn (TP), 110 are false alarms (FP), 30 real churners slip through (FN), and 770 are correctly seen as safe (TN).

Now the metrics. Precision = 90 / 200 = 0.45; recall = 90 / 120 = 0.75. So three-quarters of churners are caught — exactly what we wanted — at the cost of a 45% precision, meaning less than half of our flagged customers would truly have left. Given a cheap intervention, that's a fine trade: we will happily send a retention offer to some people who weren't going to leave anyway. If the intervention were expensive, we would raise the threshold to lift precision and spend only on near-certain churners.

1. Frame the yes/no question and name the positive class (the thing you are trying to detect).
2. Fit a logistic regression; read each coefficient as an odds ratio via e^{β}.
3. Get predicted probabilities on a held-out validation or test set — never the training data.
4. Pick the threshold from the real costs of false alarms versus misses, not out of habit at 0.5.
5. Build the confusion matrix and report precision and recall (plus the PR or ROC curve) — not accuracy alone.
6. Sanity-check for overfitting before you trust any of it on genuinely new data.