Logistic Regression
Fit a yes/no outcome to one or more predictors, with odds ratios and their intervals.
0.2–0.4 already indicates a good fit
16 of 20 classified correctly
log-likelihood -9.0216
| Term | Coefficient | SE | z | p | Odds ratio | 95% CI |
|---|---|---|---|---|---|---|
| Intercept | -2.96471 | 1.42208 | -2.085 | 0.0371 | 0.0516 | [0.003, 0.837] |
| hours | 0.31509 | 0.13344 | 2.361 | 0.0182 | 1.3704 | [1.055, 1.78] |
An odds ratio above 1 means the odds of the outcome rise with that predictor; below 1, they fall. A confidence interval spanning 1 is the same statement as p above 0.05.
At a 0.5 threshold
The fit
Logistic regression models a yes/no outcome. It fits a straight line not to the probability but to the log-odds, which keeps every prediction between 0 and 1. Exponentiating a coefficient turns it into an odds ratio: the factor by which the odds change per unit.
Why not ordinary regression
Fit a straight line to a 0/1 outcome and it will happily predict −0.3 or 1.4, which are not probabilities. It also assumes constant error variance, which a binary outcome cannot have — the variance of a coin flip depends on how biased the coin is. Both problems come from modelling the probability directly.
The fix is to model the log-odds instead. Odds are p ÷ (1 − p), running from 0 to infinity; their logarithm runs from −∞ to +∞, which is exactly the range a linear predictor can cover. Whatever the line produces maps back to a probability between 0 and 1 through the logistic curve, and the shape of that curve is where the familiar S comes from.
Reading the coefficients
A coefficient is a change in log-odds per unit of the predictor, which is not intuitive. Exponentiate it and you get the odds ratio, which is: multiply the odds by this for every one-unit increase. In the worked example an extra hour of study has a coefficient of 0.315, so the odds ratio is e^0.315 = 1.37 — each extra hour multiplies the odds of passing by about 1.37, a 37% increase in the odds. Not in the probability: those are different statements, and conflating them is the commonest error in reporting.
fitted by iteratively reweighted least squares, since there is no closed form
- 1 Check the model does better than nothing. McFadden’s pseudo-R² compares the fit against an intercept-only model. 0.2 to 0.4 already counts as a good fit — it does not run on the same scale as an ordinary R².
- 2 Look at each coefficient’s p-value. The Wald z is the coefficient over its standard error. A p below 0.05 means the predictor carries information beyond the others.
- 3 Convert to an odds ratio. e raised to the coefficient. A coefficient of 0.315 gives 1.37, so each unit multiplies the odds by 1.37.
- 4 Check whether its interval crosses 1. An odds ratio interval spanning 1 says the same thing as p above 0.05 — the direction is not established.
- 5 Treat accuracy with suspicion. If 90% of cases are zeros, a model predicting zero every time scores 90% and is useless. Compare against that base rate before being impressed.
Reading an odds ratio
The coefficient is on the log-odds scale; the odds ratio is what you report.
| Coefficient b | Odds ratio e^b | Meaning per unit increase |
|---|---|---|
| −0.693 | 0.50 | Halves the odds |
| −0.223 | 0.80 | Cuts the odds by a fifth |
| 0 | 1.00 | No association |
| 0.315 | 1.37 | Raises the odds by 37% |
| 0.693 | 2.00 | Doubles the odds |
| 1.609 | 5.00 | Multiplies the odds by five |
Separation, and the trap it sets
If a predictor splits the outcome perfectly — everyone above some threshold passed, everyone below failed — the likelihood has no maximum. The fitting algorithm pushes the coefficient upward forever, because a steeper curve always fits a little better, and stops only when it hits an iteration limit. What comes out looks spectacular: an enormous odds ratio, an enormous standard error, and a model with 100% accuracy.
It is an artefact. The data simply does not contain enough overlap to estimate where the curve should turn, and the honest answer is that the effect is unbounded rather than huge. This page detects the condition and says so instead of printing the numbers as findings; the standard remedy is a penalised fit such as Firth’s method, or collecting enough cases to give the two outcomes some overlap.
Two related cautions. The rule of thumb is at least ten cases of the rarer outcome per predictor — twenty events and three predictors is already thin. And odds ratios are not risk ratios: when the outcome is common, an odds ratio of 2 corresponds to a much smaller increase in probability than most readers assume, which is why medical journals increasingly ask for both.