01Input space & score distribution
02Outcomes
| Metric | Definition | Value |
|---|---|---|
| TPR (Recall)how well the classifier "recalls" all actual positives | $\text{TP}/(\text{TP}+\text{FN})$ | — |
| FPRhow often the classifier raises a false alarm on negatives | $\text{FP}/(\text{FP}+\text{TN})$ | — |
| Precisionhow trustworthy a positive prediction is | $\text{TP}/(\text{TP}+\text{FP})$ | — |
| F1harmonic mean of precision & recall | $2\cdot\text{Prec}\cdot\text{Rec}/(\text{Prec}+\text{Rec})$ | — |
| Specificityhow well the classifier dismisses true negatives | $\text{TN}/(\text{TN}+\text{FP}) = 1-\text{FPR}$ | — |
| Accuracyoverall fraction of correct predictions | $(\text{TP}+\text{TN})/N$ | — |
| Error Rateoverall fraction of incorrect predictions | $(\text{FP}+\text{FN})/N = 1-\text{Acc}$ | — |
03Threshold sweep
Why precision and recall trade off
Both share TP in the numerator, but their denominators pull in opposite directions as you move the threshold.
Lower the threshold → more positives predicted → recall rises, precision falls (more false alarms).
Raise the threshold → fewer positives predicted → precision rises, recall falls (more misses).
Predict positive on everything and recall is perfect — but precision collapses. Keep only the highest-scored examples and precision is usually high — but you miss many true positives. The F1 score and the PR curve make this trade-off explicit so you can pick the operating point that fits your cost structure.
Average Precision (AP) summarises the stepwise PR curve as $\sum_n (R_n-R_{n-1})P_n$: every gain in recall is weighted by the precision at that threshold. Examples tied at the same score enter together. By convention the curve starts at recall $0$, precision $1$; the outcomes table instead shows precision as undefined when no positives are predicted. The dashed horizontal reference is the positive-class prevalence — the expected precision of a random ranking.
What the ROC-AUC actually measures: $\Pr\!\big[\,s(\text{binder}) > s(\text{non-binder})\,\big]$
The AUC has a threshold-free interpretation that needs no confusion matrix at all. Draw one binder (positive) and one non-binder (negative) at random. The AUC is exactly the probability that the model scores the binder higher:
$\mathrm{AUC} = \Pr\!\big[\,s(X^{+}) > s(X^{-})\,\big] + \tfrac{1}{2}\Pr\!\big[\,s(X^{+}) = s(X^{-})\,\big]$
where $X^{+}$ is a random binder and $X^{-}$ a random non-binder (the second term just splits ties evenly). So AUC $= 0.5$ means the model ranks a binder above a non-binder no better than a coin flip, and AUC $= 1$ means every binder outscores every non-binder. It is a measure of ranking quality, independent of where you place the decision threshold.
Why the area under the ROC equals that probability — the plain-language version. Every point on the ROC curve comes from one threshold $\tau$. Its horizontal position, $\mathrm{FPR}(\tau)$, is "what fraction of non-binders score above $\tau$"; its height, $\mathrm{TPR}(\tau)$, is "what fraction of binders score above $\tau$." Now pick a single non-binder at random. Its score marks one spot along the horizontal axis — and here is how a score becomes an FPR position: feed the non-binder's own score in as the threshold, and $\mathrm{FPR}(\text{its score})$ is the fraction of non-binders scoring above it. In other words, a non-binder's spot on the FPR axis is simply its rank among the non-binders — high scorers on the left (few above them, small FPR), low scorers on the right. Because FPR counts the share of non-binders cleared, each of the $n_{-}$ non-binders owns an equal strip of width $1/n_{-}$: as you lower the threshold past exactly one non-binder, FPR ticks up by $1/n_{-}$. So this non-binder claims a thin vertical strip of the curve — the strip's width is its equal $1/n_{-}$ share of the FPR axis, and the curve's height right there is exactly the chance a random binder outscores this particular non-binder. Add up (width × height) for every non-binder and you have filled in the whole area under the curve — which is therefore the average chance a binder beats a non-binder, i.e. $\Pr[s(X^{+}) > s(X^{-})]$.
Step-by-step proof
Let $f_{-}$ be the score density of the non-binders (the smooth version of their histogram). We make each step of the plain-language argument exact.
- Read the two axes as probabilities. Fix a threshold $\tau$. By definition, $$\mathrm{TPR}(\tau)=\Pr[\,s(X^{+})>\tau\,], \qquad \mathrm{FPR}(\tau)=\Pr[\,s(X^{-})>\tau\,].$$ As $\tau$ slides from $+\infty$ down to $-\infty$, the point $(\mathrm{FPR}(\tau),\,\mathrm{TPR}(\tau))$ sweeps the ROC curve from $(0,0)$ up to $(1,1)$.
- Write the area as an integral of height against width. "Area under the curve" means summing the height $\mathrm{TPR}$ over every sliver of width $\mathrm{d}(\mathrm{FPR})$: $$\mathrm{AUC}=\int_{0}^{1}\mathrm{TPR}\;\mathrm{d}(\mathrm{FPR}).$$
- Turn the width $\mathrm{d}(\mathrm{FPR})$ into a statement about scores. Since $\mathrm{FPR}(\tau)=\Pr[\,s(X^{-})>\tau\,]=\int_{\tau}^{\infty}f_{-}(t)\,\mathrm{d}t$, nudging the threshold by a tiny $\mathrm{d}\tau$ changes the FPR by $$\mathrm{d}(\mathrm{FPR}) = -\,f_{-}(\tau)\,\mathrm{d}\tau.$$ The minus sign just says "lower the threshold and more non-binders clear it, so FPR grows." The quantity $f_{-}(\tau)\,\mathrm{d}\tau$ is simply the chance a random non-binder scores in the sliver $[\tau,\tau+\mathrm{d}\tau]$ — the strip width from the picture above.
- Substitute and flip the limits. Swapping the variable of integration from $\mathrm{FPR}$ to $\tau$ sends the limits $0\to 1$ into $+\infty\to-\infty$; reversing them back to $-\infty\to+\infty$ cancels the minus sign: $$\mathrm{AUC}=\int_{-\infty}^{\infty}\mathrm{TPR}(\tau)\,f_{-}(\tau)\,\mathrm{d}\tau=\int_{-\infty}^{\infty}\underbrace{\Pr[\,s(X^{+})>\tau\,]}_{\text{height}}\;\underbrace{f_{-}(\tau)\,\mathrm{d}\tau}_{\text{width}}.$$
- Recognise the law of total probability. The integrand multiplies "the non-binder's score lands near $\tau$" by "a binder beats that value $\tau$." Because the binder and the non-binder are drawn independently, summing this product over all possible $\tau$ is exactly the total chance the binder outscores the non-binder: $$\int_{-\infty}^{\infty}\Pr[\,s(X^{+})>\tau\,]\,f_{-}(\tau)\,\mathrm{d}\tau=\Pr\!\big[\,s(X^{+})>s(X^{-})\,\big]. \qquad\blacksquare$$
Sanity check with real numbers (no calculus). With $n_{+}$ binders and $n_{-}$ non-binders, form all $n_{+}\!\cdot n_{-}$ possible binder–non-binder pairs. Then $$\mathrm{AUC}=\frac{\#\{\text{pairs where the binder scores higher}\}+\tfrac12\,\#\{\text{ties}\}}{n_{+}\cdot n_{-}}.$$ This is the Mann–Whitney $U$ / Wilcoxon rank-sum statistic — it is what plotting software actually computes, and it is the discrete twin of the integral above.
This ranking view is why the population ROC is invariant to class prevalence and to any monotonic rescaling of the scores — both leave the pairwise ordering untouched. A finite sampled ROC can still move slightly when the class mix changes. Watch it live: as you slide the class centres apart in section 01, the two score histograms separate, more binder–non-binder pairs land in the correct order, and the AUC climbs toward 1.
04Youden's J — balanced operating point
What the J-optimal threshold balances
$J(\tau) = \mathrm{TPR}(\tau) - \mathrm{FPR}(\tau)$ measures separation from random ranking at a given cut-off. A population-level random ranking has $J = 0$; finite samples fluctuate around it, while a perfect classifier can reach $J = 1$.
The J-optimal threshold $\tau^\star = \arg\max_\tau J(\tau)$ maximises the vertical distance between the ROC curve and the diagonal chance line.
This is a balanced criterion, not a universally optimal decision rule: it weights sensitivity and specificity equally and ignores class prevalence and error costs. If a false negative is much costlier than a false positive (e.g. cancer screening), you may prefer a lower threshold than $\tau^\star$ even though $J$ is slightly smaller there.
05Low false-alarm regime
Why low-false-alarm metrics matter
The full AUC integrates sensitivity uniformly over the entire FPR axis — but many production systems can only tolerate a tiny false-alarm budget. Fraud alerts, medical screening, content moderation queues: every flag costs a human review, so only the very low-FPR slice of the ROC is operationally relevant. Two classifiers with identical AUC can behave very differently in that slice.
Partial AUC $\mathrm{pAUC}(\alpha)$ integrates the ROC only over $\mathrm{FPR} \in [0, \alpha]$ — the shaded strip on the left. The displayed standardised value uses the McClish correction, $\tfrac12\!\left[1+\frac{\mathrm{pAUC}-\alpha^2/2}{\alpha-\alpha^2/2}\right]$, so chance maps to $0.5$ and a perfect classifier to $1$. The raw area is also shown.
Success@K reframes the same idea from the analyst's seat: "of the top $K$ items the model flagged, how many were real?" (precision) and "how many of all the positives did we catch?" (recall). The marker tracks the current threshold's $K$ — slide the probability cut-off to see how the candidate list grows. If a score tie straddles $K$, the curve reports the expected result under random ordering within the tied group.