Beyond the AUC: How to evaluate a clinical prediction model

A high AUC can make a clinical prediction model look impressive. Short for the area under the receiver operating characteristic curve, AUC measures discrimination: how well a model ranks people with a particular outcome above those without it. It does not establish whether individual estimates are accurate or whether using the model improves clinical decisions.

That distinction framed the Ottawa Medical AI Research Institute’s second seminar, held in person on October 28, 2026. In Beyond the AUC: Developing and Validating Clinical Prediction Models, Dr. Steven Hawken, senior scientist at the Ottawa Hospital Research Institute and associate professor at the University of Ottawa, examined what researchers and clinicians need to know before relying on a model’s predictions. An opening slide posed a question: would an AUC of 0.95 be enough to use a model in a clinic?

Through simulated examples and analyses of real newborn screening data, the presentation showed why answering that question requires more than a performance score.

Newborn screening and clinical prediction models

Start with what the model is meant to do

The case study concerned models that estimate postnatal gestational age—how far a pregnancy had progressed when an infant was born. This information supports newborn care and estimates of preterm birth rates, but accurate dating ultrasounds are not always available. Developed using Ontario newborn screening data, the models combine clinical information with metabolic markers measured in dried blood spots. They have been externally validated in several countries.

The presentation distinguished three uses: estimating an individual infant’s gestational age, identifying infants born preterm for follow-up, and estimating a population’s preterm birth rate. Each requires a different assessment.

For an estimate expressed in weeks, mean absolute error describes the average size of the errors, regardless of direction. Reporting how often estimates fall within one or two weeks of the reference measurement also helps clinicians interpret performance. Researchers should examine systematic over- or underestimation and subgroup errors, rather than relying on an overall average.

Ranking well is not the same as estimating accurately

In the seminar’s preterm-birth examples, the AUC represents the probability that a randomly selected preterm infant receives a higher predicted risk than a randomly selected term infant. That describes the ordering of predictions, not whether the probabilities themselves are right.

Calibration addresses the second question. Among infants assigned a 20 per cent probability of having been born preterm, is the observed proportion approximately 20 per cent? Other checks examine whether average predictions are correct and whether estimates are too extreme or too compressed.

A simulated example in the presentation made the distinction concrete. At a new site, a model ranked infants about as well as it did in the development data, yet overestimated the probability of preterm birth. Its AUC did not reveal the problem.

Another example exposed a different misleading measure. Labelling every infant as born at term produced 91.6 per cent classification accuracy in the simulated data—while identifying none of the preterm infants. When one outcome is much more common, a high percentage of correct classifications can conceal a failure to identify the cases that matter.

Connect predictions to clinical decisions

The session then moved from statistical performance to potential clinical usefulness. Suppose a model is used to identify infants for additional follow-up. At what predicted probability should an infant be flagged, and how should the benefit of identifying a preterm infant be weighed against unnecessarily flagging an infant born at term?

Decision-curve analysis examines these trade-offs across different thresholds for action. It calculates net benefit by weighing correctly identified cases against false positives. The presentation stressed that clinicians should help define a relevant range of thresholds before researchers examine the results.

In the simulated examples, the model combining clinical information and metabolic markers offered greater net benefit than the clinical-only model across plausible thresholds. The analysis also compared model-guided decisions with flagging everyone or flagging no one. These comparisons connect model performance to a proposed use, rather than assuming that a strong AUC makes a model useful.

Test what happens in a different setting

Internal and external validation answer different questions. Internal validation uses the development data to estimate how much a model’s apparent performance may be overstated. External validation tests whether it works elsewhere, where the population, laboratory, sample type or reference measurement may differ.

The real-data examples illustrated why those differences matter. In the analyses presented, the Ontario model’s mean absolute error using heel-prick samples was about 0.8 weeks in both Bangladesh and Zambia, compared with about 0.7 weeks in Ontario. Performance was worse with cord blood. Measurements from different sample types were not interchangeable simply because they came from the same infants.

The presentation also examined uncertainty around performance estimates. The Zambia heel-prick group included only 11 infants born preterm. Planning validation therefore requires attention to the number of relevant outcomes, not just the total number of participants.

A separate simulated example showed why the reference measurement itself needs scrutiny. A model appeared less accurate at a site using later ultrasounds to establish gestational age. Because the true ages were known in the simulation, the analysis could show that most of the apparent decline came from the less reliable reference measurement, not the model. In real studies, that distinction is harder to establish.

Treat data preparation as part of the model

One demonstration using real data showed that an infant’s predicted gestational age could change depending on which other infants were included in the same batch. The original process standardised each dataset separately before generating predictions. As a result, the composition of the batch could influence an individual estimate.

The lesson extended beyond the model’s mathematical formula. Data-preparation settings should be learned from the training data, stored with the model and tested to ensure predictions do not depend on the batch. Where site-specific standardisation is used deliberately, it needs to be documented and tested as part of the model.

Updating a model for a new setting is a separate, deliberate step. In another simulated example, an adjustment using local data substantially improved calibration without changing the AUC. The ordering of predictions remained the same, but the estimates became more appropriate for that setting.

Build evidence for the intended use

The importance of intended use also surfaced in population surveillance. Counting infants whose predicted gestational age falls below 37 weeks can underestimate the preterm birth rate because predictions tend to be pulled towards the more common term values. For surveillance, the approach presented was to average calibrated probabilities of preterm birth instead of counting classifications. A method suited to individual estimation is not automatically suited to estimating a population rate.

Hawken’s closing checklist brought these examples back to the decisions facing researchers and clinicians. Model development should begin with a defined purpose. Evaluation should examine calibration as well as discrimination, test performance in the intended setting, and assess net benefit at thresholds that make clinical sense.

For someone reading a research paper or considering a model for clinical use, that changes the questions to ask. How wrong are the estimates? Does performance hold in the relevant population? What happens when the predictions guide action? A high AUC answers only part of that assessment. Before a model can support a clinical decision, the evidence needs to support that particular use, not simply show that the model ranks cases well.

Explore the materials

The October 28 session was the second in OMARI’s four-session, in-person series of seminars. The seminar’s GitHub repository includes the presentation slides, companion R code, worked examples and publicly available datasets. Readers can review the examples without running code or follow the instructions to reproduce the analyses.

For information on the full series, visit the OMARI events page.