Larger sample sizes are needed when developing a clinical prediction model using machine learning in oncology: methodological systematic review

Biruk Tsegaye; Kym Ie Snell; Lucinda Archer; Shona Kirtley; Richard D Riley; Matthew Sperrin; Ben Van Calster; Gary S Collins; Paula Dhiman

doi:10.1016/j.jclinepi.2025.111675

Larger sample sizes are needed when developing a clinical prediction model using machine learning in oncology: methodological systematic review

J Clin Epidemiol. 2025 Jan 13:111675. doi: 10.1016/j.jclinepi.2025.111675. Online ahead of print.

Authors

Biruk Tsegaye¹, Kym Ie Snell², Lucinda Archer², Shona Kirtley³, Richard D Riley², Matthew Sperrin⁴, Ben Van Calster⁵, Gary S Collins³, Paula Dhiman³

Affiliations

¹ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK. Electronic address: biruk.tsegaye@csm.ox.ac.uk.
² Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, B15 2TT, United; National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK.
³ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK.
⁴ Division of Imaging, Informatics and Data Science, University of Manchester, Manchester Academic Health Science Centre, Manchester, M13 9PL, United Kingdom.
⁵ Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, the Netherlands; Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium.

PMID: 39814217
DOI: 10.1016/j.jclinepi.2025.111675

Abstract

Background: Having a sufficient sample size is crucial when developing a clinical prediction model. We reviewed details of sample size in studies developing prediction models for binary outcomes using machine learning (ML) methods within oncology and compared the sample size used to develop the models with the minimum required sample size needed when developing a regression-based model (N_min).

Methods: We searched the Medline (via OVID) database for studies developing a prediction model using ML methods published in December 2022. We reviewed how sample size was justified. We calculated N_min, which is the minimum required sample size needed when developing a regression-based model, and compared this with the sample size that was used to develop the models.

Results: Only one of 36 included studies justified their sample size. We were able to calculate N_min for 17 (47%) studies. 5/17 studies met N_min, allowing to precisely estimate the overall risk and minimise overfitting. There was a median deficit of 302 participants with the event (n= 17; range: -21331 to 2298) when developing the ML models. An additional three out of the 17 studies met the required sample size to precisely estimate the overall risk only.

Conclusion: Studies developing a prediction model using ML in oncology seldom justified their sample size and sample sizes were often smaller than N_min. As ML models almost certainly require a larger sample size than regression models, the deficit is likely larger. We recommend that researchers consider and report their sample size and at least meet the minimum sample size required when developing a regression-based model.

Keywords: machine learning; methodology; oncology; prediction model; sample size; systematic review.