Machine learning models for acute kidney injury (AKI) often report high predictive performance, but accuracy depends heavily on when prediction is made relative to AKI onset. This complicates comparisons across studies and may overestimate clinical utility. We evaluated how prediction timing influences model performance, interpretability, and clinical applicability.

Using the publicly available University of Kansas Health System cohort (2007–2016), we reproduced the four prediction perspectives proposed by He et al. Perspectives 1–3 included 76,957 admissions (9.4% AKI), while Perspective 4 included 72,846 admissions (2.2% AKI). These represent four clinical scenarios: prediction before AKI onset, admission-time prediction of any in-hospital AKI, admission-time prediction within fixed risk windows, and daily surveillance for next-day AKI. Logistic regression (LR), random forest (RF), and a soft-voting LR+RF ensemble were trained using 10-fold stratified cross-validation. Permutation importance was used to assess model interpretability.

The ensemble consistently achieved the best performance. Admission- and surveillance-based models closely reproduced the original study, with AUCs of 0.720, 0.744, and 0.670 versus the reported 0.734, 0.764, and 0.679. In contrast, the onset-anchored model achieved a substantially higher AUC (0.802 vs. 0.744), likely reflecting differences in feature availability rather than superior prediction. Interpretability revealed distinct learning patterns. The onset-anchored model relied predominantly on blood urea nitrogen and white blood cell count, whereas admission- and surveillance-based models integrated a broader range of clinically meaningful predictors, including renal and cardiovascular comorbidities, major surgery, admission diagnoses, and illness severity, yielding more balanced and clinically plausible risk profiles.

Prediction timing fundamentally shapes both the apparent performance and clinical reasoning of AKI prediction models. Although onset-anchored prediction may achieve higher discrimination, its dependence on future knowledge limits clinical usefulness and can inflate performance estimates. Admission-based and continuous surveillance strategies provide more realistic, interpretable, and clinically deployable approaches for AKI early warning and should serve as the benchmark for future model development.