This study establishes an open, multi-language machine learning benchmark to predict subjective cognitive decline (SCD) using Cognitive Decline modules from the Behavioral Risk Factor Surveillance System (BRFSS). The framework emphasizes temporal validity by using chronological training, validation, and locked-test splits to assess model generalization to future survey years. Authors implemented multiple end-to-end pipelines and interpretability methods and made the code and data usage reproducible with public BRFSS datasets.
Analyses used two cohorts drawn from BRFSS: a pooled national sample (n = 298,944) and a New York state sample (n = 30,366). Data were partitioned chronologically to reflect temporal drift common in public health surveillance: training years 2015–2019; validation years 2020–2022 nationally (2020–2021 for New York); and a locked test period covering 2023–2024. The BRFSS datasets are public and de-identified; secondary analysis did not require institutional review board approval according to the authors.
Stable predictors were identified using nested LASSO. The authors benchmarked sixteen machine learning algorithms using year-grouped cross-validation. Synthetic Minority Over-sampling Technique (SMOTE) was applied only within training folds to address class imbalance without leaking information into validation or test data. The study also examined post-hoc generalized linear models (GLMs)—unweighted for the national cohort and design-weighted for New York—to support interpretability alongside machine learning explanations.
Four end-to-end pipelines were implemented in Python and R, representing single-model and soft-voting ensemble approaches. Validation-only isotonic calibration was applied to predicted probabilities, followed by Youden thresholding for classification decision thresholds. Primary reporting pipelines were prespecified before unlocking the locked test set: an R tidymodels single-model pipeline for the national cohort and a Python single-model pipeline for the New York cohort. Within each prespecified pipeline, the final algorithm choice was based on validation ROC-AUC performance.
Locked-test discrimination was consistent across implementations, with ROC-AUC values around 0.76–0.77. The prespecified national pipeline (R gradient boosting) achieved a test ROC-AUC of 0.770 (95% CI 0.767–0.773). The New York prespecified pipeline (Python AdaBoost) achieved a test ROC-AUC of 0.762 (95% CI 0.746–0.777). Soft-voting ensemble pipelines produced similar discrimination (national 0.770; New York 0.757) and were treated as sensitivity analyses.
Predicted probabilities exhibited reasonable calibration as measured by Brier scores: 0.118 nationally and 0.112 in New York. Importantly, calibrated scores remained separated by SCD status across survey years, with higher predicted probabilities persisting among respondents who reported SCD.
Multiple interpretability methods converged on a similar set of important predictors. Permutation importance, SHAP values, and GLMs consistently ranked items related to difficulty deciding (DECIDE), mental health, and functional health among the top predictors for SCD. The authors reported that respondents who indicated no difficulty deciding (DECIDE = 2) had substantially lower odds of SCD compared with those who reported difficulty deciding (DECIDE = 1); the adjusted odds ratio was approximately 0.13 with a false discovery rate below 0.05. Training-only knowledge graphs similarly placed difficulty deciding nearest to SCD in both cohorts, reinforcing this feature's prominence in the learned relationships.
By locking test years and prespecifying analysis pipelines, the benchmark assesses model performance on future surveillance data and reduces risk of optimistic bias from temporal leakage. Comparable ROC-AUC across multiple implementations and reasonable calibration indicate that the benchmark yields stable SCD risk ranking and usable probability scores for population surveillance. The open, multi-language implementation supports reproducible surveillance-oriented machine learning workflows for cognitive health monitoring.
All data used in the analyses are publicly available via the CDC BRFSS portal. Because BRFSS is a public, de-identified surveillance dataset, the authors reported that institutional review board approval was not required for this secondary analysis. The authors declared no competing interests and stated that all relevant ethical guidelines were followed.