Investigators evaluated eight classifiers on the public DARWIN dataset using ten repetitions of five-outer/three-inner stratified nested cross-validation, yielding 50 held-out fold estimates per model. The analytic matrix consisted of 450 predictors (18 engineered features for each of 25 handwriting tasks) and a binary outcome coded as control = 0, AD = 1. Preprocessing—including median imputation, ANOVA SelectKBest (k ∈ {30,60,90}), and z-score scaling—was performed strictly within training partitions so that outer test folds remained unseen during selection and tuning. The study team reported that Random Forest produced the highest descriptive ROC-AUC across the 50 held-out folds (0.938 ± 0.033). At the conventional 0.50 probability threshold Random Forest accuracy averaged 0.856 ± 0.048, with sensitivity 0.866 ± 0.069 and specificity 0.847 ± 0.095. Stacking attained the highest observed accuracy and macro-F1 on single folds but these ranks were descriptive rather than definitive.
The authors implemented a secondary, training-only operating-point selection for Random Forest in every outer fold. Three-fold out-of-fold probabilities from the outer training partition identified the threshold maximizing sensitivity while enforcing at least 80% specificity; ties favoured the higher threshold. Median selected threshold across folds was 0.461 (IQR 0.406–0.522). Applied to untouched outer test folds, mean sensitivity was 0.869 ± 0.070 and specificity 0.831 ± 0.113. The study team emphasized that these operating characteristics are exploratory and that variation across folds precludes declaring a single retrospective cutoff as clinically actionable.
Task-wise discrimination varied substantially. The study team found Task 23 (telephone number under dictation) achieved the highest single-task AUC with Random Forest (AUC 0.871), followed by Task 17 (copy six words into boxes; AUC 0.864) and Task 19 (postal-order fields; AUC 0.853). Task 22 reached AUC 0.830 and was described as moderate rather than top-performing. These task-wise estimates were presented as descriptive; they were not promoted as independently validated clinical batteries.
To examine reduced-task batteries, the authors implemented task-level greedy forward selection with swap refinement (GFS-SR). All 18 predictors for a task were added or removed together. A same-cohort search reproduced a previously reported seven-task combination (Tasks 2, 5, 9, 10, 19, 21, 24) with an optimistic same-cohort AUC of 0.9804 versus full-battery 0.9617. Repeating selection fully nested—so that outer-fold participants were excluded from selection—yielded a held-out AUC of 0.916 ± 0.037 for the seven-task set, compared with 0.956 ± 0.030 for the 25-task Random Forest. The authors quantified selection instability: mean Jaccard overlap across folds was 0.195, indicating weak reproducibility of chosen subsets. They also transferred the fixed same-cohort seven-task set to multiple classifiers (Logistic Regression, SVM-RBF, Random Forest, XGBoost) in exploratory transfer tests, explicitly noting the transfer analysis is not an unbiased validation because the set was chosen using the full cohort.
Explainability used foldwise SHAP generated for Random Forest and XGBoost. For each of five outer folds, feature selection, scaling, and model fitting used only outer training data; SHAP values were calculated exclusively for held-out participants. Absolute SHAP values were aligned by feature name and aggregated; zero contribution was assigned when a feature was not selected in a fold. The cross-validated SHAP profile emphasized temporal feature families. A further sensitivity analysis recomputed SHAP using all 450 predictors—bypassing ANOVA preselection—and the authors reported that individual feature ranks depended partly on the univariate preselection step. The team cautioned that these explanation analyses describe model associations rather than causal mechanisms or validated biomarkers.
Hyperparameter selection and calibration were nested inside outer training folds; isotonic calibration with three-fold internal splits was applied before testing. Implementations used scikit-learn, XGBoost, and SHAP on a CPU-only Linux environment. Median per-fold training times ranged from fractions of a second up to about 10 seconds on the stated hardware; median inference was below 1.4 ms per participant for every model. For comparative inference the Friedman omnibus test indicated overall differences among classifiers (χ2(7) = 45.70, p < 0.001). Despite Random Forest having the lowest mean rank, pairwise Wilcoxon signed-rank tests with Holm correction left no paired comparison statistically significant (minimum adjusted p = 0.055). The authors interpreted statistical variability conservatively given that the ten repetitions reuse the same participant pool rather than independent cohorts.
The analysis used the DARWIN file from the UCI Machine Learning Repository; the authors provided code, fixed configuration, fold-level predictions, and figure-workflow as supporting materials. The released dataset contains exact linear relationships among some predictors (for example, total_time equals air_time plus paper_time), and the authors noted that correlated predictors can divide importance across substitutable features. The released DARWIN file did not include participant-level age, education, sex, or gender fields; consequently demographic-balance checks and covariate sensitivity analyses could not be performed. The study team therefore restricted interpretation to model-association findings and to benchmarking statements.
The authors framed the contribution as methodological: an integrated, leakage-controlled benchmark and hypothesis generator rather than a new classifier or a validated screening test. The analysis is limited by cohort size, absence of demographic covariates in the released file, correlated predictors in the feature matrix, and selection instability for reduced-task subsets. External prospective validation on independent cohorts, with available demographic and clinical covariates, is required before any clinical application or policy consideration.