---
title: "Handwriting-based Alzheimer’s classification on the DARWIN dataset: nested validation and explaina"
id: "plos-one-19-handwriting-based-alzheimer-s-disease-classification-on-the-darwin-dataset"
canonical_url: "https://medichelpline.com/clinical-feed/plos-one-19-handwriting-based-alzheimer-s-disease-classification-on-the-darwin-dataset"
content_type: "clinical_feed_article"
specialty: "Neurology"
source_name: "PLOS ONE (Medicine)"
source_url: "https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0361117"
published_at: "2026-10-26T14:00:00.000Z"
evidence_level: "Journal Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Handwriting-based Alzheimer’s classification on the DARWIN dataset: nested validation and explaina
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/plos-one-19-handwriting-based-alzheimer-s-disease-classification-on-the-darwin-dataset
- **Specialty:** [Neurology](https://medichelpline.com/clinical-feed/neurology.md)
- **Primary Source:** PLOS ONE (Medicine)
- **Source URL:** [Original Journal Publication](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0361117)
- **Published At:** 2026-10-26T14:00:00.000Z
- **Evidence Rating:** Journal Feed
## Executive GIST (TL;DR)
- The study reanalysed the public **DARWIN dataset**, containing 174 participants (89 Alzheimer’s disease, 85 controls) and 450 predictors derived from 25 handwriting tasks. - Investigators compared eight classifiers (including **Random Forest**, XGBoost, SVM, Logistic Regression, MLP, KNN, Gaussian NB, and a stacking ensemble) using ten repetitions of 5-outer/3-inner stratified nested cross-validation, producing 50 held-out evaluations per model. - Preprocessing steps (median imputation, ANOVA SelectKBest with k ∈ {30,60,90}, z-score scaling), hyperparameter tuning, and calibration were performed inside outer training folds only; outer test folds remained untouched. - Random Forest produced the highest descriptive ROC-AUC across held-out folds (0.938 ± 0.033). The Friedman test found overall model differences (p < 0.001), but no paired comparison survived Holm correction. - The authors derived a training-only operating threshold per fold (median 0.461) yielding held-out sensitivity ~0.869 and specificity ~0.831; they explicitly note these are exploratory, not clinical cutoffs. - Task-reduction via a task-level greedy forward selection with swap refinement (GFS-SR) yielded a same-cohort seven-task AUC of 0.9804, but fully nested seven-task evaluation attained AUC 0.916 ± 0.037 versus 0.956 ± 0.030 for all 25 tasks; selected subsets were unstable (mean Jaccard 0.195). - Cross-validated **SHAP** analyses emphasized temporal features; an all-450-feature sensitivity run showed ANOVA preselection affected individual feature ranks. - The code, fold-level predictions, and data provenance are provided; the authors framed the work as a leakage-controlled benchmark and hypothesis generator, not a validated diagnostic instrument. - Limitations explicitly reported: no participant-level demographics in the released file (age, sex, education missing), modest cohort size, correlated predictors (exact linear relationships), and the need for external prospective validation before clinical use.
## Clinical Analysis & Structured Key Points
Handwriting-based Alzheimer’s disease classification on the DARWIN dataset: Repeated nested validation, task-level reduction, and cross-validated explainability | PLOS One Browse Subject Areas ? Click through the PLOS taxonomy to find articles in your field. For more information about PLOS Subject Areas, click here . Article Authors Metrics Comments Media Coverage Peer Review Reader Comments Figures Figures Abstract Online handwriting provides a low-burden source of cognitive–motor measurements, but small datasets can yield optimistic results when preprocessing, task selection, or explanation is not separated from evaluation. We evaluated eight classifiers on the DARWIN dataset (174 participants; 25 tasks; 450 predictors) using ten repetitions of 5-outer/3-inner-fold nested stratified cross-validation. All preprocessing, hyperparameter selection, and calibration were restricted to outer training data. We added a training-derived operating point, fully nested seven-task selection, cross-model subset transfer, foldwise SHAP analysis, and an all-feature SHAP sensitivity analysis. Random Forest had the highest descriptive ROC-AUC (0.938 ± 0.033 across 50 held-out folds). The Friedman omnibus test indicated model differences (p < 0.001), but no one of 28 paired comparisons remained significant after Holm correction. Training-derived thresholds (median 0.461) yielded held-out sensitivity 0.869 and specificity 0.831; these are exploratory operating characteristics, not a clinical cutoff. Nested seven-task selection achieved AUC 0.916 ± 0.037, compared with 0.956 ± 0.030 for all 25 tasks, and selected subsets were unstable (mean Jaccard 0.195). Cross-validated SHAP emphasized temporal variables, although an all-450-feature sensitivity analysis showed that individual ranks depend partly on ANOVA preselection. The study contributes an integrated, leakage-controlled evaluation rather than a new classifier or a validated diagnostic instrument; external prospective validation remains necessary. Citation: Fatima T, Lashari TA, Moetesum M, Lashari SA (2026) Handwriting-based Alzheimer’s disease classification on the DARWIN dataset: Repeated nested validation, task-level reduction, and cross-validated explainability. PLoS One 21(10): e0361117. https://doi.org/10.1371/journal.pone.0361117 Editor: Paulo Jorge Simões Coelho, Polytechnic Institute of Leiria: Instituto Politecnico de Leiria, PORTUGAL Received: June 7, 2026; Accepted: October 6, 2026; Published: October 26, 2026 Copyright: © 2026 Fatima et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The DARWIN dataset is publicly available from the UCI Machine Learning Repository at https://archive.ics.uci.edu/dataset/732/darwin (DOI: https://doi.org/10.24432/C55D0K ). The complete code, fixed configuration, fold-level predictions, machine-readable result tables, and figure-generation workflow underlying the revised analysis are provided with Supporting information files as S1 Code . Funding: The author(s) received no specific funding for this work. Competing interests: The authors have read the journal’s policy and declare that there are no competing interests associated with this manuscript. The authors have declared that no competing interests exist. 1. Introduction Alzheimer’s disease (AD) is a progressive neurodegenerative disorder associated with decline in memory, reasoning, and everyday functioning and is a leading cause of dementia worldwide [ 1 ]. Biomarker-supported diagnosis can involve imaging or cerebrospinal fluid procedures that are resource intensive and not suited to population-scale preliminary assessment [ 2 ]. Accessible behavioural measurements may therefore complement, but cannot replace, clinical evaluation. Handwriting integrates attention, visuospatial processing, language, motor planning, and fine motor control. AD and related cognitive impairment have been associated with changes in writing speed, pausing, pressure, and stroke organisation [ 3 , 4 ]. Digitising tablets make these temporal and kinematic processes measurable, creating a possible non-invasive input for retrospective classification research. The DARWIN benchmark contains online handwriting measurements from 174 participants (89 with AD and 85 healthy controls) who performed 25 tasks [ 5 ]. Prior studies have evaluated conventional classifiers, ensembles, feature-selection methods, deep models, and explainability approaches, but their reported performance is difficult to compare because data splits, tuning, task scope, and metrics differ. Three methodological questions remain particularly important on this small cohort. First, apparent model rankings may change with the outer split and require repeated, paired evaluation. Second, a task subset chosen and scored on the same participants can overstate reduction performance. Third, feature explanations can become outcome-informed when ANOVA selection is performed once on the full labelled dataset. Threshold-free discrimination also does not by itself describe the sensitivity–specificity trade-off relevant to screening. The novelty of this work is methodological and integrative, not algorithmic. Nested cross-validation, Random Forest, XGBoost, grouped greedy selection, and SHAP are established methods. The contribution is their auditable combination in one leakage-controlled analysis that (i) compares eight classifiers over 50 held-out folds with multiplicity-adjusted paired statistics; (ii) evaluates a complete task-selection procedure on participants excluded from selection; (iii) derives an operating threshold from training data only; and (iv) aligns foldwise SHAP values and tests their sensitivity to univariate preselection. These analyses are intended as a rigorous benchmark and hypothesis-generating assessment, not a clinically validated screening protocol. 2. Related work Cilia et al. introduced DARWIN and reported an SVM-based benchmark [ 5 ]. Mitra and Rehman subsequently evaluated stacking under repeated validation [ 6 ], while Moreira et al. combined dimensionality reduction with an Explainable Boosting Machine [ 7 ]. Pannala et al. reported a voting ensemble with ANOVA selection [ 8 ]. These studies demonstrate the benchmark’s utility but do not provide head-to-head estimates under a common resampling design. More recent DARWIN analyses include a hybrid transformer focused on a single task [ 9 ], stroke-level modelling [ 10 ], CNN explanation with DeepSHAP [ 11 ], and broader reviews of LIME and SHAP in AD modelling [ 12 ]. Other work has examined word semantics and phonology [ 13 ], PCA-based classification [ 14 ], sequential handwriting representations [ 15 ], and one-dimensional convolutional networks [ 16 ]. Handwriting assessment for neurological conditions has also been reviewed at the visual and procedural levels [ 17 ], and studies of Parkinson’s disease illustrate related feature and task-design considerations [ 18 , 19 ]. Generative augmentation [ 20 ], hybrid SHAP-SVM modelling [ 21 ], digital-biomarker reviews [ 22 ], and self-attention mechanisms [ 23 ] further show the diversity of available methods. Their validation protocols range from one train–test split to repeated or nested cross-validation, so numerical results should not be treated as a common leaderboard. Accordingly, the present study does not claim a new classifier or feature-selection algorithm. It complements earlier work through a controlled repeated evaluation, honest task-selection assessment, training-only threshold derivation, and foldwise explanation workflow. 3. Materials and methods 3.1. Dataset and ethics We analysed the public DARWIN data file [ 5 ] from the UCI Machine Learning Repository. After excluding the identifier and class columns, the analytic matrix contained 450 predictors: 18 engineered features for each of 25 tasks. The outcome was encoded as healthy control = 0 and AD = 1. The released file contained 85 controls and 89 participants with AD and no missing predictor values. It did not contain participant-level age, education, sex, or gender fields; therefore, demographic balance and covariate sensitivity analyses could not be performed. Ethics and consent. This study is a secondary analysis of a publicly released, de-identified dataset. The original investigators reported institutional ethics approval and informed consent from participants or their legal representatives [ 5 ]. We recruited no participants, collected no new identifiable information, and performed no participant contact; no additional review was required for this secondary analysis. 3.2. Preprocessing and correlated predictors For every inner and outer training split, a single pipeline fitted median imputation, an ANOVA F-test selector (SelectKBest; k ∈ {30, 60, 90}), and z-score scaling before fitting the classifier. The value of k and all model hyperparameters were chosen jointly by the inner cross-validation loop. The corresponding outer test fold was never used to estimate medians, F statistics, selected features, scaling parameters, calibration, or hyperparameters. The 450 predictors are not independent. In particular, for every task and every participant, total_time equalled air_time plus paper_time exactly in the released file. This perfect linear relationship, together with other correlated kinematic measures, can divide tree and SHAP importance among substitutable predictors. We therefore interpret individual attribution ranks conditionally on the fitted pipeline and place greater weight on recurrent feature families than on one isolated rank. 3.3. Classifiers and hyperparameter search We evaluated Logistic Regression [ 24 ] with C ∈ {0.5, 1, 2}; an RBF Support Vector Machine [ 25 ] with C ∈ {1, 5}, gamma = scale; KNN [ 26 ] with neighbours ∈ {3, 5} and weights ∈ {uniform, distance}; Gaussian Naïve Bayes without additional hyperparameters; Random Forest [ 27 ] with 300 trees and maximum depth ∈ {None, 12}; an MLP with hidden layers (64, 32), maximum 1,000 iterations, and alpha ∈ {10 −4 , 10 −3 }; and XGBoost [ 28 ] with 300 trees, maximum depth ∈ {4, 6}, learning rate 0.1, subsample 0.9, and column subsample 0.9. The stacking model [ 29 ] used the seven fixed base estimators, three-fold internal out-of-fold predictions, and Logistic Regression as the final estimator; its inner grid tuned ANOVA k. Implementations used scikit-learn [ 30 ] and the XGBoost Python API. 3.4. Repeated nested validation, thresholds, and statistics Evaluation used stratified nested cross-validation [ 31 ]: five outer folds estimated performance and three inner folds selected hyperparameters by ROC-AUC [ 32 ]. The complete outer procedure was repeated with seeds 42–51, yielding 50 held-out evaluations per classifier. Within each outer training partition, the selected model was calibrated by three-fold isotonic calibration [ 33 , 34 ]. The conventional probability threshold of 0.50 was prespecified for the primary classification metrics. ROC-AUC remained the primary threshold-independent discrimination measure. A secondary Random Forest operating point was learned separately in every outer fold. Three-fold out-of-fold probabilities from the outer training partition were used to select the threshold with maximum sensitivity among thresholds attaining at least 80% specificity; ties used the higher threshold. The selected threshold was then applied unchanged to the untouched outer test fold. This procedure estimates a screening-oriented trade-off without using test labels and does not establish a clinical cutoff. Sigmoid calibration is described by Platt [ 35 ], but was not used in the final reported pipeline. We reported accuracy, macro-F1, sensitivity, specificity, Brier score [ 36 ], and expected calibration error in addition to ROC-AUC. For classifier comparison, each seed contributed one mean outer-fold AUC for every model. A Friedman omnibus test [ 37 ] was followed by all 28 paired two-sided Wilcoxon signed-rank comparisons with Holm family-wise correction [ 38 ]. The ten repetitions reuse the same participants and are not independent cohorts; tests and variability summaries are therefore interpreted conservatively. Training time included inner search, refitting, and calibration. Inference time was the median per-participant predict_proba time over repeated calls. Computations used Python 3.12.13, NumPy 2.4.6, pandas 3.0.3, SciPy 1.18.0, scikit-learn 1.9.0, XGBoost 3.2.0, SHAP 0.52.0, and Matplotlib 3.10.8 on Linux with nine logical CPUs available (AMD EPYC 9V74) and 15.9 GiB RAM. No GPU was used. 3.5. Task-level greedy forward selection with swap refinement The original analysis was labelled “SFFS-style,” but its procedure does not implement classical Sequential Forward Floating Selection, which interleaves conditional exclusions with inclusions [ 39 ]. We therefore renamed it task-level greedy forward selection with swap refinement (GFS-SR) and make no algorithmic novelty claim. All 18 predictors for a task entered or left together. For an exact task budget k, GFS-SR began with an empty set and added the task giving the highest inner three-fold AUC until k tasks were present. It then evaluated every one-for-one exchange between a selected and unselected task, accepted the best strictly improving swap, and repeated until no swap improved AUC. Candidate subsets were cached; deterministic task-number order resolved equal scores. The search classifier was a lightweight Random Forest with 100 trees and maximum depth 5. The same-cohort searches for k = 1–10 are retained only to document how the original seven-task hypothesis arose. For honest evaluation conditional on the originally proposed seven-task burden, the full GFS-SR search was rerun within each of five outer training folds using three-fold inner scores. The seven tasks selected without the outer participants were fitted on the outer training data and evaluated on the untouched fold. A full 25-task Random Forest with the same fixed search-model configuration was evaluated on the same outer splits. Task-selection frequency and pairwise Jaccard overlap quantified stability. Finally, the fixed same-cohort seven-task set (Tasks 2, 5, 9, 10, 19, 21, and 24) was transferred without reselecting tasks to Logistic Regression, SVM-RBF, Random Forest, and XGBoost in five-fold nested model evaluation. Because that set was chosen using the full cohort, the transfer analysis is exploratory and tests model dependence; it is not an unbiased validation of the fixed subset. 3.6. Cross-validated SHAP analysis SHAP [ 40 ] explanations were generated for Random Forest and XGBoost. In each of five outer folds, imputation, ANOVA k selection, scaling, tree hyperparameter selection, and fitting used only outer training data; SHAP values were calculated only for held-out participants. Selected k was 60 or 90. Absolute SHAP values were aligned by feature name, with zero contribution assigned when a feature was not selected in a fold, and then aggregated across all held-out participants. SHAP was applied to the tuned tree estimator before the separate calibration wrapper so that TreeExplainer described the fitted tree decision function. We reported feature-, task-, and feature-type-level normalized mean absolute importance and contrasted correct with misclassified Random Forest predictions. To assess whether ANOVA could exclude interaction-relevant predictors, we repeated the five-fold held-out SHAP workflow with all 450 predictors and compared the combined rank profile and top-20 overlap. These explanation analyses describe model associations, not causal mechanisms or validated biomarkers. 4. Results 4.1. Repeated classifier comparison and operating point Table 1 reports performance across 50 held-out outer folds. Random Forest had the highest observed AUC (0.938 ± 0.033; 2.5th–97.5th fold percentiles 0.868–0.992). At threshold 0.50 its accuracy was 0.856 ± 0.048, sensitivity 0.866 ± 0.069, and specificity 0.847 ± 0.095. Stacking had the highest observed accuracy (0.870) and macro-F1 (0.869). These are descriptive ranks. Download: PNG larger image TIFF original image Table 1. Repeated nested cross-validation performance at threshold 0.50. Values are mean ± SD across 50 held-out outer folds; the bracketed interval in the ROC-AUC column is the 2.5th–97.5th percentile envelope, not a confidence interval. https://doi.org/10.1371/journal.pone.0361117.t001 The Friedman test indicated an overall difference among the eight classifiers (χ 2 (7) = 45.70, p < 0.001). Random Forest had the lowest mean rank (1.70), but none of the 28 paired Wilcoxon comparisons remained significant after Holm correction (minimum adjusted p = 0.055; S1 Table ). Table 2 summarises rank, calibration, and computation. On the stated CPU, median outer-fold training time ranged from 0.115 s to 10.08 s; median inference was below 1.4 ms per participant for every model. Download: PNG larger image TIFF original image Table 2. Calibration, mean AUC rank, and computational timing. Brier and ECE are mean ± SD across 50 held-out folds. Timing is median [IQR] per outer fold. https://doi.org/10.1371/journal.pone.0361117.t002 The repeated held-out Random Forest ROC curves, accumulated threshold-0.50 confusion matrix, and pooled calibration plot are shown in Figs 1 – 3 . Each participant contributed one held-out prediction per repetition to Figs 2 and 3 ; the counts are prediction instances rather than unique participants. Download: PNG larger image TIFF original image Fig 1. Random Forest ROC curves across 50 held-out outer folds. The dark line is the mean interpolated ROC; the shaded region is the 2.5th–97.5th percentile fold envelope. Thin lines show individual folds. https://doi.org/10.1371/journal.pone.0361117.g001 Download: PNG larger image TIFF original image Fig 2. Accumulated Random Forest confusion matrix at threshold 0.50 across ten repeated five-fold evaluations. Rows are normalised; counts are held-out prediction instances. https://doi.org/10.1371/journal.pone.0361117.g002 Download: PNG larger image TIFF original image Fig 3. Random Forest calibration plot from repeated held-out predictions. Points use quantile bins; the dashed line indicates perfect calibration. https://doi.org/10.1371/journal.pone.0361117.g003 For the secondary operating-point analysis, fold-specific training-derived thresholds had median 0.461 (IQR 0.406–0.522; range 0.322–0.667). Applied to outer test folds, mean sensitivity was 0.869 ± 0.070 and specificity was 0.831 ± 0.113. The variation across folds precludes presenting one retrospectively fitted threshold as a clinical cutoff. 4.2. Task-wise performance Single-task discrimination varied across the 25 tasks ( Table 3 ). Task 23 (telephone number under dictation) had the highest task-wise AUC (0.871 with Random Forest), followed by Task 17 (copy six words into boxes; AUC 0.864) and Task 19 (postal-order fields; AUC 0.853). Task 22 achieved AUC 0.830 and is therefore reported as moderate, not as one of the strongest tasks. These task-wise estimates are descriptive and were not used as independent evidence of a clinical battery. Download: PNG larger image TIFF original image Table 3. Best model for each handwriting task under the submitted task-wise nested cross-validation analysis. Values are mean ± SD across five outer folds where available. https://doi.org/10.1371/journal.pone.0361117.t003 4.3. Task reduction and stability The same-cohort GFS-SR search produced the previously reported maximum AUC of 0.9804 for Tasks 2, 5, 9, 10, 19, 21, and 24, compared with a same-cohort full-battery score of 0.9617. However, the fully nested, fixed-seven-task procedure achieved held-out AUC 0.916 ± 0.037, while the full 25-task comparator achieved 0.956 ± 0.030 on the
## Related Clinical Research

- [U.S. POINTER: Structured lifestyle program slows brain aging and cardiometabolic risk over 24 mont](https://medichelpline.com/clinical-feed/medical-news-today-0-combining-physical-mental-and-social-habits-slows-brain-aging-in-2-year-trial.md)
- [Psychotropic drug prescribing trends in Dutch nursing home residents with dementia, 2016–2023](https://medichelpline.com/clinical-feed/plos-one-17-trends-in-psychotropic-drug-use-in-nursing-home-residents-with-dementia-and.md)
- [High-resolution map of the basal ganglia reveals cell types tied to Parkinson’s and schizophrenia](https://medichelpline.com/clinical-feed/stat-news-1-stat-scientists-debut-map-of-basal-ganglia-a-key-brain-region-involved-in-an.md)
- [fMRI-based ARC tool differentiates SuperAgers from Alzheimer’s with strong AUC](https://medichelpline.com/clinical-feed/medical-news-today-2-new-brain-imaging-tool-distinguishes-between-alzheimer-s-disease-and-superagers.md)
- [Spatial single-cell atlas maps bilateral glial reactivity and neuronal growth programs after corti](https://medichelpline.com/clinical-feed/biorxiv-1-spatially-resolved-single-cell-atlas-reveals-cortical-glial-reactivity.md)

## Navigation
- [← Back to Neurology Feed](https://medichelpline.com/clinical-feed/neurology.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.