The authors analyzed 1,566 published response signatures coming from 1,235 papers across 2,045 treated patients. Their finding was stark. The collective predictive ability of those signatures was essentially at chance, with an reported AUC of 0.515. Short, blunt fact: many contemporary computational signature pipelines did not discriminate responders from non-responders in this aggregated cohort. The study team also reported that older, pre-existing immune scores trailed these newer signatures less often than expected—those legacy scores outperformed the wave of recent signatures in many comparisons. Data indicated that the validation filters themselves have low specificity: a median 44% of random gene sets passed the same validation used to endorse real signatures.
The authors examined how a widely used tool, TIDE, influenced signature labeling. TIDE interprets features of exhausted T cells—the targets of PD-1 antibodies—as evidence of failed immunity. That interpretation, investigators reported, redirected classification for many signatures: of 1,524 evaluated, 994 were oriented by TIDE to label immune-infiltrated tumors as non-responders. Yet the cohort-level outcomes showed those immune-rich tumors responded more often. TIDE’s readout therefore introduced a systematic bias in computational validation that flipped the expected relationship between immune infiltration and therapeutic response.
The authors developed and applied ResponseVerdict to reorient published scores by measured response in independent treated cohorts. They trained or selected scores using only data that were publicly available before each target, avoiding information leakage. Compared with TIDE, ResponseVerdict raised AUROC by 0.14. The new approach was tested in many pairwise comparisons: of 2,000 paired evaluations, ResponseVerdict was significantly better than the published predictors, immune scores and learning programs in 1,178 comparisons, and worse in none of them according to the reported analyses. In a blinded, untouched cohort, ResponseVerdict improved responder recovery by 10.2 patients per 100 compared with TIDE. Against the original authors’ own signatures across 2,179 patients it recovered 16.2 additional responders per 100 enrollment slots. When restricted to data available before the target signature, ResponseVerdict still added 9.5 responders per 100 versus GEP18.
The preprint documents a worrying discordance between computational claims and subsequent bench experiments. Of 392 experiments that the authors reviewed which ostensibly tested immune- or stromal-cell signals, 376 were performed in other cell types—different from the cell context implied by the computational signature. That mismatch raises questions about the biological rationale driving many follow-up studies. The paper also notes that a low-TMEscore rule, derived computationally, has been set as an eligibility criterion in a randomized trial. The investigators reported these translational choices to illustrate how computational misdirection can propagate into clinical trial design.
Investigators did not confine their analysis to performance metrics alone. They translated differences into prospective clinical consequences. When ResponseVerdict was used to allocate treatment across 14 public randomized immunotherapy trials—using only earlier trials as the scoring source—the algorithm outperformed every program evaluated on the same trials. The reported benefit was an added 67.3 progression-free months per 100 patients over random allocation. Those aggregate numbers were accompanied by per-100-patient gains versus other predictors: 16.2 more responders compared with authors’ original signatures, 9.5 more versus GEP18 under earlier-data constraints, and a 10.2-responder advantage compared with TIDE in a blind-scored cohort.
The authors declared no competing interests. The manuscript is a preprint and has not been peer-reviewed; the study team provided supplementary material and trial-level comparisons. The public report documents methodological critiques of existing validation pipelines, the specific bias introduced by TIDE, and numerical gains achieved by ResponseVerdict. Remaining open questions include independent external replication under prospective conditions and peer review scrutiny of the algorithmic choices and cohort selection. The source did not report prospective clinical deployment details, regulatory milestones, or randomized trial protocols initiated specifically to validate ResponseVerdict—those elements remain to be addressed in subsequent work and formal review.