The authors conducted a scoping review following PRISMA-ScR guidelines. They searched four databases covering January 2022 through February 2026. The protocol was registered on PROSPERO (CRD420251082465). This review focused on text-based and text-derived neurology tasks addressed with Large Language Models. The stated objective: chart the evidence on use cases, evaluation approaches, metrics, and safety concerns so clinicians and researchers can assess readiness for clinical integration.
Thirty-two studies met inclusion criteria; they were published between 2023 and 2026. The cohort of papers included three broad design classes: nine simulation or feasibility studies, seventeen analyses of retrospective clinical data, and six prospective evaluations. Investigators applied models across seven neurology subspecialties. The distribution of study types suggests early-phase work dominated by retrospective and simulated assessments rather than large-scale prospective validation.
Most included studies used proprietary models. Five implemented retrieval-augmented generation approaches to supplement model outputs with external text, and one study employed a multiagent framework. The source did not list all specific model names or parameter settings for each study; those granular model-level details were not comprehensively reported in the abstract. Authors noted that proprietary systems predominated over open, fully reproducible model releases.
Reported performance varied markedly by task type. Constrained tasks achieved the strongest metrics: binary diagnostic classification showed AUC values ranging from 0.75 to 0.94. Information extraction tasks reported F1 scores around 0.89–0.90. Patient-education question answering accuracy ranged from 68% to 97%. Decision support for ischemic stroke thrombectomy reached an AUC of 0.92 in one report. By contrast, open-ended case-based reasoning produced substantially lower accuracy (42%–54%). These results indicate that LLMs perform better when the task and answer space are narrow and well-defined, and weaker when complex clinical reasoning and unconstrained outputs are required.
Across studies the review team identified recurrent safety concerns. Hallucinated or fabricated citations appeared in multiple reports. Calibration problems manifested as overconfidence; the abstract gives a concrete example where an epilepsy diagnostic task produced 100% sensitivity but only 27% specificity, indicating inappropriate certainty coupled with poor discriminative balance. Authors also documented a systematic bias toward recommending interventions rather than conservative management. These safety signals were consistently noted across study designs and pose tangible risks if models are used without clinician oversight.
The investigators adapted QUADAS-AI for charting study quality. Results showed one study with low overall risk of bias (3%), sixteen studies with moderate risk (50%), and fifteen with high risk (47%). The principal driver of elevated bias was absent external validation: 31 of 32 studies (97%) lacked it. Benchmarked against high-reliability and Six Sigma standards, the authors stated reported error rates remain orders of magnitude too high for autonomous clinical operation.
Based on the compiled evidence, the authors recommended narrow, clinician-supervised applications for current LLMs in neurology. Examples identified as reasonable include documentation assistance and patient education. The review explicitly cautioned against autonomous diagnostic or therapeutic use. The heterogeneous evidence base and persistent safety signals underpin that caution. The abstract did not provide specific operational guidelines, vendor-agnostic deployment frameworks, or validated clinical workflows; those elements were not reported in the summary.
The review team identified several clear evidence gaps. Prospective studies and external validation were scarce. Standardized reporting of methods and safety assessments was lacking across the literature. The authors called for rigorous, prospective, externally validated studies that incorporate standardized outcome metrics and explicit ethical and safety evaluations. They argued these steps are necessary before broader clinical adoption can be responsibly considered. The abstract did not enumerate timelines, regulatory milestones, or specific study designs to address these gaps; such details were not provided in the source.
Limitations reported within the abstract include the early and heterogeneous nature of the evidence and the near-universal absence of external validation. Open clinical questions that remain—based on the authors’ synthesis—include how to measure acceptable error rates for autonomous tasks, which evaluation frameworks best capture safety in real-world deployment, and how to mitigate documented calibration and intervention biases prior to clinical use.