Frontier artificial intelligence (AI) models have advanced rapidly through training on internet-scale public data, yet such systems lack access to private clinical data. Neuroimaging is underrepresented in the public domain due to identifiable facial features within magnetic resonance imaging (MRI) and computed tomography (CT) scans, restricting model performance in clinical medicine. Here we show that frontier models underperform on neuroimaging tasks and that learning directly from uncurated data generated during routine clinical care at health systems, a paradigm we call ‘health system learning’, yields high-performance, generalist neuroimaging models. We introduce NeuroVFM, a visual foundation model trained on 5.24 million clinical MRI and CT volumes using a scalable volumetric predictive architecture. NeuroVFM learns comprehensive representations of brain anatomy and pathology, achieving state-of-the-art performance across multiple clinical tasks, including radiologic diagnosis and report generation. The model embeds MRI and CT scans into a shared neuroanatomic latent space and grounds diagnostic findings. When paired with open-source language models, NeuroVFM generates radiology reports that surpass frontier models in accuracy, clinical triage and expert preference.
Frontier artificial intelligence (AI) models have advanced rapidly through training on internet-scale public data, yet such systems lack access to private clinical data. Neuroimaging is underrepresented in the public domain due to identifiable facial features within magnetic resonance imaging (MRI) and computed tomography (CT) scans, restricting model performance in clinical medicine. Here we show that frontier models underperform on neuroimaging tasks and that learning directly from uncurated data generated during routine clinical care at health systems, a paradigm we call ‘health system learning’, yields high-performance, generalist neuroimaging models. We introduce NeuroVFM, a visual foundation model trained on 5.24 million clinical MRI and CT volumes using a scalable volumetric predictive architecture. NeuroVFM learns comprehensive representations of brain anatomy and pathology, achieving state-of-the-art performance across multiple clinical tasks, including radiologic diagnosis and report generation. The model embeds MRI and CT scans into a shared neuroanatomic latent space and grounds diagnostic findings. When paired with open-source language models, NeuroVFM generates radiology reports that surpass frontier models in accuracy, clinical triage and expert preference. NeuroVFM reduces hallucinated findings and critical errors, offering safer clinical decision support. These results establish health system learning as a paradigm for building generalist medical AI and provide a scalable framework for clinical foundation models.
Multimodal large language models (MLLMs) derive much of their capability from learning on internet-scale data, enabling these models to approximate the breadth of human experience across language, images and video. Clinical medicine, however, is underrepresented on the public internet. MLLMs trained exclusively on public data lack access to the rich private information embedded in real-world patient care, which fundamentally limits their performance on clinical tasks. We propose ‘health system learning’ as a new paradigm in which medical foundation models learn directly from uncurated data generated during clinical operations at health systems. By learning in the same complex, nuanced environment in which expert clinicians themselves train, rather than second-hand internet descriptions used by MLLMs, AI models can acquire rich, medical representations grounded in anatomy, pathology and clinical workflows. MLLMs know the map; health system learners know the territory.
To demonstrate the strength of health system learning, we introduce NeuroVFM, a generalist neuroimaging visual foundation model trained on all clinical MRI and CT studies from a large academic health system. Unlike previous models that rely on data curation, human annotations or radiology report supervision 1 , 2 , NeuroVFM is optimized for general neuroimaging through a self-supervised vision-only algorithm called Volumetric Joint-Embedding Predictive Architectures (Vol-JEPA) 3 . Our method enforces representation learning across imaging modalities and disease spectra, capturing both global and fine-grained neuroanatomic and pathologic features. Health system learning with Vol-JEPA enables NeuroVFM to achieve state-of-the-art performance, surpassing leading proprietary and open-source frontier models across multiple clinical tasks, including radiologic diagnosis and report generation. NeuroVFM predictions are diagnostically grounded, with pathologic image regions mapped to neurologic diagnoses. When integrated with open-source language models, NeuroVFM acts as a visual perception module that outperforms GPT-5 and Claude Sonnet 4.5 on neuroimaging interpretation and triage.
To train NeuroVFM, we assembled UM-NeuroImages, a multicenter, multimodal dataset comprising 566,915 CT and MRI studies (5.24 million three-dimensional (3D) volumes) of the brain, head, neck, face and orbits from over two decades of routine clinical care at Michigan Medicine (Fig. 1 and Extended Data Fig. 1 ). We defined a diagnostic ontology of 74 MRI and 82 CT diagnoses spanning neoplastic, traumatic, infectious, inflammatory and other major pathologic categories. Diagnostic labels were automatically assigned from radiology reports using a validated large language model (LLM) pipeline 4 , with a subset verified by expert neuroradiologists (Supplementary Table 1 ). These labels were not used for self-supervised pretraining and were used solely to train and evaluate supervised diagnostic heads. Detailed dataset characteristics, including sequence types, image resolutions and diagnosis distributions, are provided in Extended Data Fig. 2 .
a , Health system learning directly models the data-generating process of clinical operations at large health systems. The UM-NeuroImages dataset comprises 5.24 million volumes from 566,915 studies, acquired over 20 years at Michigan Medicine. Age and sex distributions are shown below. b , NeuroVFM was trained using Vol-JEPA, a scalable volumetric self-supervised method that learns a unified latent space for CT and MRI. A 3D volume is partitioned into a small context and larger masked target with the background removed. The context is encoded by an online (‘student’) 3D vision transformer. A predictor combines context latents with position encodings of the masked target region to predict the masked region latents. Ground truth latents for the masked region are generated by an offline (‘teacher’) encoder updated by an exponential moving average, with gradients stopped through the teacher. Training minimizes the distance between the predicted and teacher latents using a smooth L1 loss. c , At inference, NeuroVFM encodes all volumes in a neuroimaging study into latent visual tokens for downstream tasks. The same visual tokens can be used to fine-tune an open-source multimodal language model (that is, Qwen3-14B, LLaVA-1.5-style) to generate radiology reports. Illustrative findings and corresponding grounded attention maps are shown. The findings can then be passed to a frontier reasoning LLM (that is, GPT-5-thinking) for interpretation and triage. NC, non-contrast.
JEPAs are self-supervised learning methods that, given a context region within a data sample, predict a non-overlapping target region in a learned latent space. JEPAs have achieved state-of-the-art performance on image and video data but have not been applied to volumetric medical images 5 , 6 , 7 . We adapted this framework for volumetric neuroimaging (Vol-JEPA), designing a neuroanatomy-informed masking strategy that exploits the shared spatial structure in neuroimages (Fig. 1b and Extended Data Fig. 3a ). We hypothesized that this objective would require the encoder to learn representations invariant to imaging protocol and equivariant to neuroanatomy and pathology. Because Vol-JEPA learns through latent prediction, it does not require voxel-level augmentations, negative pairs, generative decoders or paired radiology reports, enabling efficient scaling to large, uncurated clinical datasets. Vol-JEPA outperformed alternative self-supervised strategies, including contrastive and reconstruction-based objectives, on downstream diagnostic tasks (Extended Data Fig. 3c ).
To compare health system learning with existing approaches, we curated a temporally held-out diagnostic cohort comprising all consecutively evaluated patients who underwent a CT or MRI scan of the head or neck between 1 June 2023 and 31 May 2024, without exclusion, yielding more than 21,000 CT and 29,000 MRI studies (Supplementary Table 2 ). We compared NeuroVFM against five baselines spanning report-supervised health system pretraining (HLIP 8 and Prima 4 ), voxel reconstruction self-supervision on the same health system data (NeuroMAE), internet-scale self-supervised learning (DINOv3 (ref. 9 )) and biomedical vision-language pretraining (BiomedCLIP 10 ). For each model, we standardized evaluation by training identical study-level attentive probes on frozen encoder embeddings.
On the primary endpoint (macro-averaged area under the receiver operating characteristic (AUROC) across all 156 diagnostic tasks), NeuroVFM achieved 92.68 (95% confidence interval (CI) 92.27−93.08) on CT and 92.49 (95% CI 92.14−92.82) on MRI (Fig. 2a ). Because PRIMA, HLIP, NeuroMAE and NeuroVFM were all pretrained on UM-NeuroImages, these comparisons control for data source, and differences are attributed to pretraining objective and model design. NeuroVFM outperformed both report supervision (HLIP: +0.98, 95% CI 0.76−1.20; PRIMA: +3.87, 95% CI 3.53−4.21) and voxel reconstruction (NeuroMAE: +1.55, 95% CI 1.35−1.75). The largest margins were over internet-scale baselines (DINOv3: +2.24, 95% CI 1.97−2.50; BiomedCLIP: +2.88, 95% CI 2.62−3.15), where differences in both objective and pretraining data compound. For individual pathology categories, NeuroVFM exceeded baselines broadly after Benjamini−Hochberg correction ( q 2b,c ). NeuroVFM significantly outperformed DINOv3, BiomedCLIP and NeuroMAE in at least 12 of 17 CT categories and 14 of 18 MRI categories. Against HLIP, NeuroVFM won 10 of 17 CT categories with no losses, but only five of 18 MRI categories, with HLIP winning the inflammatory category and the rest being non-significant. NeuroVFM also demonstrated better label efficiency, requiring fewer labeled positive examples than each baseline (31.5–55.9% fewer CT scans; 6.5–37.9% fewer MRI scans) to reach equivalent performance (Fig. 2d ). Per-task and per-category results are in Supplementary Tables 3 and 4 , respectively.
a , NeuroVFM performance over 82 CT and 74 MRI diagnostic tasks, compared to both health system-scale (HLIP, PRIMA) and internet-scale (DINOv3, BiomedCLIP) models. NeuroVFM outperforms models trained with language supervision and those trained on public internet data. PRIMA was trained and evaluated only on MRI scans. Results are mean ± 95% CI. b , c , Performance across diagnostic ontologies, such as traumatic, congenital and ischemic lesions, is shown for both CT ( c ) and MRI ( b ). NeuroVFM consistently outperforms other baselines. Results are mean ± 95% CI. Significance was computed using paired bootstrap, corrected for multiple hypotheses using Benjamini−Hochberg with a false discovery rate of 0.05. n refers to the number of tasks per ontology. ENT stands for ear, nose and throat. d , We discovered an empirical log-linear scaling relationship between the number of positive training examples and model performance. This relationship held across at least four orders of magnitude, imaging modalities and models (Extended Data Figs. 4 and 5 ). We plot label equivalence factors for baselines relative to NeuroVFM, which requires fewer training positives to reach the same performance. Results are mean ± 95% CI.
Performance scaled with both pretraining data and encoder capacity, improving by +7−10 AUROC points from 5% to 100% of UM-NeuroImages on both modalities (Extended Data Figs. 4c and 5c ). These trends showed no evidence of saturation, suggesting continued benefit from additional health system data. Multimodal pretraining on both CT and MRI scans was non-inferior to unimodal models, indicating that learning a shared latent space did not degrade within-modality performance. NeuroVFM also maintained performance across MRI manufacturers, magnetic field strengths, demographic subgroups and medical centers (Extended Data Fig. 6 ).
On public neuroimaging benchmarks spanning brain age estimation, neurodegenerative and neurodevelopmental disorders and traumatic brain injury (Extended Data Fig. 7 ), NeuroVFM achieved the strongest performance on tasks whose labels are not directly encoded in radiology reports. Using classifiers trained on the Alzheimerʼs Disease Neuroimaging Initiative (ADNI) cognitively normal versus Alzheimerʼs disease task, NeuroVFM generalized strongly to external cohorts, achieving AUROCs of 93.49 (95% CI 90.03−96.40) on AIBL and 88.09 (95% CI 80.47−94.24) on OASIS-1 compared to 76.18 (95% CI 70.16−81.77) and 69.08 (95% CI 58.84−78.56) for HLIP, respectively. HLIP outperformed NeuroVFM on most CQ500 tasks, a small CT benchmark with label definitions closely aligned to report descriptions. This advantage diminished on the larger RSNA-ICH benchmark, where NeuroVFM exceeded HLIP. Together, these results provide evidence that report-supervised models can perform well within specific diagnoses, whereas NeuroVFM representations generalize better as task diversity increases. Full results across all evaluated public benchmarks are in Supplementary Table 5 .
Self-supervised learning on large-scale natural image and video datasets can give rise to representations with rich spatial and semantic features 9 , 11 . Achieving similar results in medical foundation models remains a major challenge. NeuroVFM patch embeddings reveal spatially ordered neuroanatomic clusters, without needing segmentation or report supervision during pretraining (Fig. 3a ). These results indicate that NeuroVFM has learned a neuroanatomic manifold that encodes semantic and spatial information. We quantified these properties through three evaluations of increasing complexity: anatomical matching across patients, pathology retrieval across patients and MRI sequences and zero-shot diagnostic transfer across patients and modalities.
a , Patch embeddings from a T1-weighted MRI sequence plotted using t-SNE and colored by neuroanatomical region from SynthSeg segmentation 28 . NeuroVFM representations cluster by anatomy without explicit supervision during pretraining. Language-supervised baselines were not evaluated here because their training objectives do not explicitly learn dense patch-level representations, and cross-modal alignment is an expected result of CLIP. b , Cross-protocol anatomical matching accuracy across seven landmarks compared to NeuroMAE, a voxel reconstruction baseline. Results are mean ± two-way cluster-robust standard error. c , One-shot tumor retrieval over BraTS21 sequences using paired glioma segmentation masks 12 . Given a query patch, the nearest neighbor in a support image transfers its label. NeuroVFM achieves IoU greater than 0.5, even across sequence types. Results are mean ± two-way cluster-robust standard error. d , Zero-shot cross-modal diagnostic transfer performance, with CT-trained attentive classifier evaluated on MRI across nine shared diagnostic labels. NeuroVFM exhibits less than a 5-point AUROC drop, whereas other baselines degrade substantially, indicating that their representations do not generalize across modalities. Results are mean ± 95% CI. Horizontal dotted lines in c and d indicate reference benchmarks for interpretation. Cor., coronal; Sag., sagittal; sim., similarity; t-SNE, t-distributed stochastic neighbor embedding.
To test whether NeuroVFM encodes shared neuroanatomy across imaging protocols, we performed nearest-neighbor patch matching of anatomical landmarks across image pairs differing in both sequence and orientation (for example, coronal T2 and sagittal T1 with contrast). Across seven landmarks, NeuroVFM achieved a 44.2% lower mean localization error compared to NeuroMAE (2.27 cm versus 4.07 cm), indicating that neuroanatomic information is invariant to acquisition parameters. This is consistent with the fine-grained spatial localization and grounded diagnosis achieved by NeuroVFM-based classifiers on both MRI and CT (Extended Data Fig. 8 ).
To determine whether this invariance extended to pathology, we designed a one-shot tumor retrieval task using BraTS21 (ref. 12 ) segmentation masks: given a query patch, we retrieve its nearest-neighbor patch in a support image and transfer its label (Fig. 3c ). Pairs were selected such that tumors occupied different anatomical locations, and performance was evaluated across MRI sequences (for example, fluid-attenuated inversion recovery (FLAIR) to T2). NeuroVFM achieved intersection over union (IoU) exceeding 50% regardless of transfer direction, whereas NeuroMAE performed worse within the same sequence and degraded further across sequences. To test whether this generalizes to study-level diagnosis, we evaluated CT-trained attentive classifiers on MRI, measuring the AUROC drop relative to within-modality performance (Fig. 3d ). Across nine shared diagnostic tasks, NeuroVFM exhibited less than a 5-point drop in AUROC, outperforming NeuroMAE and DINOv3 on eight of nine tasks. These results demonstrate that NeuroVFM has learned a modality-agnostic representation of neuroanatomy and disease. These properties do not emerge from alternative training objectives applied to the same data, indicating that they arise at the intersection of volumetric latent prediction and large-scale neuroimaging datasets.
Generating accurate radiology findings from the raw clinical imaging stream is a prerequisite for enabling real-time worklist prioritization, reducing reporting delays and extending expert-level interpretation to settings without subspecialty coverage 13 , 14 . Unlike curated benchmarks, report generation requires operating on the full diversity of studies encountered in routine practice: motion artifact, variable protocols, incidental findings and the long tail of rare pathology. We tested whether NeuroVFM could support this task by training a simple generative model to produce structured findings from ‘uncurated clinical neuroimaging studies’ (Fig. 4a ).
a , Overview of the study design and workflow for report generation and acuity assessment. Step 1: a generative multimodal model, such as NeuroVFM-LLaVA or GPT-5, is prompted to generate image key findings. Step 2: an LLM, GPT-5 or Claude Sonnet 4.5, is then used as a standardized judge to evaluate the study or acuity level based on the findings generated in step 1. Step 3: ‘Blinded’ clinical experts then evaluate whether the study was stratified correctly, the quality of the generated reports and which report they prefer. NeuroVFM outperformed the other frontier models on overall acuity performance ( b ), urgent findings generation ( c ) and all natural language processing metrics ( d and Extended Data Fig. 9 ). Results are mean ± 95% CI. e , Blinded clinical experts noted that NeuroVFM was more likely to generate correct key findings, less likely to hallucinate and much less likely to make a laterality error compared to GPT-5. f , Clinical experts also preferred NeuroVFM reports more than 2:1 over GPT-5 reports. g − i , Illustrative examples of urgent ( g ), routine ( h ) and unremarkable ( i ) neuroimaging studies in the expert-annotated testing set. Results are mean ± 95% CI. acc., accuracy; CSF, cerebrospinal fluid.
We paired the frozen NeuroVFM encoder with Qwen3-14B via LLaVA-1.5-style visual instruction tuning 15 , 16 and fine-tuned the language model to generate structured key findings from UM-NeuroImages studies. Generated findings were then passed to a fixed reasoning model that assessed acuity (unremarkable, routine or urgent), providing a model-agnostic evaluation of visual perception quality (Supplementary Fig. 1 ). We validated this protocol by confirming that frontier reasoning models achieved greater than 96% accuracy when given the ground truth radiology findings, establishing that acuity differences between models reflect perception quality rather than reasoning ability (Extended Data Fig. 9b ). Multimodal frontier models (GPT-5 and Claude Sonnet 4.5) with ‘provider-imposed input constraints’ were the strongest available comparators, as existing open-source medical vision-language models (for example, MedGemma) failed to produce clinically usable findings (Extended Data Fig. 9 and Supplementary Table 6 ).