Analyzing the performance of deep learning splice prediction algorithms

by Nathan Fortier, Gabe Rudy, Andreas Scherer SpliceAI is the leading tool for predicting splice-altering variants, but restrictive licensing limits clinical adoption. While open-source implementations have been published with author-reported comparisons, independent benchmarking across diverse datasets is needed to establish equivalence.

We compared the original SpliceAI with two open-source implementations (OpenSpliceAI and CI-SpliceAI) and a legacy ensemble baseline across six datasets: a curated set of 1,316 validated variants, 213 variants with splice-assay data, 99,601 variants from the SPiP splicing prediction study, 242 manually curated deep intronic pathogenic variants, and two ClinVar-derived datasets comprising 53,600 intronic variants and 58,064 variants spanning all genomic contexts. The deep learning models were also evaluated against an ensemble of four legacy splice-prediction tools.

Across all datasets, the deep learning algorithms outperformed the legacy ensemble. All three deep learning algorithms showed similar performance on the larger datasets dominated by canonical splice site variants (balanced accuracies 0.889-0.977).

On the deep intronic benchmark, the original SpliceAI achieved the highest balanced accuracy (0.940), outperforming both CI-SpliceAI (0.890) and OpenSpliceAI (0.841).