OOD Benchmark Exposes Fragility in Genetic Perturbation Prediction
Genetic perturbation prediction models are being positioned as in silico stand-ins for genome-scale screening, but a new benchmark suggests they are far less dependable once the test set moves beyond familiar territory. The preprint introduces EL-PERTURB, a framework built to evaluate whether models that predict transcriptional responses to gene inactivation can hold up when asked to generalize outside the data they were trained on.
The central finding is blunt. Performance fell substantially under out-of-distribution conditions.
That result matters because random train-test splits can make a model appear stronger than it really is. If the training and test examples are too similar, a model can look accurate without truly learning the biology needed for a fresh perturbation setting. EL-PERTURB was designed to avoid that trap by using a stratified out-of-distribution splitting strategy that partitions train and test sets into functionally distinct regions of transcriptional space.
Twenty Models, Six Architecture Classes, and a Harder Test of Generalization
The benchmark examined 20 perturbation prediction models spanning six architecture classes across two major screen types: essential gene datasets and genome-wide Perturb-seq screens. The design was intentionally broad, not limited to a single favored framework or a single experimental setting. By comparing many models under the same harsher evaluation rules, the study aimed to distinguish real generalization from performance that depends on familiar data geometry.
In that setting, architectural sophistication did not translate into a consistent advantage. Even models built with single-cell foundation model embeddings failed to show a reliable edge over simpler, carefully tuned baselines. The implication is direct: adding complexity does not automatically buy biological portability.
A model can be elaborate and still fail when the data shift.
This point is especially important for drug discovery workflows that hope to use perturbation prediction as a shortcut around wet-lab screening. If the model cannot predict responses outside its training distribution, then its value as a screening surrogate narrows quickly. The benchmark does not claim these systems are useless. It shows that their apparent strength depends heavily on how they are tested.
Why Random Splits Can Hide the Real Problem
The weakness EL-PERTURB targets is familiar across machine learning, but it has a sharper edge in functional genomics. Random splits often mix train and test examples that are very close in transcriptional space, which can inflate apparent accuracy. In a perturbation setting, that may mean the model is being asked to interpolate between nearly adjacent biological states rather than infer the effect of a genuinely new perturbation.
EL-PERTURB changes that setup by separating functionally distinct regions of transcriptional space. The result is a more demanding and more realistic test of whether the model has learned perturbation logic or merely memorized patterns that recur in the training set. Under those conditions, the performance drop was substantial, and the gap between headline model complexity and practical utility became harder to ignore.
This is not just a technical concern. In drug discovery, the difference between interpolation and extrapolation determines whether a model can guide the selection of new targets or merely restate what has already been measured. A system that performs well only on nearby examples may still be useful for prioritization, but it cannot be treated as a robust replacement for experiment.
Foundation Models Did Not Rescue Generalization
One of the more consequential findings is the lack of a dependable boost from single-cell foundation model embeddings. These systems are often assumed to provide a richer representation of cellular state and therefore a stronger basis for downstream prediction. EL-PERTURB did not find that assumption borne out in a way that consistently overcame the out-of-distribution challenge.
The practical lesson is not that foundation models lack value. It is that representation quality alone is not enough when the evaluation regime forces a model to confront biologically novel perturbations. A representation may be rich, but if the predictive head cannot translate that richness into reliable extrapolation, the downstream output still fails where it matters most.
For researchers building perturbation models, this creates a sharper benchmark for progress. Better results on random splits are no longer sufficient evidence of readiness for screening applications. The field needs methods that can survive stricter distribution shifts and preserve performance when the test set no longer resembles the training set in simple, forgiving ways.
What EL-PERTURB Means for Wet-Lab Replacement Claims
EL-PERTURB does more than compare model scores. It sets a higher bar for claims that perturbation models can act as surrogates for wet-lab experiments. The framework establishes a gradient of generalization difficulty, allowing performance to be judged across increasingly challenging conditions rather than in a single easy-to-game configuration.
That matters for translational use cases. Genome-scale screening is expensive and time-consuming, which is why there is such interest in computational substitutes. But a substitute must be reliable when the biological context shifts. If a model breaks down outside the distribution it has seen before, its role shifts from surrogate to supplement.
The preprint also provides a practical signal for future method development. Simpler baselines, when properly tuned, may remain surprisingly competitive. Future progress may depend less on layering in more elaborate architectures and more on building evaluation strategies, training objectives, and biological priors that explicitly reward true extrapolation.
A Benchmark That Reframes Model Readiness
The strongest contribution of EL-PERTURB is methodological. It formalizes a way to ask whether perturbation prediction models are actually ready for the kinds of novel biological questions that drug discovery demands. The answer, at least across the benchmarked settings, is sobering: out-of-distribution prediction remains a major failure point, and architectural complexity has not yet closed that gap.
The preprint is still awaiting peer review, so the field will want to see independent scrutiny of the benchmark design and the specific model comparisons. Even so, the message is already clear enough to influence how future work is judged. Claims of generalization will need to survive stricter splits, not just familiar ones, if they are to support real screening decisions.