Benchmark audits reveal a ceiling effect hidden inside single-cell perturbation scoring
Benchmarks for single-cell perturbation models are supposed to answer a simple question: can a model predict how a cell will respond when its state is altered? This preprint shows that some common evaluation setups can answer something else entirely. By stress-testing an existing harness across two datasets, three models, and three uncertainty methods, the investigators found several ways in which the metric, the split design, and even the inference convention can distort the apparent ranking of methods.
One result is blunt. When each test cell is fed back into the model as input and then used as the target for scoring, an identity predictor that simply returns the input unchanged can attain a perturbation-effect correlation of 1.0 across all six splits. In other words, the benchmark can reward echoing the answer rather than predicting a biological response. For a field that relies on these scores to compare scVI, CPA, and a linear Ridge baseline, that ceiling is not a small technical nuisance. It changes what the metric is actually measuring.
Split design, label handling, and inference conventions drive the ranking
The harness covered two datasets, Norman 2019 and Replogle 2022, and four split types across six dataset-split combinations. The surface appearance is broad coverage, but the internal logic is uneven. One split held out sequencing batches instead of perturbations, which weakens the claim that it truly tested out-of-distribution perturbation generalization. Another split scored no genuinely novel perturbation after label collapse, so the model was not being pushed into the kind of extrapolation that the benchmark title implies.
The most consequential distortions were not all statistical. Unseen perturbations were silently replaced with a negative-control sentinel at inference, which changes the problem from prediction of a missing perturbation to substitution with a special fallback class. In a separate convention-dependent decision-value entry, a change in library-size handling moved the result by 29.2 percentage points even though the all-gene mean squared error shifted by just 0.01%. Small preprocessing choices can therefore produce large swings in the headline interpretation while barely moving a global loss metric.
A constant baseline outperformed a model in one dataset and collapsed in another
Comparison against a constant train-mean predictor exposed a stark dataset split. scVI beat that baseline on 99.7% of Norman cells, yet on Replogle cells it did so on only 0.2%. The models are not operating in a vacuum, and this kind of reversal signals strong dependence on dataset-specific structure, perturbation composition, and perhaps the distribution of cell states available during training. In practical terms, a model can appear nearly dominant in one cohort and nearly irrelevant in another without any change in architecture.
That kind of swing should not be brushed off as an exotic edge case. It points to a deeper issue in single-cell perturbation benchmarking: the same method may be evaluated on what looks like a common task while actually facing a substantially different data regime. If the benchmark does not preserve the biological novelty of the held-out perturbations, the comparison becomes less about causal generalization and more about recovery of familiar expression patterns.
Ranking uncertainty by what it prevents, not just by what it covers
The paper also takes aim at how uncertainty is scored. Instead of treating coverage as the only target, the investigators evaluated each uncertainty signal by the differential-expression gene error it avoided in the top k prioritized predictions. That shift matters because many experimental decisions are not made on the entire prediction set. They are made on a shortlist, where ordering determines which genes get follow-up validation and which are left behind.
Under that lens, the worst-calibrated method became the best ranker, while a benchmark that reported only coverage ordered the methods in the opposite direction. This is a direct warning against assuming that marginal calibration and decision-level utility are interchangeable. A predictor can produce intervals that look reasonable in aggregate and still fail to place the most biologically useful cases near the top of a ranked queue. In a lab workflow, that mismatch can waste validation slots on low-value candidates.
Conformal prediction survives, but only inside a narrower biological boundary
Not every finding was a failure. One result remained usable, though only within a defined boundary condition: split-conformal coverage calibrated on K562 residuals under-covered on RPE1, falling from 0.900 to 0.847. The same framework also showed that a nominal 0.900 marginal coverage can correspond to only 0.271 to 0.342 of the differential-expression-gene intervals that a prioritization decision would actually rely on. Those numbers matter because they separate interval validity from interval usefulness in downstream decision-making.
The biological implication is straightforward. Different cellular contexts can reshape the residual structure of a model, which means conformal calibration cannot safely be treated as universal across cell types. K562 and RPE1 do not share the same error geometry, so a calibration layer learned in one context may under-protect predictions in another. In a perturbation setting, where the goal is to identify signal after a gene or pathway is perturbed, poor calibration can lead to misplaced confidence precisely where the model is most likely to be wrong.
Six failure modes and a transferable audit template
The authors document six failure modes, each with a mechanism, a diagnostic, and a detection procedure intended to transfer to other benchmarking harnesses. The value of that audit is not limited to these specific datasets. Any evaluation system that mixes biological prediction with preprocessing conventions, label transformations, fallback sentinels, and decision-oriented uncertainty scoring can inherit the same vulnerabilities.
What makes this preprint especially consequential is its refusal to treat benchmark outputs as self-validating. A benchmark can be internally consistent and still be misaligned with the scientific question it claims to ask. Here, the claimed object is perturbation response under distribution shift, yet parts of the harness reward identity copying, obscure whether perturbations are truly unseen, and let a library-size convention swing a decision metric by almost a third of its percentage scale. For single-cell perturbation modeling, the next step is not merely better models. It is cleaner evaluation, recalibrated per cellular context, and reporting that distinguishes marginal coverage from the gene-set level decisions researchers actually make.