Cross-species annotation is where single-cell foundation models separate from simpler tools
Single-cell foundation models are being marketed as general-purpose engines for cell annotation, perturbation prediction, and gene-regulatory inference. This benchmark asks a narrower question with immediate methodological value: where do these models actually outperform simpler approaches, and where do they merely keep pace with them? The answer is mixed, and the biggest gains appear only when the task demands transfer across species or across contexts that are not well covered by the training data.
The analysis compared five foundation models — scGPT, Geneformer V1, Geneformer V2, Nicheformer, and scFoundation — against CellOracle, supervised perturbation models including STATE, and task-matched simple baselines. The same framework was applied across K562 Perturb-seq screens, primary cells, healthy-donor and isogenic leukodystrophy brain organoids, a public organoid CRISPR screen, and lung cancer biopsies. That design matters because it forces the models to answer the same biological question under different levels of distribution shift, rather than benefiting from a favorable benchmark built around any one algorithm.
The clearest advantage emerged in cross-species cell-type annotation. When labels had to move from human to mouse lung, Nicheformer reached 0.98 accuracy, while Geneformer V2 and scFoundation each landed around 0.90. The best baseline reached 0.68. In that setting, foundation-model embeddings appear to capture conserved transcriptional structure that survives species boundaries, while conventional feature reductions struggle to preserve enough of the biology for reliable transfer. Within a species, the story was less dramatic: foundation-model embeddings annotated cells about as accurately as principal components. In other words, for routine within-species annotation, the expensive representation did not clearly outperform a simpler one.
Perturbation prediction improved when the model had to infer direction, not exact magnitude
The perturbation benchmarks showed a similar pattern. Zero-shot in silico perturbations captured the direction of real responses in K562 cells, with up to 75% of perturbations shifted in the expected direction for scFoundation, with p < 1e-8. That is a meaningful result, but it is also a narrowly defined one: the models were strongest at getting the sign of the response right rather than reconstructing a fully quantitative molecular phenotype.
For cell therapy development and disease modeling, directionality can still be useful. A model that consistently predicts whether a gene knockout pushes a transcriptional program up or down can help prioritize interventions before a wet-lab screen, especially when the library is large and expensive to run. The mechanism is straightforward enough: if the latent representation has learned enough of the regulatory grammar of the cell, it can infer which axes of transcription are likely to bend after a perturbation. But the benchmark does not suggest that foundation models have already solved perturbation biology. They performed well on a subset of responses, and the stronger claims remain bounded by the experimental systems chosen here.
Fine-tuning changed classification accuracy, but it did not alter knockout rankings. That split is important because it suggests the models can be adapted to specific labels without necessarily changing the underlying prioritization of perturbation effects. In practical terms, supervised refinement improved some downstream classification behavior while leaving the order of predicted knockouts largely intact.
Regulatory networks recovered tumor signals better than simple co-expression
Regulatory inference was the third domain where foundation models gained ground, though again not everywhere. In lung cancer biopsies, Geneformer V2 networks recovered patient-level tumor differential expression better than co-expression networks, with fold enrichment of 2.9 versus 2.0. Co-expression remains a useful baseline because it is simple, transparent, and easy to compute, but it cannot distinguish correlation from directionality or regulatory dependence. A model that encodes broader single-cell context has a better chance of separating genuine tumor programs from background correlation structure.
Here, the biological rationale is compelling. Tumor tissue is a heterogeneous mixture of malignant and nonmalignant cells, and differential expression at the patient level can reflect shifts in composition as much as changes inside the cancer cells themselves. A regulatory model that better reconstructs patient-level signals is doing more than ranking genes; it is learning which transcriptional relationships are preserved across cells and which emerge from disease state. Even so, the enrichment numbers remain modest, and the benchmark does not justify treating these networks as a replacement for orthogonal validation.
Organoid knockout effects were often driven by expression context first
The organoid experiments place an important constraint on how far these tools can currently be trusted. In healthy-donor and isogenic leukodystrophy brain organoids, predicted knockout effects were largely explained by where the target gene is expressed. An expression-matched control test isolated the gene-specific part of the effect, and that component accounted for only 12% to 35% of the observed changes. This is a sober result. Much of what looks like a perturbation signature may simply reflect baseline expression geography rather than a deep causal effect tied to gene function.
The isogenic Canavan and Krabbe organoids, along with 4H patient organoids, were especially revealing because they were meant to test whether methods could disentangle biological mechanism from clone and donor differences. They could not, at least not yet. With one library per line and time point, the systems did not provide enough replication to separate method performance from variability introduced by clones and donors. That is not a minor technical footnote; it defines the minimum evidence needed before these models can be validated in a more clinically relevant setting.
Benchmarking, not model worship, is the point of the release
The benchmark package, baselines, pipeline checks, and isogenic reference task were released as fmaudit. That makes the paper more than a comparative exercise. It offers a shared framework for testing future single-cell foundation models on the specific tasks where they already appear to help, while also exposing the places where simpler methods still hold their own.
The most defensible conclusion is selective, not sweeping. Foundation models add the clearest value when labels must move across species, when perturbation direction is the goal, and when regulatory inference must recover patient-level disease structure from noisy single-cell data. They are less convincing for ordinary within-species annotation and remain sensitive to baseline expression patterns in organoid perturbation settings. The remaining challenge is not just to build larger models, but to design validation systems with enough replication to separate true biological signal from clone-to-clone and donor-to-donor variation. Until that is done, the strongest claims will stay bounded by the benchmark that produced them.