BenchmarkHub tackles the reproducibility gap in healthcare LLM scoring
Large language models are increasingly being measured by benchmark scores, yet those scores can be difficult to trust when the prompt, inference settings, and model build are not fully documented. In healthcare, that weakness is more than an academic nuisance. A model that appears strong on paper may fall apart when the exact grading rules, answer format, or multimodal inputs change. BenchmarkHub was built to close that gap by turning benchmark evaluation into an auditable process rather than a one-off performance claim.
The platform is an open-source Python web application built with Django 5.2 and SQLite, released under the MIT licence and available through GitHub. It can be used through a browser interface, command-line tools, or a REST API, and the same evaluation logic runs across all three access paths. Fixed seeds control question sampling, models can be run directly or as tool-using agents inside a code sandbox, and every prompt, reply, and tool call is stored. In practical terms, that means a result can be reconstructed later instead of being accepted on trust.
One sentence captures the core problem: benchmark scores without a full audit trail are fragile.
Fixed seeds, stored prompts, and identical records across workflows
BenchmarkHub was designed so interactive use and scripted pipelines produce identical records. That matters because reproducibility failures often begin with small implementation differences: a hidden prompt template, a changed decoder setting, a different question subset, or a grading step that silently depends on free-text parsing. By sampling questions with fixed seeds and preserving the complete interaction history, the platform makes each run inspectable at the level of individual model decisions.
The grading logic is deterministic, which removes another common source of drift. Instead of relying on subjective reranking or opaque post hoc interpretation, the system applies fixed answer-checking rules and then stores the outcome alongside the original exchange. The interface also includes a leaderboard so users can compare models without programming expertise, lowering the barrier for clinicians, investigators, and data scientists who need to examine whether one model truly outperforms another.
That design fits the central tension in medical AI evaluation: performance claims are only as credible as the methods used to produce them.
Performance shifted sharply across 15 biomedical benchmarks
BenchmarkHub was used to evaluate 12 open-weight models across five experiments on 15 biomedical benchmarks, including 11 text benchmarks, two multimodal benchmarks, and two computational biology benchmarks, plus two general benchmarks. Across the pooled runs, accuracy ranged from 46.1% to 80.9%, with confidence intervals spanning 43.0% to 49.3% at the low end and 78.3% to 83.3% at the high end. Those are not marginal differences. They point to a substantial spread in model reliability even before the evaluation protocol is adjusted.
The platform also exposed how fragile benchmark rankings can be when grading rules change. Re-grading responses with simple answer-extraction rules lowered one model’s accuracy by as much as 33.1 percentage points and reordered the leaderboard. In a clinical setting, this kind of movement would raise immediate concerns about assay stability; in model evaluation, it should raise the same concern about metric stability. A score that depends heavily on superficial formatting or token parsing is not a stable measure of biomedical reasoning.
Published benchmark claims often exceeded the BenchmarkHub results
Published scores were available for 23 of the 143 benchmark/model combinations tested. In 19 of those 23 comparisons, the accuracy measured within BenchmarkHub was lower than the published result. The discrepancy does not automatically mean earlier reports were wrong, but it does show how benchmark claims can lose comparability when prompts, model versions, or grading conventions are not pinned down. A single number without the surrounding protocol is too thin a basis for deciding whether one system is genuinely better than another.
Prompting strategy had measurable effects as well. Four prompting protocols were tested on MedQA: a default prompt, asking the model to answer with the letter only, chain-of-thought prompting, and three-shot prompting. Accuracy shifted by up to 11.3 points, although the difference was not significant after multiple-testing correction across 150 questions. Even without statistical significance after correction, the size of the swing is clinically relevant for anyone trying to benchmark a model intended for medical question answering. Prompt design is not a cosmetic detail; it is part of the intervention.
Visual input and tool use changed outcomes in opposite directions
Ablation testing on MMMU showed how multimodal models can lose ground when images are removed. For the two strongest models, stripping images lowered accuracy by 18.0 and 19.3 points. The result is unsurprising biologically and computationally: imaging often carries the key diagnostic signal, and a text-only input can erase the spatial information needed for correct interpretation. In radiology-adjacent or image-rich tasks, any benchmark that ignores visual input risks measuring something other than the intended capability.
Tool use told a different story. When agentic benchmarks used a sandboxed interpreter, accuracy on computational biology tasks increased by 36 to 63 points. That gain suggests some benchmark domains are better matched to agents that can execute code, inspect outputs, and iterate on intermediate results. In those settings, the model is not merely answering from memory; it is reasoning through a computational workflow. For biomedical research tasks that involve data manipulation, that distinction can determine whether a system is merely fluent or actually useful.
Checkable claims matter more when models stay on institutional systems
BenchmarkHub’s authors frame the platform as a way to make benchmark claims checkable without moving data outside the institution. That is a meaningful operational advantage for healthcare, where privacy, governance, and local infrastructure often limit how models can be tested. A self-hosted system that records every choice and parameter allows teams to reproduce results internally, compare models under the same conditions, and maintain an audit trail that supports review later on.
The platform also accepts user-uploaded datasets, which expands its usefulness beyond canned benchmarks. For hospital groups, research centers, and translational teams, this creates a path to evaluate models on locally relevant data rather than assuming that performance on public benchmarks will transfer cleanly to clinical work. The exact benchmark score remains only one part of the picture. The more important question is whether the entire evaluation chain can be inspected, repeated, and defended when results are challenged.
This preprint does not answer how any one model will perform in bedside use, and it does not claim to solve clinical validation. It does show that the way LLMs are benchmarked can move results by double digits, reorder rankings, and widen or narrow apparent model advantages depending on prompts, grading rules, and multimodal access. For healthcare AI, that is a methodological problem with direct consequences for model selection, internal review, and downstream study design.