Training overlap blunts claims of stronger clinical reasoning
Fine-tuned medical language models can look markedly better on paper than they are in practice, especially when benchmark questions overlap with the data used to train them. In this evaluation, an 8.2-billion-parameter QLoRA fine-tune of Qwen3-8B was re-examined under a fixed generative chain-of-thought harness, and the results showed how quickly apparent progress can collapse once contamination is measured directly.
The central problem was simple: several public medical question-answering sets were not cleanly separated from the model’s 97,041-item fine-tuning corpus. Exact-stem overlap exceeded 70% for PubMedQA, MedQA and MMLU Professional Medicine, making those scores poor evidence for real capability gains. On those benchmarks, the line between training and evaluation was too blurred to support confident claims about generalization.
One result stood out immediately. On multiple overlapping benchmarks, the unmodified parent checkpoint matched or even outperformed the fine-tuned model, with the clearest separation on MedQA. In a clinical language model, that is not a minor technicality; it means extra supervision can look impressive while leaving the underlying reasoning unchanged, or even degraded, once the test items are not effectively memorized.
A cleaner read on MedXpertQA still showed only a modest lift
The most informative comparison came from the 1,838 MedXpertQA items that were not flagged by exact-stem matching. There, the fine-tuned model scored 20.8%, compared with 15.6% for the parent checkpoint, a paired improvement of 5.2 percentage points. The statistical signal was strong, with McNemar exact P = 7.9 × 10⁻⁷, but the absolute performance remained low by any standard used for bedside-relevant medical reasoning.
The original reported gain had been 7.1 points, so the independently re-evaluated effect was smaller once the fixed harness and overlap-aware filtering were applied. Even after the contamination-sensitive subset was isolated, the model’s advantage was not transformed into a large clinical leap. It was a measurable improvement, not a wholesale shift in reasoning quality.
A model can win on a benchmark and still answer too few questions correctly to matter clinically.
Fixed evaluation exposes how much of the gain survives methodological tightening
The study did not stop at the headline comparison. A bundled ablation changed reasoning traces, sampling weights, curriculum staging, token exposure and output-format supervision at the same time, then asked how much of the improvement remained. Roughly 70% of the observed gain persisted across that bundled modification, which means the learning signal was not an artifact of one isolated training choice. Still, the remaining 1.6-point difference was not statistically significant, with P = 0.175, and it could not be assigned to any single component.
For medical AI, that detail matters because training recipes are often described as if every ingredient contributes independently to measured performance. Here, the evaluation design did not permit that kind of clean attribution. A model trained with several simultaneous changes may improve, but the available data cannot tell whether the advantage came from the reasoning traces, the exposure schedule, the supervision format, or some interaction among them.
The fixed generative chain-of-thought harness also mattered because it reduced one familiar source of noise: decoding variability. Greedy decoding at temperature zero made the comparison more reproducible, and the released weights were verified byte-identical across four shards by SHA-256. In a domain where small differences in prompt structure or sampling can distort results, that level of control strengthens the claim that the remaining uncertainty is methodological rather than accidental.
Contamination is a statistical problem with clinical consequences
Benchmark contamination is not just a machine-learning nuisance. In medical settings, a model that appears to improve by recalling overlapping question stems can be mistaken for one that has learned transferable reasoning about symptoms, diagnoses, differential diagnosis, or treatment choices. That distinction is crucial when these systems are proposed as decision-support tools or as surrogates for educational assessment.
The overlap analysis used 13-gram, exact-stem and longest-common-subsequence matching against the fine-tuning corpus. By those checks, MedMCQA was effectively unflagged because it had been evaluated on a held-out split, while several other benchmarks were compromised to the point that their raw scores should not be treated as capability claims. In practical terms, this means a benchmark can still be useful for internal debugging, but it should not be sold as proof of general clinical competence if the questions are already sitting inside the training distribution.
The issue is especially sharp for public medical QA datasets, which are widely reused, circulated and folded back into model development. Once a benchmark begins to function as a memory test, the metric rewards exposure rather than reasoning. A fine-tuned model may look better simply because it has seen more of the answer space in advance.
What the evaluation can and cannot support
The paper draws a hard boundary around what these results justify. The released weights, benchmark loaders, per-item scoring outputs and evaluation code allow reproduction of the accuracy results and the statistical comparisons under the specified software environment and request-stream protocol. The deposited overlap flags and exact-stem-unflagged item lists also allow audit and reuse of the reported subsets.
Reproducibility stops short at one point: the overlap analysis itself cannot be independently regenerated by third parties because the training corpus used for matching is not redistributable. That is a real limitation, not a cosmetic caveat. It means readers can verify the downstream results, but they cannot fully repeat the contamination audit without access to the original corpus.
For clinical AI developers, the message is straightforward. Performance on public medical question-answering benchmarks is only as persuasive as the separation between training and test data, and that separation has to be demonstrated rather than assumed. When overlap is high, the score may describe memorization more than competence. When the evaluation harness is fixed and the contaminated items are stripped away, the apparent gain can shrink to something much smaller, and much easier to interpret.