AI-Assisted Systematic Reviews Fell Short on Methodological Adherence
Artificial intelligence is moving quickly into the machinery of evidence synthesis, but the newest matched meta-research analysis sends a cautionary signal: published AI-assisted systematic reviews scored worse on methodological adherence and reported their processes less completely than matched reviews with no declared AI use. In a cohort of 231 health systematic reviews, the AI-assisted group had a mean modified 13-item AMSTAR-2 score of 7.58, compared with 9.63 in the non-AI comparison group. After multivariable adjustment, the AI-assisted reviews scored 1.64 points lower, with a 95% confidence interval from −2.37 to −0.91, and the prespecified noninferiority margin was not met.
The signal is difficult to ignore because systematic reviews sit near the top of the evidence hierarchy for clinical practice and policy. When the workflow depends on automated search, screening, extraction, synthesis, or drafting, even small process failures can propagate into a polished final manuscript. A faster review is not automatically a stronger review. In this dataset, speed-enhancing tools came with measurable losses in core methodological adherence.
What the Matched Cohort Actually Compared
Investigators assembled a matched-cohort meta-research study of English-language, peer-reviewed health systematic reviews published from 2023 through 2025. Each AI-assisted review was matched 1:2 to reviews with no reported AI use through individualized MEDLINE searches that aligned publication timing, clinical domain, review type, and meta-analysis status. When more than two eligible comparators were available, two were randomly selected. AI assistance counted if it was reported during searching, screening, data extraction, risk-of-bias assessment, evidence synthesis, or substantive drafting.
The primary endpoint was a prespecified modified 13-item AMSTAR-2 methodological-adherence score, with a range from 0 to 13 and a noninferiority margin of −1 point. Secondary outcomes included PRISMA 2020 reporting completeness and ROBIS risk of bias. The analysis also examined transparency around AI use itself, including whether the model version or date, prompts or parameters, and data-governance or ethics considerations were disclosed.
This design matters because it does not compare AI-assisted reviews against an abstract ideal. It compares them against contemporaneous published reviews in similar clinical and methodological settings. The gap therefore reflects something real in current practice, not a historical artifact.
Lower Scores, Less Complete Reporting, and a Narrower Margin for Error
The difference in AMSTAR-2 adherence was not subtle. AI-assisted reviews averaged nearly two points lower than comparators before adjustment, and the adjusted deficit remained at 1.64 points. On a 13-point scale intended to capture essential review behaviors, that is a material loss of rigor. It suggests that AI use, as currently implemented and disclosed in the literature examined here, may be linked to more frequent omissions in the procedures that protect against bias and error.
PRISMA completeness told a similar story. In primary adjusted analyses, AI-assisted reviews were estimated to be 4.7 percentage points lower in reporting adherence, and the deficit widened to 6.4 points in matched-set fixed-effects analyses. Reporting quality is not cosmetic in systematic review science. Missing details about search strategy, selection flow, synthesis methods, or analytic choices make it harder for readers to judge whether the review can be trusted, reproduced, or updated.
One sentence captures the central concern: AI did not buy higher-quality evidence synthesis in this study.
Why Faster Drafting Can Still Leave the Review Weaker
The findings fit a plausible mechanism. Large language models and other AI tools can streamline repetitive tasks, but they can also flatten nuance, miss protocol-specific requirements, or introduce silent inconsistencies if human oversight is thin. In systematic reviews, rigor lives in the details: the exact search string, the eligibility logic, the handling of duplicates, the calibration of reviewers, the justification for risk-of-bias judgments, and the clarity of synthesis decisions. If AI is used as a drafting accelerator without disciplined verification, the final manuscript may look complete while containing procedural gaps.
Transparency was also limited. Among AI-assisted reviews, model version or date was reported in 42.9%, prompts or parameters in 55.8%, and data-governance or ethics considerations in only 9.1%. Those numbers point to a field that is still documenting AI use unevenly. Without model versioning, prompt provenance, and governance details, readers cannot reconstruct how the tool was used or whether the output was stable across time and context. That uncertainty weakens accountability.
Risk-of-Bias Ratings Looked Better, but the Pattern Needs Careful Reading
An unexpected result emerged in the ROBIS assessment. Among reviews with assessable ROBIS ratings, 75.0% of AI-assisted reviews and 65.6% of comparators were rated low risk of bias, with an adjusted difference of +17.8 percentage points and a 95% confidence interval from 2.7 to 33.0. On the surface, that looks favorable for the AI-assisted group.
Interpretation needs restraint. A more favorable ROBIS rating does not erase the lower AMSTAR-2 adherence or the weaker PRISMA completeness. The apparent ROBIS advantage may reflect differences in how the final reviews were structured, how eligible domains were judged, or how the study sample sorted out after matching and assessability constraints. It is also possible for a review to receive a relatively favorable bias judgment while still documenting its methods poorly. In evidence synthesis, clean-looking conclusions do not substitute for transparent process.
A Practical Warning for Evidence Synthesis Teams
The study’s bottom line is not that AI should be excluded from systematic review work. It is that current AI-assisted reviews, as published here, showed weaker methodological adherence and less complete reporting, with very limited disclosure of how the tools were used. For clinicians, guideline developers, and policymakers who depend on systematic reviews, this raises a straightforward concern: speed gains should not be mistaken for evidentiary reliability.
Human verification remains the safeguard that matters most. Search strategies need manual checking, eligibility decisions need audit trails, extracted data need independent confirmation, and any AI-assisted drafting needs line-by-line review against the underlying evidence. Without those controls, AI can amplify efficiency while quietly eroding the very disciplines that make a review trustworthy.
This was a preprint, so the findings have not yet undergone peer review. Even so, the pattern is strong enough to justify closer scrutiny of how AI enters the review pipeline, how its use is disclosed, and which safeguards are required before the field treats machine assistance as routine rather than experimental.