GPT-5 Came Closest to Radiologist Labels Across Most Incidental Findings
Large language models are moving into a part of radiology work that has been stubbornly manual: sorting through narrative CT reports and identifying incidental findings that matter in lung cancer screening. In a retrospective comparison of 300 chest CT reports from a Dutch academic hospital, investigators tested whether several models could recognize nine specific incidental findings, or SIFs, using the radiologist’s labels as the reference standard. The task was not simple document classification in the abstract. It was a clinically grounded exercise in turning unstructured prose into structured data that could support screening programs, follow-up pathways, and research registries.
GPT-5 produced the strongest overall performance in this dataset, reaching substantial or almost perfect agreement for seven of the nine SIFs. The strongest agreement appeared for bronchiectasis, coronary artery calcification, emphysema, and thyroid abnormalities, while aortic valve disease, consolidation, and mediastinal lymph node abnormalities also reached substantial agreement. In practical terms, that means a model can read the kind of free-text radiology language that often varies from report to report and still recover the same clinical signal a trained reader identified. The performance was not uniform across every target, and that unevenness matters because incidental findings are not interchangeable; some are straightforward descriptors, while others depend on context, thresholds, and nuanced phrasing.
Nine Incidental Findings, One Reference Standard
The cohort included CT reports generated between 19 March 2003 and 28 December 2020, a long time span that likely captured shifts in reporting style, scanner technology, and screening practice. Each report was labeled for the presence or absence of nine SIFs by a radiologist, and that label served as the benchmark against which the automated approaches were judged. The models under comparison included GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, and a keyword-based search method. Each model was tested using four prompting strategies: two without chain-of-thought reasoning and two with it, including one zero-shot and one few-shot version.
A design like this is important because model behavior can change sharply depending on how the prompt is framed. A model that performs well under one instruction set may lose precision when asked to reason step by step, especially if the target is a short, formulaic radiology statement. In this study, the non-chain-of-thought prompts generally produced the highest agreement for most SIFs. The result fits the operational reality of report labeling: when the task is direct extraction rather than open-ended interpretation, extra reasoning steps can introduce noise rather than improve accuracy.
The report also makes clear that one-size-fits-all automation was not achieved. No single model-prompt combination topped every endpoint. Some findings were easy targets for the best-performing models, while others remained only moderately aligned with the radiologist labels.
Where Agreement Was Strong and Where It Slipped
For seven SIFs, GPT-5 delivered performance that would be attractive for downstream automation workflows. The reported Cohen’s kappa values were 0.81 for aortic valve disease, 0.97 for bronchiectasis, 0.71 for consolidation, 0.92 for coronary artery calcification, 0.91 for emphysema, 0.71 for mediastinal lymph node abnormalities, and 0.92 for thyroid abnormalities. Those values span substantial to almost perfect agreement, suggesting that the model can reliably identify several of the incidental findings most likely to recur in lung cancer screening reports.
Two targets resisted stronger alignment. For interstitial lung abnormalities, MedGemma-4B-IT reached a Cohen’s kappa of 0.48, and for mediastinal mass, GPT-4-Turbo reached 0.42. Both fall in the moderate range, far below the top-tier results seen with several other findings. The drop-off is clinically plausible. Interstitial lung abnormalities can be described in subtle, variable language that may overlap with mild scarring, dependent atelectasis, or nonspecific parenchymal change. Mediastinal masses can also be under-described or embedded in broader narrative phrasing, making them harder to classify from text alone. These are the kinds of edge cases where language models still face the same ambiguity that challenges human abstractors.
The comparison with a keyword-based method is also instructive. Keyword searches are fast and transparent, but they are brittle when report wording shifts or when negation, synonymy, and context matter. A model with broader semantic understanding can bridge some of those gaps, which helps explain why the strongest systems outperformed simple search for several findings.
Why Structured Report Labeling Matters in Screening Programs
Lung cancer screening generates more than nodule measurements. It also surfaces incidental findings that may alter follow-up, prompt referral, or affect risk stratification. Coronary artery calcification, emphysema, bronchiectasis, and thyroid abnormalities each point toward different care pathways, and those pathways are often missed when reports remain unstructured text buried in a PACS or database. Automated labeling can convert those reports into usable data for clinical audit, registry building, secondary analysis, and quality improvement.
The biological and clinical logic is straightforward. Coronary artery calcification reflects atherosclerotic burden and may identify patients who need cardiovascular risk assessment. Emphysema and bronchiectasis may flag chronic obstructive or airway disease that never made it into a problem list. Thyroid abnormalities and mediastinal findings can trigger entirely different diagnostic routes. When these findings are present in screening CT reports, they are not incidental to the health system, even if they are incidental to the scan order.
This is where an automated labeler could become valuable. A model that can reliably identify the majority of SIFs can reduce manual abstraction burden and support large-scale retrospective research, while also enabling screening programs to audit how often these findings are mentioned and acted upon. The remaining gap is reliability across every target category, especially the less common or more ambiguously worded ones.
Limits of the Preprint and the Remaining Validation Gap
The study is retrospective and limited to 300 CT reports from a single Dutch academic center. That makes the results informative, but not definitive. Report style, local terminology, and institutional workflow can all shape model performance, and those factors may not transfer cleanly to other hospitals or languages. The dataset was also not publicly available, which constrains external reproducibility and independent benchmarking.
Another boundary is the outcome metric itself. Cohen’s kappa, sensitivity, specificity, and F1 score provide useful agreement statistics, but they do not tell the full story about downstream clinical utility. A model can achieve respectable agreement while still misclassifying rare but important cases, or while failing to generalize when embedded in a live reporting pipeline. The preprint therefore supports a narrower conclusion: these models can label many incidental findings well, yet their performance remains uneven enough that deployment would require careful local validation.
For now, the strongest signal is methodological rather than operational. LLMs can extract clinically relevant incidental findings from radiology narratives with impressive accuracy in selected categories, but the variation across findings and prompts shows that this is still a task for supervised implementation, not blind automation. The unresolved question is not whether the models can help; it is which findings, in which reporting environments, and under what validation standard they are safe to trust.