Weak Labels Fell Short of a Reported-Only Baseline in Urine Culture Prediction
In a two-site machine learning experiment focused on Escherichia coli urine cultures, rule-derived labels did not consistently improve prediction of antibiotic non-susceptibility when reported labels were scarce. The practical question was simple but important: if manually reported labels are limited, can weakly generated labels from rules expand the training set without degrading probability estimates? Across the tested budgets and model families, the answer was largely no.
Investigators evaluated non-susceptibility to cefepime, cefoxitin, and ceftazidime in the Stanford ARMD dataset, and to cefepime and ceftazidime in the UT Southwestern ARMD-UTSW dataset. Reported-label budgets were set at 10%, 25%, 50%, and 100% of development patients, with five patient-disjoint outer splits used to structure the comparison. Logistic regression and histogram gradient boosting formed the core model set. Each was trained either on reported labels alone or on reported labels plus rule-derived labels, with several weighting strategies layered on top: equal weights, a tuning-selected weight, class-proportion adjustment, and prior-probability correction.
The central comparator was not an artificially starved model. A full reported-only baseline used every budgeted reported label for training, giving the weak-supervision arms a fair and demanding benchmark.
Calibration Deteriorated When Rule-Derived Labels Entered the Mix
Equal-weight augmentation performed poorly from a calibration standpoint. Predicted risk fell far below observed prevalence, and the gap was stark. At Stanford for cefepime with the 10% budget, mean predicted risk was 0.35% against an observed prevalence of 4.85%. In a clinical setting, a model that suppresses risk this aggressively can understate the probability of resistance and encourage false reassurance at the point of empiric antibiotic selection.
The same pattern carried through the broader comparison. Equal-weight augmentation had a higher Brier score than the all-budget reported-only baseline in every drug, model, and budget summary. In prediction work, the Brier score matters because it penalizes both poor discrimination and poor calibration; a model can rank patients reasonably well and still be unreliable if its absolute probabilities drift away from observed outcomes. Here, the weakly supervised strategy did not deliver a stable probability model.
A model that sounds more data-rich can still be less trustworthy.
Tuning Did Not Rescue the Weak-Supervision Strategy
The tuning-selected weight was supposed to adapt the contribution of rule-derived labels instead of treating them as equals. It still did not reliably improve on the all-budget reported-only baseline. Across 40 drug, model, and budget summaries, the median Brier difference favored the reported-only approach in 38. Paired bootstrap intervals were above zero in 100 of 296 Stanford analyses and 31 of 198 UT Southwestern analyses, while falling below zero in only a small minority of comparisons.
The tuning process itself repeatedly chose no added weak labels. A weight of zero was selected in 146 of 296 Stanford analyses and 153 of 198 UT Southwestern analyses. That pattern is clinically meaningful even though the study is methodological. It suggests the tuning routine often found that the safest move was to ignore the rule-derived labels entirely. In other words, the added data frequently behaved more like noise than signal once calibration was judged against a fully used reported-label baseline.
Class-proportion adjustment and prior-probability correction were included to mitigate imbalance, but the abstract does not show those methods changing the broad result. The core finding remained intact: adding rule-derived labels did not consistently improve probability prediction over the strongest reported-only comparator.
Site Differences Shaped What the Models Could Learn
The two datasets were not identical in design, label source, or feature availability. Stanford used inferred labels released with the dataset and included a feature set with resistance history. UT Southwestern used researcher-derived labels translated from Stanford rule text, and those labels were not validated for that site. The primary UT Southwestern model also relied on age only, a much thinner clinical feature set than the Stanford formulation.
That asymmetry matters because label quality and feature richness both shape learning. A weak label built from one hospital’s conventions may not transfer cleanly to another hospital with different ordering patterns, susceptibility practices, or documentation structure. The age-only UT Southwestern models showed little discrimination, with median AUROC values between 0.49 and 0.54. When discrimination is that flat, most of the comparison is really about calibration rather than ranking ability, and the weak-label methods still did not create a compelling advantage.
Why Weak Supervision Underperformed Here
The likely mechanism is not mysterious. Rule-derived labels can expand sample size, but they also import systematic misclassification, especially when the rule is tuned to one institution’s microbiology workflow and then moved across sites. In antimicrobial resistance prediction, prevalence is often low, class balance is uneven, and the consequences of overconfident probability estimates are immediate. If weak labels pull the training distribution away from the observed patient mix, the model may learn a distorted version of resistance risk rather than a better one.
This is where calibration becomes more than a statistical nicety. A model used for resistance prediction should estimate how likely non-susceptibility really is, not just sort cases into rough risk strata. When predicted probabilities sink below observed prevalence, antibiotic decision support can become overly conservative about resistance or deceptively optimistic about susceptibility, depending on how the probabilities are operationalized.
What This Means for Limited-Label Model Development
The study sets a clear methodological guardrail for future weak-supervision work in antimicrobial resistance. Rule-derived labels should not be assumed to outperform a model trained on every available reported label, even when reported labels are scarce. Any evaluation of weak supervision should include that full-budget reported-only baseline, not just a reduced-label comparator, because otherwise the apparent gain can be an artifact of withholding usable labels from the control arm.
The analysis also argues for reporting calibration alongside discrimination. AUROC alone would have left an incomplete picture, especially at UT Southwestern where discrimination was limited from the outset. Brier score differences, bootstrap intervals, and the frequency with which tuning selected a zero weight all pointed in the same direction: in this design, the weak labels were not reliable enough to justify routine augmentation.
This was an exploratory label-budget experiment on de-identified datasets, not a prospective clinical deployment. The findings still have direct relevance for anyone building antimicrobial resistance prediction tools from sparse reports: more labels are not automatically better when those labels are inferred rather than observed, and a larger training set can still be the wrong training set if its probability estimates drift away from reality.