Multimodal AI Outperforms Current Automated Cause-of-Death Assignment in South African Verbal Autopsies
Timely cause-of-death assignment remains one of the weakest links in mortality surveillance across much of Africa, where fewer than 10% of deaths are medically certified. In that setting, verbal autopsy has become essential: families answer a structured symptom questionnaire and often provide a free-text narrative describing the circumstances, symptoms, and course of illness before death. The challenge is not a shortage of information, but a mismatch between what systems collect and what they can use. Most automated verbal autopsy tools were built around the structured response set, leaving the narrative text outside the algorithmic workflow even though physicians routinely use it when assigning a cause.
This national benchmark study from South Africa tested whether machine learning could close that gap. Investigators analyzed 4,999 verbal autopsy records for deaths at age 12 years or older from a national sample of registered deaths and used physician-assigned causes of death as the reference standard. They compared unimodal pipelines built on narratives alone or structured responses alone with multimodal approaches that fused both inputs at the data, feature, or decision level. A locally deployed generative large language model was also prompted on the same inputs without parameter updates. The operational comparator was InSilicoVA, the current automated practice in routine use.
The central finding was straightforward and clinically important. Every supervised pipeline improved on InSilicoVA by 8.5 to 15.9 percentage points in top-1 accuracy and by 3.5 to 7.1 points in cause-specific mortality fraction accuracy. In this context, the gains were not academic increments; they changed whether the assigned cause resembled the physician reference and whether the aggregate mortality profile tracked the underlying population pattern. Supervised models also showed lower expected calibration error than InSilicoVA, indicating that their probabilities tracked reality more closely rather than merely producing sharper guesses.
Narrative Text Carried Its Own Diagnostic Signal
The study design separated two kinds of evidence that often get collapsed together in automated mortality work: the structured checklist and the story. The two unimodal supervised pipelines performed similarly overall, but they were not interchangeable. Identical predictions appeared in 63.9% of deaths, while 11.6% were correct only from the narrative pipeline and 13.6% only from the questionnaire pipeline. In other words, each modality captured information the other missed.
A cause-specific pattern emerged from that split. The narrative pipeline performed better for noncommunicable causes, while the questionnaire pipeline did better for communicable causes. That difference is biologically plausible. Chronic noncommunicable disease trajectories often unfold over longer periods, with symptoms, functional decline, medication use, and behavioral context described more naturally in open text. Infectious deaths, by contrast, are frequently characterized by more standardized symptom clusters and temporally compressed features that map well to structured questions. When the cause is encoded in language as well as in checkbox responses, the model can exploit both.
Fusion Strategies Helped Different Clinical Problems
Not all multimodal approaches improved the same aspect of performance. Decision-fusion pipelines, which combine the outputs of separate models, added 4.2 to 5.4 percentage points in top-1 accuracy against the unimodal representatives, with gains concentrated in the most common causes. Feature fusion with cross-modal attention behaved differently. It improved the less common causes instead, adding 6.6 points in balanced accuracy and 5.5 points in macro F1, and it reached the highest cause-specific mortality fraction accuracy at 95.7%. The paired differences from the better unimodal representative ranged from −0.2 to +3.4 points, with all intervals including zero.
That pattern matters because the hardest classification problems in mortality surveillance are often the rare causes, the ambiguous deaths, and the syndromes that do not fit neatly into a single checklist. Cross-modal attention can weigh narrative phrases and structured findings jointly, allowing the model to redistribute attention when one modality is sparse or misleading. Decision fusion, by contrast, can favor the most common classes because it aggregates the outputs of already-trained predictors. The study therefore shows that multimodal design is not a single choice but a set of tradeoffs between common-cause discrimination and rare-cause recovery.
Zero-Shot Large Language Models Shifted the Balance Toward Rare Causes
The locally deployed large language model, used without parameter updates, outperformed InSilicoVA on individual-level metrics and achieved the highest balanced accuracy of any pipeline at 60.6%. Supervised pipelines still held the edge in cause-specific mortality fraction accuracy, which means the large language model was better at spreading attention across causes while the trained models better preserved the overall population distribution. In practical terms, one system favored rare-case sensitivity and another better reconstructed the mortality mix at the population level.
This split reflects a familiar tension in cause-of-death surveillance. A tool can be excellent at identifying uncommon causes yet still drift when asked to represent a death registry as a whole. Alternatively, it can mirror the population distribution well while missing clinically important outliers. In a national surveillance system, both forms of error matter, but they do not carry the same operational consequence. A rare cancer death misassigned as an infection distorts individual-level attribution; a systematic drift in common cardiovascular or infectious deaths warps planning, budgeting, and policy response.
Provincial Generalization Was Maintained Across South Africa
The gains were not confined to a single training geography. Supervised improvements held across all nine held-out provinces in South Africa, including every top-1 comparison and nearly every macro F1 and cause-specific mortality fraction comparison. That result is especially relevant for a national mortality platform because local dialects, reporting habits, health system access, and disease mixtures can all change how a verbal autopsy is described. A model that performs well only where it was trained has limited public-health value. Performance that persists across provinces is far closer to what a national system needs.
The study also frames narratives as a core input rather than an optional supplement. The raw language collected in a verbal autopsy interview is not just context around the structured form; it is diagnostic data. In settings where physician certification is scarce, that distinction affects how mortality surveillance systems should be built, validated, and deployed.
What the Benchmark Does and Does Not Resolve
Even with strong performance, the comparison remains bounded by its reference standard and setting. Physician-assigned cause of death served as the benchmark, and the analysis focused on deaths at age 12 years or older from South Africa. External validation in other countries, health systems, age groups, and death registration environments will be necessary before broader implementation can be justified. The preprint also does not settle which fusion strategy should become operational standard, because the best choice depends on whether the system prioritizes individual-level assignment, population distribution, or rare-cause sensitivity.
The most important practical implication is narrower than a headline about artificial intelligence replacing existing tools. Narrative text should no longer be treated as expendable in automated verbal autopsy. The evidence here shows that supervised models using stories alone can substantially outperform current practice, and multimodal models can improve further when the goal is precise cause assignment. For mortality surveillance systems built in low-certification settings, the text field is not decorative metadata; it is a clinically informative record of how death unfolded.