Frontiers in Digital Health · 2026 · Accepted

MedBayes-Lite:
A Clinical Uncertainty Governance Layer for Risk-Aware Medical Decision Support

Elias Hossain1, Md Mehedi Hasan Nipu2, Maleeha Sheikh3, Tasfia Nuzhat1, Rajib Rana4, Subash Neupane5, Björn W. Schuller6,7, Niloofar Yousefi1

1University of Central Florida · 2North South University · 3Purdue University Fort Wayne · 4University of Southern Queensland · 5Meharry Medical College · 6Technical University of Munich · 7Imperial College London

Clinical language models are often confidently wrong, and most dangerously so on high-severity questions and on data unlike their training set. MedBayes-Lite wraps an existing model in a layer that measures its uncertainty, recalibrates its confidence, and sends low-confidence cases to a clinician. It needs no retraining and adds no parameters.

arXiv 📄 PDF 💻 Code 📚 BibTeX
0.23–0.33
Calibration error removed
21% → ~0%
Confident errors under shift
0
Retraining or new parameters

Explainer

The paper in one minute


At a Glance

A governance layer, not a new model

MedBayes-Lite sits between a trained clinical predictor and the clinician. It changes what the system does with the model's confidence: answer when the calibrated confidence is high enough, and escalate to a person when it is not.

1

Harmful errors drop to near zero

On MedMCQA and MedQA-USMLE, calibration error falls by 0.23–0.33 and confident, wrong, high-severity answers fall toward zero (paired bootstrap, p ≈ 0).

2

Strongest under domain shift

Trained on MedMCQA and tested on MedQA-USMLE, the base models are confidently wrong on about 21% of high-severity questions. With the layer: 0.2% and 0.0%.

3

Honest about its limits

Temperature scaling alone gets most of the calibration gain, deep ensembles rank risk better, coverage drops, and well-calibrated models gain nothing. All of it is reported.


The Problem

Confidence that cannot be trusted

Transformer-based clinical predictors assign high confidence to incorrect answers, and the miscalibration is worst on ambiguous, high-severity, and out-of-distribution cases, exactly where a confident error costs the most. Existing remedies each leave a gap.

≈

Post-hoc calibration only rescales

Temperature scaling makes the confidence scale honest, but on its own it does not decide anything. A deployed system still answers every question.

$

Ensembles are expensive

A five-member deep ensemble of Bio_ClinicalBERT needs 541M parameters and 2.85 GB of GPU memory, against 108M and 0.43 GB for the single model MedBayes-Lite wraps.

!

Accuracy hides severity

A wrong answer about a medication interaction is not equivalent to a wrong answer on a low-stakes item. Standard metrics weight all errors equally.


Method

Three stages, no training

The layer applies to open-weight encoders (BERT-base, BioBERT, Bio_ClinicalBERT, PubMedBERT) with full internal access. A sampling-only contrast with Qwen2.5-7B-Instruct is included for comparison.

1

Estimate

Run M = 15 stochastic forward passes with inference dropout 0.2. Their average is the predictive distribution; their spread is the uncertainty signal.

2

Calibrate

Fit a temperature on a held-out calibration set so that reported confidence matches empirical accuracy.

3

Decide

Convert the calibrated distribution into a normalized confidence and answer only above a threshold τ = 0.70; otherwise defer to human review.

C(p) = 1 − H(p) / log K,    H(p) = −∑k pk log pk,    answer only if C(p) ≥ τ Dividing the entropy by its maximum, log K, maps confidence to [0, 1]. The paper also proposes the Clinical Uncertainty Score (CUS), a severity-weighted calibration error over 13 rule-based clinical risk categories, used as an evaluation summary rather than a training signal.

Results

Fewer confident, high-severity errors

Calibration and harmful errors

harmful@0.8 is the rate of answers that are confident (≥ 0.8), wrong, and high-severity. Selective accuracy and coverage are measured at τ = 0.70. Temperature scaling is shown as a strong, cheap baseline, not a strawman.

SettingMethodECE ↓harmful@0.8 ↓Selective acc. ↑CoverageAURC ↓
MedMCQA · PubMedBERTBase0.2550.0560.5710.5410.376
Temperature scaling0.0810.0120.7210.2170.375
MedBayes-Lite0.0230.0030.8180.1000.392
MedMCQA · BioBERTBase0.3170.0670.4570.5080.517
Temperature scaling0.0850.0030.6180.0830.515
MedBayes-Lite0.0210.0000.7140.0210.521
MedQA-USMLE · PubMedBERTBase0.3250.2130.4370.5410.539
Temperature scaling0.0940.0150.5830.1060.533
MedBayes-Lite0.0460.0020.5390.0260.554

Table 1. Selected rows from the paper's Table 8 (mean over three seeds). MedBayes-Lite gives the lowest calibration error and removes nearly all harmful overconfident errors, but answers fewer questions and does not improve AURC. On MedQA-USMLE with BioBERT (not shown), selective accuracy falls from 0.288 to 0.143.

Under domain shift

Models trained on MedMCQA are evaluated on MedQA-USMLE. "Drift" is the change in calibration error from in-distribution to out-of-distribution.

ModelMethodID ECEOOD ECEECE driftOOD harmful@0.8
PubMedBERTBase0.2550.325+0.0700.213
PubMedBERTMedBayes-Lite0.0300.055+0.0260.002
BioBERTBase0.3170.410+0.0930.217
BioBERTMedBayes-Lite0.0150.084+0.0690.000

Table 2. From the paper's Table 9. A model that is most overconfident exactly when it is most likely to be wrong is the failure the layer is designed to contain.

Where the harmful errors were

Each question is assigned to one or more of 13 rule-based clinical risk categories. The base model's harmful overconfidence concentrates in high-severity, diagnosis, treatment, and medication questions; the layer drives it toward zero in each.

Bar chart of harmful overconfident error rate by clinical risk category. The base model reaches about 0.18 to 0.21 on rare disease, diagnosis, high severity, medication and drug interaction, treatment recommendation and risk prognosis; MedBayes-Lite is below 0.01 in every category.

Figure 1. Harmful overconfident error rate by clinical risk category, base model (blue) versus MedBayes-Lite (orange). From the paper's Figure 6.

r

CUS tracks clinical harm

Across 739 model, method, and category groups, the Clinical Uncertainty Score correlates with the harmful overconfident error rate at r = 0.88. Adding it to ECE raises the variance explained in harmful error from R² = 0.60 to 0.79.

✓

It defers the right cases

At τ = 0.70, the layer defers 97–99% of incorrect predictions, against roughly 55% for the base model.

⏱

Cost scales with samples

On Bio_ClinicalBERT, one pass takes 32 ms; ten Monte Carlo passes take 60 ms, at the same 0.43 GB of memory and the same 108M parameters.


Limitations

What this does not claim

MedBayes-Lite is positioned as a deployable calibration-and-abstention layer, not as a new uncertainty-aware architecture and not as a guarantee of clinical safety.

↕

No better risk ranking

AURC is statistically unchanged in three of four settings and marginally worse in one. On PubMedQA, deep ensembles achieve the best AURC.

≈

A cheap baseline is close

Temperature scaling, a single parameter, captures most of the calibration gain (ECE 0.255 → 0.081 on PubMedBERT).

%

Safety costs coverage

On MedMCQA with PubMedBERT, coverage at τ = 0.70 falls from 0.541 to 0.100: the model answers far less often.

0

No gain when already calibrated

The fine-tuned PubMedQA encoders are already well calibrated, and the layer neither helps nor hurts them.

R

Retrospective benchmarks only

All results are on public question-answering benchmarks with three seeds and one shift pair; clinical deployment was not evaluated, and the optional attention pathway stayed inactive (λ = 0).


Citation

Cite MedBayes-Lite

@article{hossain2026medbayes,
  title   = {MedBayes-Lite: A Clinical Uncertainty Governance Layer
             for Risk-Aware Medical Decision Support},
  author  = {Hossain, Elias and Nipu, Md Mehedi Hasan and Sheikh, Maleeha and
             Nuzhat, Tasfia and Rana, Rajib and Neupane, Subash and
             Schuller, Bj{\"o}rn W. and Yousefi, Niloofar},
  journal = {Frontiers in Digital Health},
  year    = {2026},
  note    = {Accepted. arXiv:2511.16625}
}