CareBench: When Correct Diagnoses Come from Unsafe Reasoning
Elias Hossain1, Tasfia Nuzhat Ornee1, Shubhashis Roy Dipta2, Andrea Rivera-Sepulveda3, Pegah Khosravi1, Niloofar Yousefi1
1University of Central Florida · 2University of Maryland, Baltimore County · 3Nemours Children’s Health
Medical-agent leaderboards score one thing: did the agent reach the right diagnosis. That hides a measurable failure mode — an agent can be right by way of a process no clinician would endorse. CareBench measures the process, not just the answer.
Across 1,000 trajectories from five frontier models, nearly four in five correct diagnoses were reached without any invoked safety check (pooled B-LCR = 0.788), replicating an earlier preliminary study at 0.795.
An agent that commits to a diagnosis before gathering the necessary evidence, or that skips a red-flag check, can still land on the right answer. On an outcome-only leaderboard that trajectory is indistinguishable from careful clinical reasoning. We call the gap lucky correctness, and it is invisible to any metric that reads only the final answer.
◔
Outcome hides process
Diagnostic accuracy is one bit per case. It cannot distinguish an agent that ordered the right workup from one that guessed early and got lucky, even though only one of those is safe to deploy.
⇅
Rankings invert
The model that ranks first on diagnostic accuracy in our evaluation ranks near the bottom on process safety. Choosing a model by outcome alone can select the least safe reasoner in the pool.
⌾
Policy stays hidden
Each case ships a JSON clinical policy — necessary evidence, red flags, accepted diagnoses and treatments, closure threshold — visible only to the scoring layer, never to the agent or the patient simulator.
What It Measures
Outcome and process, measured separately
Every trajectory is scored on both what the agent concluded and how it got there. CareBench ships as a third clinical domain of the tau-bench tool-agent-user framework, alongside retail and airline, reusing its agent loop, tool-calling interface, user simulator, and pass^k pipeline.
Metric
What it captures
DxAcc
Diagnostic accuracy, via a conservative layered diagnosis matcher
ENR
Evidence Necessity Recall: fraction of case-required evidence actually gathered
PE
Pathway Efficiency: fraction of gathered evidence that was necessary
PCR
Premature Closure Rate: committed before sufficient evidence
RFCR
Red-Flag Capture Rate: clinically relevant red flags acknowledged
NOSAFETY
Reference-free action-log signal: detects commitment without a required safety check
LCR
Lucky Correctness Rate: correct diagnosis reached through an unsafe process
B-LCR
Behavioral variant, policy-independent, read only from the action log
The two primary trajectory-level signals are ENR (evidence coverage) and NOSAFETY (the reference-free action-log signal); the remaining pathway metrics are reported as supporting diagnostics. B-LCR is the correctness-conditioned summary of NOSAFETY, and it is the headline number precisely because it is policy-independent: computed from what the agent did, it cannot be gamed by disagreement over what a case policy should have required.
The outcome-process partition
Safe · correct
Right diagnosis, defensible pathway. The only quadrant an outcome-only metric and a clinician would both accept.
Lucky · correct
Right diagnosis, unsafe pathway. Scored as a win by every outcome-only leaderboard.
Safe · wrong
Wrong diagnosis, sound process. Penalised identically to reckless failure by outcome-only scoring.
Unsafe · wrong
Wrong on both counts.
A composite pass^k requires both a correct outcome and a safe process, so a model cannot buy the score with lucky correctness.
Results
Outcome-only and process-aware selection disagree
Five frontier models, 200 trajectories each. Read the first two numeric columns together: the ordering by diagnostic accuracy is close to the reverse of the ordering by process safety.
Model
DxAcc ↑
B-LCR ↓
PE ↑
ENR ↑
Safe-correct
Lucky-correct
gpt-4o-mini
0.120
1.000
0.725
0.225
0
24
llama-70b
0.075
0.733
0.812
0.238
1
14
deepseek-chat
0.065
0.385
0.850
0.263
3
10
gpt-4o
0.055
0.818
0.855
0.269
2
9
claude-sonnet
0.015
1.000
0.455
0.141
0
3
Outcome-only vs process-aware model selection.n = 200 trajectories per model. gpt-4o-mini ranks first on diagnostic accuracy while scoring B-LCR = 1.000 — every one of its 24 correct diagnoses came from an unsafe pathway, and it produced zero safe-correct trajectories. deepseek-chat ranks third on accuracy but first on process safety, with the most safe-correct trajectories in the pool. Pick by accuracy and you ship the least safe reasoner.
↺
Replicated
The pooled B-LCR of 0.788 reproduces an earlier preliminary study at 0.795, on independently generated trajectories. The effect is not an artifact of one run or one scoring pass.
⊟
Evidence is thin everywhere
ENR sits between 0.141 and 0.269 across all five models: even the best of them gathers barely a quarter of the evidence its cases require before concluding.
↻
Regenerable offline
Every table and figure rebuilds from saved trajectories with no further model calls, so the analysis is reproducible without paying for inference twice.
Quick Start
Run the benchmark
The case pool is 50 clinical cases: 19 translated from MedChain and 31 from PMC-Patients. The harness routes through LiteLLM, so any provider it supports can be evaluated.
install.shBash
# Clone and install
git clone https://github.com/eliashossain001/carebench.git
cd carebench
pip install -e .
# Pull the case pool and trajectories
huggingface-cli download EliasHossain/carebench \
--repo-type dataset \
--local-dir data/carebench/
--num-trials sets k for pass^k; --agent-strategy is one of tool-calling, act, react, or few-shot.
⚠
Research use only. CareBench is an evaluation benchmark, not a clinical tool. Its cases and policies exist to measure agent behaviour, and nothing here is validated for patient care. Cases derive from MedChain and PMC-Patients; please cite those datasets alongside CareBench when using the corresponding cases.
Citation
Cite CareBench
If CareBench or the lucky-correctness metrics are useful in your work, please consider citing it.
@misc{hossain2026carebench,
title = {CareBench: When Correct Diagnoses Come from Unsafe Reasoning},
author = {Hossain, Elias and Ornee, Tasfia Nuzhat and Dipta, Shubhashis Roy
and Rivera-Sepulveda, Andrea and Khosravi, Pegah
and Yousefi, Niloofar},
year = {2026},
note = {Process-aware evaluation for clinical LLM agents.
Code: \url{https://github.com/eliashossain001/carebench}
Data: \url{https://huggingface.co/datasets/EliasHossain/carebench}},
}