Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Authors Elias Hossain, Niloofar Yousefi, Ser-Nam Lim University of Central Florida
AI models are now routinely used to grade the answers of other AI models, and a panel of such "judges" that agrees looks like strong evidence. But agreement is only strong evidence when the judges make their mistakes independently. If they tend to fail on the same questions, a unanimous vote tells you much less than it seems, for the same reason that ten survey answers from one household are not ten independent opinions.
Each row is a judge and each column is a question. A coloured square is a wrong verdict. Every judge is wrong on about 1 question in 5; only how much the mistakes line up changes.
Filled circles show the effective number of judges, n / (1 + (n − 1)ρ). In this grid the majority is wrong on 0 of 30 questions. The simulation and formula assume every pair of judges is equally correlated; the paper estimates the full correlation matrix, so its numbers for a given bank differ slightly.
Across ten open-weight judges the average error correlation is 0.21, so the panel is worth about 3.5 independent judges. Frontier judges from three different companies are even more correlated (0.56) despite being far more accurate.
Counting every vote as independent declares one AI system significantly better than another in 79% of resampled comparisons; treating each question as one observation does so in 52%. In 28% of comparisons the verdict depends on that choice alone.
Report how correlated the judges are alongside how accurate they are, and run statistical tests on questions rather than votes. In our experiments a cleverer voting rule was not a reliable fix, even when the error pattern was known.
Alignment pipelines increasingly use panels of LLM judges to decide which of two responses is preferred, and keep an example only when enough judges agree. The intuition is that agreement among many judges is stronger evidence than any single judge. That intuition assumes the judges make their errors independently. LLM judges share pretraining data, post-training recipes, evaluation templates, and blind spots, so they often make the same mistakes. This project measures how much that dependence weakens consensus.
We estimate the pairwise correlation of judge errors and convert it into an effective number of independent judges, so that a bank's evidence is reported in the same units as its size.
We compare a pooled-vote test, which treats every judge vote as independent, with an item-level test that uses each preference pair as one observation, and count how often they disagree.
We separate errors shared by most judges from errors concentrated in a smaller group, and test which way of combining judge decisions works under each pattern.
The number of judge votes should not be read as the amount of independent statistical evidence. Correlated errors shrink a bank's evidence well below its nominal size, can change statistical verdicts, and are stronger, not weaker, among the most accurate judges.
In the main ten-judge bank, the mean pairwise error correlation is 0.21. This inflates the variance of the mean error rate by a factor of 2.85, so the ten judges carry about as much information as 3.5 independent ones. The same holds on two more datasets and 73 alternative bank compositions.
On resampled 100-pair panels, a pooled-vote test finds a significant system difference in 79% of panels, while an item-level test does so in 52%. In 28% of panels the verdict depends only on whether shared errors are ignored.
Frontier judges from three different providers reach 91–93% accuracy, yet make the same error together at 7.7× the rate expected under independence. Cross-provider judges are about as correlated as judges from one provider. Adding open-weight judges to the frontier bank raises the effective number of judges from 1.88 to 3.83.
When many judges fail together, measured correlation rises. When failures are concentrated in a vulnerable subgroup, correlation can fall, from 0.22 to 0.07 in our experiments, even as majority decisions get worse. The two patterns favor different filters, so correlation magnitude alone is not enough.
Open-weight and frontier judges are evaluated on the same preference pairs, so that dependence can be compared across banks, datasets, and capability tiers on identical items.
| Component | Coverage |
|---|---|
| Main open-weight bank | 10 judges: Llama-3.1-8B, Qwen-2.5-7B, Gemma-2-9B, Mistral-7B-v0.3, Phi-3.5-mini × 2 prompt styles |
| Frontier judges | 6: three Gemini models, plus GPT, Claude, and Grok |
| Preference datasets | RewardBench v2 · UltraFeedback · PKU-SafeRLHF |
| Calibration set | 1,195 stratified RewardBench pairs (1,133 answered by all ten judges) |
| Robustness | 73 alternative bank compositions; label-noise sweep of 1–10% |
| Training-regime control | DPO- and GRPO-trained open-weight judges |
| Transfer screen | 8 LLM-AggreFact components, plus factuality and code judging |
| Filters compared | majority · 0.75 supermajority · Automatic-Jury-Consensus · Bias-Cluster · learned selector |
Filters are compared on held-out splits at matched retention, and no threshold or other parameter is tuned on the test split.
The contribution is a measurement and validation protocol, not a new aggregation method that replaces majority voting. The negative results are reported with the same prominence as the positive ones.
No single fixed rule works best under both error patterns. None of the 11 screened factuality and code settings met the criteria for a filter to help, and on all eight LLM-AggreFact components neither Automatic-Jury-Consensus nor Bias-Cluster beats plain majority.
The learned selector helps mainly when the best rule shifts between datasets. It guards against transferring an unsuitable filter, and its gains over a fixed rule are small.
The global and subgroup patterns are created by controlled interventions and do not establish how often each occurs naturally, although subgroup failures do appear in the unmodified data.
The repository contains the code for the experiments in the paper. The method-level recipe is simple enough to apply to any judge bank: