Preprint · Under Review · 2026

Knowing When a VLM Is Wrong Is Not Enough:
When Does Extra Compute Help?

Anonymous authors · under double-blind review

Adaptive test-time compute methods let a vision-language model decide, per question, whether to look closer or to reason first. We ask how much of the reported gain comes from that per-question decision itself. Measured against an equal-cost random allocation of the same actions, very little: confidence tells a router that an answer is wrong, not whether extra compute will fix it.

Key findings 🧪 Evaluation protocol 🧭 Takeaways

Paper, code, and per-sample outcome records will be released after review

5
Small VLMs
600
Questions
4.8–9.8
Oracle Headroom (pts)
≤ 2.1
Router Gain (pts)

Explainer

The paper in one minute


At a Glance

Wrong is not the same as fixable

A VLM can spend extra inference compute by encoding the image with more visual tokens (LOOK), by reasoning step by step before answering (THINK), or both. Neither helps every question. The best action depends on the question, some questions stay wrong under every action, and the signal a router has at inference time identifies errors far better than it identifies fixable errors.

Four panels: (a) four inference actions B, LOOK, THINK, LOOK+THINK; (b) example HR-Bench questions where a different action is the cheapest fix, and one where nothing fixes it; (c) confidence detects wrong answers with AUROC 0.77 to 0.84 but fixable ones only at 0.55 to 0.64; (d) trained routers gain at most 2.1 points over the equal-cost random mix while the oracle gains 7.6 to 14.9.

Figure 1. (a) Four inference actions. (b) The best action varies by question; some questions remain wrong under every action. (c) Confidence detects incorrect answers more reliably than it identifies errors another action will correct. (d) Trained routers recover only a small part of the improvement available to an oracle that knows which action works for each question.

1

The choice has real headroom

LOOK and THINK fix different questions. A held-out oracle that picks each question's action gains 4.8–9.8 points over the best fixed action, or matches baseline accuracy at 0.44–0.51× its FLOPs.

2

Routers recover little of it

Trained routers gain at most 2.1 points over the equal-cost random mix, under one-fifth of the oracle's gain. The published VisionThink does not beat an equal-FLOPs mixture of its own fixed configurations.

3

Because confidence predicts error, not fixability

Confidence detects wrong answers (AUROC 0.77–0.84) but barely predicts fixable ones (0.55–0.64). Knowing which errors an action fixes is worth 8.0–9.5 points more than knowing which answers are wrong.


The Problem

Reported gains mix three effects

When an adaptive method beats a fixed setting, the gain may come from the actions themselves, from the training that produced the policy, or from the per-question decision. An audit of the seven most closely related papers (VisionThink, AVR, AVIS, SPARC, DIRECT, Thinkless, VL-RouterBench) finds that published comparisons do not separate them, for three reasons.

⚖

Baseline

An adaptive policy is usually compared with one fixed setting, so it may win simply by spending more compute on average. The baseline that isolates the decision is a random allocation of the same fixed actions at the same measured cost. None of the seven papers reports it.

W

Weights

Fixed baselines often use the untrained base model or a separately trained one, so a gain may come from training rather than from the choice. Fixed actions should use the adaptive policy's own trained weights.

↑

Headroom

Oracle estimates that select an action because one sampled answer is correct, and then score that same answer, reward lucky draws. With one sample per action this overstates the headroom by 10–20 points.


Protocol

Isolating the value of the decision

For every question and action we draw K = 20 independent samples (10 for DeepSeek-VL2-Small). A sample's correctness is never counted as an extra question; the fraction of correct samples estimates the action's success rate on that question. A policy must beat the cost-matched random mix below to show that its question-specific decisions add anything.

maxw ∑a wa pa   s.t.   ∑a wa = 1,   ∑a wa ca ≤ C The random mix assigns fixed action a (accuracy pa, measured cost ca) to a fraction wa of questions, independently of their content. Its value traces the upper hull of the fixed actions' cost and accuracy, and is evaluated at each router's measured average cost.
A

Actions

B: direct answer, ≤256 visual tokens, ≤32 generated tokens. LOOK: 1280 visual tokens (2.6–2.8× FLOPs). THINK: step-by-step reasoning, ≤512 tokens (1.3–1.5× FLOPs). LOOK+THINK: both (3.0–3.2×). B64 and B128 use fewer visual tokens.

½

Disjoint selection and scoring

Whenever samples inform a decision (picking an oracle action, training a router) each question's samples are split into a selection half and a scoring half. A decision is never scored on a sample that informed it; results average over up to 200 random splits.

R

Routers

Pre-routers pick an action from question and image features before any answer exists. Cascades run a cheap action first and use its answer and confidence (token log-probabilities, entropy, margin, optional hidden-state probe) to keep it or escalate. Cost includes both actions.

$

Costs and intervals

Costs are reported in both FLOPs and latency (batch size 1 on a T4, R² ≥ 0.994 linear fit), relative to B. Confidence intervals resample whole images, since several questions can share one image.

Models. Three dense VLMs (Qwen2.5-VL-3B, Qwen3-VL-2B, LFM2.5-VL-3B) and two mixture-of-experts VLMs (DeepSeek-VL2-Tiny and -Small). Data. 600 questions from 364 images: 200 each from ChartQA (exact chart values), MMStar (small images), and the 4K split of HR-Bench (median 4032 × 4032 pixels). Case study. The published VisionThink, re-run with its released weights and a token-exact port of its two-turn inference code.


Results

Large headroom, small realized gain

Finding 1 · THINK helps hard questions and hurts easy ones

THINK's average effect ranges from −8.4 to +7.2 points across models, but on all five models it helps where the baseline usually fails (+7.2 to +27.9 points on the hardest questions) and hurts where it usually succeeds (−8.2 to −36.1 on the easiest). LOOK also helps most on hard questions but rarely hurts. THINK costs only 1.3–1.5× the FLOPs yet takes 13–19× longer, because it generates many tokens one at a time.

ActionQwen2.5-VL-3BFLOPsQwen3-VL-2BFLOPsLFM2.5-VL-3BFLOPs
B6436.40.35×34.30.36×45.70.36×
B12849.10.57×49.20.58×60.30.59×
B (baseline)59.31.00×53.91.00×64.21.00×
LOOK63.42.75×56.72.64×68.92.62×
THINK50.91.32×61.21.51×65.91.37×
LOOK+THINK56.03.07×65.13.15×69.12.99×

Table 1. Accuracy (%) on 600 questions, 20 samples per question; FLOPs relative to each model's baseline B. No single fixed action is best across models.

Same-sample oracles overstate the headroom

With one sample per action, scoring the oracle on its own selection sample gives 85.2%, 78.9%, and 85.3% on the three dense models, 10 to 20 points above the held-out estimate. On Qwen2.5-VL-3B it claims +20.8 points over the best fixed action where the held-out oracle finds +1.2. Selecting on 10 samples and scoring on 10 others, most of the real gain comes from adding THINK to the menu.

PolicyQwen2.5-VL-3BQwen3-VL-2BLFM2.5-VL-3B
Best fixed action64.4 (2.62×)65.1 (3.15×)69.1 (2.99×)
Oracle over {B, LOOK}64.7 (1.55×)58.6 (1.40×)70.8 (1.40×)
Oracle over all four69.2 (1.69×)71.1 (1.78×)78.9 (1.60×)
Oracle − best fixed (pts)+4.8 [3.5, 6.3]+6.0 [4.4, 7.9]+9.8 [7.7, 12.2]

Table 2. Held-out oracles: accuracy (%) and relative FLOPs. The oracle uses ground-truth answers, so it is an upper bound on what routing can add, not a deployable policy. A cost-aware oracle matches baseline accuracy at 0.44–0.51× its FLOPs, almost entirely by giving fewer visual tokens to questions that do not need them.

Finding 2 · Trained routers and VisionThink recover little

At 1.25–2× the baseline FLOPs, trained cascades (logistic regression and gradient-boosted trees) beat the best random mix by 1.1–1.5 points on Qwen2.5-VL-3B, against 7.6–9.2 for the oracle, and by at most 1.4 and 2.1 points on the other two models, against 10.5–14.9. The gain is consistent across five repeats but small. Under the same protocol, VisionThink's decision carries signal against a random choice between its own two resolutions, but not against a mixture of all its fixed configurations.

VisionThink routing gain vs. equal-cost random mix of…at equal FLOPsat equal latency
its two resolutions (low, low→full)+2.4 [1.1, 3.8]+1.7 [0.6, 2.9]
all of its fixed configurations+0.7 [−2.0, 3.1]−4.5 [−8.3, −0.7]

Table 3. VisionThink (RL-trained Qwen2.5-VL-7B) re-run on our 600 questions with its released weights. Giving the model the full image first is 4.8 points more accurate at the same latency, because escalation pays for a second decoding phase. Comparisons with the untrained base model cannot reveal this.

Finding 3 · Confidence finds errors, not fixable errors

LOOK, THINK, and LOOK+THINK correct a wrong baseline answer only 17–42% of the time, and among questions the baseline almost always gets wrong, 35–44% stay wrong under every other action. Holding the escalation action (LOOK+THINK) and budget (1.5× FLOPs) fixed and changing only which answers are escalated isolates what each kind of knowledge is worth.

Escalate… (gain in pts over the row above)Qwen2.5-VL-3BQwen3-VL-2BLFM2.5-VL-3B
at random (equal-cost mix)referencereferencereference
least confident answers (deployable)+0.7 to +1.6below the mix−0.2 to +1.7
all wrong answers (perfect error detection)+3.4+2.6+4.3
fixable errors only (perfect fixability)+8.3+9.5+8.0

Table 4. Escalation rules at 1.5× baseline FLOPs. The last two rows use true outcomes and are diagnostic upper bounds. A question-level version that knows only each question's success rate from the other 19 samples still adds 4.4–7.4 points over perfect error detection: half to four-fifths of the fixability gain is a property of the question, not of a lucky sample.


Takeaways

What to do differently

⚙

For practitioners deploying VLMs

Before deploying a router, compare it with a random mixture of the same fixed actions, using the same weights and the router's measured average cost. Report both FLOPs and latency: saving FLOPs does not imply lower latency when a policy generates long answers or decodes twice. When the gain is small or uncertain, prefer a fixed action or a validated fixed-action mixture.

✎

For researchers building adaptive systems

Evaluate whether an action corrects an error separately from whether a policy predicts that correction. Select and score oracles on separate samples. Train routers to predict whether a given action will correct an answer, not whether the answer is uncertain; an action's held-out success rate per question is a natural training target.


Limitations

What this does not claim

The results do not show that adaptive computation cannot help. They show that its benefit has to be measured against the same actions allocated without looking at the question.

S

Scope

Five small VLMs (at most 3.75B dense parameters), three benchmarks, one prompted reasoning style with a 512-token limit, and the routers tested here. Larger models or other signals may behave differently.

V

VisionThink re-run

One greedy sample per question, float16, full images capped at 1280 visual tokens, and our own scorer; only the within-model comparison is meaningful.

∑

Upper bounds and costs

The oracle and fixability rules use ground-truth outcomes and are not deployable; predicting fixability is the open problem. FLOP counts omit attention, and the dense versus MoE comparison is confounded by model family and quantization.