One-Line Summary
A controlled study of 13 multimodal LLMs on two knowledge-conflict datasets showing that MLLMs are not modality-robust: the same evidence yields different answers depending on whether it arrives as text or as a rendered image. Contrary to the widely reported "text bias," models follow contradicting evidence more readily in image form, and when text and image conflict, the preferred modality flips arbitrarily with input order, model, and dataset (8/13 models flip sign on ConflictQA). This instability raises image-rendered attack success by +6.99 points on average, and among several remedies only conflict-aware SFT offers partial mitigation.
Background & Motivation
Multimodal LLMs (MLLMs) are increasingly deployed in retrieval-augmented and agentic workflows where evidence arrives in heterogeneous surface forms: plain text, webpages, PDFs, screenshots, or scanned documents. These pipelines implicitly assume that semantically identical content is processed consistently regardless of modality. That assumption is not guaranteed, because text and image inputs follow distinct internal pathways starting from modality-specific tokenization.
The risk is amplified under knowledge conflict (KC), where external evidence contradicts the model's parametric knowledge or multiple sources support competing answers. If which source the model trusts is governed by the form of the evidence rather than its content, reliability is severely compromised.
Gaps in Prior Work:
- Modality bias not tested under conflict: Studies reporting a "Blind Faith in Text" tendency in MLLMs never examined knowledge-conflict scenarios.
- Natural images, not documents: Existing multimodal KC work focuses on scenes or faces, even though knowledge is typically organized in document form.
- Uncontrolled parametric knowledge: Prior work either skipped verifying the model's internal belief or assumed parametric memory is always correct, leaving findings potentially confounded.
This paper fills these gaps by formalizing modality robustness as an evidence-authority problem: holding the semantic content of evidence fixed, presenting it as text or as a rendered image of that text, and asking whether the model resolves the conflict the same way.
Proposed Method: Controlled Modality-Robustness Evaluation
The framework evaluates two conflict configurations across 13 MLLMs (Claude Sonnet 4.5, GPT-5.4, GPT-4o; Qwen3-Omni, Qwen2.5-Omni, MiniCPM-o2.6, OmniVinci; Qwen2.5-VL-7B/32B, InternVL3-8B/38B, LLaVA-v1.6-34B, LLaVA-vicuna-7B) and two datasets: ConflictQA (LLM-generated counter-memory passages) and NQ-Swap (entity-substituted Natural Questions passages).
Experimental Results
The dominant pattern across models, datasets, and orders is instability: modality is not a passive container of evidence but a factor that changes how conflicts are resolved.
Normalized Image-over-Text Gap G (%) on ConflictQA (selected models)
| Model | Single | Multi I→T | Multi T→I |
|---|---|---|---|
| Claude Sonnet 4.5 | +42.86 | +79.41 | +84.62 |
| GPT-5.4 | +15.29 | +1.80 | +63.37 |
| GPT-4o | +9.75 | +4.40 | −19.05 |
| Qwen3-Omni | +1.64 | −38.24 | +79.25 |
| OmniVinci | −1.81 | −64.66 | +86.58 |
| Qwen2.5-VL-7B | +4.09 | −68.98 | +73.82 |
| InternVL3-8B | −1.27 | −93.03 | +58.77 |
| LLaVA-v1.6-34B | −5.34 | −81.64 | −85.65 |
| LLaVA-vicuna-7B | −9.17 | −95.45 | −94.86 |
Positive = image preference, negative = text preference. On NQ-Swap, Claude Sonnet 4.5 shows +22.63 (single) and GPT-4o reverses to −14.74; full results for all 13 models are in the paper's Table 1.
Mitigation on Qwen2.5-VL-7B (ConflictQA, G in %)
| Method | Multi I→T | Multi T→I | Single (misleading) | Single (reliable) |
|---|---|---|---|---|
| Original | −68.97 | +73.82 | +2.54 | +3.45 |
| + Explicit prompting | −17.54 | +45.69 | – | – |
| + Conflict-aware SFT | −67.07 | +52.04 | +0.25 | +1.08 |
| + DPO (vs. −68.89 / +68.47 base) | −88.08 | +65.57 | – | – |
| Steering, scale 5 (vs. −38.64 / +77.77 base) | −86.96 | +60.00 | – | – |
Only conflict-aware SFT moves every gap toward zero; DPO and steering reduce the image-dominant gap but inflate the text-dominant one. Prompting amplifies bias for GPT-5.4 (+1.80 → +71.26 under I→T).
- Image preference, not text bias, in single-evidence conflict: Proprietary models show the strongest positive gaps (Claude Sonnet 4.5: +42.86 on ConflictQA, +22.63 on NQ-Swap; GPT-5.4: +15.29 / +3.80), reversing the "Blind Faith in Text" consensus. Only the LLaVA family stays consistently text-dominant.
- Order flips the preferred modality: With image first (I→T), most open-source models lean on text; reversing the order shifts them to image. 8/13 models on ConflictQA and 6/13 on NQ-Swap change the sign of G across the two orders.
- Dataset matters: ConflictQA, whose counter-evidence diverges semantically, produces far larger gaps than NQ-Swap, which only swaps the answer entity (OmniVinci: +86.58 vs. near zero).
- RAG consequence: Rendering the gold passage as an image mitigates the lost-in-the-middle effect; Gemini 2.5 Pro gains +6.59 ROUGE-L when the image-rendered gold passage is in the middle of nine distractors.
- Safety consequence: Moving the full harmful instruction into the image raises Attack Success Rate by +6.99 points on average over 156 settings, positive in 126/156 (80.8%) comparisons, largest against the Text-only baseline (+11.60) and across Qwen (+3.74), InternVL (+7.93), and LLaVA (+9.30).
- Preprocessing shapes reliance: Applying LLaVA-style 336×336 preprocessing to Qwen2.5-VL moves its NQ-Swap single-evidence gap from +9.71 to −37.23, within three points of LLaVA's native −34.92; the reverse intervention shifts LLaVA only from −34.92 to −30.91.
Why It Matters
As MLLMs are wired into RAG and agentic pipelines that ingest PDFs, screenshots, and scanned documents alongside plain text, the system rarely controls how evidence arrives. This paper shows that this upstream representation choice silently changes which source a model treats as authoritative, so modality preference is better understood as an artifact of the evaluation setup than as a fixed property of the model.
- Reassesses the consensus on modality bias: The widely reported text bias in MLLMs does not survive controlled knowledge-conflict evaluation; the direction often reverses, and dissolves into arbitrariness once both modalities conflict.
- Exposes an exploitable vulnerability and a usable lever: The same instability enables stronger image-rendered jailbreaks (+6.99 ASR) and, conversely, suggests rendering salient passages as images to steer attention in long RAG contexts.
- Distinguishes controlling preference from improving robustness: DPO and steering merely shift reliance toward text; only conflict-aware SFT reduces imbalance in both directions, and even then only partially, indicating the problem is fundamental and must be addressed at multiple training stages.