One-Line Summary
A hybrid document retrieval framework that leverages a vision-language model (VLM) to generate rich textual descriptions of visually complex documents — capturing information carried by figures, tables, and layout that plain text extraction misses — and fuses these descriptions with conventional text and dense-vector retrieval signals to improve retrieval quality. This work received an Outstanding Paper Award at the Korea Computer Congress 2026 (KCC 2026).
Background & Motivation
Modern document collections are rarely plain text. Reports, scientific papers, manuals, and financial filings routinely convey essential information through figures, charts, tables, and spatial layout. Conventional retrieval pipelines, however, operate over text extracted by OCR or parsers — a process that flattens or discards precisely the visual signals that make these documents informative. As a result, a query whose answer lives inside a chart or a table can fail to retrieve the page that actually contains it.
Key Challenge: Visually rich documents encode meaning not only in their words but in figures, tables, and layout. Text-only retrieval systems — whether sparse lexical (e.g., BM25) or dense-vector — index a degraded, text-only view of each page and therefore systematically miss content that is expressed visually. The core question is how to inject the missing visual semantics into a retrieval pipeline without abandoning the efficiency and maturity of existing text-based methods.
One line of prior work tackles this with fully vision-based retrieval, embedding rendered page images directly. Such approaches capture visual content but discard the strengths of established text retrieval — exact lexical matching, interpretability, and low-cost indexing — and can struggle with fine-grained textual queries. This paper instead pursues a hybrid route: keep the proven text and dense-vector retrieval machinery, and augment it with a compact, searchable textual surrogate of each document's visual content produced by a VLM.
Text-Only vs. VLM-Augmented Retrieval
| Aspect | Text-Only Retrieval | VLM-Augmented Hybrid (This Work) |
|---|
| Indexed content | OCR / parsed text only | Text + VLM description of figures, tables, layout |
| Visual information | Lost or degraded | Recovered as natural-language description |
| Retrieval signals | Lexical and/or dense-vector | Lexical + dense-vector + description-based |
| Compatibility | Standard index | Reuses existing text/vector infrastructure |
| Queries about charts/tables | Often unanswerable | Made retrievable via generated descriptions |
Proposed Method
The approach adds a description-generation stage in front of an otherwise standard retrieval pipeline, then fuses the resulting signals at scoring time. It has two conceptual pillars: (1) turning visual content into searchable text with a VLM, and (2) combining that new signal with existing text and dense-vector retrieval.
1
VLM-Based Description Generation
Each document (or page) is rendered as an image and passed to a vision-language model, which produces a natural-language description of its visual content — summarizing what figures depict, transcribing and paraphrasing tables, and noting layout cues such as headings, captions, and the relationships between visual elements. The VLM effectively translates non-textual information into a form that text-based retrievers can index and match against, recovering semantics that OCR or parsers would otherwise drop.
2
Enriched Document Representation
The generated descriptions are combined with the document's originally extracted text to form an enriched representation of each page. This unified representation carries both the literal textual content and a verbalized account of the visual content, so that a single document entry can be matched by queries targeting either modality. The enrichment is applied at indexing time, keeping query-time behavior lightweight.
3
Hybrid Retrieval Fusion
Retrieval combines complementary signals over the enriched representation: sparse lexical matching for exact terms and dense-vector similarity for semantic matching, together with the description-derived evidence. By fusing these scores, the system benefits from the precision of lexical matching, the generalization of dense retrieval, and the newly added visual coverage from the VLM descriptions — rather than relying on any single signal alone.
Design Principles:
- Augment, don't replace: The VLM description is added on top of existing text and dense-vector retrieval, preserving their efficiency, interpretability, and mature tooling.
- Verbalize the visual: Rather than embedding raw page images, visual content is converted into searchable natural language, making it directly compatible with standard text retrievers.
- Complementary fusion: Lexical, dense, and description-based signals cover different failure modes, so combining them is more robust than optimizing any one in isolation.
Key Points
- Recovers visual semantics for retrieval: By having a VLM verbalize figures, tables, and layout, the method makes information that is otherwise invisible to text retrievers searchable.
- Hybrid over single-signal: Fusing sparse lexical, dense-vector, and VLM-description evidence combines the strengths of each and mitigates the blind spots of any individual retrieval method.
- Reuses existing infrastructure: Because visual content is expressed as text, the approach plugs into standard text and dense-vector indexes without requiring a specialized image-based retrieval stack.
- Targets visually rich documents: The method is aimed squarely at documents where meaning is carried by charts, tables, and layout — the setting where conventional pipelines most often fail.
- Indexing-time enrichment: Descriptions are generated when documents are indexed, so the added VLM cost is paid up front and does not burden every query.
- Recognized contribution: The work was selected for an Outstanding Paper Award at KCC 2026, reflecting the practical relevance of bridging visual document understanding and information retrieval.
Why It Matters
Retrieval quality is the foundation of many downstream applications, from search interfaces to retrieval-augmented generation (RAG). When the underlying documents are visually rich, text-only retrieval imposes a hidden ceiling: no matter how strong the reader or generator, it cannot use evidence that was never retrieved because it lived inside a figure or table. By recovering that evidence as searchable text, this work raises the ceiling on what downstream systems can access.
- Bridges vision and retrieval pragmatically: Instead of replacing mature text retrieval with heavy image-based systems, it verbalizes visual content and folds it into existing pipelines — a route that is both effective and easy to deploy.
- Strengthens RAG on real documents: Enterprise, scientific, and financial corpora are dominated by visually rich formats; improving retrieval over them directly improves the quality of the answers RAG systems can produce.
- Complementary, not competitive, signals: The hybrid design shows that VLM-generated descriptions are most valuable as an additional signal alongside lexical and dense retrieval, offering a principled way to integrate multimodal understanding into IR.
Broader Insight: The bottleneck for retrieval over visually rich documents is not the retriever's algorithm but the impoverished, text-only view it is given. By using a vision-language model to translate figures, tables, and layout into natural language, the visual and textual worlds are unified within a single, familiar retrieval framework — delivering the coverage of multimodal understanding while retaining the efficiency and reliability of established text-based information retrieval.
Information Retrieval
Multimodal