Multimodal Retrieval Augmented Generation via Embedding Space Alignment

Abstract

Retrieval-augmented generation (RAG) systems answer questions by first retrieving supporting evidence and then generating a response conditioned on that evidence. In visually rich document collections, such as Swedish municipal PDF archives, relevant information is often spread across text, maps, diagrams, and photographs. Many deployed pipelines are built around strong text embedding models like E5, and fully migrating to a new multimodal retriever can be costly and disruptive. This thesis investigates whether visual retrieval can be added to an existing E5- based text pipeline by aligning image representations to its embedding space, instead of re-embedding the corpus with a unified multimodal model. Concretely, a neural “bridge” is trained to project SigLIP image embeddings into the E5 text space, using a staged procedure (MAPS) that gradually shapes the target space, leverages paired question–image and question–text examples, and learns calibrated joint ranking between images and text. A visual-question-answering style benchmark is constructed from Swedish municipal documents, where each natural-language question is paired with the images or text passages from which it is answerable. On a corpus of 235,653 documents, 901,571 text chunks, and 427,843 images, naïve combinations of E5 and SigLIP scores fail to retrieve relevant images, yielding essentially no high-ranked image hits. The learned bridge, in contrast, raises image-only retrieval performance from single-digit to high-20% top-ranked hit rates and achieves comparable gains in a joint image–text setting, while while largely preserving text retrieval beyond the very top rank. These results suggest that embedding-space alignment could be a practical strategy for extending text-centric RAG infrastructure with visual search over document images, without replacing the underlying text encoder or index.