Using LoRA to Improve Retrieval in Structured Document Collections

Abstract

This thesis explores different roles for Low-Rank Adaptation (LoRA) in a question answering (QA) system over structured document collections, and aims to shed light on benefits and limitations as well as rank requirement interactions with dataset size and confusability. Three roles are investigated on synthetic document collections mimicking company data. First, LoRA as parametric memory to internalize new knowledge directly into the model weights; second, LoRA as a tool for embedding adaptation in a retrieval-augmented generation (RAG) system to improve retrieval performance by improving the vector representations of chunks; third, LoRA as a tool for context routing in a RAG system to improve retrieval by narrowing the search space. The findings suggest that LoRA is a flexible tool that can improve QA performance in all three roles, but that each role comes with its own characteristics and trade-offs between for example performance, flexibility, and robustness. In the specific settings tested, parametric memory is shown to be effective and highly rank sensitive; embedding adaptation is shown to give robust retrieval and generation improvements already at low ranks with gradual benefits of increased rank; context routing is shown to give large retrieval benefits at low rank requirements but introduces certain brittleness in both the retrieval and generation step.