Towards Reliable Retrieval Systems: Design and Evaluation of an Agentic GraphRAG Pipeline

Abstract

Large language models are increasingly used for knowledge-intensive question answering, but their reliability remains limited when answers must be grounded in domain-specific documents. This limitation is particularly important in high-assurance environments, where systems must be controllable, traceable, and deployable without relying on external infrastructure. This thesis investigates the design and evaluation of an agentic GraphRAG system for document-grounded question answering, using scientific literature as a controlled proxy for technical internal documentation. The implemented system combines document ingestion, section-aware chunking, knowledge graph construction, vector indexing, graph traversal, reranking, and languagemodel- based answer generation. Two retrieval architectures are compared under shared conditions: a VectorRAG baseline using iterative hybrid search and a GraphRAG system using community-first hierarchical traversal over an explicit knowledge graph. The systems are evaluated across multiple retrieval configurations, generation models, and query sets using automated RAG evaluation metrics, statistical tests, and pairwise LLM-as-judge comparisons. The results show a clear divergence between retrieval-oriented metrics and answerlevel evaluation. VectorRAG achieves stronger automated retrieval scores, particularly on context precision and recall, while GraphRAG is preferred in holistic answer comparisons and manual validation. This suggests that chunk-level retrieval metrics do not always capture the usefulness of structurally retrieved evidence for downstream answer generation. The findings indicate that graph-based retrieval can improve answer quality and interpretability in agentic RAG systems, but also highlight important limitations related to dataset construction, evaluator dependence, corpus scale, graph quality, and agentic control.