Domain-Aware Reasoning with Lightweight Models via Knowledge Distillation

Abstract

Large Language Models (LLMs) offer powerful reasoning capabilities but their computational demands often hinder deployment in domain-specific applications like cybersecurity. This thesis investigates the efficacy of knowledge distillation for transferring advanced reasoning from a large teacher model (DeepSeek-R1) to a lightweight student model (Meta-Llama-3.1-8B-Instruct) within a Retrieval-Augmented Generation (RAG) framework for cybersecurity intelligence. Using a dataset of real-world queries and RAG-retrieved context, the student model was fine-tuned via Supervised Fine-Tuning (SFT) on the teacher’s generated reasoning chains, employing Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA). Comprehensive evaluation, incorporating both AI-assisted analysis and blind domainexpert assessments, demonstrated that the distilled model significantly outperformed both the existing production RAG system at Recorded Future and its base Meta- Llama-3.1-8B-Instruct model. The distilled model exhibited superior contextual accuracy, a marked reduction in hallucinations, and higher overall response quality. This research successfully validates knowledge distillation as a potent strategy for creating computationally efficient, yet highly capable, domain-aware reasoning models, offering a practical pathway to enhance AI-driven solutions in specialized fields.