Self-Improving LLM Contexts for Agentic AI

Loading...
Thumbnail Image

Date

Type

Examensarbete för masterexamen
Master's Thesis

Model builders

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

Large language model (LLM) agents have become increasingly capable systems for solving complex tasks through reasoning, tool use, and interaction with external environments. Recent work has shown that the LLM agents can be improved not only through parameter fine-tuning, but also through modifying the context. This thesis investigates context adaptation for domain-specific LLM agents in a threat intelligence environment. In collaboration with Recorded Future, we implement and evaluate a framework based on Agentic Context Engineering (ACE) for optimizing a search-oriented LLM agent through iterative generation, reflection and curation. The framework is further extended with ReAct-style reflection and the incorporation of external domain documentation during optimization. The approach is evaluated on datasets constructed from realistic threat intelligence tasks and compared against baseline agents with either minimal or handcrafted context. Experimental results show that the context adaptation substantially improves agent performance, achieving higher search accuracy and grounding. The best performing framework, with the new contributions and without ground truth supervision, achieves 15.1 percentage points increase in pass@1. Overall, the findings demonstrate that context engineering is an effective approach for adapting LLM agents to specialized enterprise domains. Furthermore, indication is given that ground truth supervision might be substitutable for environment interaction.

Description

Keywords

Agentic context engineering (ACE), large language models (LLM), AI agents, context adaptation, prompt optimization.

Citation

Architect

Location

Type of building

Build Year

Model type

Scale

Material / technology

Index

Endorsement

Review

Supplemented By

Referenced By