Data-Driven Automated Reporting Solution for External Collaborations - LLM-driven KPI Definition

Abstract

This thesis presents a proof-of-concept, developed with AstraZeneca (AZ), that explores automating progress reporting for external collaborations by testing whether a large language model (LLM)-driven system can extract objectives from contracts and translate them into tailor-made key performance indicators (KPIs). Objective extraction is quite reliable, reaching several highs of accuracy around the 85%-mark, but converting objectives into KPIs that stakeholders judge as relevant, clear, actionable, and measurable, is substantially less solid. Fewer than half of the KPIs met each quality criterion on average, and 39% met none. Survey responses noted that KPIs were often unclear, overly generic, or poorly timed, and skewed toward simple counts (e.g., “number of models”) that miss quality and impact. From interviews conducted at AZ, a set of general KPIs, that were deemed meaningful to measure in a collaboration project, could be demonstrated. The final evaluation suggests that these KPIs (e.g., external engagement and budget coherence) outperform collaboration-specific KPIs generated directly from objectives. This underscores the difficulty of creating bespoke target measures in diverse contexts. Despite these issues, the approach offers practical value. In principle, the pipeline should be better suited for agreements with explicit milestones (e.g., business or commercialisation contracts), where more clearly defined expected outcomes support better-formed KPIs. However, this cannot be conclusively established by the implementation in this thesis, due to limited data. Ultimately, translating qualitative objectives into quantitative, decision-grade KPIs remains inherently difficult. Contemporary LLMs are capable across many aspects of automation, but evidently less reliable for high-judgement and context-specific KPI design that balances relevance, clarity, actionability, and measurability, at least by following the approach outlined in this thesis. Therefore, the most defensible nearterm usefulness is in metadata extraction and recommendation, while still requiring a human-in-the-loop as a safeguard. In turn, this can improve customer relationship management (CRM) metadata completeness and enable collaboration health insights and automated reporting.