Automating Financial Report Analysis and Generation using LLMs

Abstract

Background: The emergence of Large Language Models (LLMs) has enabled automation of complex natural language processing across a wide range of domains. Still, their application and research on designing them in the financial domain remains limited.

Objective: This study explored how LLMs can be integrated into financial report analysis and commentary generation, with a particular focus on the software engineering challenges encountered during the design and implementation of such solutions.

Method: A Design Science Research methodology was used where an exploratory case study was conducted. Two LLM-based systems were iteratively designed and evaluated: one employing local open-source models in a multi-agent workflow, and the other utilizing GPT-4o. Both solutions were evaluated through expert assessments of a real-world financial reporting use case.

Results: It was observed that LLMs had great potential to automate tasks within financial reporting workflows, yet their integration presents challenges. Through iterative development and expert evaluation, several issues were identified, including prompt design, contextual dependency, and trade-offs between implementation options. Cloud-based models were found to offer greater fluency and ease of use, but raised concerns related to data privacy and reliance on external services. In contrast, local open-source models provide stronger data control and compliance but require substantially more engineering effort to ensure reliability and usability.

Conclusion: LLMs showed strong potential for automating financial reporting, but their integration requires careful attention to architecture, prompt design, and system reliability. Successful implementation depends on addressing domain-specific challenges through tailored validation mechanisms and engineering strategies that strike a balance between accuracy, control, and compliance.