User feedback to actionable insight: Generative AI for user feedback analysis

Abstract

Software teams rely heavily on user feedback to understand how people experience their applications. However, the unstructured and high-volume nature of app reviews makes manual analysis increasingly difficult. Although recent research has shown that Large Language Models are able to perform classification tasks related to user feedback, it is still unclear if it is possible to leverage such models to produce actionable insights that are applicable to software development. This thesis explores the capability of generative AI to extract actionable insights from user feedback through a mixed-methods approach. In the qualitative phase, semi-structured interviews with five industry practitioners are conducted to establish a grounded definition of an actionable insight and identify the operational challenges of manual triage. In the experimental phase, multiple LLMs, including ChatGPT, Gemini, and Mistral AI are evaluated using prompt engineering techniques on the public REFSQ-2026 app review dataset. The research examines how well these LLMs perform when processing raw or unclassified data as opposed to classified data. Finally, the AI-generated insights are compared directly against a human analyst. This baseline was established by manually extracting insights from the dataset using the four-point criteria developed in the qualitative phase of the study, allowing for a systematic comparison to identify areas of alignment, divergence, and potential AI blind spots. From the results it is clear that the LLMs can successfully match human expert baselines across issue recall, semantic similarity, and structural formatting, reducing processing time from hours to seconds. Furthermore, the evaluation reveals that classification labels impact each model differently: ChatGPT achieves maximum is sue recall but degrades in formatting, Mistral AI performs optimally on raw text and degrades under structural constraints, and Gemini trades slight semantic precision for higher bug discovery. This research provides an evidence-based understanding of how software teams can reliably select and deploy LLMs to translate noisy user feedback into structured, developer-ready tasks.