Contrasting LLM-Assisted and Traditional UX Research in a B2B SaaS Environment - LLM-Assisted UX Research using Naturally Occurring Organizational Data and Synthetic Users

Abstract

User Experience research in Business-to-Business Software-as-a-Service (B2B SaaS) organizations relies on methods that are resource-intensive to sustain alongside continuous product development. Meanwhile, these organizations continuously generate large volumes of naturally occurring data such as customer support conversations, internal feedback channels, and design artifacts, that are rarely treated as systematic research input. While large language models (LLMs) are increasingly used to analyze data elicited for research, their ability to surface UX insight from naturally occurring organizational data, and how that insight compares to traditional methods, have received little systematic attention. This thesis investigates that question through a two-track study at Teamtailor, a B2B SaaS applicant tracking system (ATS) used by over 12,000 organizations. Track 1 conducted manual user research: a survey of 437 respondents and seven usability tests on a proposed redesign of the job creation flow. Track 2 developed a multi-agent LLM system that analyzed the same design context using different data. For the current product it drew on naturally occurring Slack feedback and Intercom support conversations; for the redesign it used synthetic-persona evaluation of the prototype in Figma. The two tracks were then systematically contrasted. The approaches surfaced different but complementary insights. LLM-assisted analysis was more effective at detecting silent system behaviors and recurring workflow constraints distributed across large data volumes; manual research was more effective at surfacing domain knowledge, missing functionality, and insights requiring contextual reasoning grounded in direct user engagement. The two tracks converged most strongly on the most critical usability issues, all of which informed design changes. Output quality depended heavily on conditions set before analysis: personas grounded in prior research, prompts framed with neutral domain context, and data structured consistently enough to carry meaningful distinctions. These findings support a hybrid model in which LLM-assisted analysis serves as a continuous breadth-first layer and human research provides the depth-first layer of contextual and interpretive understanding.