Error and Reliability Analysis of Open Source LLMs for Text-to-SQL Generation Across Query Complexities

Abstract

Retrieving the correct data from databases quickly and accurately using Structured Query Language (SQL) is a crucial task in Software Engineering, but one that is time-consuming if done manually, and complicated to navigate if done through an application. However, asking a specific question about the database in natural language and using a Large Language Model (LLM) to generate a SQL query that retrieves the desired data is more time-efficient and intuitive for practitioners. This Text-to-SQL process is nonetheless difficult for LLMs, as it requires high-reasoning and domain-knowledge capabilities. This study aims to gain a better understanding for failures of LLMs in this context by exploring how the complexity of a Text-to-SQL task, and the active parameter count of LLMs, affect the type of failures, how often they occur, and how consistent the failure rates are. To achieve this, three randomly sampled sets of Text-to-SQL questions of different complexities from the BIRD and LiveSQLBench-Base-Lite datasets are run on the Qwen3 A3B, Qwen3 A22B and Qwen3 A35B LLMs via Amazon Bedrock using a two-step prompting strategy. The results are first analyzed descriptively and then statistically tested. The results show that selecting the correct tables to use in the generated query is the most common subcategory of failure observed across LLMs, that increases in size between the selected LLMs do not significantly affect performance on any complexity, and that none of the examined LLMs are significantly different in failure reliability across complexities. With that being said, all three LLMs achieve absolute failure consistency, in at least 76% of the Text-to-SQL questions, indicating that failures are likely systematic due to an inability to produce a correct answer.