Impact of Generative AI on Learning Programming

Abstract

Generative AI has become a powerful tool, particularly in code generation for nonprogrammers. Despite its potential, non-experienced programmers may face challenges and limited knowledge acquisition. This study analyzes whether generative AI, specifically ChatGPT, can assist individuals with limited programming knowledge in solving programming tasks in Java and if they learn from the experience. Additionally, it examines the strategies participants use when leveraging ChatGPT 3.5. A controlled experiment was conducted to observe the effects of ChatGPT on problem-solving and learning. Participants completed three programming tasks: a baseline task, a task using ChatGPT, and a final task without ChatGPT to assess knowledge retention. This revealed varying degrees of success in task completion with ChatGPT. While participants retained conceptual knowledge, practical application without ChatGPT remained challenging. Thematic analysis of participant prompts and exit interviews revealed three main strategies: learning-focused, task-focused leveraging ChatGPT, and conversational interaction with ChatGPT. The learning-focused strategy involves using the AI to gain knowledge but may be ineffective if rushed, hindering comprehension. In contrast, the task-focused strategy prioritizes task completion but may lead to overreliance on AI, limiting long-term learning. The conversational strategy is distinct because it does not serve a specific purpose like the other two. Instead, it involves a back-and-forth interaction with the AI, where participants build upon ChatGPT’s previous responses. Overall, participants had a positive attitude towards generative AI, recognizing both its benefits and limitations. The study highlights the importance of prior knowledge and effective prompt engineering to formulate informed prompts and obtain quality responses, thereby improving the overall experience with AI.