AI-Wolf in Sheep’s Clothing Distinguishing between Swedish humans and AI wannabes

Abstract

This report is investigating whether it is possible for Artificial Intelligence (AI) chatbots and AI detectors to detect texts created by AI in Swedish and in English. Mainly focusing on texts created by the AI chatbot ChatGPT 4, the performances of the AI detectors Smodin and Copyleaks are investigated. The research is motivated by a scarce previous research about AI in Swedish and an articulated need from Swedish schools to understand if it is possible to detect AI created content in homework and essays. By prompting the AI detectors with 400 articles written by humans in Swedish, 400 articles written by humans in English, 400 texts created by AI in Swedish, and 400 texts created by AI in English the AI detectors were thorogly examined. What could be proven was that Smodin showed an accuracy between 73.61% - 73.89% on a 99% confidence level on Swedish content, and an accuracy between 94.68% - 94.82% on a 99% confidence level on English content. On the other hand, Copyleaks showed an accuracy between 94.80% - 94.95% on a 99% confidence level on Swedish content, and an accuracy between 100% on a 99% confidence level on English content. This indicates that, whilst always sucessfully detecting texts written by humans, it is possible to detect Swedish text created by AI nine out of ten times.