A Swedish Sign Language Chatbot

Abstract

Being unable to hear is a huge obstacle to smooth and natural communication, especially in interactions with non-sign language speakers. This communication barrier makes hearing-impaired people more vulnerable in various emergency situations where the ability to be understood quickly can be the difference between life and death. A significant amount of the previous sign language recognition research has been focused on the translation from signs to words, through sensory gloves and cameras, focused on the signer. This previous research can be seen as circumventing the core problem, that non-hearing impaired people lack the ability to understand sign language. The thesis aims to expand this research field through the development of a sign language chatbot, a tool that hopefully allows for faster learning of sign language. This study presents the development of a sign recognition neural network that predicts sign language made by a person through a webcamera. This sign recognition network achieved a 99% accuracy in recognizing signs from a test dataset, demonstrating the effectiveness of the chosen neural network architecture and training methodology and keypoint estimation. However, the real-world evaluation showed less optimal results, but still sufficiently high, due to a lower generalization of unseen data and novice signers as test subjects. No proper evaluation was conducted to see which of the two reasons contributed the most to the lower real-world performance. This sign recognition network has been paired with a chatbot to enable sign-language conversations between a user and a computer, potentially enabling the system to be used as a sign-language training tool.