Lightweight CNN and Grad-CAM Integration for Interpreting Arabic Sign Language Hijaiyyah Letter Classification

Abstract

Communication plays an important role in social life as a means of conveying information and building interactions between humans. For individuals who are deaf and mute, sign language serves as a visual communication medium that supports the process of conveying messages. One of the sign language systems is Arabic Sign Language (ArSL), which utilises the Hijaiyyah letters through the fingerspelling method to spell out certain terms or vocabulary. However, the limited understanding of sign language within society still poses a barrier to social interaction. This research develops an Arabic Sign Language (ArSL) Hijaiyyah letter classification system based on a Lightweight Convolutional Neural Network (CNN) with the aim of producing a model that has high performance and low computational requirements. Additionally, the Explainable Artificial Intelligence (XAI) method using Gradient-weighted Class Activation Mapping (Grad-CAM) is applied to enhance the transparency of classification results through the visualisation of important areas in the images. Based on the test results, the developed model achieved an accuracy of 0.84, a precision of 0.84, a recall of 0.84, an F1-score of 0.84 and a an Specificity of  0.99. The Grad-CAM visualisation results show that the model focuses on the relevant hand areas during the prediction process. The findings of this study indicate that the system is capable of performing classification with good and measurable performance, as well as supporting real-time sign language recognition with a short response time, thereby improving communication accessibility for individuals with disabilities.