Sign Language and Face Expression Recognition Using Neural Networks: Deep Learning Approach to Break Communication Barriers

Notice

This is an unedited manuscript accepted for publication and provided as an Article in Press for early access at the author’s request. The article will undergo copyediting, typesetting, and galley proof review before final publication. Please be aware that errors may be identified during production that could affect the content. All legal disclaimers of the journal apply.

Year : 2024 | Volume : 14 | Issue : 03 | Page : 1 10
    By

    Aditya Mohan Dixit,

  • Swati Dixit,

  • Jayashree Mahajan,

  • Namrata Soni,

  1. Student,, Department of Computer Engineering, Indira College of Engineering and Management Pune, Maharashtra, India
  2. Student,, Department of Computer Engineering, Indira College of Engineering and Management Pune, Maharashtra, India
  3. Student,, Department of Computer Engineering, Indira College of Engineering and Management Pune, Maharashtra, India
  4. Student,, Department of Computer Engineering, Indira College of Engineering and Management Pune, Maharashtra, India

Abstract

Our study proposes a multimodal gesture recognition system specifically designed to aid communication for the deaf community. By employing neural network concepts, we utilize 3D convolutional neural networks (3D CNNs) to extract features from both hand and face images, focusing on relevant regions. Preprocessing techniques are applied to isolate these areas of interest prior to feature extraction. Unique 3D CNN architectures are then trained for each modality to capture the spatial and temporal characteristics of common gestures used in sign language and non-verbal communication among the deaf. An important aspect of our approach involves integrating information from both hand and face modalities through fusion strategies, enhancing recognition accuracy. Through early and late fusion techniques, we combine features or decisions from individual modalities to ensure robust performance, even in challenging scenarios such as obscured or ambiguous gestures. Experimental evaluations confirm the effectiveness of our system, showing significant improvements in recognition accuracy across various scenarios. This research contributes to advancing inclusive technology tailored to the needs of the deaf community, with potential applications in communication aids, education, and assistive technologies, empowering deaf users to interact effectively with computing devices in the digital age.

Keywords: Neural Network Model, Sign Language Recognition, Sign language, muteness, deep learning models, Custom Sign Language. Region of Interest, Media Pipe model, Support Vector Machine, Artificial Neural Network, American Sign Language, Vietnamese Sign Language

[This article belongs to Current Trends in Signal Processing ]

How to cite this article:
Aditya Mohan Dixit, Swati Dixit, Jayashree Mahajan, Namrata Soni. Sign Language and Face Expression Recognition Using Neural Networks: Deep Learning Approach to Break Communication Barriers. Current Trends in Signal Processing. 2024; 14(03):1-10.
How to cite this URL:
Aditya Mohan Dixit, Swati Dixit, Jayashree Mahajan, Namrata Soni. Sign Language and Face Expression Recognition Using Neural Networks: Deep Learning Approach to Break Communication Barriers. Current Trends in Signal Processing. 2024; 14(03):1-10. Available from: https://journals.stmjournals.com/ctsp/article=2024/view=183865


References

  1. Sahoo AK, Mishra GS, Ravulakollu KK. Sign language recognition: State of the art. ARPN Journal of Engineering and Applied Sciences. 2014 Feb;9(2):116-34.
  2. Adeyanju IA, Bello OO, Adegboye MA. Machine learning methods for sign language recognition: A critical review and analysis. Intelligent Systems with Applications. 2021 Nov 1;12:200056.
  3. Tolentino LK, Juan RS, Thio-ac AC, Pamahoy MA, Forteza JR, Garcia XJ. Static sign language recognition using deep learning. International Journal of Machine Learning and Computing. 2019 Dec;9(6):821-7.
  4. Garimella M. Sign Language Recognition with Advanced Computer Vision (2023). Medium. com.
  5. Kumar S, Kumar P, Mishra P, Tewari P. A Robust Sign Language and Hand Gesture Recognition System Using Convolutional Neural Networks. In2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) 2023 Dec 15 (pp. 395-399). IEEE.
  6. Subramanian B, Olimov B, Naik SM, Kim S, Park KH, Kim J. An integrated mediapipe-optimized GRU model for Indian sign language recognition. Scientific Reports. 2022 Jul 13;12(1):11964.
  7. Trivedi K, Gaikwad P, Soma M, Bhore K, Agarwal P. Improve the recognition accuracy of sign language gesture. International Journal for Research in Applied Science and Engineering Technology.;10:4343-7.
  8. Lu C, Kozakai M, Jing L. Sign Language Recognition with Multimodal Sensors and Deep Learning Methods. 2023 Nov 29;12(23):4827.
  9. Rekha J, Bhattacharya J, Majumder S. Shape, texture and local movement hand gesture features for indian sign language recognition. In3rd international conference on trendz in information sciences & computing (TISC2011) 2011 Dec 8 (pp. 30-35). IEEE.
  10. Shanableh T, Assaleh K, Al-Rousan M. Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 2007 May 15;37(3):641-50.
  11. Brashear H, Starner T, Lukowicz P, Junker H. Using multiple sensors for mobile sign language recognition. InSeventh IEEE International Symposium on Wearable Computers, 2003. Proceedings. 2003 Oct 1 (pp. 45-45). IEEE Computer Society..
  12. Gao W, Ma J, Wu J, Wang C. Sign language recognition based on HMM/ANN/DP. International journal of pattern recognition and artificial intelligence. 2000 Aug;14(05):587-602.
  13. Al-Hammadi M, Muhammad G, Abdul W, Alsulaiman M, Bencherif MA, Mekhtiche MA. Hand gesture recognition for sign language using 3DCNN. IEEE access. 2020 Apr 27;8:79491-509.
  14. Byeon YH, Kwak KC. Facial expression recognition using 3d convolutional neural network. International journal of advanced computer science and applications. 2014;5(12).
Regular Issue Subscription Review Article
Volume 14
Issue 03
Received 09/10/2024
Accepted 18/10/2024
Published 18/11/2024
320

Login


My IP

PlumX Metrics