Ankitha Bekal1,,
Abhay Shetty M.,
Gourav,
Maneesh Shetty M,
- Assistant Professor, Department of Computer Science Engineering, PA College of Engineering (Affiliated to Visvesvaraya Technological University), Mangalore, Karnataka, India
- Student, Department of Computer Science Engineering, PA College of Engineering (Affiliated to Visvesvaraya Technological University), Mangalore, Karnataka, India
- Student, Department of Computer Science Engineering, PA College of Engineering (Affiliated to Visvesvaraya Technological University), Mangalore, Karnataka, India
- Student, Department of Computer Science Engineering, PA College of Engineering (Affiliated to Visvesvaraya Technological University), Mangalore, Karnataka, India
Abstract
Lip reading, the ability to interpret spoken language by observing lip movements, is a valuable skill that can aid in various applications, particularly in enhancing speech recognition systems. This project explores the implementation of a deep learning-based lip-reading model to improve the accuracy and robustness of speech recognition in challenging environments, such as noisy or audio-limited settings. The proposed lip-reading system leverages Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to effectively capture temporal and spatial features from lip movement sequences. A comprehensive data set comprising diverse speakers and linguistic contexts is used to train and evaluate the model, ensuring its generalization across different scenarios. The key steps of the project include data preprocessing, feature extraction, model training, and integration with existing speech recognition systems. The model is trained on synchronized audio-visual datasets to learn the correlation between speech signals and corresponding lip movements. To address real-world challenges, the system is designed to handle variations in lighting conditions, facial expressions, and speaker accents the performance of the lip reading-enhanced speech recognition system is evaluated through rigorous testing, comparing its accuracy and efficiency against traditional audio- only systems. The results demonstrate the potential of incorporating lip reading as a supplementary modality to improve the overall robustness and accuracy of speech recognition, especially in scenarios where audio signals alone may be insufficient. This research contributes to the growing field of multimodal deep learning and highlights the practical applications of lip reading in enhancing human-computer interaction, accessibility, and communication systems. The findings open avenues for future research in developing more advanced and context-aware models that can further bridge the gap between visual and auditory information processing in intelligent systems.
Keywords: CNN, Deep Learning, Lip-reading, speech recognition systems, RNN, facial expressions
[This article belongs to Current Trends in Signal Processing(ctsp)]
Browse Figures
References
- Assael YM, Shillingford B, Whiteson S, De Freitas N. Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv:1611.01599. 2016 Nov 5.
- Wand M, Schmidhuber J. Improving speaker-independent lipreading with domain-adversarial training. arXiv preprint arXiv:1708.01565. 2017 Aug 4.
- Hong J, Kim M, Park SJ, Ro YM. Speech reconstruction with reminiscent sound via visual voice memory. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2021 Nov 17;29:3654-67.
- Stafylakis T, Tzimiropoulos G. Combining residual networks with LSTMs for lipreading. arXiv preprint arXiv:1703.04105. 2017 Mar 12.
- Chung JS, Zisserman A. Learning to lip read words by watching videos. Computer Vision and Image Understanding. 2018 Aug 1;173:76-85.
- Prajwal KR, Afouras T, Zisserman A. Sub-word level lip reading with visual attention. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition 2022 (pp. 5162-5172).
- Feng D, Yang S, Shan S, Chen X. Learn an effective lip-reading model without pains. arXiv preprint arXiv:2011.07557. 2020 Nov 15.
- Kanamaru T, Arakane T, Saitoh T. Isolated single sound lip-reading using a frame-based camera and event-based camera. Frontiers in Artificial Intelligence. 2023 Jan 11; 5:1070964.
- Kulkarni AH, Kirange D. Artificial intelligence: a survey on lip-reading techniques. In2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT) 2019 Jul 6 (pp. 1-5). IEEE.
- Fenghour S, Chen D, Guo K, Li B, Xiao P. Deep learning-based automated lip-reading: A survey. IEEE Access. 2021 Aug 25;9:121184-205.
- Yargıc A, Dogan M. A lip-reading application on MS Kinect camera. In2013 IEEE INISTA 2013 Jun 19 (pp. 1-5). IEEE.
- Fenghour S. Viseme-based Lip-Reading using Deep Learning (Doctoral dissertation, London South Bank University).
- Rahmani MH, Almasganj F. Lip-reading via a DNN-HMM hybrid system using combination of the image-based and model-based features. In2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA) 2017 Apr 19 (pp. 195-199). IEEE.
- Shin J, Lee J, Kim D. Real-time lip-reading system for isolated Korean word recognition. Pattern Recognition. 2011 Mar 1;44(3):559-71.
- Morade SS, Patnaik S. A novel lip-reading algorithm by using localized ACM and HMM: Tested for digit recognition. optik. 2014 Sep 1;125(18):5181-6.
- Dupont S, Luettin J. Audio-visual speech modeling for continuous speech recognition. IEEE transactions on multimedia. 2000 Sep;2(3):141-51.
Current Trends in Signal Processing
| Volume | 14 |
| Issue | 01 |
| Received | June 4, 2024 |
| Accepted | July 4, 2024 |
| Published | August 16, 2024 |