An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples

Year : 2024 | Volume : 11 | Issue : 03 | Page : 19 27
    By

    Devang Mulye,

  • Parakh Pawar,

  • Aditya Yadav,

  • S. Poornima,

  1. Student, Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai, Maharashtra, India
  2. Student, Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai, Maharashtra, India
  3. Student, Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai, Maharashtra, India
  4. Associate Professor, Department of Information Technology, SIES Graduate School of Technology, Navi Mumbai, Maharashtra, India

Abstract

In today’s rapidly evolving digital landscape, deepfake technology stands as both a marvel and a threat to privacy and security. Deepfakes, hyper-realistic synthetic media created using artificial intelligence (AI), can deceive and manipulate on an unprecedented scale, from political propaganda to compromising videos of public figures. This research navigates deepfake detection, focusing on two advanced methodologies: the vision transformers (ViT) image classifier and the Meso4 method. The ViT model utilizes convolutional neural networks (CNNs) for image analysis, while the Meso4 method examines images at a mesoscopic level. These methodologies are evaluated for their effectiveness in differentiating genuine content from altered media. Motivated by the threats posed by deepfakes to individuals and society, this project evaluates the ViT model using the Open Forensics dataset and the Meso4 model using a combination of self-generated content and the Face2Face dataset. The ViT model achieved 99.9% accuracy within its dataset and 85% accuracy for subtly deceptive images. Conversely, the Meso4 model showed a 60% accuracy and biases, particularly towards female subjects, likely due to its training on adult content. To address privacy and security concerns, five additional features are proposed: eye movement error detection, lip sync inconsistency, facial expression analysis, facial texture inconsistency, and background inconsistency. These features enhance detection accuracy and reliability and are integrated into a customized model, augmenting the ViT model’s strengths and addressing its limitations. The proposed model improves accuracy and computation speed, making it practical for real-world applications. This research highlights the urgency of advancing robust and unbiased deepfake detection methods to safeguard privacy and security in an era where truth can be easily manipulated.

Keywords: Deepfake detection, Meso4, ViT, eye movement, skin texture, facial expression, lip sync

[This article belongs to Journal of Image Processing & Pattern Recognition Progress ]

How to cite this article:
Devang Mulye, Parakh Pawar, Aditya Yadav, S. Poornima. An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples. Journal of Image Processing & Pattern Recognition Progress. 2024; 11(03):19-27.
How to cite this URL:
Devang Mulye, Parakh Pawar, Aditya Yadav, S. Poornima. An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples. Journal of Image Processing & Pattern Recognition Progress. 2024; 11(03):19-27. Available from: https://journals.stmjournals.com/joipprp/article=2024/view=171885


References

  1. Taeb M, Chi H. Comparison of deepfake detection techniques through deep learning. J Cybersecur Priv. 2022;2:89–106. DOI: 10.3390/jcp2010007.
  2. Ramachandran S, Nadimpalli AV, Rattani A. An experimental evaluation on deepfake detection using deep face recognition. 2021 International Carnahan Conference on Security Technology (ICCST), Hatfield, United Kingdom, 2021, pp. 1–6. DOI: 10.1109/ICCST49569.2021.9717407.
  3. Lyu S. Deepfake detection: Current challenges and next steps. IEEE Int Conf Multimed Expo Workshops (ICMEW). IEEE Publications; 2020. p. 1–6. DOI: 10.1109/ICMEW46912.2020.9105
  4. Khichi M. Deepfake or real image prediction using Mesonet [dissertation]. Delhi Technological University. 2021. Available from: http://dspace.dtu.ac.in:8080/jspui/handle/repository/18952.
  5. Liang J, Wang D, Ling X. Image classification for soybean and weeds based on ViT. J Phys Conf Ser. 2021;2002:012068. DOI: 10.1088/1742-6596/2002/1/012068.
  6. Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer. [Preprint]. ArXiv:2102.11126. 2021.
  7. Pan D, Sun L, Wang R, Zhang X, Sinnott RO. Deepfake detection through deep learning. 2020 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT), Leicester, UK, 2020. pp. 134–43. DOI: 10.1109/BDCAT50828.2020.00001.
  8. Aghasanli A, Kangin D, Angelov P. Interpretable-through-prototypes deepfake detection for diffusion models. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Paris, France, 2023, pp. 467–74. DOI: 10.1109/ICCVW60793.2023.00053.
  9. Turan SG. Deepfake and digital citizenship: A long-term protection method for children and youth. In: Deep Fakes, Fake News, and Misinformation in Online Teaching and Learning Technologies. IGI Global; 2021. p. 124–42. DOI: 10.4018/978-1-7998-6474-5.ch006.
  10. Poornima S, Subramanian S. Unconstrained iris authentication through fusion of RGB channel information. Int J Pattern Recognit Artif Intell. 2014;28:1456010. DOI: 10.1142/S0218001414
  11. Poornima S, Subramanian S. An efficient feature level fusion for a multimodal biometric system using correlation filter. Int J Appl Eng Res. 2014;9:8975–8.
  12. Poornima S, Thaya IM. Comparison analysis of feature level fusion in multimodal biometrics system. Int J Appl Eng Res. 2014;9:8908-8912.
  13. Poornima, S., Fathima Nadheen, M. Feature level fusion in multimodal biometric authentication system. Int J Comput Appl. 2013;69(18):36-40.
  14. Poornima S. Performance study of fusion in multimodal biometric verification using ear and iris features. In: Proceedings of the International Conference on Research Trends in Computer Technologies (ICRTCT); Feb 2013. ICRTCT, 4:133-136.
  15. Poornima S, Rajavelu C, Subramanian S. Comparison and a neural network approach for iris localization. Procedia Comput Sci. 2010;2:127–32. DOI: 10.1016/j.procs.2010.11.016.
  16. George AS, George AH. Deepfakes: The evolution of hyper-realistic media manipulation. Partners Univ Innov Res. 2023;1:58–74.
  17. Westerlund M. The emergence of deepfake technology: A review. Technol Innov Manag Rev. 2019;9:39–52. DOI: 10.22215/timreview/1282.
  18. Shahzad HF, Rustam F, Flores ES, Luís Vidal Mazón J, de la Torre Diez I, Ashraf I. A review of image processing techniques for deepfakes. Sensors. 2022;22:4556. DOI: 10.3390/s22124556.
  19. Wang Z, Guo Y, Zuo W. Deepfake forensics via an adversarial game. IEEE Trans Image Process. 2022;31:3541–52. DOI: 10.1109/TIP.2022.3172845.
  20. Silva SH, Bethany M, Votto AM, Scarff IH, Beebe N, Najafirad P. Deepfake forensics analysis: An explainable hierarchical ensemble of weakly supervised models. Forensic Sci Int Synergy. 2022;4:100217. DOI: 10.1016/j.fsisyn.2022.100217.
  21. Pashine S, Mandiya S, Gupta P, Sheikh R. Deep fake detection: Survey of facial manipulation detection solutions. [Preprint] ArXiv:2106.12605. 2021.
  22. Joseph Z, Nyirenda C. Deepfake detection using a two-stream capsule network. 2021 IST-Africa Conference (IST-Africa), South Africa, South Africa, 2021, pp. 1–8.
  23. Heo YJ, Yeo WH, Kim BG. DeepFake detection algorithm based on improved vision transformer. Appl Intell. 2023;53:7512–27. DOI:10.1007/s10489-022-03867-9.
  24. Afchar D, Nozick V, Yamagishi J, Echizen I. Mesonet: A compact facial video forgery detection network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, China. 2018, pp. 1–7. DOI: 10.1109/WIFS.2018.8630761.
  25. Boesch G. (2023). Vision Transformers (ViT) in Image Recognition – 2024 Guide. [online] viso.ai. Available from: https://viso.ai/deep-learning/vision-transformer-vit/
  26. Ahmed OB, Asghar AH, Bahwerth FS, Assaggaf HM, Bamaga MA. [RETRACTED ARTICLE]: The prevalence of aminoglycoside-resistant genes in Gram-negative bacteria in tertiary hospitals. Appl Nanosci. 2023;13:1093–3. DOI: 10.1007/s13204-021-01887-4.
  27. Das S, Seferbekov S, Datta A, Islam MS, Amin MR. Towards solving the deepfake problem: An analysis on improving deepfake detection using dynamic face augmentation. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, 2021, pp. 3769–78. DOI: 10.1109/ICCVW54120.2021.00421.
  28. Coccomini DA, Messina N, Gennaro C, Falchi F. Combining EfficientNet and vision transformers for video deepfake detection. In: Sclaroff S, Distante C, Leo M, Farinella GM, Tombari F, editors. Image Analysis and Processing – ICIAP 2022. Cham (Germany): Springer; 2022. p. 239–51. DOI: 10.1007/978-3-031-06433-3_19.
Regular Issue Subscription Review Article
Volume 11
Issue 03
Received 27/06/2024
Accepted 05/08/2024
Published 12/09/2024
593

Login


My IP

PlumX Metrics