An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples

Year : 2024 | Volume :11 | Issue : 03 | Page : –
By

Devang Mulye,

Parakh Pawar,

Aditya Yadav,

S. Poornima,

  1. Student, Department of Information Technology, SIES Graduate School of Technology, Maharashtra, India
  2. Student, Department of Information Technology, SIES Graduate School of Technology, Maharashtra, India
  3. Student, Department of Information Technology, SIES Graduate School of Technology, Maharashtra, India
  4. Associate Professor, Department of Information Technology, SIES Graduate School of Technology, Maharashtra, India

Abstract

‘]

In today’s rapidly evolving digital landscape, deepfake technology stands as both a marvel and a threat to privacy and security. Deepfakes, hyper-realistic synthetic media created using artificial intelligence (AI), can deceive and manipulate on an unprecedented scale, from political propaganda to compromising videos of public figures. This research navigates deepfake detection, focusing on two advanced methodologies: the VIT image classifier and the Meso4 method. The VIT model utilizes convolutional neural networks (CNNs) for image analysis, while the Meso4 method examines images at a mesoscopic level. These methodologies are evaluated for their effectiveness in differentiating genuine content from altered media. Motivated by the threats posed by deepfakes to individuals and society, this project evaluates the VIT model using the Open Forensics dataset and the Meso4 model using a combination of self-generated content and the Face2Face dataset. The VIT model achieved 99.9% accuracy within its dataset and 85% accuracy for subtly deceptive images. Conversely, the Meso4 model showed a 60% accuracy and biases, particularly towards female subjects, likely due to its training on adult content. To address privacy and security concerns, five additional features are proposed: Eye Movement Error Detection, Lip Sync Inconsistency, Facial Expression Analysis, Facial Texture Inconsistency, and Background Inconsistency. These features enhance detection accuracy and reliability and are integrated into a customized model, augmenting the VIT model’s strengths and addressing its limitations. The proposed model improves accuracy and computation speed, making it practical for real-world applications. This research highlights the urgency of advancing robust and unbiased deepfake detection methods to safeguard privacy and security in an era where truth can be easily manipulated.

Keywords: Deepfake detection, Meso4, ViT, Eye Movement, Skin Texture, Facial Expression, Lip Sync

[This article belongs to Journal of Image Processing & Pattern Recognition Progress (joipprp)]

How to cite this article:
Devang Mulye, Parakh Pawar, Aditya Yadav, S. Poornima. An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples. Journal of Image Processing & Pattern Recognition Progress. 2024; 11(03):-.
How to cite this URL:
Devang Mulye, Parakh Pawar, Aditya Yadav, S. Poornima. An Analysis of Multimodal Fusion in Deepfake Detection for Video Samples. Journal of Image Processing & Pattern Recognition Progress. 2024; 11(03):-. Available from: https://journals.stmjournals.com/joipprp/article=2024/view=171885



Fetching IP address…

References ‘]

  1. Taeb, Maryam, and Hongmei Chi. “Comparison of deepfake detection techniques through deep learning.” Journal of Cybersecurity and Privacy 2.1 (2022): 89- 106.
  2. Lyu S. Deepfake detection: Current challenges and next steps. In2020 IEEE international conference on multimedia & expo workshops (ICMEW) 2020 Jul 6 (pp. 1-6). IEEE.
  3. Khichi, Manish. “Deepfake or Real Image Prediction  using Mesonet.” PhD diss., 2021. Available from: https://www.grafiati.com/en/literature-selections/deepfake/dissertation/
  4. Liang, J., Wang, D., & Ling, X. (2021, August). Image classification for soybean and weeds based on VIT. In Journal of Physics: Conference Series (Vol. 2002, No. 1, p. 012068). IOP Publishing.
  5. Westerlund M. The emergence of deepfake technology: A review. Technology innovation management review. 2019;9(11).
  6. George, A. S., & George, A. H. (2023). Deepfakes: The Evolution of Hyper realistic Media Manipulation. Partners Universal Innovative Research Publication, 1(2), 58-74.
  7. Turan, S. G. (2021). Deepfake and digital citizenship: A long-term protection method for children and youth. In Deep fakes, fake news, and misinformation in online teaching and learning technologies (pp. 124-142). IGI Global.
  8. Shahzad, H. F., Rustam, F., Flores, E. S., Luís Vidal Mazón, J., de la Torre Diez, I., & Ashraf, I. (2022). A Review of Image Processing Techniques for Deepfakes. Sensors, 22(12), 4556.
  9. Silva, S. H., Bethany, M., Votto, A. M., Scarff, I. H., Beebe, N., & Najafirad, P. (2022). Deepfake forensics analysis: An explainable hierarchical ensemble of weakly supervised models. Forensic Science International: Synergy, 4, 100217.
  10. Wang, Z., Guo, Y., & Zuo, W.(2022). Deepfake forensics via an adversarial game. IEEE Transactions on Image Processing, 31, 3541-3552.
  11. Joseph, Z., & Nyirenda, C. (2021, May). Deepfake detection using a twostream capsule network. In 2021 ISTAfrica Conference (IST-Africa) (pp. 18). IEEE.
  12. Wodajo, D., & Atnafu, S. (2021). Deepfake video detection using convolutional vision transformer. arXiv preprint arXiv:2102.11126.
  13. Afchar D, Nozick V, Yamagishi J, Echizen I. Mesonet: a compact facial video forgery detection network. In2018 IEEE international workshop on information forensics and security (WIFS) 2018 Dec 11 (pp. 1-7). IEEE.
  14. Boesch G. Vision Transformers (ViT) in Image Recognition – 2024 Guide – viso.ai . viso.ai. 2023 . Available from: https://viso.ai/deep-learning/vision-transformer-vit/ ‌
  15. Coccomini, D.A., Messina, N., Gennaro, C., Falchi, F. (2022). Combining EfficientNet and Vision Transformers for Video Deepfake Detection. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13233. Springer, Cham. https://doi.org/10.1007/978-3-031-06433-3_19
  16. Pashine, S., Mandiya, S., Gupta, P. and Sheikh, R., 2021. Deep fake detection: Survey of facial manipulation detection solutions. arXiv preprint arXiv:2106.12605.
  17. Pan, L. Sun, R. Wang, X. Zhang and R. O. Sinnott, “Deepfake Detection through Deep Learning,” 2020 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT), Leicester, UK, 2020, pp. 134-143, doi: 10.1109/BDCAT50828.2020.00001.
  18. Ahmed, S.R.A. and Sonuç, E., 2023. Deepfake detection using rationale-augmented convolutional neural network. Applied Nanoscience, 13(2).
  19. Poornima and S. Subramanian, Unconstrained iris authentication through fusion of RGB channel information, Int. J. Patt. Recogn. Artif. Intell. Vol.28, No.5,1456010 (August 2014) [18 pages] DOI: 10.1142/S0218001414560102.
  20. S, S. Subramanian, An Efficient Feature Level Fusion for a Multimodal Biometric System using Correlation Filter, Int. Journal Applied Engg. Research, Vol 9, No. 26, 2014, pp. 8975-8978.
  21. S, Thaya.I.M, Comparison Analysis of Feature Level Fusion in Multimodal Biometrics System, Int. Journal Applied Engg. Research, Vol 9, No. 26, 2014, 8908-8912.
  22. Fathima Nadheen, Poornima.S, Feature Level Fusion in Multimodal Biometric Authentication System, International Journal of Computer Applications (0975 – 8887), May 2013, Volume 69– No.18, 36-40.
  23. S, Performance Study of Fusion in Multimodal Biometric Verification using Ear and Iris Features, Proceedings published in International Journal of Computer Applications® (IJCA) (0975 – 8887), Feb 2013, ICRTCT(4), 13-16.
  24. S, Rajavelu, S. Subramanian, Comparison and a neural network approach for iris localization, Elsevier Procedia Computer Science, ISSN: 1877-0509, DOI:10.1016/j.procs.2010.11.016, 2010, 2, 127-132.
  25. Das, S., Seferbekov, S., Datta, A., Islam, M.S. and Amin, M.R., 2021. Towards solving the deepfake problem: An analysis on improving deepfake detection using dynamic face augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3776-3785).
  26. Aghasanli, A., Kangin, D. and Angelov, P., 2023. Interpretable-through-prototypes deepfake detection for diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 467-474).
  27. Ramachandran, A. V. Nadimpalli and A. Rattani, “An Experimental Evaluation on Deepfake Detection using Deep Face Recognition,” 2021 International Carnahan Conference on Security Technology (ICCST), Hatfield, United Kingdom, 2021, pp. 1-6, doi: 10.1109/ICCST49569.2021.9717407.
  28. Heo, YJ., Yeo, WH. & Kim, BG. DeepFake detection algorithm based on improved vision transformer. Appl Intell 53, 7512–7527 (2023). https://doi.org/10.1007/s10489-022-03867-9
Regular Issue Subscription Review Article
Volume 11
Issue 03
Received June 27, 2024
Accepted August 5, 2024
Published September 12, 2024
Views: 35

Check Our other Platform for Workshops in the field of AI, Biotechnology & Nanotechnology.
Learn more
Check Out Platform for Webinars in the field of AI, Biotech. & Nanotech.
Learn more