Reliability-Aware Multimodal Sentiment Analysis Using Speech, Visual and Textual Signals: A CMU-MOSEI Benchmark Re-analysis

Notice

This is an unedited manuscript accepted for publication and provided as an Article in Press for early access at the author’s request. The article will undergo copyediting, typesetting, and galley proof review before final publication. Please be aware that errors may be identified during production that could affect the content. All legal disclaimers of the journal apply.

Year : 2026 | Volume : 17 | 03 | Page :
By

Purva Chandan Khatavkar,

Geetika Narang,

  1. U.G Student, Department of Computer Engineering, Trinity College of Engineering and Research, Maharashtra, India
  2. Assistant Professor, Department of Computer Engineering, Trinity College of Engineering and Research, Maharashtra, India

Abstract

Multimodal sentiment analysis integrates linguistic, acoustic, and visual cues, but reported gains often assume that every modality is present and equally reliable. This study develops a reliability-aware multimodal sentiment-analysis framework and performs a quantitative re-analysis of verified CMU- MOSEI benchmark results to determine how modality availability changes predictive performance. The conceptual architecture combines text, acoustic, and visual encoders with quality-conditioned gated fusion. For the quantitative component, complete- and missing-modality results for MulT, MISA, MAG-BERT, Self-MM, MMIM, and MissModal are harmonized under a common CMU- MOSEI evaluation source; relative F1 gains, MAE reduction, correlation change, and F1-retention are then recomputed. With complete modalities, binary F1 increases from 83.9% for MulT to 85.8% for MissModal, while MAE decreases from 0.564 to 0.533, corresponding to a 5.50% relative reduction. Modality analysis shows 84.6% F1 for text alone, compared with 48.5% for audio and 49.2% for vision, while complete fusion adds 1.2 percentage points over text-only inference. At 90% text missingness, F1 decreases to 54.0% (62.94% retention), whereas 90% acoustic and visual missingness retain 84.9% and 85.6% F1, respectively. These findings highlight the importance of considering modality reliability when designing multimodal systems. The analysis supports adaptive fusion strategies that account for missing or less informative inputs, rather than assigning equal importance to all modalities. The results also emphasize the need to evaluate model robustness under incomplete- modality conditions alongside standard complete-input performance.

Keywords: Multimodal sentiment analysis, CMU-MOSEI, reliability-aware fusion, missing modalities, text, speech, facial expressions, robustness

How to cite this article: Purva Chandan Khatavkar, Geetika Narang. Reliability-Aware Multimodal Sentiment Analysis Using Speech, Visual and Textual Signals: A CMU-MOSEI Benchmark Re-analysis. Journal of Electronic Design Technology. 2026; 17(03):-.
How to cite this URL: Purva Chandan Khatavkar, Geetika Narang. Reliability-Aware Multimodal Sentiment Analysis Using Speech, Visual and Textual Signals: A CMU-MOSEI Benchmark Re-analysis. Journal of Electronic Design Technology. 2026; 17(03):-. Available from: https://journals.stmjournals.com/joedt/article=2026/view=261092

References

  1. Baltrusaitis T, Ahuja C, Morency LP. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis & Machine Intelligence. 2019 Feb 1;41(02):423-43. doi: 10.1109/TPAMI.2018.2798607
  2. Zadeh A, Chen M, Poria S, Cambria E, Morency LP. Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP); 2017; Copenhagen, Denmark. p. 1103–14. doi: 10.18653/v1/D17-1115.
  3. Zadeh AB, Liang PP, Poria S, Cambria E, Morency LP. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) 2018 Jul (pp. 2236-2246). doi: 10.18653/v1/P18-1208
  4. Tsai YH, Bai S, Liang PP, Kolter JZ, Morency LP, Salakhutdinov R. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the 57th annual meeting of the association for computational linguistics 2019 Jul (pp. 6558-6569). doi: 10.18653/v1/P19-1656.
  5. Rahman W, Hasan MK, Lee S, Zadeh AB, Mao C, Morency LP, Hoque E. Integrating multimodal information in large pretrained transformers. InProceedings of the 58th annual meeting of the association for computational linguistics 2020 Jul (pp. 2359-2369). doi: 10.18653/v1/2020.acl- main.214.
  6. Hazarika D, Zimmermann R, Poria S. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. InProceedings of the 28th ACM international conference on multimedia 2020 Oct 12 (pp. 1122-1131). doi: 10.1145/3394171.3413678
  7. Yu W, Xu H, Yuan Z, Wu J. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. InProceedings of the AAAI conference on artificial intelligence 2021 May 18 (Vol. 35, No. 12, pp. 10790-10797).
  8. Han W, Chen H, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. InProceedings of the 2021 conference on empirical methods in natural language processing 2021 Nov (pp. 9180-9192). doi: 10.18653/v1/2021.emnlp-main.723.
  9. Lin R, Hu H. Missmodal: Increasing robustness to missing modality in multimodal sentiment analysis. Transactions of the Association for Computational Linguistics. 2023 Dec 19;11:1686- 702. doi: 10.1162/tacl_a_00628
  10. Guo Z, Jin T, Zhao Z. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) 2024 Aug (pp. 1726-1736). doi: 10.18653/v1/2024.acl-long.94
  11. Wang H, Luo S, Hu G, Zhang J. Gradient-guided modality decoupling for missing-modality robustness. InProceedings of the AAAI Conference on Artificial Intelligence 2024 Mar 24 (Vol. 38, No. 14, pp. 15483-15491). doi: 10.1609/aaai.v38i14.29474
  12. Li X, Cheng X, Miao D, Zhang X, Li Z. TF-Mamba: Text-enhanced Fusion Mamba with Missing Modalities for Robust Multimodal Sentiment Analysis. InEMNLP (Findings) 2025 May 20 (pp. 11252-11267). doi: 10.18653/v1/2025.findings-emnlp.602
  13. Zou S, Wei W, Zhang Y. DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities. InFindings of the Association for Computational Linguistics: ACL 2026 2026 Jul (pp. 30344-30361). doi: 10.18653/v1/2026.findings- acl.1517
Ahead of Print Subscription Original Research
Volume 17
03
Received 12/10/2026
Accepted 16/10/2026
Published 21/10/2026
Publication Time 9 Days

Login

My IP

PlumX Metrics

Support