Multi-modal speech emotion recognition: Improving accuracy through fusion of vggish and bert features with multi-head attention
Published in International Conference on Industrial Networks and Intelligent Systems, 2023
Abstract (excerpt). Recent research has shown that multi-modal learning is a successful method for enhancing classification performance by mixing several forms of input, notably in speech-emotion recognition (SER) tasks. However, the difference between the modalities may affect SER performance. To overcome this problem, a novel approach for multi-modal SER called 3M-SER is proposed in this paper. The 3M-SER leverages multi-head attention to fuse information from multiple feature embeddings, including audio and text features. The 3M-SER approach is based on the SERVER approach but includes an additional fusion module that improves the integration of text and audio features, leading to improved classification performance. To further enhance the correlation between the modalities, a LayerNorm is applied to audio features prior to fusion. Our approach achieved an unweighted accuracy (UA) and weighted accuracy …
Recommended citation: Phuong-Nam Tran, Thuy-Duong Thi Vu, Duc Ngoc Minh Dang, Nhat Truong Pham, Anh-Khoa Tran. (2023). "Multi-modal speech emotion recognition: Improving accuracy through fusion of vggish and bert features with multi-head attention." International Conference on Industrial Networks and Intelligent Systems, 148-158.
Download Paper