Enhancing multimodal emotion recognition with dynamic fuzzy membership and attention fusion

Abstract (excerpt). Multimodal learning has been shown to enhance classification outcomes in speech emotion recognition (SER). Despite this advantage, multimodal approaches in SER often face key challenges, including limited robustness to uncertainty, difficulty generalizing across diverse emotional contexts, and inefficiencies in integrating heterogeneous modalities. To overcome these constraints, we propose a multimodal emotion recognition architecture, named FleSER, which leverages dynamic fuzzy membership and attention-based fusion. Unlike most previous SER studies that apply fuzzy logic at the decision level, FleSER introduces a feature-level, rule-based dynamic fuzzy membership mechanism that adaptively refines modality representations prior to fusion. The FleSER architecture leverages audio and textual modalities, employing self-modality and cross-modality attention mechanisms with the α interpolation to …

Read the full paper

Recommended citation: Nhut Minh Nguyen, Trung Minh Nguyen, Thanh Trung Nguyen, Phuong-Nam Tran, Nhat Truong Pham, Linh Le, Alice Othmani, Abdulmotaleb El Saddik, Duc Ngoc Minh Dang. (2026). "Enhancing multimodal emotion recognition with dynamic fuzzy membership and attention fusion." Engineering Applications of Artificial Intelligence, 165, 113396.
Download Paper