pipette
ENEnglish

AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning

Chuan Han Li, Yun Chien Cheng

Preprint

En palabras de los autores

This study develops an audio-visual emotion recognition system that integrates automatic utterance segmentation with multimodal fusion. Unlike conventional methods that rely on manually annotated utterance boundaries, the proposed system automatically generates candidate segments from continuous dialogue and extracts the corresponding audio and visual features for emotion classification. To address the challenges introduced by automatic segmentation, the proposed AGF-Net integrates Audio-Guided Fusion (AGF), Multimodal Negative Learning (MNL), and Predictive Dynamic Fusion (PDF) to improve cross-modal fusion, modality balance, and robustness to varying modality quality. Experimental results show that AGF-Net achieves 64.92% Accuracy and 65.47% F1-score under clean conditions, 62.13% Accuracy and 63.08% F1-score under moderate noise, and 58.19% Accuracy and 59.41% F1-score under higher noise. Overall, the proposed framework reduces reliance on manually annotated utterance boundaries and maintains competitive performance under automatically segmented and noisy audio-visual inputs.

Resultado principalEl resumen no menciona limitaciones.

Apareció: viernes, 23 de octubre. arXiv. Preprint, todavía sin revisión por pares.