Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention.

Overview

1. School of Information Engineering, Dalian University, Dalian 116622, China
2. Communication and Network Laboratory, Dalian University, Dalian 116622, China

Abstract

Highlights: What are the main findings?

We propose a Multi-Scale Dilated Attention Convolution network specifically designed for facial expression; it efficiently captures both local micro-expression details and global semantic features, leading to substantially improved facial emotion recognition performance.

We introduce a self-learning weight fusion mechanism for decision-level fusion of EEG and facial modalities; compared with fixed or simple weighted fusion, this method adaptively selects the optimal fusion weighting and yields superior multimodal classification results.

What is the implication of the main finding?

The proposed method was separately evaluated on the FER2013, CK+, and DEAP datasets, where it demonstrated superior performance over existing approaches in facial single modal, EEG single modal, and multimodal fusion settings.

Abstract: Background/Objectives: Emotions can be recognized through external behavioral cues and internal physiological signals. Owing to the inherently complex psychological and physiological nature of emotions, models relying on a single modality often suffer from limited robustness. This study aims to improve emotion recognition performance by effectively integrating electroencephalogram (EEG) signals and facial expressions through a multimodal framework. Methods: We propose a multimodal emotion recognition model that employs a Multi-Scale Dilated Attention Convolution (MSDAC) network tailored for facial expression recognition, integrates an EEG emotion recognition method based on three-dimensional features, and adopts a self-learning decision-level fusion strategy. MSDAC incorporates Multi-Scale Dilated Convolutions and a Dual-Branch Attention (D-BA) module to capture discontinuous facial action units. For EEG processing, raw signals are converted into a multidimensional time–frequency–spatial representation to preserve temporal, spectral, and spatial information. To overcome the limitations of traditional stitching or fixed-weight fusion approaches, a self-learning weight fusion mechanism is introduced at the decision level to adaptively adjust modality contributions. Results: The facial analysis branch achieved average accuracies of 74.1% on FER2013, 99.69% on CK+, and 98.05% (valence)/96.15% (arousal) on DEAP. On the DEAP dataset, the complete multimodal model reached 98.66% accuracy for valence and 97.49% for arousal classification. Conclusions: The proposed framework enhances emotion recognition by improving facial feature extraction and enabling adaptive multimodal fusion, demonstrating the effectiveness of combining EEG and facial information for robust emotion analysis.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above.

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

• 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
• 0 scripts, each with its path and the digest of its content;
• no match between paragraphs and code yet;
• neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data Availability Statement

The original code and data presented in the study are openly available at https://github.com/yy11z/SL_MER_MSDA (accessed on 13 February 2026).

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 30 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 2 authors, 8 keywords, 1 funder, 31 references.

Cite

This paper

Du, X., & Zhu, L. (2026). Self-Learning Multimodal Emotion Recognition Based on Multi-Scale Dilated Attention. Brain sciences, 16(4), 350. https://doi.org/10.3390/brainsci16040350

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.12688/f1000research.171170.2

A Review of Deep Learning Techniques for EEG-Based Emotion Recognition: Models, Methods, and Datasets.

Journal: F1000Research

In common: EEG, cognitive, 5 references

[2] doi:10.7717/peerj-cs.1977

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition.

Journal: PeerJ. Computer science

In common: EEG, cognitive, 4 references

[3] doi:10.3390/brainsci14040367

Electroencephalographic Signal Data Augmentation Based on Improved Generative Adversarial Network.

Journal: Brain sciences

In common: EEG, 1 reference, author Xiuli Du

[4] doi:10.3390/s25072029

MSDSANet: Multimodal Emotion Recognition Based on Multi-Stream Network and Dual-Scale Attention Network Feature Representation.

Journal: Sensors (Basel, Switzerland)

In common: EEG, cognitive, 3 references

[5] doi:10.3390/brainsci14040375

A Data Augmentation Method for Motor Imagery EEG Signals Based on DCGAN-GP Network.

Journal: Brain sciences

In common: EEG, author Xiuli Du

[6] doi:10.7717/peerj-cs.2649 [code]

Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning.

Journal: PeerJ. Computer science

In common: EEG, cognitive, 3 references

[7] doi:10.1007/s11571-023-10004-w [code]

EEG-based emotion recognition using a temporal-difference minimizing neural network.

Journal: Cognitive neurodynamics

In common: EEG, cognitive, 3 references

[8] doi:10.3390/e24121830 [code]

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

Journal: Entropy (Basel, Switzerland)

In common: EEG, cognitive, 3 references

[9] doi:10.3390/brainsci16020145 [code]

EEG-fNIRS Cross-Subject Emotion Recognition Based on Attention Graph Isomorphism Network and Contrastive Learning.

Journal: Brain sciences

In common: EEG, cognitive, 2 references

[10] doi:10.3390/s26020582

SAWGAN-BDCMA: A Self-Attention Wasserstein GAN and Bidirectional Cross-Modal Attention Framework for Multimodal Emotion Recognition.

Journal: Sensors (Basel, Switzerland)

In common: EEG, cognitive, 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.3390/brainsci16040350.svg)](https://openscicode.org/badge/doi/10.3390/brainsci16040350)
<!-- oscr-badge:end -->

.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.3390/brainsci16040350.svg
   :target: https://openscicode.org/badge/doi/10.3390/brainsci16040350
   :alt: OSCR

.. oscr-badge:end

<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.3390/brainsci16040350"><img src="https://openscicode.org/badge/doi/10.3390/brainsci16040350.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.