Eye-Movement-Assisted Time-Frequency EEG Decoding for Multimodal Robotic Arm Control.

Overview

1. School of Electronics and Information, Changchun University, Changchun 130022, China; (X.S.); (W.Z.); (J.W.); (X.X.)
2. Key Laboratory of Intelligent Rehabilitation and Barrier-Free for the Disabled (Ministry of Education), Changchun University, Changchun 130022, China

Abstract

Brain–computer interface (BCI) technology has shown potential for future rehabilitation-related and assistive control applications. Nevertheless, single-modality electroencephalography-based motor imagery (EEG-MI) signals are susceptible to interference, whereas existing algorithmic models suffer from limited classification accuracy and insufficient actionable control commands for interactive devices, thereby impeding their practical deployment. To tackle these limitations, this study presents a multimodal human–computer interaction control scheme that integrates eye-movement command encoding with EEG motor imagery decoding. Self-collected EEG-MI and eye-movement datasets were established to support the proposed multimodal control framework. In this framework, eye movements are not used merely as auxiliary inputs, but are encoded as discrete commands for start, stop, grasp, and release, thereby reducing the command burden of EEG-MI decoding. The EEG-TransNet model is enhanced by integrating a time–frequency feature branch and replacing the original convolutional encoder with an adaptive multi-branch EEG feature gating module, strengthening the representation and fusion of multi-domain features. The model yields average classification accuracies of 86.96% and 88.73% on the BCI IV-2a dataset and the self-collected EEG dataset, respectively. Four independent SVM binary classifiers are adopted to identify four eye movement patterns. The EEG and eye movement classification results are binary-encoded to generate hardware-compatible control commands. Robotic-arm grasping experiments with healthy trained participants showed an average task completion time of 17 s, and the repeated grasping success-rate results further provide preliminary evidence for the real-time feasibility of the multimodal control framework under controlled laboratory conditions.

Reproduced under the paper's license (CC BY), from the paper cited above.

Code

The paper links to its data, not to its authors' code: see the Data section.

The paper's code and data availability statement is in the Data section.

Tracing map

A tracing map links a paper to the code its authors published: this paper has none, so it has no map.

Data

Datasets cited

• bbci.de/competition/iv, at bbci.de; found in the text, “1. Introduction”

Data Availability Statement

The public BCI IV-2a dataset used in this study is publicly available through public channels. The self-collected EEG-MI and eye-movement datasets are not publicly available because they contain private physiological and behavioral information from participants and are subject to ethical and privacy restrictions. De-identified data or controlled-access materials may be available from the corresponding author upon reasonable academic request, subject to institutional approval, ethical compliance, and a necessary data-use agreement. The complete source code is not publicly released at this stage because the model, robotic-arm control program, and data-processing pipeline are still being optimized and integrated. Key implementation details have been added to the revised manuscript to improve methodological transparency.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 5 authors, 4 keywords, 1 funder, 23 references.

Cite

This paper

Sun, X., Zhang, W., Wu, J., Xiong, X., & Mei, H. (2026). Eye-Movement-Assisted Time-Frequency EEG Decoding for Multimodal Robotic Arm Control. Journal of eye movement research, 19(4), 74. https://doi.org/10.3390/jemr19040074

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi: [code]

Self-Rectifying Integrate-and-Fire Neuron and Collaborative Trim Training Framework for SNN-Based EEG Motor Imagery Classification

Journal: Brain sciences

In common: bbci.de/competition/iv, EEG, 2 references

[2] doi:10.3390/bioengineering13050578

EEG-ShuffleFormer: A Multi-View Hybrid Network Integrating Time-Frequency and Raw Signal Representations for Few-Channel Motor Imagery EEG Classification.

Journal: Bioengineering (Basel, Switzerland)

In common: bbci.de/competition/iv, EEG, 2 references

[3] doi:10.1186/s40708-026-00321-1

EEG-DBNet: a dual-branch framework for temporal-spectral representation learning of motor imagery electroencephalography.

Journal: Brain informatics

In common: bbci.de/competition/iv, methods / tools, EEG, 1 reference

[4] doi:10.1038/s41598-026-47627-y [code]

QuantumNeuroXAI: a quantum-inspired deep learning framework with explainability for brain signal analysis and neurological disorder detection.

Journal: Scientific reports

In common: bbci.de/competition/iv, methods / tools, EEG, 1 reference

[5] doi:10.3390/s26051730 [code]

SFE-GAT: Structure-Feature Evolution Graph Attention Network for Motor Imagery Decoding.

Journal: Sensors (Basel, Switzerland)

In common: bbci.de/competition/iv, EEG, 1 reference

[6] doi:10.1038/s41746-026-02778-0 [code]

Trust-gated synthetic EEG augmentation reduces performance drops when generalizing to new patients.

Journal: NPJ digital medicine

In common: bbci.de/competition/iv, EEG, 1 reference

[7] doi:10.3390/biomimetics11060377

PG-MCTFormer: A Prior-Guided Multi-Scale Convolutional Transformer for Interpretable Motor Imagery EEG Classification.

Journal: Biomimetics (Basel, Switzerland)

In common: bbci.de/competition/iv, EEG, 1 reference

[8] doi:10.3389/fnhum.2026.1755549

Dynamic graph based attention spectral network for motor imagery-brain computer interface.

Journal: Frontiers in human neuroscience

In common: bbci.de/competition/iv, EEG, 1 reference

[9] doi:10.1371/journal.pone.0354976 [code]

Improved motor imagery BCI performance via task-unaware compression in the BELT Bayesian Edge-Cloud architecture.

Journal: PloS one

In common: bbci.de/competition/iv, methods / tools, EEG

[10] doi:10.3390/s26134045

Brain Signal for Secure EEG Biometric Authentication: A Comprehensive Survey.

Journal: Sensors (Basel, Switzerland)

In common: bbci.de/competition/iv, methods / tools, EEG

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.