Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR.

Overview

1. The First Clinical Medical College, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
2. Fujian Medical University, Fuzhou, China
3. Department of Medical Imaging, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China

Abstract

Objective: This study investigates optimal training strategies for YOLOv8s in brain tumor MRI detection, systematically evaluates the effects of key hyperparameters on model performance, and compares the adaptability and trade-offs between convolution-based dense prediction and Transformer-based sparse query mechanisms in medical image detection. Methods: Experiments were conducted on publicly available Kaggle MRI datasets of meningioma and glioma. YOLOv8s was adopted as the baseline model, and a systematic hyperparameter search was performed over learning rates (0.01, 0.001, and 0.0001) and bounding box regression loss weights (5.0, 7.5, and 10.0), forming nine configurations evaluated via five-fold cross-validation. Based on the optimal setting (learning rate = 0.01, loss weight = 7.5), the original YOLOv8s detection head was replaced with an RT-DETR Transformer-based decoder head. Performance and generalization were assessed on both internal and external test sets, and Score-CAM was employed for model interpretability analysis. Results: The hyperparameter analysis demonstrated that the learning rate was the primary factor affecting detection performance, whereas the influence of the bounding box regression loss weight was relatively limited within a reasonable range. Using the optimal hyperparameter combination (learning rate = 0.01 and box loss weight = 7.5), the native YOLOv8s detection head achieved the best overall performance, with a precision of 91.7%, a recall of 89.2%, and of 92.6%, outperforming the RT-DETR detection head ( = 91.1%). On the external test set, YOLOv8s also demonstrated superior cross-domain generalization ( = 79.0%), exceeding RT-DETR by 10.1%. Further NMS IoU sensitivity analysis showed that the RT-DETR detection head maintained more stable performance across different NMS IoU thresholds, whereas YOLOv8s was more sensitive to variations in NMS post-processing. Score-CAM visualization further revealed distinct attention patterns, with YOLOv8s exhibiting broader lesion responses and RT-DETR focusing more selectively on lesion regions. Conclusion: Appropriate hyperparameter optimization can substantially improve the performance of brain tumor MRI detection models. The dense prediction mechanism of YOLOv8s provides superior lesion coverage, resulting in better detection performance and cross-domain generalization, whereas the sparse query mechanism of RT-DETR exhibits greater robustness to NMS parameter variations during inference. These findings highlight the complementary characteristics of the two detection paradigms and provide practical guidance for detection head selection and inference strategy optimization in medical object detection

Reproduced under the paper's license (CC BY), from the paper cited above.

Code

The paper links to its data, not to its authors' code: see the Data section.

Tracing map

A tracing map links a paper to the code its authors published: this paper has none, so it has no map.

Data

Datasets cited

• kaggle.com/datasets/masoudnickparvar, at Kaggle; found in “Data availability statement”
• kaggle.com/datasets/sartajbhuvaji, at Kaggle; found in the references

Data availability statement

Publicly available datasets were analyzed in this study. This data can be found here: https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset~https://www.kaggle.com/datasets/sartajbhuvaji/brain-tumor-classification-mri.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 6 authors, 8 keywords, 33 references.

Cite

This paper

Xie, L., Chen, J., Sun, Z., Huang, Y., Ye, S., & Ye, C. (2026). Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR. Frontiers in medicine, 13, 1905061. https://doi.org/10.3389/fmed.2026.1905061

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1186/s40708-026-00325-x

CAE-BrainNet: a statistically validated class-adaptive attention ensemble model for explainable brain tumor classification from MRI.

Journal: Brain informatics

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

[2] doi:10.3390/diagnostics16111745

Brain Tumor Classification and Segmentation in MR Images Using EfficientNet and U-Net++ Models.

Journal: Diagnostics (Basel, Switzerland)

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

[3] doi:10.3390/jimaging12060233 [code]

Brain Tumor Classification in MRI Images Using Combined Transfer Learning and Convolutional Neural Networks.

Journal: Journal of imaging

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

[4] doi:10.1038/s41598-026-51236-0

A minimal-net CNN model for an IoT-based brain tumor detection and monitoring system.

Journal: Scientific reports

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

[5] doi:10.3390/brainsci16050468

Advancing Brain Tumor Diagnosis Using Deep Learning: A Systematic and Critical Review on Methodological Approaches to Glioma Segmentation and Classification Through Multiparametric MRI.

Journal: Brain sciences

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

[6] doi:

NeuroTrustNet: a cost-effective multimodal ensemble framework for brain tumor classification under cross-dataset variability

Journal: Frontiers in artificial intelligence

In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, other condition

[7] doi:10.3389/frai.2026.1849051

Query-guided learning for efficient and interpretable multi-class brain tumor classification in MRI.

Journal: Frontiers in artificial intelligence

In common: kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 2 references

[8] doi:

Toward reliable computer-aided brain tumor diagnosis: a contrast-enhanced deep learning approach with hybrid KNN classification

Journal: Frontiers in human neuroscience

In common: kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 2 references

[9] doi:10.1002/acm2.70560 [code]

Hybrid CNN-GCN framework for brain tumor MRI classification: A graph-based approach to smart healthcare diagnostics.

Journal: Journal of applied clinical medical physics

In common: kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 2 references

[10] doi:10.1371/journal.pone.0352353

Deep convolutional GAN and hypernet-based neural architecture search for brain tumor diagnosis detection and classification.

Journal: PloS one

In common: kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.