Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR.
Overview
- The First Clinical Medical College, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
- Fujian Medical University, Fuzhou, China
- Department of Medical Imaging, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
Abstract
Objective: This study investigates optimal training strategies for YOLOv8s in brain tumor MRI detection, systematically evaluates the effects of key hyperparameters on model performance, and compares the adaptability and trade-offs between convolution-based dense prediction and Transformer-based sparse query mechanisms in medical image detection. Methods: Experiments were conducted on publicly available Kaggle MRI datasets of meningioma and glioma. YOLOv8s was adopted as the baseline model, and a systematic hyperparameter search was performed over learning rates (0.01, 0.001, and 0.0001) and bounding box regression loss weights (5.0, 7.5, and 10.0), forming nine configurations evaluated via five-fold cross-validation. Based on the optimal setting (learning rate = 0.01, loss weight = 7.5), the original YOLOv8s detection head was replaced with an RT-DETR Transformer-based decoder head. Performance and generalization were assessed on both internal and external test sets, and Score-CAM was employed for model interpretability analysis. Results: The hyperparameter analysis demonstrated that the learning rate was the primary factor affecting detection performance, whereas the influence of the bounding box regression loss weight was relatively limited within a reasonable range. Using the optimal hyperparameter combination (learning rate = 0.01 and box loss weight = 7.5), the native YOLOv8s detection head achieved the best overall performance, with a precision of 91.7%, a recall of 89.2%, and of 92.6%, outperforming the RT-DETR detection head ( = 91.1%). On the external test set, YOLOv8s also demonstrated superior cross-domain generalization ( = 79.0%), exceeding RT-DETR by 10.1%. Further NMS IoU sensitivity analysis showed that the RT-DETR detection head maintained more stable performance across different NMS IoU thresholds, whereas YOLOv8s was more sensitive to variations in NMS post-processing. Score-CAM visualization further revealed distinct attention patterns, with YOLOv8s exhibiting broader lesion responses and RT-DETR focusing more selectively on lesion regions. Conclusion: Appropriate hyperparameter optimization can substantially improve the performance of brain tumor MRI detection models. The dense prediction mechanism of YOLOv8s provides superior lesion coverage, resulting in better detection performance and cross-domain generalization, whereas the sparse query mechanism of RT-DETR exhibits greater robustness to NMS parameter variations during inference. These findings highlight the complementary characteristics of the two detection paradigms and provide practical guidance for detection head selection and inference strategy optimization in medical object detection
Reproduced under the paper's license (CC BY), from the paper cited above.
Code
The paper links to its data and to no code of its authors: see the Data section.
Tracing map
A tracing map links a paper to the code its authors published: this paper has none, so it has no map.
Data
Datasets cited
- kaggle.com/
datasets/ , at Kaggle; found in “Data availability statement”masoudnickparvar - kaggle.com/
datasets/ , at Kaggle; found in the referencessartajbhuvaji
Data availability statement
Publicly available datasets were analyzed in this study. This data can be found here: https://
Reproduced under the paper's license (CC BY), from the paper cited above.
Versions
The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.
Version 1, 27 September 2026: the first record
Recorded: type, language, journal, volume, pages, dates, 6 authors, 8 keywords, 33 references.
Cite
This paper
Xie, L., Chen, J., Sun, Z., Huang, Y., Ye, S., & Ye, C. (2026). Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR. Frontiers in medicine, 13, 1905061. https://
BibTeX
@article{xie2026performa
author = {Xie, Lianxin and Chen, Jinghui and Sun, Zhipeng and Huang, Yuhan and Ye, Shan and Ye, Chengbin},
title = {{Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR}},
journal = {Frontiers in medicine},
year = {2026},
month = aug,
volume = {13},
pages = {1905061},
publisher = {Frontiers Media SA},
issn = {2296-858X},
doi = {10.3389/
url = {https://
pmid = {42741587},
pmcid = {PMC13572972}
}
RIS
TY - JOUR
AU - Xie, Lianxin
AU - Chen, Jinghui
AU - Sun, Zhipeng
AU - Huang, Yuhan
AU - Ye, Shan
AU - Ye, Chengbin
TI - Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR
T2 - Frontiers in medicine
J2 - Front Med (Lausanne)
PY - 2026
DA - 2026/
VL - 13
SP - 1905061
SN - 2296-858X
PB - Frontiers Media SA
DO - 10.3389/
UR - https://
LA - en
ER -
CSL-JSON
{
"id": "10.3389/
"type": "article-journal",
"title": "Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR",
"container-title": "Frontiers in medicine",
"author": [
{
"family": "Xie",
"given": "Lianxin"
},
{
"family": "Chen",
"given": "Jinghui"
},
{
"family": "Sun",
"given": "Zhipeng"
},
{
"family": "Huang",
"given": "Yuhan"
},
{
"family": "Ye",
"given": "Shan"
},
{
"family": "Ye",
"given": "Chengbin"
}
],
"container-title-short":
"volume": "13",
"page": "1905061",
"DOI": "10.3389/
"PMID": "42741587",
"PMCID": "PMC13572972",
"ISSN": "2296-858X",
"publisher": "Frontiers Media SA",
"URL": "https://
"language": "en",
"issued": {
"date-parts": [
[
2026,
8,
31
]
]
}
}
Similar papers
The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.
- [1] doi:10.3389/fmed.2025.1660445
- Application research on YOLOv5 model based on Lightweight Atrous Attention Module in brain tumor MRI image segmentation.Journal: Frontiers in medicineIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 3 references
- [2] doi:10.3389/fnins.2024.1510175
- Application of MRI image segmentation algorithm for brain tumors based on improved YOLO.Journal: Frontiers in neuroscienceIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
- [3] doi:10.1038/s41746-026-02365-3 [code]
- DARE-FUSE: domain aligned evidence guided learning for joint brain tumor MRI segmentation and classification.Journal: NPJ digital medicineIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
- [4] doi:10.3390/jimaging11020031
- Design of an Optimal Convolutional Neural Network Architecture for MRI Brain Tumor Classification by Exploiting Particle Swarm Optimization.Journal: Journal of imagingIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
- [5] doi:10.3390/diagnostics14040383
- A Novel Ensemble Framework for Multi-Classification of Brain Tumors Using Magnetic Resonance Imaging.Journal: Diagnostics (Basel, Switzerland)In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
- [6] doi:10.1371/journal.pone.0361135
- A method for brain tumor detection and localization by means of deep learning and reliable prediction explainability.Journal: PloS oneIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
- [7] doi:10.1186/s40708-026-00325-x
- CAE-BrainNet: a statistically validated class-adaptive attention ensemble model for explainable brain tumor classification from MRI.Journal: Brain informaticsIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
- [8] doi:10.3390/diagnostics16111745
- Brain Tumor Classification and Segmentation in MR Images Using EfficientNet and U-Net++ Models.Journal: Diagnostics (Basel, Switzerland)In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
- [9] doi:10.3390/jimaging12060233 [code]
- Brain Tumor Classification in MRI Images Using Combined Transfer Learning and Convolutional Neural Networks.Journal: Journal of imagingIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
- [10] doi:10.1038/s41598-026-51236-0
- A minimal-net CNN model for an IoT-based brain tumor detection and monitoring system.Journal: Scientific reportsIn common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
Contribute
The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.
Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.
Claim this paper
Correct its record
Say what each link of this record is, remove the ones that are not the paper's, add the ones that are missing. The correction becomes a new version of the record, in its Versions section.
Request its removal
To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).
Discussion, reproductions, activity
Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.
Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.
Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.
