OSCR

Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR.

Overview

Authors: Lianxin Xie1, Jinghui Chen1, Zhipeng Sun1, Yuhan Huang1, Shan Ye2, Chengbin Ye3
  1. The First Clinical Medical College, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
  2. Fujian Medical University, Fuzhou, China
  3. Department of Medical Imaging, The Affiliated People's Hospital of Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
Journal: Frontiers in medicine, volume 13, article 1905061
Dates: received 10 June 2026; accepted 11 August 2026; published online 31 August 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.3389/fmed.2026.1905061 · PMID 42741587 · PMCID PMC13572972 · OpenAlex W7204837167
Open access: gold, a free copy (OpenAlex)
Categories: structural MRI / diffusion (modality), other condition (population)
Methods: Machine learning
Keywords: artificial intelligence, brain tumor, hyper-parameter optimization, magnetic resonance imaging (MRI), object detection, performance trade-off, RT-DETR, YOLOv8s
Topic: Brain Tumor Detection and Classification (Computer Vision and Pattern Recognition, Computer Science), according to OpenAlex
Citations: not cited yet (Europe PMC); 43 references in the paper

Abstract

Objective: This study investigates optimal training strategies for YOLOv8s in brain tumor MRI detection, systematically evaluates the effects of key hyperparameters on model performance, and compares the adaptability and trade-offs between convolution-based dense prediction and Transformer-based sparse query mechanisms in medical image detection. Methods: Experiments were conducted on publicly available Kaggle MRI datasets of meningioma and glioma. YOLOv8s was adopted as the baseline model, and a systematic hyperparameter search was performed over learning rates (0.01, 0.001, and 0.0001) and bounding box regression loss weights (5.0, 7.5, and 10.0), forming nine configurations evaluated via five-fold cross-validation. Based on the optimal setting (learning rate = 0.01, loss weight = 7.5), the original YOLOv8s detection head was replaced with an RT-DETR Transformer-based decoder head. Performance and generalization were assessed on both internal and external test sets, and Score-CAM was employed for model interpretability analysis. Results: The hyperparameter analysis demonstrated that the learning rate was the primary factor affecting detection performance, whereas the influence of the bounding box regression loss weight was relatively limited within a reasonable range. Using the optimal hyperparameter combination (learning rate = 0.01 and box loss weight = 7.5), the native YOLOv8s detection head achieved the best overall performance, with a precision of 91.7%, a recall of 89.2%, and of 92.6%, outperforming the RT-DETR detection head ( = 91.1%). On the external test set, YOLOv8s also demonstrated superior cross-domain generalization ( = 79.0%), exceeding RT-DETR by 10.1%. Further NMS IoU sensitivity analysis showed that the RT-DETR detection head maintained more stable performance across different NMS IoU thresholds, whereas YOLOv8s was more sensitive to variations in NMS post-processing. Score-CAM visualization further revealed distinct attention patterns, with YOLOv8s exhibiting broader lesion responses and RT-DETR focusing more selectively on lesion regions. Conclusion: Appropriate hyperparameter optimization can substantially improve the performance of brain tumor MRI detection models. The dense prediction mechanism of YOLOv8s provides superior lesion coverage, resulting in better detection performance and cross-domain generalization, whereas the sparse query mechanism of RT-DETR exhibits greater robustness to NMS parameter variations during inference. These findings highlight the complementary characteristics of the two detection paradigms and provide practical guidance for detection head selection and inference strategy optimization in medical object detection

Reproduced under the paper's license (CC BY), from the paper cited above.

Code

The paper links to its data and to no code of its authors: see the Data section.

Tracing map

A tracing map links a paper to the code its authors published: this paper has none, so it has no map.

Data

Datasets cited

Data availability statement

Publicly available datasets were analyzed in this study. This data can be found here: https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset~https://www.kaggle.com/datasets/sartajbhuvaji/brain-tumor-classification-mri.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 6 authors, 8 keywords, 33 references.

Cite

This paper

Xie, L., Chen, J., Sun, Z., Huang, Y., Ye, S., & Ye, C. (2026). Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR. Frontiers in medicine, 13, 1905061. https://doi.org/10.3389/fmed.2026.1905061

BibTeX

@article{xie2026performance,
author = {Xie, Lianxin and Chen, Jinghui and Sun, Zhipeng and Huang, Yuhan and Ye, Shan and Ye, Chengbin},
title = {{Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR}},
journal = {Frontiers in medicine},
year = {2026},
month = aug,
volume = {13},
pages = {1905061},
publisher = {Frontiers Media SA},
issn = {2296-858X},
doi = {10.3389/fmed.2026.1905061},
url = {https://doi.org/10.3389/fmed.2026.1905061},
pmid = {42741587},
pmcid = {PMC13572972}
}

RIS

TY - JOUR
AU - Xie, Lianxin
AU - Chen, Jinghui
AU - Sun, Zhipeng
AU - Huang, Yuhan
AU - Ye, Shan
AU - Ye, Chengbin
TI - Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR
T2 - Frontiers in medicine
J2 - Front Med (Lausanne)
PY - 2026
DA - 2026/08/31
VL - 13
SP - 1905061
SN - 2296-858X
PB - Frontiers Media SA
DO - 10.3389/fmed.2026.1905061
UR - https://doi.org/10.3389/fmed.2026.1905061
LA - en
ER -

CSL-JSON

{
"id": "10.3389/fmed.2026.1905061",
"type": "article-journal",
"title": "Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR",
"container-title": "Frontiers in medicine",
"author": [
{
"family": "Xie",
"given": "Lianxin"
},
{
"family": "Chen",
"given": "Jinghui"
},
{
"family": "Sun",
"given": "Zhipeng"
},
{
"family": "Huang",
"given": "Yuhan"
},
{
"family": "Ye",
"given": "Shan"
},
{
"family": "Ye",
"given": "Chengbin"
}
],
"container-title-short": "Front Med (Lausanne)",
"volume": "13",
"page": "1905061",
"DOI": "10.3389/fmed.2026.1905061",
"PMID": "42741587",
"PMCID": "PMC13572972",
"ISSN": "2296-858X",
"publisher": "Frontiers Media SA",
"URL": "https://doi.org/10.3389/fmed.2026.1905061",
"language": "en",
"issued": {
"date-parts": [
[
2026,
8,
31
]
]
}
}

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.3389/fmed.2025.1660445
Application research on YOLOv5 model based on Lightweight Atrous Attention Module in brain tumor MRI image segmentation.
Journal: Frontiers in medicine
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 3 references
[2] doi:10.3389/fnins.2024.1510175
Application of MRI image segmentation algorithm for brain tumors based on improved YOLO.
Journal: Frontiers in neuroscience
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
[3] doi:10.1038/s41746-026-02365-3 [code]
DARE-FUSE: domain aligned evidence guided learning for joint brain tumor MRI segmentation and classification.
Journal: NPJ digital medicine
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
[4] doi:10.3390/jimaging11020031
Design of an Optimal Convolutional Neural Network Architecture for MRI Brain Tumor Classification by Exploiting Particle Swarm Optimization.
Journal: Journal of imaging
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
[5] doi:10.3390/diagnostics14040383
A Novel Ensemble Framework for Multi-Classification of Brain Tumors Using Magnetic Resonance Imaging.
Journal: Diagnostics (Basel, Switzerland)
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition, 1 reference
[6] doi:10.1371/journal.pone.0361135
A method for brain tumor detection and localization by means of deep learning and reliable prediction explainability.
Journal: PloS one
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
[7] doi:10.1186/s40708-026-00325-x
CAE-BrainNet: a statistically validated class-adaptive attention ensemble model for explainable brain tumor classification from MRI.
Journal: Brain informatics
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
[8] doi:10.3390/diagnostics16111745
Brain Tumor Classification and Segmentation in MR Images Using EfficientNet and U-Net++ Models.
Journal: Diagnostics (Basel, Switzerland)
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
[9] doi:10.3390/jimaging12060233 [code]
Brain Tumor Classification in MRI Images Using Combined Transfer Learning and Convolutional Neural Networks.
Journal: Journal of imaging
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition
[10] doi:10.1038/s41598-026-51236-0
A minimal-net CNN model for an IoT-based brain tumor detection and monitoring system.
Journal: Scientific reports
In common: kaggle.com/datasets/sartajbhuvaji, kaggle.com/datasets/masoudnickparvar, structural MRI / diffusion, other condition

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.