OSCR

Visual prompt engineering for multimodal and irregularly sampled medical data.

Code ↔ Paper

7 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 7 matches
  1. [1] § Results › Interpretability with attention visualization ↔ patient_journey.ipynb, lines 93–209 · score 0.86 · Invasive Blood Pressure, Respiratory Rate, Heart Rate, ICU patients, vital signs, Diastolic
  2. [2] § Methods › Data and benchmarks › MIMIC-IV ↔ data.py, lines 15–78 · score 0.59 · acute myocardial infarction, chronic, shock, Phenotype
  3. [3] § Methods › Data and benchmarks › MIMIC-IV ↔ data2.py, lines 1–36 · score 0.59 · acute myocardial infarction, chronic, shock, Phenotype
  4. [4] § Methods › Architecture with vision transformers ↔ model.py, lines 6–87 · score 0.56 · text model, linear, RoBERTa, pretrained, classification, Swin
  5. [5] § Methods › Data and benchmarks › CDSL ↔ main.py, lines 201–231 · score 0.54 · HM Hospitales, hospital mortality, multimodal, COVID, CDSL, PhysioNet
  6. [6] § Methods › Data and benchmarks › CDSL ↔ data2.py, lines 174–281 · score 0.51 · HM Hospitales, hospital mortality, multimodal, COVID, CDSL, diagnoses
  7. [7] § Results › Uni-modal training › MIMIC In-hospital mortality ↔ eval_inhospital_mortality.ipynb, lines 164–182 · score 0.51 · Bal.Acc, balanced accuracy, AUPRC, mortality

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 348 lines · 12 KB · no license · 2 matches

  1. from pathlib import Path
  2. from typing import List
  3. from PIL import Image
  4. import pandas as pd
  5. import torch
  6. from torch.utils.data import DataLoader
  7. from torchvision import transforms
  8. from transformers import ViTFeatureExtractor, AutoTokenizer
  10. PHENOTYPING_COLUMNS = [
  11. 'Acute and unspecified renal failure',
  12. 'Acute cerebrovascular disease',
  13. 'Acute myocardial infarction',
  14. 'Cardiac dysrhythmias',
  15. 'Chronic kidney disease',
  16. 'Chronic obstructive pulmonary disease and bronchiectasis',
  17. 'Complications of surgical procedures or medical care',
  18. 'Conduction disorders',
  19. 'Congestive heart failure; nonhypertensive',
  20. 'Coronary atherosclerosis and other heart disease',
  21. 'Diabetes mellitus with complications',
  22. 'Diabetes mellitus without complication',
  23. 'Disorders of lipid metabolism',
  24. 'Essential hypertension',
  25. 'Fluid and electrolyte disorders',
  26. 'Gastrointestinal hemorrhage',
  27. 'Hypertension with complications and secondary hypertension',
  28. 'Other liver diseases',
  29. 'Other lower respiratory disease',
  30. 'Other upper respiratory disease',
  31. 'Pleurisy; pneumothorax; pulmonary collapse',
  32. 'Pneumonia (except that caused by tuberculosis or sexually transmitted disease)',
  33. 'Respiratory failure; insufficiency; arrest (adult)',
  34. 'Septicemia (except in labor)',
  35. 'Shock'
  36. ]
  38. def number_to_bucket(value):
  39. """Convert continuous number to bucket classification"""
  40. # if pd.isna(value):
  41. # return np.nan
  42. if 0 <= value < 1:
  43. return 0
  44. elif 1 <= value < 2:
  45. return 1
  46. elif 2 <= value < 3:
  47. return 2
  48. elif 3 <= value < 4:
  49. return 3
  50. elif 4 <= value < 5:
  51. return 4
  52. elif 5 <= value < 6:
  53. return 5
  54. elif 6 <= value < 7:
  55. return 6
  56. elif 7 <= value < 8:
  57. return 7
  58. elif 8 <= value <= 14:
  59. return 8
  60. else: # > 14
  61. return 9
  63. class MIMICDataset:
  64. def __init__(
  65. self,
  66. task,
  67. split,
  68. modalities,
  69. with_diagnoses,
  70. root='/mnt/hdd/data/ViTiMM_data2/data/',
  71. cxr_root='/mnt/sds/sd20i001/malte/data/physionet.org/files/mimic-cxr-jpg',
  72. non_empty=[],
  73. timeframe=48,
  74. image_size=384,
  75. max_length=128,
  76. img_model_name='microsoft/swin-large-patch4-window12-384-in22k',
  77. *args, **kwargs
  78. ):
  79. self.root = Path(root)
  80. self.src = self.root / 'plots' / task / split if task in ['in-hospital-mortality','phenotyping'] else self.root / task / str(timeframe) / split
  81. self.cxr_root = Path(cxr_root)
  82. self.task = task
  83. self.split = split
  84. self.modalities = modalities
  85. self.with_diagnoses = with_diagnoses
  87. self.df = pd.read_csv(self.root / f'{task}_{split}.csv', index_col=0)
  88. # if non_empty:
  89. self.non_empty = non_empty
  90. if len(non_empty):
  91. self.df = self.df.dropna(subset=[f'{m}_fname' for m in non_empty], how='any')
  93. self.tokenizer_img = ViTFeatureExtractor.from_pretrained(img_model_name) # 'google/vit-base-patch16-224')
  94. self.cxr_train_transform = transforms.Compose([
  95. transforms.RandomHorizontalFlip(),
  96. transforms.RandomAffine(degrees=45, scale=(.85, 1.15), shear=0, translate=(0.15, 0.15))
  97. ])
  99. self.image_size = image_size
  100. self.img_transform = transforms.Compose([
  101. transforms.Resize((image_size, image_size)),
  102. transforms.ToTensor(),
  103. transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
  104. ])
  106. self.text_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
  107. self.max_length = max_length
  108. # self.empty_ecg_path = self.root / 'ecgs2' / 'empty_ecg.png'
  110. def __getitem__(self, i):
  111. sample = self.df.iloc[i]
  113. if self.task == 'phenotyping':
  114. label = sample[PHENOTYPING_COLUMNS].astype(float).tolist()
  115. elif self.task == 'length-of-stay':
  116. label = sample.y_true
  117. label = number_to_bucket(label)
  118. elif self.task == 'decompensation':
  119. label = [sample.y_true]
  120. else: # inhospital_mortality
  121. label = [sample.y_true]
  123. get_text = lambda k: sample[f'{k}_text'] if type(sample[f'{k}_text']) == str else ''
  124. text = get_text('meta')
  125. if self.with_diagnoses:
  126. text += ' ' + get_text('diagnoses')
  127. if type(sample.ecg_text) == str:
  128. text += ' ' + get_text('ecg')
  129. if type(sample.cxr_text) == str:
  130. text += ' ' + get_text('cxr')
  131. if type(sample.med_text) == str:
  132. text += ' ' + get_text('med')
  134. data = {'y': torch.tensor(label, dtype=torch.float32)}
  136. if 'lab' in self.modalities:
  137. data['lab'] = self.load_image(sample.lab_fname)
  139. if 'med' in self.modalities:
  140. data['med'] = self.load_image(sample.med_fname)
  142. if 'cxr' in self.modalities:
  143. data['cxr'] = self.load_image(sample.cxr_fname,'cxr')
  145. if 'ecg' in self.modalities:
  146. data['ecg'] = self.load_image(sample.ecg_fname)
  148. text_inputs = self.text_tokenizer(
  149. text,
  150. padding='max_length',
  151. max_length=self.max_length,
  152. return_tensors='pt',
  153. truncation=True
  154. )
  155. data['input_ids'] = text_inputs['input_ids'].squeeze()
  156. data['attention_mask'] = text_inputs['attention_mask'].squeeze()
  158. return data
  160. def load_image(self, fname, modality=None):
  161. if type(fname) == str:
  162. if not fname.endswith('.png') and not fname.endswith('.jpg'):
  163. fname += '.png'
  164. fname2 = self.root / fname if modality != 'cxr' else self.cxr_root / fname
  165. img = Image.open(fname2).convert('RGB')
  166. img = self.img_transform(img)
  167. else:
  168. img = torch.zeros((3,self.image_size, self.image_size)).float()
  169. return img
  171. def __len__(self):
  172. return len(self.df)
  174. class CovidDataset:
  175. def __init__(
  176. self,
  177. task,
  178. split,
  179. modalities,
  180. with_diagnoses,
  181. root='/mnt/hdd/data/covid-data-for-shared-learning-cdsl-a-comprehensive-multimodal-covid-19-dataset-from-hm-hospitales-1.0.0/',
  182. non_empty=[],
  183. image_size=384,
  184. max_length=128,
  185. img_model_name='microsoft/swin-large-patch4-window12-384-in22k',
  186. *args, **kwargs
  187. ):
  188. self.root = Path(root)
  189. self.src = self.root / 'plots' #task / split if task in ['in-hospital-mortality','phenotyping'] else self.root / task / str(timeframe) / split
  190. self.task = task
  191. self.split = split
  192. self.modalities = modalities
  193. self.with_diagnoses = with_diagnoses
  195. self.df = pd.read_csv(self.root / 'meta.csv', index_col=0)
  196. self.df = self.df.dropna(subset=['inhospital_mortality','length_of_stay'], how='any')
  197. self.df = self.df[self.df.split==split]
  198. self.df = self.df.dropna(subset=['med_fname','lab_fname','cxr_fname'], how='all')
  199. self.df = self.df[self.df.length_of_stay>=5]
  200. self.non_empty = non_empty
  201. if len(non_empty):
  202. self.df = self.df.dropna(subset=[f'{m}_fname' for m in non_empty], how='any')
  204. self.tokenizer_img = ViTFeatureExtractor.from_pretrained(img_model_name) # 'google/vit-base-patch16-224')
  205. self.cxr_train_transform = transforms.Compose([
  206. transforms.RandomHorizontalFlip(),
  207. transforms.RandomAffine(degrees=45, scale=(.85, 1.15), shear=0, translate=(0.15, 0.15))
  208. ])
  210. self.image_size = image_size
  211. self.img_transform = transforms.Compose([
  212. transforms.Resize((image_size, image_size)),
  213. transforms.ToTensor(),
  214. transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
  215. ])
  217. self.text_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
  218. self.max_length = max_length
  219. # self.empty_ecg_path = self.root / 'ecgs2' / 'empty_ecg.png'
  221. def __getitem__(self, i):
  222. sample = self.df.iloc[i]
  224. # if self.task == 'phenotyping':
  225. # label = sample[PHENOTYPING_COLUMNS].astype(float).tolist()
  226. if self.task == 'length-of-stay':
  227. label = sample.length_of_stay
  228. label = number_to_bucket(label) - 5
  229. else: # inhospital_mortality
  230. label = [sample.inhospital_mortality]
  232. get_text = lambda k: sample[f'{k}_text'] if type(sample[f'{k}_text']) == str else ''
  233. text = get_text('meta')
  234. if self.with_diagnoses:
  235. text += ' ' + get_text('diag')
  236. # if type(sample.ecg_text) == str:
  237. # text += ' ' + get_text('ecg')
  238. # if type(sample.cxr_text) == str:
  239. # text += ' ' + get_text('cxr')
  240. if type(sample.med_text) == str:
  241. text += ' ' + get_text('med')
  243. data = {'y': torch.tensor(label, dtype=torch.float32)}
  245. if 'lab' in self.modalities:
  246. data['lab'] = self.load_image(sample.lab_fname)
  248. if 'med' in self.modalities:
  249. data['med'] = self.load_image(sample.med_fname)
  251. if 'cxr' in self.modalities:
  252. data['cxr'] = self.load_image(sample.cxr_fname)
  254. # if 'ecg' in self.modalities:
  255. # data['ecg'] = self.load_image(sample.ecg_fname)
  257. text_inputs = self.text_tokenizer(
  258. text,
  259. padding='max_length',
  260. max_length=self.max_length,
  261. return_tensors='pt',
  262. truncation=True
  263. )
  264. data['input_ids'] = text_inputs['input_ids'].squeeze()
  265. data['attention_mask'] = text_inputs['attention_mask'].squeeze()
  267. return data
  269. def load_image(self, fname):
  270. if type(fname) == str:
  271. if not fname.endswith('.png') and not fname.endswith('.jpg'):
  272. fname += '.png'
  273. fname = self.src / fname
  274. img = Image.open(fname).convert('RGB')
  275. img = self.img_transform(img)
  276. else:
  277. img = torch.zeros((3,self.image_size, self.image_size)).float()
  278. return img
  280. def __len__(self):
  281. return len(self.df)
  283. def dataloaders(
  284. dataset: str,
  285. task: str,
  286. modalities: List[str],
  287. non_empty: bool = False,
  288. image_size: int = 224,
  289. img_model_name: str = 'microsoft/swin-base-patch4-window7-224-in22k',
  290. batch_size: int = 16,
  291. root: str = './data',
  292. cxr_root: str = './cxrs',
  293. with_diagnoses: bool = True
  294. ):
  295. if dataset == 'covid' and task in ['decompensation','phenotyping']:
  296. raise ValueError
  298. dataset = MIMICDataset if dataset == 'mimic' else CovidDataset
  299. trainset = dataset(
  300. task=task,
  301. split='train',
  302. modalities=modalities,
  303. non_empty=non_empty,
  304. with_diagnoses=with_diagnoses,
  305. root=root,
  306. cxr_root=cxr_root,
  307. timeframe=48,
  308. max_length=512,
  309. image_size=image_size,
  310. img_model_name=img_model_name
  311. )
  312. from sklearn.model_selection import train_test_split
  313. train_df = trainset.df
  314. train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)
  315. valset = dataset(
  316. task=task,
  317. split='train',
  318. modalities=modalities,
  319. non_empty=non_empty,
  320. with_diagnoses=with_diagnoses,
  321. root=root,
  322. cxr_root=cxr_root,
  323. timeframe=48,
  324. max_length=512,
  325. image_size=image_size,
  326. img_model_name=img_model_name
  327. )
  328. valset.df = val_df
  329. trainset.df = train_df
  331. testset = dataset(
  332. task=task,
  333. split='test',
  334. modalities=modalities,
  335. non_empty=non_empty,
  336. with_diagnoses=with_diagnoses,
  337. root=root,
  338. cxr_root=cxr_root,
  339. timeframe=48,
  340. max_length=512,
  341. image_size=image_size,
  342. img_model_name=img_model_name
  343. )
  345. trainloader = DataLoader(trainset, batch_size=batch_size, shuffle=True)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  346. valloader = DataLoader(valset, batch_size=batch_size)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  347. testloader = DataLoader(testset, batch_size=batch_size)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  348. return trainloader, valloader, testloader

data2.py at commit 4ae9b54, no license · at the source

Overview

Authors: Malte Tölle1,2,3, Mohamad Scharaf1, Samantha Fischer1,2,3, Christoph Reich1,2,3, Silav Zeid4,5,6, Christoph Dieterich1,2,3, Benjamin Meder1,2,3, Norbert Frey1,2,3, Philipp Wild4,5,6,7, Sandy Engelhardt1,2,3
ORCID iDs: Malte Tölle, Christoph Reich, Christoph Dieterich, Benjamin Meder, Norbert Frey, Philipp Wild
  1. Department of Cardiology, Angiology and Pneumology, Heidelberg University Hospital, Heidelberg, Germany
  2. Informatics for Life Institute, Heidelberg, Germany
  3. DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany
  4. Preventive Cardiology and Preventive Medicine, Department of Cardiology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany
  5. Clinical Epidemiology and Systems Medicine, Center for Thrombosis and Hemostasis, University Medical Center Mainz, Johannes Gutenberg University Mainz, Mainz, Germany
  6. DZHK (German Centre for Cardiovascular Research), Partner Site Rhine-Main, Mainz, Germany
  7. Systems Medicine, Institute of Molecular Biology (IMB), Mainz, Germany
Institutions: Heidelberg University (Germany); University Hospital Heidelberg (Germany); German Centre for Cardiovascular Research (Germany); Johannes Gutenberg University Mainz (Germany); University Medical Center of the Johannes Gutenberg University Mainz (Germany); Institute of Molecular Biology (Germany)
Journal: Communications medicine, volume 6, issue 1, article 474
Dates: received 13 March 2025; accepted 16 July 2026; published online 1 September 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s43856-026-01817-x · PMID 42680823 · PMCID PMC13534428 · OpenAlex W7204889893
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: clinical / translational (subfield)
Methods: Statistics, Machine learning
Keywords: Medical imaging, Diagnosis
Topic: Multimodal Machine Learning Applications (Computer Vision and Pattern Recognition, Computer Science), according to OpenAlex
Funding: Deutsche Forschungsgemeinschaft (INST 35/1314-1 FUGG, INST35/1503‐1FUGG, INST 35/1314‐1, 35/1503-1, 35/1503-1 FUGG, 35/1314-1 FUGG, INST 35, INST 35/1503‐1, 35/1314-1)
Citations: not cited yet (Europe PMC); 63 references in the paper

Abstract

Background: A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure.

Methods: We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), simplifies data preprocessing and modeling by unifying clinical measurements, medications, X-ray images, and electrocardiography scans into a single visual representation.

Results: ViTiMM outperforms current state-of-the-art methods in predicting in-hospital mortality, phenotyping, and decompensation on two datasets, the MIMIC-IV and COVID Data for Shared Learning (CDSL) dataset.

Conclusions: We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code is publicly available at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 7 matches between paragraphs and lines of code.

Zenodo 20786553

License: CC-BY-4.0
State: the link answers, verified on 26 September 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: “Code availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (11 files), pandas (11 files), PyTorch (11 files), Matplotlib (9 files), seaborn (6 files), Pillow (5 files), Hugging Face Transformers (4 files), scikit-learn (3 files), SciPy (3 files), OpenCV (1 file)
Availability: 1 check, the latest on 26 September 2026: the link answers (HTTP 200)
  • 26 September 2026: the link answers (HTTP 200)
17 files
At the source:

Cardio-AI/ViTiMM

License: none: the authors keep all their rights
State: the link answers, verified on 26 September 2026
Evidence: files inventoried
Commit: 4ae9b544665e1a8b5be2ee81b1622c21ca4fe115, 5 August 2025
Languages: Jupyter (10), Python (6)
Size: 25 files, 16 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, environment (requirements.txt), 10 notebooks
Not found: license file, CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (11 files), pandas (11 files), PyTorch (11 files), Matplotlib (9 files), seaborn (6 files), Pillow (5 files), Hugging Face Transformers (4 files), scikit-learn (3 files), SciPy (3 files), OpenCV (1 file)
Availability: 1 check, the latest on 26 September 2026: the link answers
  • 26 September 2026: the link answers
17 files
At the source: github.com/Cardio-AI/ViTiMM

Code availability

Following the FAIR criteria (findability, accessibility, interoperability, and reusability) in scientific research all code used in this study will be made publicly available. The code for cohort creation, data preprocessing, i.e., plot generation, training, and evaluation is open source available at https://doi.org/10.5281/zenodo.20786553 and at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 32 scripts, each with its path and the digest of its content;
  • 7 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability Statement

All data is publicly available from the MIMIC database22,34 on PhysioNet (MIMICIV: https://physionet.org/content/mimiciv/3.1, MIMIC-CXR-JPG: https://physionet.org/content/mimic-cxr-jpg/2.0.0, and MIMIC-IV-ECG: https://physionet.org/content/mimic-iv-ecg/1.0). The COVID Data for Shared Learning (CDSL) dataset is available at https://physionet.org/content/covid-data-shared-learning/1.0.0/. The code to extract the chest radiographs and corresponding clinical parameters can be found in the GitHub repository linked in the code availability section.

Following the FAIR criteria (findability, accessibility, interoperability, and reusability) in scientific research all code used in this study will be made publicly available. The code for cohort creation, data preprocessing, i.e., plot generation, training, and evaluation is open source available at https://doi.org/10.5281/zenodo.20786553 and at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 3, 28 September 2026

  • Funding: added Deutsche Forschungsgemeinschaft: INST 35/1314-1 FUGG, INST35/1503‐1FUGG, INST 35/1314‐1, 35/1503-1, 35/1503-1 FUGG, 35/1314-1 FUGG, INST 35, INST 35/1503‐1, 35/1314-1

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 10 authors, 2 keywords, 40 references.

Cite

This paper

Tölle, M., Scharaf, M., Fischer, S., Reich, C., Zeid, S., Dieterich, C., Meder, B., Frey, N., Wild, P., & Engelhardt, S. (2026). Visual prompt engineering for multimodal and irregularly sampled medical data. Communications medicine, 6(1), 474. https://doi.org/10.1038/s43856-026-01817-x

BibTeX

@article{tolle2026visual,
author = {Tölle, Malte and Scharaf, Mohamad and Fischer, Samantha and Reich, Christoph and Zeid, Silav and Dieterich, Christoph and Meder, Benjamin and Frey, Norbert and Wild, Philipp and Engelhardt, Sandy},
title = {{Visual prompt engineering for multimodal and irregularly sampled medical data}},
journal = {Communications medicine},
year = {2026},
month = sep,
volume = {6},
number = {1},
pages = {474},
publisher = {Nature Publishing Group},
issn = {2730-664X},
doi = {10.1038/s43856-026-01817-x},
url = {https://doi.org/10.1038/s43856-026-01817-x},
pmid = {42680823},
pmcid = {PMC13534428}
}

RIS

TY - JOUR
AU - Tölle, Malte
AU - Scharaf, Mohamad
AU - Fischer, Samantha
AU - Reich, Christoph
AU - Zeid, Silav
AU - Dieterich, Christoph
AU - Meder, Benjamin
AU - Frey, Norbert
AU - Wild, Philipp
AU - Engelhardt, Sandy
TI - Visual prompt engineering for multimodal and irregularly sampled medical data
T2 - Communications medicine
J2 - Commun Med (Lond)
PY - 2026
DA - 2026/09/01
VL - 6
IS - 1
SP - 474
SN - 2730-664X
PB - Nature Publishing Group
DO - 10.1038/s43856-026-01817-x
UR - https://doi.org/10.1038/s43856-026-01817-x
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s43856-026-01817-x",
"type": "article-journal",
"title": "Visual prompt engineering for multimodal and irregularly sampled medical data",
"container-title": "Communications medicine",
"author": [
{
"family": "Tölle",
"given": "Malte"
},
{
"family": "Scharaf",
"given": "Mohamad"
},
{
"family": "Fischer",
"given": "Samantha"
},
{
"family": "Reich",
"given": "Christoph"
},
{
"family": "Zeid",
"given": "Silav"
},
{
"family": "Dieterich",
"given": "Christoph"
},
{
"family": "Meder",
"given": "Benjamin"
},
{
"family": "Frey",
"given": "Norbert"
},
{
"family": "Wild",
"given": "Philipp"
},
{
"family": "Engelhardt",
"given": "Sandy"
}
],
"container-title-short": "Commun Med (Lond)",
"volume": "6",
"issue": "1",
"page": "474",
"DOI": "10.1038/s43856-026-01817-x",
"PMID": "42680823",
"PMCID": "PMC13534428",
"ISSN": "2730-664X",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s43856-026-01817-x",
"language": "en",
"issued": {
"date-parts": [
[
2026,
9,
1
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1016/j.isci.2026.116135 [code]
Predicting ICU in-hospital mortality from text-encoded structured EHR data using adaptive transformer layer fusion.
Journal: iScience
In common: Hugging Face Transformers, PyTorch, seaborn, 4 other tools, physionet.org/content/mimiciv, clinical / translational, 3 references
[2] doi:10.1371/journal.pone.0344600 [code]
Robust disease prognosis via diagnostic knowledge preservation: A sequential learning approach.
Journal: PloS one
In common: OpenCV, Pillow, PyTorch, 5 other tools, clinical / translational, 2 references
[3] doi:10.1093/braincomms/fcag253 [code]
Disease detection and classification in temporal lobe epilepsy: step-wise versus simultaneous AI decision models in a multisite neuroimaging study.
Journal: Brain communications
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools, clinical / translational
[4] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools
[5] doi:10.1038/s41467-026-76837-1 [code]
Drug screen and machine learning predict neuroprotective agents in a preclinical human model of childhood dementia.
Journal: Nature communications
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools
[6] doi:10.1038/s41467-026-74358-5 [code]
Brain-inspired spatial intelligence for embodied agents.
Journal: Nature communications
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools
[7] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools
[8] doi:10.1002/hbm.70469 [code]
VarCoNet: A Variability-Aware Self-Supervised Framework for Functional Connectome Extraction From Resting-State fMRI.
Journal: Human brain mapping
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools
[9] doi:10.1038/s41598-026-50791-w [code]
Hybrid Vi+ECNN framework for advanced ADHD diagnostic accuracy in medical imaging.
Journal: Scientific reports
In common: Hugging Face Transformers, OpenCV, Pillow, 6 other tools, clinical / translational
[10] doi:10.1016/j.patter.2026.101538 [code]
A multi-modal foundation model for brain disease diagnosis and medical imaging.
Journal: Patterns (New York, N.Y.)
In common: Hugging Face Transformers, OpenCV, Pillow, 6 other tools, clinical / translational

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.