OSCR

Visual prompt engineering for multimodal and irregularly sampled medical data.

Code ↔ Paper

7 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

5 blocks of code here have a permanent identifier (oscr:b:…), listed with the matches below: citable for good, with a page of its own. Select a match and press Cite to copy it. What a block identifier is.

The 7 matches
  1. [1] § Results › Interpretability with attention visualization ↔ patient_journey.ipynb, lines 93 to 209 · score 0.86 · Invasive Blood Pressure, Respiratory Rate, Heart Rate, ICU patients, vital signs, Diastolic · Report a mismatch
  2. [2] § Methods › Data and benchmarks › MIMIC-IV ↔ data.py, lines 15 to 78 · score 0.59 · acute myocardial infarction, chronic, shock, Phenotype · block oscr:b:3C8P1HB5K4F1N-5 · Report a mismatch
  3. [3] § Methods › Data and benchmarks › MIMIC-IV ↔ data2.py, lines 1 to 36 · score 0.59 · acute myocardial infarction, chronic, shock, Phenotype · block oscr:b:N619YRDDZMMT2-W · Report a mismatch
  4. [4] § Methods › Architecture with vision transformers ↔ model.py, lines 6 to 87 · score 0.56 · text model, linear, RoBERTa, pretrained, classification, Swin · block oscr:b:JADDZ7B5WBVE6-V · Report a mismatch
  5. [5] § Methods › Data and benchmarks › CDSL ↔ main.py, lines 201 to 231 · score 0.54 · HM Hospitales, hospital mortality, multimodal, COVID, CDSL, PhysioNet · block oscr:b:TG71A7FY80PP7-K · Report a mismatch
  6. [6] § Methods › Data and benchmarks › CDSL ↔ data2.py, lines 174 to 281 · score 0.51 · HM Hospitales, hospital mortality, multimodal, COVID, CDSL, diagnoses · block oscr:b:AXZXR0GY4Z6RD-J · Report a mismatch
  7. [7] § Results › Uni-modal training › MIMIC In-hospital mortality ↔ eval_inhospital_mortality.ipynb, lines 164 to 182 · score 0.51 · Bal.Acc, balanced accuracy, AUPRC, mortality · Report a mismatch

Paper

Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 348 lines · 12 KB · no license · 2 matches

  1. from pathlib import Path
  2. from typing import List
  3. from PIL import Image
  4. import pandas as pd
  5. import torch
  6. from torch.utils.data import DataLoader
  7. from torchvision import transforms
  8. from transformers import ViTFeatureExtractor, AutoTokenizer
  9. PHENOTYPING_COLUMNS = [
  10. 'Acute and unspecified renal failure',
  11. 'Acute cerebrovascular disease',
  12. 'Acute myocardial infarction',
  13. 'Cardiac dysrhythmias',
  14. 'Chronic kidney disease',
  15. 'Chronic obstructive pulmonary disease and bronchiectasis',
  16. 'Complications of surgical procedures or medical care',
  17. 'Conduction disorders',
  18. 'Congestive heart failure; nonhypertensive',
  19. 'Coronary atherosclerosis and other heart disease',
  20. 'Diabetes mellitus with complications',
  21. 'Diabetes mellitus without complication',
  22. 'Disorders of lipid metabolism',
  23. 'Essential hypertension',
  24. 'Fluid and electrolyte disorders',
  25. 'Gastrointestinal hemorrhage',
  26. 'Hypertension with complications and secondary hypertension',
  27. 'Other liver diseases',
  28. 'Other lower respiratory disease',
  29. 'Other upper respiratory disease',
  30. 'Pleurisy; pneumothorax; pulmonary collapse',
  31. 'Pneumonia (except that caused by tuberculosis or sexually transmitted disease)',
  32. 'Respiratory failure; insufficiency; arrest (adult)',
  33. 'Septicemia (except in labor)',
  34. 'Shock'
  35. ]
  36. def number_to_bucket(value):
  37. """Convert continuous number to bucket classification"""
  38. # if pd.isna(value):
  39. # return np.nan
  40. if 0 <= value < 1:
  41. return 0
  42. elif 1 <= value < 2:
  43. return 1
  44. elif 2 <= value < 3:
  45. return 2
  46. elif 3 <= value < 4:
  47. return 3
  48. elif 4 <= value < 5:
  49. return 4
  50. elif 5 <= value < 6:
  51. return 5
  52. elif 6 <= value < 7:
  53. return 6
  54. elif 7 <= value < 8:
  55. return 7
  56. elif 8 <= value <= 14:
  57. return 8
  58. else: # > 14
  59. return 9
  60. class MIMICDataset:
  61. def __init__(
  62. self,
  63. task,
  64. split,
  65. modalities,
  66. with_diagnoses,
  67. root='/mnt/hdd/data/ViTiMM_data2/data/',
  68. cxr_root='/mnt/sds/sd20i001/malte/data/physionet.org/files/mimic-cxr-jpg',
  69. non_empty=[],
  70. timeframe=48,
  71. image_size=384,
  72. max_length=128,
  73. img_model_name='microsoft/swin-large-patch4-window12-384-in22k',
  74. *args, **kwargs
  75. ):
  76. self.root = Path(root)
  77. self.src = self.root / 'plots' / task / split if task in ['in-hospital-mortality','phenotyping'] else self.root / task / str(timeframe) / split
  78. self.cxr_root = Path(cxr_root)
  79. self.task = task
  80. self.split = split
  81. self.modalities = modalities
  82. self.with_diagnoses = with_diagnoses
  83. self.df = pd.read_csv(self.root / f'{task}_{split}.csv', index_col=0)
  84. # if non_empty:
  85. self.non_empty = non_empty
  86. if len(non_empty):
  87. self.df = self.df.dropna(subset=[f'{m}_fname' for m in non_empty], how='any')
  88. self.tokenizer_img = ViTFeatureExtractor.from_pretrained(img_model_name) # 'google/vit-base-patch16-224')
  89. self.cxr_train_transform = transforms.Compose([
  90. transforms.RandomHorizontalFlip(),
  91. transforms.RandomAffine(degrees=45, scale=(.85, 1.15), shear=0, translate=(0.15, 0.15))
  92. ])
  93. self.image_size = image_size
  94. self.img_transform = transforms.Compose([
  95. transforms.Resize((image_size, image_size)),
  96. transforms.ToTensor(),
  97. transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
  98. ])
  99. self.text_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
  100. self.max_length = max_length
  101. # self.empty_ecg_path = self.root / 'ecgs2' / 'empty_ecg.png'
  102. def __getitem__(self, i):
  103. sample = self.df.iloc[i]
  104. if self.task == 'phenotyping':
  105. label = sample[PHENOTYPING_COLUMNS].astype(float).tolist()
  106. elif self.task == 'length-of-stay':
  107. label = sample.y_true
  108. label = number_to_bucket(label)
  109. elif self.task == 'decompensation':
  110. label = [sample.y_true]
  111. else: # inhospital_mortality
  112. label = [sample.y_true]
  113. get_text = lambda k: sample[f'{k}_text'] if type(sample[f'{k}_text']) == str else ''
  114. text = get_text('meta')
  115. if self.with_diagnoses:
  116. text += ' ' + get_text('diagnoses')
  117. if type(sample.ecg_text) == str:
  118. text += ' ' + get_text('ecg')
  119. if type(sample.cxr_text) == str:
  120. text += ' ' + get_text('cxr')
  121. if type(sample.med_text) == str:
  122. text += ' ' + get_text('med')
  123. data = {'y': torch.tensor(label, dtype=torch.float32)}
  124. if 'lab' in self.modalities:
  125. data['lab'] = self.load_image(sample.lab_fname)
  126. if 'med' in self.modalities:
  127. data['med'] = self.load_image(sample.med_fname)
  128. if 'cxr' in self.modalities:
  129. data['cxr'] = self.load_image(sample.cxr_fname,'cxr')
  130. if 'ecg' in self.modalities:
  131. data['ecg'] = self.load_image(sample.ecg_fname)
  132. text_inputs = self.text_tokenizer(
  133. text,
  134. padding='max_length',
  135. max_length=self.max_length,
  136. return_tensors='pt',
  137. truncation=True
  138. )
  139. data['input_ids'] = text_inputs['input_ids'].squeeze()
  140. data['attention_mask'] = text_inputs['attention_mask'].squeeze()
  141. return data
  142. def load_image(self, fname, modality=None):
  143. if type(fname) == str:
  144. if not fname.endswith('.png') and not fname.endswith('.jpg'):
  145. fname += '.png'
  146. fname2 = self.root / fname if modality != 'cxr' else self.cxr_root / fname
  147. img = Image.open(fname2).convert('RGB')
  148. img = self.img_transform(img)
  149. else:
  150. img = torch.zeros((3,self.image_size, self.image_size)).float()
  151. return img
  152. def __len__(self):
  153. return len(self.df)
  154. class CovidDataset:
  155. def __init__(
  156. self,
  157. task,
  158. split,
  159. modalities,
  160. with_diagnoses,
  161. root='/mnt/hdd/data/covid-data-for-shared-learning-cdsl-a-comprehensive-multimodal-covid-19-dataset-from-hm-hospitales-1.0.0/',
  162. non_empty=[],
  163. image_size=384,
  164. max_length=128,
  165. img_model_name='microsoft/swin-large-patch4-window12-384-in22k',
  166. *args, **kwargs
  167. ):
  168. self.root = Path(root)
  169. self.src = self.root / 'plots' #task / split if task in ['in-hospital-mortality','phenotyping'] else self.root / task / str(timeframe) / split
  170. self.task = task
  171. self.split = split
  172. self.modalities = modalities
  173. self.with_diagnoses = with_diagnoses
  174. self.df = pd.read_csv(self.root / 'meta.csv', index_col=0)
  175. self.df = self.df.dropna(subset=['inhospital_mortality','length_of_stay'], how='any')
  176. self.df = self.df[self.df.split==split]
  177. self.df = self.df.dropna(subset=['med_fname','lab_fname','cxr_fname'], how='all')
  178. self.df = self.df[self.df.length_of_stay>=5]
  179. self.non_empty = non_empty
  180. if len(non_empty):
  181. self.df = self.df.dropna(subset=[f'{m}_fname' for m in non_empty], how='any')
  182. self.tokenizer_img = ViTFeatureExtractor.from_pretrained(img_model_name) # 'google/vit-base-patch16-224')
  183. self.cxr_train_transform = transforms.Compose([
  184. transforms.RandomHorizontalFlip(),
  185. transforms.RandomAffine(degrees=45, scale=(.85, 1.15), shear=0, translate=(0.15, 0.15))
  186. ])
  187. self.image_size = image_size
  188. self.img_transform = transforms.Compose([
  189. transforms.Resize((image_size, image_size)),
  190. transforms.ToTensor(),
  191. transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
  192. ])
  193. self.text_tokenizer = AutoTokenizer.from_pretrained('roberta-base')
  194. self.max_length = max_length
  195. # self.empty_ecg_path = self.root / 'ecgs2' / 'empty_ecg.png'
  196. def __getitem__(self, i):
  197. sample = self.df.iloc[i]
  198. # if self.task == 'phenotyping':
  199. # label = sample[PHENOTYPING_COLUMNS].astype(float).tolist()
  200. if self.task == 'length-of-stay':
  201. label = sample.length_of_stay
  202. label = number_to_bucket(label) - 5
  203. else: # inhospital_mortality
  204. label = [sample.inhospital_mortality]
  205. get_text = lambda k: sample[f'{k}_text'] if type(sample[f'{k}_text']) == str else ''
  206. text = get_text('meta')
  207. if self.with_diagnoses:
  208. text += ' ' + get_text('diag')
  209. # if type(sample.ecg_text) == str:
  210. # text += ' ' + get_text('ecg')
  211. # if type(sample.cxr_text) == str:
  212. # text += ' ' + get_text('cxr')
  213. if type(sample.med_text) == str:
  214. text += ' ' + get_text('med')
  215. data = {'y': torch.tensor(label, dtype=torch.float32)}
  216. if 'lab' in self.modalities:
  217. data['lab'] = self.load_image(sample.lab_fname)
  218. if 'med' in self.modalities:
  219. data['med'] = self.load_image(sample.med_fname)
  220. if 'cxr' in self.modalities:
  221. data['cxr'] = self.load_image(sample.cxr_fname)
  222. # if 'ecg' in self.modalities:
  223. # data['ecg'] = self.load_image(sample.ecg_fname)
  224. text_inputs = self.text_tokenizer(
  225. text,
  226. padding='max_length',
  227. max_length=self.max_length,
  228. return_tensors='pt',
  229. truncation=True
  230. )
  231. data['input_ids'] = text_inputs['input_ids'].squeeze()
  232. data['attention_mask'] = text_inputs['attention_mask'].squeeze()
  233. return data
  234. def load_image(self, fname):
  235. if type(fname) == str:
  236. if not fname.endswith('.png') and not fname.endswith('.jpg'):
  237. fname += '.png'
  238. fname = self.src / fname
  239. img = Image.open(fname).convert('RGB')
  240. img = self.img_transform(img)
  241. else:
  242. img = torch.zeros((3,self.image_size, self.image_size)).float()
  243. return img
  244. def __len__(self):
  245. return len(self.df)
  246. def dataloaders(
  247. dataset: str,
  248. task: str,
  249. modalities: List[str],
  250. non_empty: bool = False,
  251. image_size: int = 224,
  252. img_model_name: str = 'microsoft/swin-base-patch4-window7-224-in22k',
  253. batch_size: int = 16,
  254. root: str = './data',
  255. cxr_root: str = './cxrs',
  256. with_diagnoses: bool = True
  257. ):
  258. if dataset == 'covid' and task in ['decompensation','phenotyping']:
  259. raise ValueError
  260. dataset = MIMICDataset if dataset == 'mimic' else CovidDataset
  261. trainset = dataset(
  262. task=task,
  263. split='train',
  264. modalities=modalities,
  265. non_empty=non_empty,
  266. with_diagnoses=with_diagnoses,
  267. root=root,
  268. cxr_root=cxr_root,
  269. timeframe=48,
  270. max_length=512,
  271. image_size=image_size,
  272. img_model_name=img_model_name
  273. )
  274. from sklearn.model_selection import train_test_split
  275. train_df = trainset.df
  276. train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)
  277. valset = dataset(
  278. task=task,
  279. split='train',
  280. modalities=modalities,
  281. non_empty=non_empty,
  282. with_diagnoses=with_diagnoses,
  283. root=root,
  284. cxr_root=cxr_root,
  285. timeframe=48,
  286. max_length=512,
  287. image_size=image_size,
  288. img_model_name=img_model_name
  289. )
  290. valset.df = val_df
  291. trainset.df = train_df
  292. testset = dataset(
  293. task=task,
  294. split='test',
  295. modalities=modalities,
  296. non_empty=non_empty,
  297. with_diagnoses=with_diagnoses,
  298. root=root,
  299. cxr_root=cxr_root,
  300. timeframe=48,
  301. max_length=512,
  302. image_size=image_size,
  303. img_model_name=img_model_name
  304. )
  305. trainloader = DataLoader(trainset, batch_size=batch_size, shuffle=True)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  306. valloader = DataLoader(valset, batch_size=batch_size)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  307. testloader = DataLoader(testset, batch_size=batch_size)#, num_workers=0, worker_init_fn=worker_init_fn, generator=g)
  308. return trainloader, valloader, testloader

data2.py at commit 4ae9b54, no license · at the source

Overview

Authors: Malte Tölle1,2,3, Mohamad Scharaf1, Samantha Fischer1,2,3, Christoph Reich1,2,3, Silav Zeid4,5,6, Christoph Dieterich1,2,3, Benjamin Meder1,2,3, Norbert Frey1,2,3, Philipp Wild4,5,6,7, Sandy Engelhardt1,2,3
  1. Department of Cardiology, Angiology and Pneumology, Heidelberg University Hospital, Heidelberg, Germany
  2. Informatics for Life Institute, Heidelberg, Germany
  3. DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany
  4. Preventive Cardiology and Preventive Medicine, Department of Cardiology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany
  5. Clinical Epidemiology and Systems Medicine, Center for Thrombosis and Hemostasis, University Medical Center Mainz, Johannes Gutenberg University Mainz, Mainz, Germany
  6. DZHK (German Centre for Cardiovascular Research), Partner Site Rhine-Main, Mainz, Germany
  7. Systems Medicine, Institute of Molecular Biology (IMB), Mainz, Germany
Journal: Communications medicine, volume 6, issue 1, article 474
Dates: received 13 March 2025; accepted 16 July 2026; published online 1 September 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s43856-026-01817-x · PMID 42680823 · PMCID PMC13534428 · OpenAlex W7204889893
Open access: gold, a free copy (OpenAlex)
Status: code verified (how the code is found and verified)
Categories: clinical / translational (subfield)
Methods: Statistics, Machine learning
Keywords: Medical imaging, Diagnosis
Topic: Multimodal Machine Learning Applications (Computer Vision and Pattern Recognition, Computer Science), according to OpenAlex
Funding: Deutsche Forschungsgemeinschaft (INST 35/1314-1 FUGG, INST35/1503‐1FUGG, INST 35/1314‐1, 35/1503-1, 35/1503-1 FUGG, 35/1314-1 FUGG, INST 35, INST 35/1503‐1, 35/1314-1)
Citations: not cited yet (Europe PMC); 63 references in the paper

Abstract

Background: A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure.

Methods: We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), simplifies data preprocessing and modeling by unifying clinical measurements, medications, X-ray images, and electrocardiography scans into a single visual representation.

Results: ViTiMM outperforms current state-of-the-art methods in predicting in-hospital mortality, phenotyping, and decompensation on two datasets, the MIMIC-IV and COVID Data for Shared Learning (CDSL) dataset.

Conclusions: We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code is publicly available at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 7 matches between paragraphs and lines of code.

Zenodo 20786553

License: CC-BY-4.0
State: the link answers, verified on 26 September 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: “Code availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (11 files), pandas (11 files), PyTorch (11 files), Matplotlib (9 files), seaborn (6 files), Pillow (5 files), Hugging Face Transformers (4 files), scikit-learn (3 files), SciPy (3 files), OpenCV (1 file)
Availability: 1 check, the latest on 26 September 2026: the link answers (HTTP 200)
  • 26 September 2026: the link answers (HTTP 200)
17 files
At the source:

Cardio-AI/ViTiMM

License: none: the authors keep all their rights
State: the link answers, verified on 26 September 2026
Evidence: files inventoried
Commit: 4ae9b544665e1a8b5be2ee81b1622c21ca4fe115, 5 August 2025
Languages: Jupyter (10), Python (6)
Size: 25 files, 16 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, environment (requirements.txt), 10 notebooks
Not found: license file, CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (11 files), pandas (11 files), PyTorch (11 files), Matplotlib (9 files), seaborn (6 files), Pillow (5 files), Hugging Face Transformers (4 files), scikit-learn (3 files), SciPy (3 files), OpenCV (1 file)
Availability: 1 check, the latest on 26 September 2026: the link answers
  • 26 September 2026: the link answers
17 files

Code availability

Following the FAIR criteria (findability, accessibility, interoperability, and reusability) in scientific research all code used in this study will be made publicly available. The code for cohort creation, data preprocessing, i.e., plot generation, training, and evaluation is open source available at https://doi.org/10.5281/zenodo.20786553 and at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 32 scripts, each with its path and the digest of its content;
  • 7 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability Statement

All data is publicly available from the MIMIC database22,34 on PhysioNet (MIMICIV: https://physionet.org/content/mimiciv/3.1, MIMIC-CXR-JPG: https://physionet.org/content/mimic-cxr-jpg/2.0.0, and MIMIC-IV-ECG: https://physionet.org/content/mimic-iv-ecg/1.0). The COVID Data for Shared Learning (CDSL) dataset is available at https://physionet.org/content/covid-data-shared-learning/1.0.0/. The code to extract the chest radiographs and corresponding clinical parameters can be found in the GitHub repository linked in the code availability section.

Following the FAIR criteria (findability, accessibility, interoperability, and reusability) in scientific research all code used in this study will be made publicly available. The code for cohort creation, data preprocessing, i.e., plot generation, training, and evaluation is open source available at https://doi.org/10.5281/zenodo.20786553 and at https://github.com/Cardio-AI/ViTiMM.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 3, 28 September 2026

  • Funding: added Deutsche Forschungsgemeinschaft: INST 35/1314-1 FUGG, INST35/1503‐1FUGG, INST 35/1314‐1, 35/1503-1, 35/1503-1 FUGG, 35/1314-1 FUGG, INST 35, INST 35/1503‐1, 35/1314-1

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 10 authors, 2 keywords, 40 references.

Cite

This paper

Tölle, M., Scharaf, M., Fischer, S., Reich, C., Zeid, S., Dieterich, C., Meder, B., Frey, N., Wild, P., & Engelhardt, S. (2026). Visual prompt engineering for multimodal and irregularly sampled medical data. Communications medicine, 6(1), 474. https://doi.org/10.1038/s43856-026-01817-x

BibTeX

@article{tolle2026visual,
author = {Tölle, Malte and Scharaf, Mohamad and Fischer, Samantha and Reich, Christoph and Zeid, Silav and Dieterich, Christoph and Meder, Benjamin and Frey, Norbert and Wild, Philipp and Engelhardt, Sandy},
title = {{Visual prompt engineering for multimodal and irregularly sampled medical data}},
journal = {Communications medicine},
year = {2026},
month = sep,
volume = {6},
number = {1},
pages = {474},
publisher = {Nature Publishing Group},
issn = {2730-664X},
doi = {10.1038/s43856-026-01817-x},
url = {https://doi.org/10.1038/s43856-026-01817-x},
pmid = {42680823},
pmcid = {PMC13534428}
}

RIS

TY - JOUR
AU - Tölle, Malte
AU - Scharaf, Mohamad
AU - Fischer, Samantha
AU - Reich, Christoph
AU - Zeid, Silav
AU - Dieterich, Christoph
AU - Meder, Benjamin
AU - Frey, Norbert
AU - Wild, Philipp
AU - Engelhardt, Sandy
TI - Visual prompt engineering for multimodal and irregularly sampled medical data
T2 - Communications medicine
J2 - Commun Med (Lond)
PY - 2026
DA - 2026/09/01
VL - 6
IS - 1
SP - 474
SN - 2730-664X
PB - Nature Publishing Group
DO - 10.1038/s43856-026-01817-x
UR - https://doi.org/10.1038/s43856-026-01817-x
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s43856-026-01817-x",
"type": "article-journal",
"title": "Visual prompt engineering for multimodal and irregularly sampled medical data",
"container-title": "Communications medicine",
"author": [
{
"family": "Tölle",
"given": "Malte"
},
{
"family": "Scharaf",
"given": "Mohamad"
},
{
"family": "Fischer",
"given": "Samantha"
},
{
"family": "Reich",
"given": "Christoph"
},
{
"family": "Zeid",
"given": "Silav"
},
{
"family": "Dieterich",
"given": "Christoph"
},
{
"family": "Meder",
"given": "Benjamin"
},
{
"family": "Frey",
"given": "Norbert"
},
{
"family": "Wild",
"given": "Philipp"
},
{
"family": "Engelhardt",
"given": "Sandy"
}
],
"container-title-short": "Commun Med (Lond)",
"volume": "6",
"issue": "1",
"page": "474",
"DOI": "10.1038/s43856-026-01817-x",
"PMID": "42680823",
"PMCID": "PMC13534428",
"ISSN": "2730-664X",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s43856-026-01817-x",
"language": "en",
"issued": {
"date-parts": [
[
2026,
9,
1
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1016/j.isci.2026.116135 [code]
Predicting ICU in-hospital mortality from text-encoded structured EHR data using adaptive transformer layer fusion.
Journal: iScience
In common: Hugging Face Transformers, PyTorch, seaborn, 4 other tools, physionet.org/content/mimiciv, clinical / translational, 3 references
[2] doi:10.3390/bioengineering12060578 [code]
PathCare: Integrating Clinical Pathway Information to Enable Healthcare Prediction at the Neuron Level.
Journal: Bioengineering (Basel, Switzerland)
In common: PyTorch, seaborn, scikit-learn, 3 other tools, physionet.org/content/mimiciv, clinical / translational, 3 references
[3] doi:10.1038/s41598-026-36296-6
Integrative multimodal hybrid data fusion for mortality prediction.
Journal: Scientific reports
In common: physionet.org/content/mimic-iv-ecg, physionet.org/content/mimiciv, clinical / translational, 2 references
[4] doi:10.1038/s41746-025-01773-1 [code]
Detection of epileptic spasms using foundational AI and smartphone videos.
Journal: NPJ digital medicine
In common: Hugging Face Transformers, OpenCV, Pillow, 6 other tools, clinical / translational, 2 references
[5] doi:10.1016/j.ebiom.2024.105047 [code]
Drop the shortcuts: image augmentation improves fairness and decreases AI detection of race and other demographics from medical images.
Journal: EBioMedicine
In common: OpenCV, Pillow, PyTorch, 5 other tools, 3 references
[6] doi:10.1016/j.patter.2025.101325 [code]
Tucano: Advancing neural text generation for Portuguese.
Journal: Patterns (New York, N.Y.)
In common: Hugging Face Transformers, Pillow, PyTorch, 5 other tools, 2 references
[7] doi:10.1038/s41746-025-02018-x [code]
GlioSurv: interpretable transformer for multimodal, individualized survival prediction in diffuse glioma.
Journal: NPJ digital medicine
In common: Hugging Face Transformers, PyTorch, pandas, 3 other tools, clinical / translational, 3 references
[8] doi:10.1097/js9.0000000000003143 [code]
Multimodal deep learning to predict postoperative major adverse cardiac and cerebrovascular events after noncardiac surgery.
Journal: International journal of surgery (London, England)
In common: PyTorch, seaborn, scikit-learn, 4 other tools, clinical / translational, 3 references
[9] doi:10.1371/journal.pone.0344600 [code]
Robust disease prognosis via diagnostic knowledge preservation: A sequential learning approach.
Journal: PloS one
In common: OpenCV, Pillow, PyTorch, 5 other tools, clinical / translational, 2 references
[10] doi:10.1093/braincomms/fcag253 [code]
Disease detection and classification in temporal lobe epilepsy: step-wise versus simultaneous AI decision models in a multisite neuroimaging study.
Journal: Brain communications
In common: Hugging Face Transformers, OpenCV, Pillow, 7 other tools, clinical / translational

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Add the badge to the code's README

OSCR: the badge of this paper

If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.

Markdown
<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x.svg)](https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x)
<!-- oscr-badge:end -->
reStructuredText
.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x.svg
   :target: https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x
   :alt: OSCR

.. oscr-badge:end
HTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x"><img src="https://openscicode.org/badge/doi/10.1038/s43856-026-01817-x.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

On GitHub, open the README, choose Edit (the pencil), paste the snippet under the title, then "Commit changes".

If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.