OSCR

Modality-agnostic decoding of vision and language from fMRI.

Code ↔ Paper

17 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 17 matches
  1. [1] § Results › Modality-agnostic decoders ↔ notebooks/modality_agnostic_decoding.ipynb, lines 485–526 · score 0.80 · dual stream, late fusion, early fusion, single stream, modality agnostic, ANOVA
  2. [2] § Methods › Modality-agnostic decoders ↔ data.py, lines 182–248 · score 0.76 · vision features, Llama2, mistral, mixtral, ResNet, SigLip
  3. [3] § Results › Modality-agnostic decoders ↔ notebooks/notebook_utils.py, lines 204–262 · score 0.69 · Dino giant, GPT2 large, language models, vision features, imagebind, multimodal
  4. [4] § Methods › Modality-agnostic decoders ↔ analyses/decoding/ridge_regression_decoding.py, lines 45–187 · score 0.69 · ridge regression, predict latent, fMRI beta, decoder training, fitting, models
  5. [5] § Methods › fMRI preprocessing ↔ preprocessing/fmri_preprocessing.py, lines 83–124 · score 0.68 · anatomical scan, SPM, nipype, slice, coregistered, realignment
  6. [6] § Appendix 2 › Feature extraction details ↔ feature_extraction/extract_bridgetower_features.py, lines 45–51 · score 0.67 · bridgetower large itm, mlm itc, Model
  7. [7] § Appendix 1 › Dataset quality metrics › Head motion ↔ analyses/visualization/plot_dataset_quality_stats.py, lines 20–108 · score 0.64 · framewise displacement, realignment parameters, absolute, sum, head, motion
  8. [8] § Results › Modality-invariant regions ↔ analyses/decoding/searchlight/searchlight_cluster_manual_corrections.py, lines 13–38 · score 0.62 · inferior parietal, inferior temporal, middle temporal, cluster, searchlight
  9. [9] § Appendix 2 › Feature extraction details ↔ notebooks/modality_agnostic_decoding.ipynb, lines 485–526 · score 0.62 · dual stream, single stream, multimodal models, BLIP2, Paligemma2, Flava
  10. [10] § Appendix 1 › Intersession alignment ↔ notebooks/intersession_alignment.ipynb, lines 24–64 · score 0.62 · normalized mutual information, alignment, Intersession, coregistered, preprocessing
  11. [11] § Appendix 2 › Feature extraction details ↔ feature_extraction/extract_siglip_features.py, lines 37–42 · score 0.61 · siglip so400m patch14, Model
  12. [12] § Methods › fMRI preprocessing ↔ preprocessing/make_spm_design_job_mat.py, lines 178–310 · score 0.57 · gray matter masks, slice, coregistered, Preprocessing, realignment, scan
  13. [13] § Appendix 1 › Intersession alignment ↔ preprocessing/fmri_preprocessing.py, lines 83–124 · score 0.56 · anatomical scan, SPM, pipeline, coregistered, preprocessing
  14. [14] § Appendix 1 › Dataset quality metrics › Head motion ↔ analyses/visualization/plot_dataset_quality_stats.py, lines 20–108 · score 0.56 · realignment parameters, pitch, roll, yaw, displacement, head
  15. [15] § Methods › Modality-agnostic decoders ↔ notebooks/modality_agnostic_decoding.ipynb, lines 702–764 · score 0.55 · cross modal, pairwise accuracy, distance, cosine, latent, predictions
  16. [16] § Methods › fMRI preprocessing ↔ preprocessing/make_spm_design_job_mat.py, lines 22–55 · score 0.55 · train captions, train images, blank, fixations, preprocessing, imagery
  17. [17] § Methods › Modality-agnostic decoders ↔ notebooks/zero_shot_cross_modal_decoding.ipynb, lines 112–154 · score 0.54 · cross modal decoding, language model, pairwise, accuracy, captions, agnostic

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 819 lines · 29 KB · MIT · 3 matches

  1. # %%
  2. import os
  3. import numpy as np
  4. import matplotlib.pyplot as plt
  5. import pandas as pd
  6. import seaborn as sns
  7. from sklearn.preprocessing import StandardScaler
  8. from utils import RESULTS_DIR, SUBJECTS, COCO_IMAGES_DIR, STIM_INFO_PATH
  9. from data import MODALITY_AGNOSTIC, TRAINING_MODES, CAPTION, IMAGE, DEFAULT_VISION_FEATURES
  10. from eval import ACC_MODALITY_AGNOSTIC, ACC_CAPTIONS, ACC_IMAGES, ACC_IMAGERY, ACC_IMAGERY_WHOLE_TEST, get_distance_matrix, dist_mat_to_pairwise_acc
  11. from notebook_utils import create_result_graph, plot_metric_catplot, load_results_data, ACC_MEAN, METRICS_ERROR_ANALYSIS, get_data_default_feats, calc_model_feat_order, FEATS_MULTIMODAL, DEFAULT_FEAT_OPTIONS
  12. from matplotlib import patches
  14. sns.set_style("ticks", {'axes.grid': True})
  15. sns.set_theme(font_scale=1.6)
  17. from data import DEFAULT_FEATURES, DEFAULT_VISION_FEATURES, DEFAULT_LANG_FEATURES, TRAINING_MODES, MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS
  20. # %%
  21. MODELS = [
  22. "random-imagebind", "vit-b-16", "vit-l-16", "vit-h-14", "resnet-18", "resnet-50", "resnet-152", "dino-base", "dino-large", "dino-giant",
  23. "bert-base-uncased", "bert-large-uncased", "llama2-7b", "llama2-13b", "mistral-7b", "mixtral-8x7b", "gpt2-small", "gpt2-medium", "gpt2-large", "gpt2-xl",
  24. "visualbert", "bridgetower", "vilt", "siglip", "paligemma2", "clip", "flava", "blip2", "imagebind"
  25. ]
  27. # for model in MODELS:
  28. # print(model, end=" ")
  31. # %%
  32. all_data = load_results_data(MODELS, recompute_acc_scores=False)
  34. all_data = all_data[all_data["mask"] == "whole_brain"]
  35. all_data_vol = all_data[all_data.surface == False].copy()
  37. all_data = all_data[all_data.surface == True].copy()
  39. multimodal_models = all_data[all_data.features.isin(FEATS_MULTIMODAL)].model.unique().tolist()
  40. vision_models = [m for m in all_data[all_data.features == "vision"].model.unique() if len(all_data[all_data.model == m].features.unique()) == 1]
  43. # %%
  44. data_default_feats = get_data_default_feats(all_data)
  46. # %%
  47. dd = data_default_feats.copy()
  49. dd = dd[dd.subject == 'sub-01']
  50. dd = dd[dd.model == 'dino-base']
  51. dd = dd[dd.metric.isin(['pairwise_acc_images', 'pairwise_acc_captions'])]
  52. dd = dd[dd.training_mode == 'agnostic']
  53. display(dd)
  56. # %%
  57. dd = data_default_feats.copy()
  59. dd = dd[dd.subject == 'sub-01']
  60. dd = dd[dd.model == 'gpt2-small']
  61. dd = dd[dd.metric.isin(['pairwise_acc_images', 'pairwise_acc_captions'])]
  62. dd = dd[dd.training_mode == 'agnostic']
  63. display(dd)
  66. # %% [markdown]
  67. # ## Feature comparison for multimodal models
  69. # %%
  70. data_mod_agnostic_train = all_data[(all_data.metric == ACC_MEAN) & (all_data.training_mode == MODALITY_AGNOSTIC)]
  73. with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
  74. # grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(count=('value', 'size'), pairwise_acc=('value', 'mean'), pairwise_acc_stddev=('value', 'std')).reset_index()
  75. grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(pairwise_acc=('value', 'mean')).reset_index()
  77. grouped = grouped[grouped.model.isin(multimodal_models)]
  78. grouped = grouped[~grouped.model.isin(["random-imagebind"])]
  80. display(grouped)
  82. # print(grouped.to_markdown())
  84. grouped = grouped.replace("n_a", "")
  85. # grouped = grouped[grouped.model.isin(multimodal_models)]
  86. # del grouped["count"]
  87. print(grouped.to_latex(index=False, escape=True, float_format="%.3f"))
  91. # %%
  92. # data_default_vision_feats = all_data.copy()
  93. # for model in all_data.model.unique():
  94. # default_vision_feats = DEFAULT_VISION_FEATURES[model]
  95. # data_default_vision_feats = data_default_vision_feats[((data_default_vision_feats.model == model) & (data_default_vision_feats.vision_features == default_vision_feats)) | (data_default_vision_feats.model != model)]
  96. #
  97. # feat_legend = {"avg": "average over tokens from vision and language streams", "fused_cls": "fused [CLS] token", "fused_mean": "average over fused tokens"}
  98. # feat_order = ["fused_cls", "fused_mean", "avg"]
  99. #
  100. # feat_order_long = [feat_legend[feat] for feat in feat_order]
  101. #
  102. # data_to_plot = data_default_vision_feats.copy()
  103. #
  104. # data_to_plot = data_to_plot[data_to_plot.model.isin(multimodal_models)]
  105. # models_excluded = ['random-imagebind', 'siglip', 'clip', 'imagebind']
  106. # data_to_plot = data_to_plot[~data_to_plot.model.isin(models_excluded)]
  107. #
  108. # data_to_plot["features"] = data_to_plot.features.replace(feat_legend)
  109. #
  110. # model_feat_order = calc_model_feat_order(data_to_plot, MODELS, feat_options=feat_order)
  111. #
  112. # metrics_order = [ACC_MEAN]
  113. # figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=feat_order_long, ylim=(0.5, 1),
  114. # legend_bbox=(0.06,1.01), height=5, legend_title="Modality-agnostic decoders projecting into multimodal model features based on ", verify_num_datapoints=False, plot_modality_specific=False)
  115. # plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_multimodal_models.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0, dpi=300)
  117. # %% [markdown]
  118. # ## Feature comparison for vision models
  120. # %%
  121. data_mod_agnostic_train = all_data[(all_data.metric == ACC_MEAN) & (all_data.training_mode == MODALITY_AGNOSTIC)]
  123. with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
  124. # grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(count=('value', 'size'), pairwise_acc=('value', 'mean'), pairwise_acc_stddev=('value', 'std')).reset_index()
  125. grouped = data_mod_agnostic_train.groupby(["model", "vision_features"]).agg(pairwise_acc=('value', 'mean')).reset_index()
  127. grouped = grouped[grouped.model.isin(vision_models)]
  128. models_excluded = ["resnet-18", "resnet-50", "resnet-152"]
  129. grouped = grouped[~grouped.model.isin(models_excluded)]
  131. display(grouped)
  133. # print(grouped.to_markdown())
  135. grouped = grouped.replace("n_a", "")
  136. # grouped = grouped[grouped.model.isin(multimodal_models)]
  137. # del grouped["count"]
  138. print(grouped.to_latex(index=False, escape=True, float_format="%.3f"))
  142. # %%
  143. # data_default_vision_feats = all_data.copy()
  144. #
  145. # feat_legend = {"vision_features_cls": "[CLS] token", "vision_features_mean": "average over patches"}
  146. # feat_order = ["vision_features_cls", "vision_features_mean"]
  147. #
  148. # feat_order_long = [feat_legend[feat] for feat in feat_order]
  149. #
  150. # data_to_plot = data_default_vision_feats.copy()
  151. #
  152. # data_to_plot = data_to_plot[data_to_plot.model.isin(vision_models)]
  153. # models_excluded = ["resnet-18", "resnet-50", "resnet-152"]
  154. # data_to_plot = data_to_plot[~data_to_plot.model.isin(models_excluded)]
  155. #
  156. # data_to_plot["vision_features"] = data_to_plot.vision_features.replace(feat_legend)
  157. #
  158. # model_feat_order = calc_model_feat_order(data_to_plot, MODELS)
  159. # metrics_order = [ACC_MEAN]
  160. # figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_variable='vision_features', hue_order=feat_order_long, ylim=(0.5, 1),
  161. # legend_bbox=(0.06,1.01), height=5, legend_title="Modality-agnostic decoders projecting into vision model features based on ", verify_num_datapoints=False, plot_modality_specific=False)
  162. # plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_vision_models.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0, dpi=300)
  164. # %% [markdown]
  165. # ## Modality-agnostic decoding vs. modality-specific decoding
  167. # %% [markdown]
  168. # ### Model features comparison
  170. # %%
  171. model_order = ['random-imagebind']
  172. model_feat_order = ['random-imagebind_avg']
  173. for features in DEFAULT_FEAT_OPTIONS:
  174. print('\nmodel feat type: ', features)
  175. dp = data_default_feats.copy()
  176. dp = dp[dp.features == features]
  177. dp = dp[dp.training_mode == MODALITY_AGNOSTIC]
  179. dp = dp[dp.metric == ACC_MEAN]
  180. for model in dp.model.unique():
  181. if len(dp[dp.model == model]) != len(SUBJECTS):
  182. print(f"unexpected number of datapoints for {model}: {len(dp[dp.model == model])}")
  183. # scores = dp.groupby("model").value.mean().sort_values()
  184. scores = dp.groupby("model_feat").value.mean().sort_values()
  185. if len(scores) > 0:
  186. print(scores)
  187. model_order.extend([mf.split('_')[0] for mf in scores.index.values])
  188. model_feat_order.extend(scores.index.values)
  190. model_order
  191. # model_feat_order
  193. # %%
  194. FEAT_ORDER = ["vision models", "language models", "multimodal models"]
  196. data_to_plot = data_default_feats.copy()
  198. data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
  199. data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
  201. # model_feat_order = calc_model_feat_order(data_to_plot)
  203. metrics_order = [ACC_MEAN]
  204. HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
  206. figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=0.85,
  207. legend_bbox=(0.68, 1.05), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
  209. figure.axes[0][0].add_patch(
  210. patches.Rectangle(
  211. (-0.5, 0.1), # (x,y)
  212. 1, # width
  213. 0.9, # height
  214. linewidth=0,
  215. clip_on=False,
  216. alpha=0.05,
  217. color='0.2'
  218. )
  219. )
  220. figure.axes[0][0].add_patch(
  221. patches.Rectangle(
  222. (0.5, 0.1), # (x,y)
  223. 9, # width
  224. 0.9, # height
  225. linewidth=0,
  226. clip_on=False,
  227. alpha=0.15,
  228. color='0.2'
  229. )
  230. )
  231. figure.axes[0][0].add_patch(
  232. patches.Rectangle(
  233. (9.5, 0.1), # (x,y)
  234. 10, # width
  235. 0.9, # height
  236. linewidth=0,
  237. clip_on=False,
  238. alpha=0.05,
  239. color='0.2'
  240. )
  241. )
  242. figure.axes[0][0].add_patch(
  243. patches.Rectangle(
  244. (19.5, 0.1), # (x,y)
  245. 9, # width
  246. 0.9, # height
  247. linewidth=0,
  248. clip_on=False,
  249. alpha=0.15,
  250. color='0.2'
  251. )
  252. )
  253. figure.axes[0][0].text(4, 0.11, 'Vision Models', fontsize=12)
  254. figure.axes[0][0].text(14, 0.11, 'Language Models', fontsize=12)
  255. figure.axes[0][0].text(23, 0.11, 'Multimodal Models', fontsize=12)
  257. plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.3, dpi=300)
  260. # %%
  261. FEAT_ORDER = ["vision models", "language models", "multimodal models"]
  263. data_to_plot = data_default_feats.copy()
  265. data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
  266. data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
  268. # model_feat_order = calc_model_feat_order(data_to_plot)
  270. metrics_order = [ACC_IMAGES, ACC_CAPTIONS]
  271. HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
  273. figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=1.07, row_title_loc='left',
  274. legend_bbox=(0.68, 1.03), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
  276. height=0.5
  277. figure.axes[0][0].add_patch(
  278. patches.Rectangle(
  279. (-0.5, 0.5), # (x,y)
  280. 1, # width
  281. height, # height
  282. linewidth=0,
  283. clip_on=False,
  284. alpha=0.05,
  285. color='0.2'
  286. )
  287. )
  288. figure.axes[0][0].add_patch(
  289. patches.Rectangle(
  290. (0.5, 0.5), # (x,y)
  291. 9, # width
  292. height, # height
  293. linewidth=0,
  294. clip_on=False,
  295. alpha=0.15,
  296. color='0.2'
  297. )
  298. )
  299. figure.axes[0][0].add_patch(
  300. patches.Rectangle(
  301. (9.5, 0.5), # (x,y)
  302. 10, # width
  303. height, # height
  304. linewidth=0,
  305. clip_on=False,
  306. alpha=0.05,
  307. color='0.2'
  308. )
  309. )
  310. figure.axes[0][0].add_patch(
  311. patches.Rectangle(
  312. (19.5, 0.5), # (x,y)
  313. 9, # width
  314. height, # height
  315. linewidth=0,
  316. clip_on=False,
  317. alpha=0.15,
  318. color='0.2'
  319. )
  320. )
  322. height=0.8
  323. figure.axes[1][0].add_patch(
  324. patches.Rectangle(
  325. (-0.5, 0.2), # (x,y)
  326. 1, # width
  327. height, # height
  328. linewidth=0,
  329. clip_on=False,
  330. alpha=0.05,
  331. color='0.2'
  332. )
  333. )
  334. figure.axes[1][0].add_patch(
  335. patches.Rectangle(
  336. (0.5, 0.2), # (x,y)
  337. 9, # width
  338. height, # height
  339. linewidth=0,
  340. clip_on=False,
  341. alpha=0.15,
  342. color='0.2'
  343. )
  344. )
  345. figure.axes[1][0].add_patch(
  346. patches.Rectangle(
  347. (9.5, 0.2), # (x,y)
  348. 10, # width
  349. height, # height
  350. linewidth=0,
  351. clip_on=False,
  352. alpha=0.05,
  353. color='0.2'
  354. )
  355. )
  356. figure.axes[1][0].add_patch(
  357. patches.Rectangle(
  358. (19.5, 0.2), # (x,y)
  359. 9, # width
  360. height, # height
  361. linewidth=0,
  362. clip_on=False,
  363. alpha=0.15,
  364. color='0.2'
  365. )
  366. )
  367. figure.axes[1][0].text(4, 0.21, 'Vision Models', fontsize=12)
  368. figure.axes[1][0].text(14, 0.21, 'Language Models', fontsize=12)
  369. figure.axes[1][0].text(23, 0.21, 'Multimodal Models', fontsize=12)
  371. plt.subplots_adjust(hspace=0.3)
  373. plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_split_by_modality.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.5, dpi=300)
  375. # %% [markdown]
  376. # # Statistical tests
  378. # %%
  379. from scipy.stats import ttest_ind
  380. import statsmodels.formula.api as smf
  383. # %% [markdown]
  384. # ## Mod agnostic vs. mod specific
  386. # %%
  387. data_default_feats_copy = data_default_feats.copy()
  389. for metric, comparison_training_mode in zip(['pairwise_acc_mean', 'pairwise_acc_mean'], ["images", "captions"]):
  390. print(f"{metric} | comparison with {comparison_training_mode}")
  391. data_filtered = data_default_feats_copy[data_default_feats_copy.metric == metric]
  392. data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
  393. assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
  395. data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
  396. assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
  398. print('mod agno: ', data_mod_agno.value.mean())
  399. print('mod spec: ', data_mod_spec.value.mean())
  401. # display(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
  403. # display(data_mod_agno)
  404. data_anova = pd.concat([data_mod_agno, data_mod_spec])
  405. data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
  407. mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
  409. print("=" * 50 + "\nANOVA\n" + "=" * 50)
  410. print(mod.summary())
  411. print('pvalues:\n', mod.pvalues)
  412. print('\n')
  415. # %% [markdown]
  416. # ## Mod agnostic vs. mod specific - imagery decoding
  418. # %%
  419. # from scipy.stats import ttest_ind
  420. # from pymer4.models import Lmer
  422. # data_ttest = data_default_feats.copy()
  424. # for metric in [ACC_IMAGERY_WHOLE_TEST, ACC_IMAGERY]:
  425. # for comparison_training_mode in ["images", "captions"]:
  426. # print(f"{metric} | comparison with {comparison_training_mode}")
  427. # data_filtered = data_ttest[data_ttest.metric == metric]
  428. # data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
  429. # # data_mod_agno = data_mod_agno[data_mod_agno.model == "imagebind"]
  430. # assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
  432. # data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
  433. # assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
  435. # print('mod agno: ', data_mod_agno.value.mean())
  436. # print('mod spec: ', data_mod_spec.value.mean())
  437. # print(len(data_mod_agno.value.values))
  438. # print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
  440. # # display(data_mod_agno)
  441. # data_glm = pd.concat([data_mod_agno, data_mod_spec])
  442. # # display(data_glm)
  443. # mod = Lmer('value ~ training_mode + (1 | subject)', data=data_glm)
  444. # # mod = Lmer('value ~ training_mode + (1 | model) + (1 | subject)', data=data_glm)
  446. # print("=" * 50 + "\nGLM\n" + "=" * 50)
  447. # fitted = mod.fit()
  448. # print(fitted)
  449. # print(fitted[["Estimate", "SE", "Sig"]])
  450. # print('\n')
  452. # %%
  453. data_ttest = data_default_feats.copy()
  455. for metric in [ACC_IMAGERY_WHOLE_TEST]:
  456. for comparison_training_mode in ["images", "captions"]:
  457. print(f"{metric} | comparison with {comparison_training_mode}")
  458. data_filtered = data_ttest[data_ttest.metric == metric]
  460. data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
  461. assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
  463. data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
  464. assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
  466. print('mod agno: ', data_mod_agno.value.mean())
  467. print('mod spec: ', data_mod_spec.value.mean())
  468. print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
  470. # display(data_mod_agno)
  471. data_anova = pd.concat([data_mod_agno, data_mod_spec])
  472. data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
  474. # print(data_anova)
  476. mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
  478. print("=" * 50 + "\nANOVA\n" + "=" * 50)
  479. print(mod.summary())
  480. print('pvalues:\n', mod.pvalues)
  482. print('\n')
  485. # %% [markdown]
  486. # ## Effect of multimodal model architecture
  488. # %%
  489. data_ttest = data_default_feats.copy()
  491. for metric in [ACC_MEAN]:
  492. print(f"{metric}")
  493. data_filtered = data_ttest[data_ttest.metric == metric]
  494. data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
  495. data_mod_agno = data_mod_agno[data_mod_agno.model != "random-imagebind"]
  497. assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
  499. data_dual_stream = data_mod_agno[data_mod_agno.model.isin(['clip', 'imagebind', 'siglip'])].copy()
  500. data_dual_stream['multimodal_type'] = 'dual_stream'
  501. data_single_stream_early_fusion = data_mod_agno[data_mod_agno.model.isin(['visualbert', 'vilt', 'paligemma2'])].copy()
  502. data_single_stream_early_fusion['multimodal_type'] = 'single_stream_early_fusion'
  503. data_single_stream_late_fusion = data_mod_agno[data_mod_agno.model.isin(['bridgetower', 'flava', 'blip2'])].copy()
  504. data_single_stream_late_fusion['multimodal_type'] = 'single_stream_late_fusion'
  507. print('data_dual_stream: ', data_dual_stream.value.mean())
  508. print(len(data_dual_stream.value.values))
  509. print('data_single_stream_early_fusion: ', data_single_stream_early_fusion.value.mean())
  510. print(len(data_single_stream_early_fusion.value.values))
  511. print('data_single_stream_late_fusion: ', data_single_stream_late_fusion.value.mean())
  512. print(len(data_single_stream_late_fusion.value.values))
  514. # print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
  516. data_anova = pd.concat([data_dual_stream, data_single_stream_early_fusion, data_single_stream_late_fusion])
  517. data_anova = data_anova[['subject', 'multimodal_type', 'value']]
  519. mod = smf.mixedlm("value ~ multimodal_type", data_anova, groups=data_anova["subject"]).fit()
  521. print("=" * 50 + "\nANOVA\n" + "=" * 50)
  522. # print(mod.random_effects)
  523. print(mod.summary())
  524. print('pvalues:\n', mod.pvalues)
  526. print('\n')
  529. # %% [markdown]
  530. # ## Comparing cross-decoding for images and captions
  532. # %%
  533. data_filtered = data_default_feats.copy()
  535. # for metric, comparison_training_mode in zip(['pairwise_acc_mean', 'pairwise_acc_mean'], ["images", "captions"]):
  536. # print(f"{metric} | comparison with {comparison_training_mode}")
  537. # data_filtered = data_default_feats_copy[data_default_feats_copy.metric == 'pairwise_acc_mean']
  538. data_cross_images = data_filtered[(data_filtered.training_mode == "captions") & (data_filtered.metric == 'pairwise_acc_images')]
  539. assert len(data_cross_images) == len(SUBJECTS) * len(data_cross_images.model.unique())
  541. data_cross_captions = data_filtered[(data_filtered.training_mode == "images") & (data_filtered.metric == 'pairwise_acc_captions')]
  542. assert len(data_cross_captions) == len(SUBJECTS) * len(data_cross_captions.model.unique())
  544. print('data_cross_images: ', data_cross_images.value.mean())
  545. print('data_cross_captions: ', data_cross_captions.value.mean())
  547. # display(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
  549. # # display(data_mod_agno)
  550. data_anova = pd.concat([data_cross_images, data_cross_captions])
  551. data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
  553. mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
  555. print("=" * 50 + "\nANOVA\n" + "=" * 50)
  556. print(mod.summary())
  557. print('pvalues:\n', mod.pvalues)
  558. print('\n')
  561. # %% [markdown]
  562. # # Imagery decoding feat comparison
  564. # %%
  565. FEAT_ORDER = ["vision models", "language models", "multimodal models"]
  567. data_to_plot = data_default_feats.copy()
  569. data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
  570. data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
  572. # model_feat_order = calc_model_feat_order(data_to_plot)
  574. metrics_order = [ACC_IMAGERY, ACC_IMAGERY_WHOLE_TEST]
  575. HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
  577. figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=0.85,
  578. legend_bbox=(0.68, 1.07), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
  580. figure.axes[1][0].add_patch(
  581. patches.Rectangle(
  582. (-0.5, 0.2), # (x,y)
  583. 1, # width
  584. 1.4, # height
  585. linewidth=0,
  586. clip_on=False,
  587. alpha=0.05,
  588. color='0.2'
  589. )
  590. )
  591. figure.axes[1][0].add_patch(
  592. patches.Rectangle(
  593. (0.5, 0.2), # (x,y)
  594. 9, # width
  595. 1.4, # height
  596. linewidth=0,
  597. clip_on=False,
  598. alpha=0.15,
  599. color='0.2'
  600. )
  601. )
  602. figure.axes[1][0].add_patch(
  603. patches.Rectangle(
  604. (9.5, 0.2), # (x,y)
  605. 10, # width
  606. 1.4, # height
  607. linewidth=0,
  608. clip_on=False,
  609. alpha=0.05,
  610. color='0.2'
  611. )
  612. )
  613. figure.axes[1][0].add_patch(
  614. patches.Rectangle(
  615. (19.5, 0.2), # (x,y)
  616. 9, # width
  617. 1.4, # height
  618. linewidth=0,
  619. clip_on=False,
  620. alpha=0.15,
  621. color='0.2'
  622. )
  623. )
  624. figure.axes[1][0].text(4, 0.21, 'Vision Models', fontsize=12)
  625. figure.axes[1][0].text(14, 0.21, 'Language Models', fontsize=12)
  626. figure.axes[1][0].text(23, 0.21, 'Multimodal Models', fontsize=12)
  628. plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_imagery.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.3, dpi=300)
  631. # %% [markdown]
  632. # ## Per-subject results
  634. # %%
  635. FEAT_ORDER = ["vision", "lang", "matched"]
  636. FEAT_PALETTE = sns.color_palette('Set2')[:3]
  638. def create_result_graph_all_subjs(data, order=model_feat_order, metrics=[ACC_CAPTIONS, ACC_IMAGES], hue_variable="training_mode", hue_order=FEAT_ORDER, ylim=None,
  639. legend_title="Modality-agnostic decoders based on features from", palette=FEAT_PALETTE, dodge=False, noise_ceilings=None, plot_modality_specific=True,
  640. row_variable="metric", col_variable=None, legend_bbox=None):
  641. for mode in TRAINING_MODES:
  642. data_mode = data[data.training_mode == mode]
  643. for x_variable_value in order:
  644. length = len(data_mode[(data_mode["model_feat"] == x_variable_value) & (data_mode.metric == metrics[0])])
  645. expected_num_datapoints = len(SUBJECTS)
  646. # if hue_variable != "features":
  647. # expected_num_datapoints *= len(data[hue_variable].unique())
  648. if (length > 0) and (length != expected_num_datapoints):
  649. message = f"unexpected number of datapoints: {length} (expected: {expected_num_datapoints}) (model_feat: {x_variable_value} {mode}"
  650. print(f"Warning: {message}")
  652. catplot_g, data_plotted, lgd = plot_metric_catplot(data, order=model_feat_order, metrics=metrics, x_variable="model_feat", legend_title=legend_title, aspect=2, legend_bbox=legend_bbox, rotation=90, cut_labels=False,
  653. hue_variable=hue_variable, row_variable=row_variable, col_variable=col_variable, hue_order=hue_order, palette=palette, ylim=ylim, noise_ceilings=noise_ceilings)
  655. for i in range(len(data.subject.unique())):
  656. catplot_g.axes[i,0].set_title(f"Subject {i+1} | Decoding of captions", fontsize=25)
  657. catplot_g.axes[i,1].set_title(f"Subject {i+1} | Decoding of images", fontsize=25)
  659. catplot_g.axes[i,0].set_ylabel('pairwise accuracy')
  661. return catplot_g, lgd
  663. data_to_plot = data_default_feats.copy()
  665. data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
  666. data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
  668. model_feat_order = calc_model_feat_order(data_to_plot, MODELS)
  669. HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
  672. figure, lgd = create_result_graph_all_subjs(data_to_plot, model_feat_order, metrics=[ACC_CAPTIONS, ACC_IMAGES], hue_order=HUE_ORDER, ylim=(0.5, 1), row_variable="subject", col_variable="metric", legend_bbox=(0.58, 1.07))
  673. plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_per_subject.png"), dpi=300, bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0)
  675. # %%
  678. # %%
  681. # %% [markdown]
  682. # #
  684. # %% [markdown]
  685. # # Error analysis
  687. # %%
  688. models = ['imagebind']
  690. data = load_results_data(models, metrics=METRICS_ERROR_ANALYSIS)
  692. # metric = ACC_IMAGES
  694. all_data = data[data.model.isin(MODELS)]
  696. all_data = all_data[all_data["mask"] == "whole_brain"]
  698. all_data = all_data[all_data.surface == True]
  700. data_default_feats_err_analysis = get_data_default_feats(all_data)
  702. # %%
  703. def all_pairwise_accuracy_scores(latents, predictions, stim_types=None, metric="cosine", standardize=True):
  704. results = dict()
  705. results['comp_mats'] = dict()
  707. for modality, acc_metric_name in zip([CAPTION, IMAGE], [ACC_CAPTIONS, ACC_IMAGES]):
  708. preds_mod = predictions[stim_types == modality].copy()
  709. latents_mod = latents[stim_types == modality]
  710. if standardize:
  711. preds_mod = StandardScaler().fit_transform(preds_mod)
  713. dist_mat = get_distance_matrix(preds_mod, latents_mod, metric)
  714. diag = dist_mat.diagonal().reshape(-1, 1)
  715. comp_mat = diag < dist_mat
  717. score = dist_mat_to_pairwise_acc(dist_mat)
  719. results[acc_metric_name] = score
  720. results['comp_mats'][acc_metric_name] = (~comp_mat).astype(int)
  722. if standardize:
  723. predictions = StandardScaler().fit_transform(predictions)
  725. dist_mat = get_distance_matrix(predictions, latents, metric)
  727. mod_agnostic_accs = []
  728. for modality in [CAPTION, IMAGE]:
  729. dist_mat_within_mod = dist_mat[stim_types == modality][:, stim_types == modality]
  730. dist_mat_cross_modal = dist_mat[stim_types == modality][:, stim_types != modality]
  731. dist_mat_min = np.min((dist_mat_within_mod, dist_mat_cross_modal), axis=0)
  732. diag = dist_mat_min.diagonal().reshape(-1, 1)
  733. comp_mat = diag < dist_mat_min
  734. score = dist_mat_to_pairwise_acc(dist_mat_min)
  735. # scores = np.mean(comp_mat, axis=0)
  736. mod_agnostic_accs.append(score)
  737. results[f"pairwise_acc_mod_agnostic_{modality}s"] = score
  738. results['comp_mats'][f"pairwise_acc_mod_agnostic_{modality}s"] = (~comp_mat).astype(int)
  740. results[ACC_MODALITY_AGNOSTIC] = np.mean(mod_agnostic_accs)
  742. return results
  744. metric = ACC_IMAGES #TODO which metric?
  745. training_type = MODALITY_AGNOSTIC
  747. all_subj_mats = []
  748. for subject in SUBJECTS:
  749. df = data_default_feats_err_analysis.copy()
  750. df_subj = df[(df.training_mode == training_type) & (df.subject == subject)]
  751. predictions = df_subj[df_subj.metric == "predictions"].value.item()
  752. latents = df_subj[df_subj.metric == "latents"].value.item()
  753. stimulus_ids = df_subj[df_subj.metric == "stimulus_ids"].value.item()
  754. stimulus_types = df_subj[df_subj.metric == "stimulus_types"].value.item()
  756. results = all_pairwise_accuracy_scores(latents, predictions, stimulus_types)
  757. print(results[ACC_IMAGES])
  758. mat = results['comp_mats'][metric]
  759. all_subj_mats.append(mat)
  761. mat = np.sum(all_subj_mats, axis=0)
  762. mat = mat - np.diag(np.diag(mat))
  764. df_err = pd.DataFrame(mat, index=stimulus_ids[:70], columns=stimulus_ids[:70])
  766. # %%
  767. # df_err = df_err.map(lambda x: 0 if x < 3 else x)
  768. # with pd.option_context('display.max_rows', None, 'display.max_columns', None):
  769. # display(df_err)
  771. # %%
  772. stimuli_info = pd.read_csv(STIM_INFO_PATH, index_col=0)
  773. stimuli_info = stimuli_info[stimuli_info.used == True]
  774. captions_dict = stimuli_info.caption.to_dict()
  776. img_paths = stimuli_info.img_path.to_dict()
  778. img_paths = {idx: os.path.join(COCO_IMAGES_DIR, path) for idx, path in img_paths.items()}
  780. from PIL import Image
  782. def display_stimuli(coco_ids):
  783. imgs = [Image.open(img_paths[img_id]).convert('RGB') for img_id in coco_ids]
  784. min_height = np.min([np.array(im).shape[0] for im in imgs])
  785. stacked = np.hstack([np.array(im)[:min_height,:,:] for im in imgs])
  786. img = Image.fromarray(stacked)
  787. # print(min_height)
  788. captions = [captions_dict[coco_id] for coco_id in coco_ids]
  789. for cap in captions:
  790. print(cap)
  791. display(img)
  793. # display_stimuli([16764, 79642])
  796. # %%
  797. for id1, row in df_err.iterrows():
  798. confusions = []
  799. for id2, count in row.items():
  800. if count > 5:
  801. confusions.append(id2)
  802. if len(confusions) > 0:
  803. print(f'common confusions involving id {id1}:\n')
  804. confusions = [id1] + confusions
  805. display_stimuli(confusions)
  806. print('\n\n')
  809. # %%
  812. # %%
  815. # %%
  818. # %%

modality_agnostic_decoding.ipynb at commit 57f7a1c, under MIT · at the source

Overview

Authors: Mitja Nikolaus1, Milad Mozafari2, Isabelle Berry1, Nicholas Asher3, Leila Reddy1, Rufin VanRullen1
ORCID iDs: Mitja Nikolaus, Milad Mozafari, Rufin VanRullen
  1. Université de Toulouse, CNRS, CerCo, Toulouse, France
  2. Torus AI, Toulouse, France
  3. Université de Toulouse, IRIT, Toulouse, France
Institutions: Centre National de la Recherche Scientifique (France); Université de Toulouse (France); Institut de Recherche en Informatique de Toulouse (France)
Journal: eLife, volume 14, article RP107933
Dates: published online 17 April 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.7554/elife.107933 · PMID 41995708 · PMCID PMC13090028 · OpenAlex W4413947960
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: fMRI (modality), human (organism), cognitive (subfield)
Methods: Connectivity, Statistics, fMRI & imaging, Machine learning
Keywords: Human
MeSH: Brain*, Language*, Magnetic Resonance Imaging*, Vision, Ocular*, Visual Perception*, Brain Mapping, Humans, Photic Stimulation (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: Agence Nationale de la Recherche (ANITI (ANR-19-PI3A-0004), AI-REPS (ANR-18-CE37-0007-01)); European Research Council (Advanced grant GLoW (101096017), 101096017)
Citations: cited by 3 papers (Europe PMC); 134 references in the paper

Abstract

Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises six subjects watching both images and short text descriptions of such images, as well as the conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders and even outperform them when decoding captions and mental imagery. Furthermore, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 17 matches between paragraphs and lines of code.

mitjanikolaus/multimodal_decoding

License: MIT
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 57f7a1c4a5da1c96887ed79f0507cbd286c727fd, 17 April 2026
Languages: Python (46), Jupyter (7), MATLAB (1)
Size: 77 files, 54 scripts
Software Heritage: archived
Found in: the references
Holds: README, license file, environment (setup.py, environment_files/environment.yml, environment_files/environment_gpu.yml), 7 notebooks
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (33 files), Matplotlib (19 files), Pillow (19 files), PyTorch (15 files), NiBabel (14 files), pandas (14 files), seaborn (14 files), SciPy (11 files), Hugging Face Transformers (10 files), Nilearn (9 files), scikit-learn (8 files), FreeSurfer (4 files), Nipype (3 files), SPM (3 files), h5py (1 file), OpenCV (1 file), scikit-image (1 file), statsmodels (1 file)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
56 files
At the source: archive.softwareheritage.org/swh:1:dir:358f81a6a99cdcdf3ba0129b4e5627b7170b3273;origin=https://github.com/mitjanikolaus/multimodal_decoding;visit=swh:1:snp:840f7a3b5e83b249f545ad61c124df2dab366bac;anchor=swh:1:rev:57f7a1c4a5da1c96887ed79f0507cbd286c727fd

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 54 scripts, each with its path and the digest of its content;
  • 17 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data availability

All fMRI data is available at https://openneuro.org/datasets/ds007272. Code for preprocessing and analyses is available at https://github.com/mitjanikolaus/multimodal_decoding (copy archived at Nikolaus, 2026).

The following dataset was generated:

Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. 2025. SemReps-8K. OpenNeuro.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 6 authors, 1 keyword, 8 MeSH terms, 2 funders, 119 references.

Cite

This paper

Nikolaus, M., Mozafari, M., Berry, I., Asher, N., Reddy, L., & VanRullen, R. (2026). Modality-agnostic decoding of vision and language from fMRI. eLife, 14, RP107933. https://doi.org/10.7554/elife.107933

BibTeX

@article{nikolaus2026modality,
author = {Nikolaus, Mitja and Mozafari, Milad and Berry, Isabelle and Asher, Nicholas and Reddy, Leila and VanRullen, Rufin},
title = {{Modality-agnostic decoding of vision and language from fMRI}},
journal = {eLife},
year = {2026},
month = apr,
volume = {14},
pages = {RP107933},
publisher = {eLife Sciences Publications, Ltd},
issn = {2050-084X},
doi = {10.7554/elife.107933},
url = {https://doi.org/10.7554/elife.107933},
pmid = {41995708},
pmcid = {PMC13090028}
}

RIS

TY - JOUR
AU - Nikolaus, Mitja
AU - Mozafari, Milad
AU - Berry, Isabelle
AU - Asher, Nicholas
AU - Reddy, Leila
AU - VanRullen, Rufin
TI - Modality-agnostic decoding of vision and language from fMRI
T2 - eLife
J2 - eLife
PY - 2026
DA - 2026/04/17
VL - 14
SP - RP107933
SN - 2050-084X
PB - eLife Sciences Publications, Ltd
DO - 10.7554/elife.107933
UR - https://doi.org/10.7554/elife.107933
LA - en
ER -

CSL-JSON

{
"id": "10.7554/elife.107933",
"type": "article-journal",
"title": "Modality-agnostic decoding of vision and language from fMRI",
"container-title": "eLife",
"author": [
{
"family": "Nikolaus",
"given": "Mitja"
},
{
"family": "Mozafari",
"given": "Milad"
},
{
"family": "Berry",
"given": "Isabelle"
},
{
"family": "Asher",
"given": "Nicholas"
},
{
"family": "Reddy",
"given": "Leila"
},
{
"family": "VanRullen",
"given": "Rufin"
}
],
"container-title-short": "eLife",
"volume": "14",
"page": "RP107933",
"DOI": "10.7554/elife.107933",
"PMID": "41995708",
"PMCID": "PMC13090028",
"ISSN": "2050-084X",
"publisher": "eLife Sciences Publications, Ltd",
"URL": "https://doi.org/10.7554/elife.107933",
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
17
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1371/journal.pcbi.1014263 [code]
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.
Journal: PLoS computational biology
In common: Hugging Face Transformers, OpenCV, scikit-image, 9 other tools, fMRI, 11 references
[2] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: FreeSurfer, Nilearn, OpenCV, 10 other tools, fMRI, 10 references
[3] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: Hugging Face Transformers, h5py, Pillow, 9 other tools, cognitive, 7 references
[4] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: Hugging Face Transformers, FreeSurfer, scikit-image, 11 other tools, 5 references
[5] doi:10.1007/s12021-026-09803-3 [code]
NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.
Journal: Neuroinformatics
In common: Nilearn, scikit-image, h5py, 8 other tools, fMRI, 7 references
[6] doi:10.1167/jov.26.5.7 [code]
Representations in vision and language converge in a shared, multidimensional space of perceived similarities.
Journal: Journal of vision
In common: Nilearn, OpenCV, h5py, 10 other tools, cognitive, 5 references
[7] doi:10.1162/nol.a.271 [code]
Compositional Complexity in Text and Images.
Journal: Neurobiology of language (Cambridge, Mass.)
In common: Nilearn, OpenCV, Pillow, 8 other tools, fMRI, 7 references
[8] doi:10.1038/s41467-026-71568-9 [code]
Convergent and selective representations of pain, appetitive processes, aversive processes, and cognitive control in the insula.
Journal: Nature communications
In common: Nipype, FreeSurfer, Nilearn, 9 other tools, cognitive, 3 references
[9] doi:10.1038/s41467-026-76452-0 [code]
Music evokes shared neural representations of imagined narratives across sensory modalities.
Journal: Nature communications
In common: FreeSurfer, Nilearn, SPM, 10 other tools, cognitive, 3 references
[10] doi:10.1162/imag.a.1299 [code]
A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Hugging Face Transformers, scikit-image, h5py, 8 other tools, 5 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.