Modality-agnostic decoding of vision and language from fMRI.
10 blocks of code here have a permanent identifier (oscr:b:…), listed with the matches below: citable for good, with a page of its own. Select a match and press Cite to copy it. What a block identifier is.
The 17 matches
- [1] § Results › Modality-agnostic decoders ↔ notebooks/modality_agnostic_decoding.ipynb, lines 485 to 526 · score 0.80 · dual stream, late fusion, early fusion, single stream, modality agnostic, ANOVA · Report a mismatch
- [2] § Methods › Modality-agnostic decoders ↔ data.py, lines 182 to 248 · score 0.76 · vision features, Llama2, mistral, mixtral, ResNet, SigLip · block
oscr:b:P92DZ60FAVT1S-4· Report a mismatch - [3] § Results › Modality-agnostic decoders ↔ notebooks/notebook_utils.py, lines 204 to 262 · score 0.69 · Dino giant, GPT2 large, language models, vision features, imagebind, multimodal · block
oscr:b:SQ6MGXNQ9HNY9-4· Report a mismatch - [4] § Methods › Modality-agnostic decoders ↔ analyses/decoding/ridge_regression_decoding.py, lines 45 to 187 · score 0.69 · ridge regression, predict latent, fMRI beta, decoder training, fitting, models · block
oscr:b:MZ5BSTEV3FZTN-F· Report a mismatch - [5] § Methods › fMRI preprocessing ↔ preprocessing/fmri_preprocessing.py, lines 83 to 124 · score 0.68 · anatomical scan, SPM, nipype, slice, coregistered, realignment · block
oscr:b:ARDP0MZDWXY2E-E· Report a mismatch - [6] § Appendix 2 › Feature extraction details ↔ feature_extraction/extract_bridgetower_features.py, lines 45 to 51 · score 0.67 · bridgetower large itm, mlm itc, Model · block
oscr:b:7EVVY13M8ME5T-L· Report a mismatch - [7] § Appendix 1 › Dataset quality metrics › Head motion ↔ analyses/visualization/plot_dataset_quality_stats.py, lines 20 to 108 · score 0.64 · framewise displacement, realignment parameters, absolute, sum, head, motion · block
oscr:b:F66TKWNP6A7FH-8· Report a mismatch - [8] § Results › Modality-invariant regions ↔ analyses/decoding/searchlight/searchlight_cluster_manual_corrections.py, lines 13 to 38 · score 0.62 · inferior parietal, inferior temporal, middle temporal, cluster, searchlight · block
oscr:b:M16437TD2TYTK-I· Report a mismatch - [9] § Appendix 2 › Feature extraction details ↔ notebooks/modality_agnostic_decoding.ipynb, lines 485 to 526 · score 0.62 · dual stream, single stream, multimodal models, BLIP2, Paligemma2, Flava · Report a mismatch
- [10] § Appendix 1 › Intersession alignment ↔ notebooks/intersession_alignment.ipynb, lines 24 to 64 · score 0.62 · normalized mutual information, alignment, Intersession, coregistered, preprocessing · Report a mismatch
- [11] § Appendix 2 › Feature extraction details ↔ feature_extraction/extract_siglip_features.py, lines 37 to 42 · score 0.61 · siglip so400m patch14, Model · block
oscr:b:1PPWVTVE84TQF-G· Report a mismatch - [12] § Methods › fMRI preprocessing ↔ preprocessing/make_spm_design_job_mat.py, lines 178 to 310 · score 0.57 · gray matter masks, slice, coregistered, Preprocessing, realignment, scan · block
oscr:b:S6H95AJY16MGB-A· Report a mismatch - [13] § Appendix 1 › Intersession alignment ↔ preprocessing/fmri_preprocessing.py, lines 83 to 124 · score 0.56 · anatomical scan, SPM, pipeline, coregistered, preprocessing · block
oscr:b:ARDP0MZDWXY2E-E· Report a mismatch - [14] § Appendix 1 › Dataset quality metrics › Head motion ↔ analyses/visualization/plot_dataset_quality_stats.py, lines 20 to 108 · score 0.56 · realignment parameters, pitch, roll, yaw, displacement, head · block
oscr:b:F66TKWNP6A7FH-8· Report a mismatch - [15] § Methods › Modality-agnostic decoders ↔ notebooks/modality_agnostic_decoding.ipynb, lines 702 to 764 · score 0.55 · cross modal, pairwise accuracy, distance, cosine, latent, predictions · Report a mismatch
- [16] § Methods › fMRI preprocessing ↔ preprocessing/make_spm_design_job_mat.py, lines 22 to 55 · score 0.55 · train captions, train images, blank, fixations, preprocessing, imagery · block
oscr:b:N21XC4BR1NE1S-S· Report a mismatch - [17] § Methods › Modality-agnostic decoders ↔ notebooks/zero_shot_cross_modal_decoding.ipynb, lines 112 to 154 · score 0.54 · cross modal decoding, language model, pairwise, accuracy, captions, agnostic · Report a mismatch
Paper
Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC
The paper is loaded when this pane is shown.
The authors' code
Jupyter notebook · 819 lines · 29 KB · MIT · 3 matches
- # %%
- import os
- import numpy as np
- import matplotlib.pyplot as plt
- import pandas as pd
- import seaborn as sns
- from sklearn.preprocessing import StandardScaler
- from utils import RESULTS_DIR, SUBJECTS, COCO_IMAGES_DIR, STIM_INFO_PATH
- from data import MODALITY_AGNOSTIC, TRAINING_MODES, CAPTION, IMAGE, DEFAULT_VISION_FEATURES
- from eval import ACC_MODALITY_AGNOSTIC, ACC_CAPTIONS, ACC_IMAGES, ACC_IMAGERY, ACC_IMAGERY_WHOLE_TEST, get_distance_matrix, dist_mat_to_pairwise_acc
- from notebook_utils import create_result_graph, plot_metric_catplot, load_results_data, ACC_MEAN, METRICS_ERROR_ANALYSIS, get_data_default_feats, calc_model_feat_order, FEATS_MULTIMODAL, DEFAULT_FEAT_OPTIONS
- from matplotlib import patches
- sns.set_style("ticks", {'axes.grid': True})
- sns.set_theme(font_scale=1.6)
- from data import DEFAULT_FEATURES, DEFAULT_VISION_FEATURES, DEFAULT_LANG_FEATURES, TRAINING_MODES, MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS
- # %%
- MODELS = [
- "random-imagebind", "vit-b-16", "vit-l-16", "vit-h-14", "resnet-18", "resnet-50", "resnet-152", "dino-base", "dino-large", "dino-giant",
- "bert-base-uncased", "bert-large-uncased", "llama2-7b", "llama2-13b", "mistral-7b", "mixtral-8x7b", "gpt2-small", "gpt2-medium", "gpt2-large", "gpt2-xl",
- "visualbert", "bridgetower", "vilt", "siglip", "paligemma2", "clip", "flava", "blip2", "imagebind"
- ]
- # for model in MODELS:
- # print(model, end=" ")
- # %%
- all_data = load_results_data(MODELS, recompute_acc_scores=False)
- all_data = all_data[all_data["mask"] == "whole_brain"]
- all_data_vol = all_data[all_data.surface == False].copy()
- all_data = all_data[all_data.surface == True].copy()
- multimodal_models = all_data[all_data.features.isin(FEATS_MULTIMODAL)].model.unique().tolist()
- vision_models = [m for m in all_data[all_data.features == "vision"].model.unique() if len(all_data[all_data.model == m].features.unique()) == 1]
- # %%
- data_default_feats = get_data_default_feats(all_data)
- # %%
- dd = data_default_feats.copy()
- dd = dd[dd.subject == 'sub-01']
- dd = dd[dd.model == 'dino-base']
- dd = dd[dd.metric.isin(['pairwise_acc_images', 'pairwise_acc_captions'])]
- dd = dd[dd.training_mode == 'agnostic']
- display(dd)
- # %%
- dd = data_default_feats.copy()
- dd = dd[dd.subject == 'sub-01']
- dd = dd[dd.model == 'gpt2-small']
- dd = dd[dd.metric.isin(['pairwise_acc_images', 'pairwise_acc_captions'])]
- dd = dd[dd.training_mode == 'agnostic']
- display(dd)
- # %% [markdown]
- # ## Feature comparison for multimodal models
- # %%
- data_mod_agnostic_train = all_data[(all_data.metric == ACC_MEAN) & (all_data.training_mode == MODALITY_AGNOSTIC)]
- with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
- # grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(count=('value', 'size'), pairwise_acc=('value', 'mean'), pairwise_acc_stddev=('value', 'std')).reset_index()
- grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(pairwise_acc=('value', 'mean')).reset_index()
- grouped = grouped[grouped.model.isin(multimodal_models)]
- grouped = grouped[~grouped.model.isin(["random-imagebind"])]
- display(grouped)
- # print(grouped.to_markdown())
- grouped = grouped.replace("n_a", "")
- # grouped = grouped[grouped.model.isin(multimodal_models)]
- # del grouped["count"]
- print(grouped.to_latex(index=False, escape=True, float_format="%.3f"))
- # %%
- # data_default_vision_feats = all_data.copy()
- # for model in all_data.model.unique():
- # default_vision_feats = DEFAULT_VISION_FEATURES[model]
- # data_default_vision_feats = data_default_vision_feats[((data_default_vision_feats.model == model) & (data_default_vision_feats.vision_features == default_vision_feats)) | (data_default_vision_feats.model != model)]
- #
- # feat_legend = {"avg": "average over tokens from vision and language streams", "fused_cls": "fused [CLS] token", "fused_mean": "average over fused tokens"}
- # feat_order = ["fused_cls", "fused_mean", "avg"]
- #
- # feat_order_long = [feat_legend[feat] for feat in feat_order]
- #
- # data_to_plot = data_default_vision_feats.copy()
- #
- # data_to_plot = data_to_plot[data_to_plot.model.isin(multimodal_models)]
- # models_excluded = ['random-imagebind', 'siglip', 'clip', 'imagebind']
- # data_to_plot = data_to_plot[~data_to_plot.model.isin(models_excluded)]
- #
- # data_to_plot["features"] = data_to_plot.features.replace(feat_legend)
- #
- # model_feat_order = calc_model_feat_order(data_to_plot, MODELS, feat_options=feat_order)
- #
- # metrics_order = [ACC_MEAN]
- # figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=feat_order_long, ylim=(0.5, 1),
- # legend_bbox=(0.06,1.01), height=5, legend_title="Modality-agnostic decoders projecting into multimodal model features based on ", verify_num_datapoints=False, plot_modality_specific=False)
- # plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_multimodal_models.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0, dpi=300)
- # %% [markdown]
- # ## Feature comparison for vision models
- # %%
- data_mod_agnostic_train = all_data[(all_data.metric == ACC_MEAN) & (all_data.training_mode == MODALITY_AGNOSTIC)]
- with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
- # grouped = data_mod_agnostic_train.groupby(["model", "features", "vision_features", "lang_features"]).agg(count=('value', 'size'), pairwise_acc=('value', 'mean'), pairwise_acc_stddev=('value', 'std')).reset_index()
- grouped = data_mod_agnostic_train.groupby(["model", "vision_features"]).agg(pairwise_acc=('value', 'mean')).reset_index()
- grouped = grouped[grouped.model.isin(vision_models)]
- models_excluded = ["resnet-18", "resnet-50", "resnet-152"]
- grouped = grouped[~grouped.model.isin(models_excluded)]
- display(grouped)
- # print(grouped.to_markdown())
- grouped = grouped.replace("n_a", "")
- # grouped = grouped[grouped.model.isin(multimodal_models)]
- # del grouped["count"]
- print(grouped.to_latex(index=False, escape=True, float_format="%.3f"))
- # %%
- # data_default_vision_feats = all_data.copy()
- #
- # feat_legend = {"vision_features_cls": "[CLS] token", "vision_features_mean": "average over patches"}
- # feat_order = ["vision_features_cls", "vision_features_mean"]
- #
- # feat_order_long = [feat_legend[feat] for feat in feat_order]
- #
- # data_to_plot = data_default_vision_feats.copy()
- #
- # data_to_plot = data_to_plot[data_to_plot.model.isin(vision_models)]
- # models_excluded = ["resnet-18", "resnet-50", "resnet-152"]
- # data_to_plot = data_to_plot[~data_to_plot.model.isin(models_excluded)]
- #
- # data_to_plot["vision_features"] = data_to_plot.vision_features.replace(feat_legend)
- #
- # model_feat_order = calc_model_feat_order(data_to_plot, MODELS)
- # metrics_order = [ACC_MEAN]
- # figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_variable='vision_features', hue_order=feat_order_long, ylim=(0.5, 1),
- # legend_bbox=(0.06,1.01), height=5, legend_title="Modality-agnostic decoders projecting into vision model features based on ", verify_num_datapoints=False, plot_modality_specific=False)
- # plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_vision_models.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0, dpi=300)
- # %% [markdown]
- # ## Modality-agnostic decoding vs. modality-specific decoding
- # %% [markdown]
- # ### Model features comparison
- # %%
- model_order = ['random-imagebind']
- model_feat_order = ['random-imagebind_avg']
- for features in DEFAULT_FEAT_OPTIONS:
- print('\nmodel feat type: ', features)
- dp = data_default_feats.copy()
- dp = dp[dp.features == features]
- dp = dp[dp.training_mode == MODALITY_AGNOSTIC]
- dp = dp[dp.metric == ACC_MEAN]
- for model in dp.model.unique():
- if len(dp[dp.model == model]) != len(SUBJECTS):
- print(f"unexpected number of datapoints for {model}: {len(dp[dp.model == model])}")
- # scores = dp.groupby("model").value.mean().sort_values()
- scores = dp.groupby("model_feat").value.mean().sort_values()
- if len(scores) > 0:
- print(scores)
- model_order.extend([mf.split('_')[0] for mf in scores.index.values])
- model_feat_order.extend(scores.index.values)
- model_order
- # model_feat_order
- # %%
- FEAT_ORDER = ["vision models", "language models", "multimodal models"]
- data_to_plot = data_default_feats.copy()
- data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
- data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
- # model_feat_order = calc_model_feat_order(data_to_plot)
- metrics_order = [ACC_MEAN]
- HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
- figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=0.85,
- legend_bbox=(0.68, 1.05), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (-0.5, 0.1), # (x,y)
- 1, # width
- 0.9, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (0.5, 0.1), # (x,y)
- 9, # width
- 0.9, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (9.5, 0.1), # (x,y)
- 10, # width
- 0.9, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (19.5, 0.1), # (x,y)
- 9, # width
- 0.9, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[0][0].text(4, 0.11, 'Vision Models', fontsize=12)
- figure.axes[0][0].text(14, 0.11, 'Language Models', fontsize=12)
- figure.axes[0][0].text(23, 0.11, 'Multimodal Models', fontsize=12)
- plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.3, dpi=300)
- # %%
- FEAT_ORDER = ["vision models", "language models", "multimodal models"]
- data_to_plot = data_default_feats.copy()
- data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
- data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
- # model_feat_order = calc_model_feat_order(data_to_plot)
- metrics_order = [ACC_IMAGES, ACC_CAPTIONS]
- HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
- figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=1.07, row_title_loc='left',
- legend_bbox=(0.68, 1.03), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
- height=0.5
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (-0.5, 0.5), # (x,y)
- 1, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (0.5, 0.5), # (x,y)
- 9, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (9.5, 0.5), # (x,y)
- 10, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[0][0].add_patch(
- patches.Rectangle(
- (19.5, 0.5), # (x,y)
- 9, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- height=0.8
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (-0.5, 0.2), # (x,y)
- 1, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (0.5, 0.2), # (x,y)
- 9, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (9.5, 0.2), # (x,y)
- 10, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (19.5, 0.2), # (x,y)
- 9, # width
- height, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[1][0].text(4, 0.21, 'Vision Models', fontsize=12)
- figure.axes[1][0].text(14, 0.21, 'Language Models', fontsize=12)
- figure.axes[1][0].text(23, 0.21, 'Multimodal Models', fontsize=12)
- plt.subplots_adjust(hspace=0.3)
- plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_split_by_modality.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.5, dpi=300)
- # %% [markdown]
- # # Statistical tests
- # %%
- from scipy.stats import ttest_ind
- import statsmodels.formula.api as smf
- # %% [markdown]
- # ## Mod agnostic vs. mod specific
- # %%
- data_default_feats_copy = data_default_feats.copy()
- for metric, comparison_training_mode in zip(['pairwise_acc_mean', 'pairwise_acc_mean'], ["images", "captions"]):
- print(f"{metric} | comparison with {comparison_training_mode}")
- data_filtered = data_default_feats_copy[data_default_feats_copy.metric == metric]
- data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
- assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
- data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
- assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
- print('mod agno: ', data_mod_agno.value.mean())
- print('mod spec: ', data_mod_spec.value.mean())
- # display(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
- # display(data_mod_agno)
- data_anova = pd.concat([data_mod_agno, data_mod_spec])
- data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
- mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
- print("=" * 50 + "\nANOVA\n" + "=" * 50)
- print(mod.summary())
- print('pvalues:\n', mod.pvalues)
- print('\n')
- # %% [markdown]
- # ## Mod agnostic vs. mod specific - imagery decoding
- # %%
- # from scipy.stats import ttest_ind
- # from pymer4.models import Lmer
- # data_ttest = data_default_feats.copy()
- # for metric in [ACC_IMAGERY_WHOLE_TEST, ACC_IMAGERY]:
- # for comparison_training_mode in ["images", "captions"]:
- # print(f"{metric} | comparison with {comparison_training_mode}")
- # data_filtered = data_ttest[data_ttest.metric == metric]
- # data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
- # # data_mod_agno = data_mod_agno[data_mod_agno.model == "imagebind"]
- # assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
- # data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
- # assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
- # print('mod agno: ', data_mod_agno.value.mean())
- # print('mod spec: ', data_mod_spec.value.mean())
- # print(len(data_mod_agno.value.values))
- # print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
- # # display(data_mod_agno)
- # data_glm = pd.concat([data_mod_agno, data_mod_spec])
- # # display(data_glm)
- # mod = Lmer('value ~ training_mode + (1 | subject)', data=data_glm)
- # # mod = Lmer('value ~ training_mode + (1 | model) + (1 | subject)', data=data_glm)
- # print("=" * 50 + "\nGLM\n" + "=" * 50)
- # fitted = mod.fit()
- # print(fitted)
- # print(fitted[["Estimate", "SE", "Sig"]])
- # print('\n')
- # %%
- data_ttest = data_default_feats.copy()
- for metric in [ACC_IMAGERY_WHOLE_TEST]:
- for comparison_training_mode in ["images", "captions"]:
- print(f"{metric} | comparison with {comparison_training_mode}")
- data_filtered = data_ttest[data_ttest.metric == metric]
- data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
- assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
- data_mod_spec = data_filtered[data_filtered.training_mode == comparison_training_mode]
- assert len(data_mod_spec) == len(SUBJECTS) * len(data_mod_spec.model.unique())
- print('mod agno: ', data_mod_agno.value.mean())
- print('mod spec: ', data_mod_spec.value.mean())
- print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
- # display(data_mod_agno)
- data_anova = pd.concat([data_mod_agno, data_mod_spec])
- data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
- # print(data_anova)
- mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
- print("=" * 50 + "\nANOVA\n" + "=" * 50)
- print(mod.summary())
- print('pvalues:\n', mod.pvalues)
- print('\n')
- # %% [markdown]
- # ## Effect of multimodal model architecture
- # %%
- data_ttest = data_default_feats.copy()
- for metric in [ACC_MEAN]:
- print(f"{metric}")
- data_filtered = data_ttest[data_ttest.metric == metric]
- data_mod_agno = data_filtered[data_filtered.training_mode == "agnostic"]
- data_mod_agno = data_mod_agno[data_mod_agno.model != "random-imagebind"]
- assert len(data_mod_agno) == len(SUBJECTS) * len(data_mod_agno.model.unique())
- data_dual_stream = data_mod_agno[data_mod_agno.model.isin(['clip', 'imagebind', 'siglip'])].copy()
- data_dual_stream['multimodal_type'] = 'dual_stream'
- data_single_stream_early_fusion = data_mod_agno[data_mod_agno.model.isin(['visualbert', 'vilt', 'paligemma2'])].copy()
- data_single_stream_early_fusion['multimodal_type'] = 'single_stream_early_fusion'
- data_single_stream_late_fusion = data_mod_agno[data_mod_agno.model.isin(['bridgetower', 'flava', 'blip2'])].copy()
- data_single_stream_late_fusion['multimodal_type'] = 'single_stream_late_fusion'
- print('data_dual_stream: ', data_dual_stream.value.mean())
- print(len(data_dual_stream.value.values))
- print('data_single_stream_early_fusion: ', data_single_stream_early_fusion.value.mean())
- print(len(data_single_stream_early_fusion.value.values))
- print('data_single_stream_late_fusion: ', data_single_stream_late_fusion.value.mean())
- print(len(data_single_stream_late_fusion.value.values))
- # print(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
- data_anova = pd.concat([data_dual_stream, data_single_stream_early_fusion, data_single_stream_late_fusion])
- data_anova = data_anova[['subject', 'multimodal_type', 'value']]
- mod = smf.mixedlm("value ~ multimodal_type", data_anova, groups=data_anova["subject"]).fit()
- print("=" * 50 + "\nANOVA\n" + "=" * 50)
- # print(mod.random_effects)
- print(mod.summary())
- print('pvalues:\n', mod.pvalues)
- print('\n')
- # %% [markdown]
- # ## Comparing cross-decoding for images and captions
- # %%
- data_filtered = data_default_feats.copy()
- # for metric, comparison_training_mode in zip(['pairwise_acc_mean', 'pairwise_acc_mean'], ["images", "captions"]):
- # print(f"{metric} | comparison with {comparison_training_mode}")
- # data_filtered = data_default_feats_copy[data_default_feats_copy.metric == 'pairwise_acc_mean']
- data_cross_images = data_filtered[(data_filtered.training_mode == "captions") & (data_filtered.metric == 'pairwise_acc_images')]
- assert len(data_cross_images) == len(SUBJECTS) * len(data_cross_images.model.unique())
- data_cross_captions = data_filtered[(data_filtered.training_mode == "images") & (data_filtered.metric == 'pairwise_acc_captions')]
- assert len(data_cross_captions) == len(SUBJECTS) * len(data_cross_captions.model.unique())
- print('data_cross_images: ', data_cross_images.value.mean())
- print('data_cross_captions: ', data_cross_captions.value.mean())
- # display(ttest_ind(data_mod_agno.value.values, data_mod_spec.value.values))
- # # display(data_mod_agno)
- data_anova = pd.concat([data_cross_images, data_cross_captions])
- data_anova = data_anova[['model', 'subject', 'training_mode', 'value']]
- mod = smf.mixedlm("value ~ training_mode", data_anova, groups=data_anova["subject"]).fit()
- print("=" * 50 + "\nANOVA\n" + "=" * 50)
- print(mod.summary())
- print('pvalues:\n', mod.pvalues)
- print('\n')
- # %% [markdown]
- # # Imagery decoding feat comparison
- # %%
- FEAT_ORDER = ["vision models", "language models", "multimodal models"]
- data_to_plot = data_default_feats.copy()
- data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
- data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
- # model_feat_order = calc_model_feat_order(data_to_plot)
- metrics_order = [ACC_IMAGERY, ACC_IMAGERY_WHOLE_TEST]
- HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
- figure, lgd = create_result_graph(data_to_plot, order=model_feat_order, metrics=metrics_order, row_order=metrics_order, hue_order=HUE_ORDER, ylim=(0.5, 1), row_title_height=0.85,
- legend_bbox=(0.68, 1.07), height=5, aspect=3.2, legend_title="", verify_num_datapoints=False)
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (-0.5, 0.2), # (x,y)
- 1, # width
- 1.4, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (0.5, 0.2), # (x,y)
- 9, # width
- 1.4, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (9.5, 0.2), # (x,y)
- 10, # width
- 1.4, # height
- linewidth=0,
- clip_on=False,
- alpha=0.05,
- color='0.2'
- )
- )
- figure.axes[1][0].add_patch(
- patches.Rectangle(
- (19.5, 0.2), # (x,y)
- 9, # width
- 1.4, # height
- linewidth=0,
- clip_on=False,
- alpha=0.15,
- color='0.2'
- )
- )
- figure.axes[1][0].text(4, 0.21, 'Vision Models', fontsize=12)
- figure.axes[1][0].text(14, 0.21, 'Language Models', fontsize=12)
- figure.axes[1][0].text(23, 0.21, 'Multimodal Models', fontsize=12)
- plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_imagery.png"), bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0.3, dpi=300)
- # %% [markdown]
- # ## Per-subject results
- # %%
- FEAT_ORDER = ["vision", "lang", "matched"]
- FEAT_PALETTE = sns.color_palette('Set2')[:3]
- def create_result_graph_all_subjs(data, order=model_feat_order, metrics=[ACC_CAPTIONS, ACC_IMAGES], hue_variable="training_mode", hue_order=FEAT_ORDER, ylim=None,
- legend_title="Modality-agnostic decoders based on features from", palette=FEAT_PALETTE, dodge=False, noise_ceilings=None, plot_modality_specific=True,
- row_variable="metric", col_variable=None, legend_bbox=None):
- for mode in TRAINING_MODES:
- data_mode = data[data.training_mode == mode]
- for x_variable_value in order:
- length = len(data_mode[(data_mode["model_feat"] == x_variable_value) & (data_mode.metric == metrics[0])])
- expected_num_datapoints = len(SUBJECTS)
- # if hue_variable != "features":
- # expected_num_datapoints *= len(data[hue_variable].unique())
- if (length > 0) and (length != expected_num_datapoints):
- message = f"unexpected number of datapoints: {length} (expected: {expected_num_datapoints}) (model_feat: {x_variable_value} {mode}"
- print(f"Warning: {message}")
- catplot_g, data_plotted, lgd = plot_metric_catplot(data, order=model_feat_order, metrics=metrics, x_variable="model_feat", legend_title=legend_title, aspect=2, legend_bbox=legend_bbox, rotation=90, cut_labels=False,
- hue_variable=hue_variable, row_variable=row_variable, col_variable=col_variable, hue_order=hue_order, palette=palette, ylim=ylim, noise_ceilings=noise_ceilings)
- for i in range(len(data.subject.unique())):
- catplot_g.axes[i,0].set_title(f"Subject {i+1} | Decoding of captions", fontsize=25)
- catplot_g.axes[i,1].set_title(f"Subject {i+1} | Decoding of images", fontsize=25)
- catplot_g.axes[i,0].set_ylabel('pairwise accuracy')
- return catplot_g, lgd
- data_to_plot = data_default_feats.copy()
- data_to_plot["features"] = data_to_plot.features.replace({"vision": "vision models", "lang": "language models"})
- data_to_plot["features"] = data_to_plot.features.replace({f: "multimodal models" for f in FEATS_MULTIMODAL})
- model_feat_order = calc_model_feat_order(data_to_plot, MODELS)
- HUE_ORDER = [MODALITY_AGNOSTIC, MODALITY_SPECIFIC_IMAGES, MODALITY_SPECIFIC_CAPTIONS]
- figure, lgd = create_result_graph_all_subjs(data_to_plot, model_feat_order, metrics=[ACC_CAPTIONS, ACC_IMAGES], hue_order=HUE_ORDER, ylim=(0.5, 1), row_variable="subject", col_variable="metric", legend_bbox=(0.58, 1.07))
- plt.savefig(os.path.join(RESULTS_DIR, f"features_comparison_per_subject.png"), dpi=300, bbox_extra_artists=(lgd,), bbox_inches='tight', pad_inches=0)
- # %%
- # %%
- # %% [markdown]
- # #
- # %% [markdown]
- # # Error analysis
- # %%
- models = ['imagebind']
- data = load_results_data(models, metrics=METRICS_ERROR_ANALYSIS)
- # metric = ACC_IMAGES
- all_data = data[data.model.isin(MODELS)]
- all_data = all_data[all_data["mask"] == "whole_brain"]
- all_data = all_data[all_data.surface == True]
- data_default_feats_err_analysis = get_data_default_feats(all_data)
- # %%
- def all_pairwise_accuracy_scores(latents, predictions, stim_types=None, metric="cosine", standardize=True):
- results = dict()
- results['comp_mats'] = dict()
- for modality, acc_metric_name in zip([CAPTION, IMAGE], [ACC_CAPTIONS, ACC_IMAGES]):
- preds_mod = predictions[stim_types == modality].copy()
- latents_mod = latents[stim_types == modality]
- if standardize:
- preds_mod = StandardScaler().fit_transform(preds_mod)
- dist_mat = get_distance_matrix(preds_mod, latents_mod, metric)
- diag = dist_mat.diagonal().reshape(-1, 1)
- comp_mat = diag < dist_mat
- score = dist_mat_to_pairwise_acc(dist_mat)
- results[acc_metric_name] = score
- results['comp_mats'][acc_metric_name] = (~comp_mat).astype(int)
- if standardize:
- predictions = StandardScaler().fit_transform(predictions)
- dist_mat = get_distance_matrix(predictions, latents, metric)
- mod_agnostic_accs = []
- for modality in [CAPTION, IMAGE]:
- dist_mat_within_mod = dist_mat[stim_types == modality][:, stim_types == modality]
- dist_mat_cross_modal = dist_mat[stim_types == modality][:, stim_types != modality]
- dist_mat_min = np.min((dist_mat_within_mod, dist_mat_cross_modal), axis=0)
- diag = dist_mat_min.diagonal().reshape(-1, 1)
- comp_mat = diag < dist_mat_min
- score = dist_mat_to_pairwise_acc(dist_mat_min)
- # scores = np.mean(comp_mat, axis=0)
- mod_agnostic_accs.append(score)
- results[f"pairwise_acc_mod_agnostic_{modality}s"] = score
- results['comp_mats'][f"pairwise_acc_mod_agnostic_{modality}s"] = (~comp_mat).astype(int)
- results[ACC_MODALITY_AGNOSTIC] = np.mean(mod_agnostic_accs)
- return results
- metric = ACC_IMAGES #TODO which metric?
- training_type = MODALITY_AGNOSTIC
- all_subj_mats = []
- for subject in SUBJECTS:
- df = data_default_feats_err_analysis.copy()
- df_subj = df[(df.training_mode == training_type) & (df.subject == subject)]
- predictions = df_subj[df_subj.metric == "predictions"].value.item()
- latents = df_subj[df_subj.metric == "latents"].value.item()
- stimulus_ids = df_subj[df_subj.metric == "stimulus_ids"].value.item()
- stimulus_types = df_subj[df_subj.metric == "stimulus_types"].value.item()
- results = all_pairwise_accuracy_scores(latents, predictions, stimulus_types)
- print(results[ACC_IMAGES])
- mat = results['comp_mats'][metric]
- all_subj_mats.append(mat)
- mat = np.sum(all_subj_mats, axis=0)
- mat = mat - np.diag(np.diag(mat))
- df_err = pd.DataFrame(mat, index=stimulus_ids[:70], columns=stimulus_ids[:70])
- # %%
- # df_err = df_err.map(lambda x: 0 if x < 3 else x)
- # with pd.option_context('display.max_rows', None, 'display.max_columns', None):
- # display(df_err)
- # %%
- stimuli_info = pd.read_csv(STIM_INFO_PATH, index_col=0)
- stimuli_info = stimuli_info[stimuli_info.used == True]
- captions_dict = stimuli_info.caption.to_dict()
- img_paths = stimuli_info.img_path.to_dict()
- img_paths = {idx: os.path.join(COCO_IMAGES_DIR, path) for idx, path in img_paths.items()}
- from PIL import Image
- def display_stimuli(coco_ids):
- imgs = [Image.open(img_paths[img_id]).convert('RGB') for img_id in coco_ids]
- min_height = np.min([np.array(im).shape[0] for im in imgs])
- stacked = np.hstack([np.array(im)[:min_height,:,:] for im in imgs])
- img = Image.fromarray(stacked)
- # print(min_height)
- captions = [captions_dict[coco_id] for coco_id in coco_ids]
- for cap in captions:
- print(cap)
- display(img)
- # display_stimuli([16764, 79642])
- # %%
- for id1, row in df_err.iterrows():
- confusions = []
- for id2, count in row.items():
- if count > 5:
- confusions.append(id2)
- if len(confusions) > 0:
- print(f'common confusions involving id {id1}:\n')
- confusions = [id1] + confusions
- display_stimuli(confusions)
- print('\n\n')
- # %%
- # %%
- # %%
- # %%
modality_agnostic_decoding.ipynb at commit 57f7a1c, under MIT · at the source
Overview
- Université de Toulouse, CNRS, CerCo, Toulouse, France
- Torus AI, Toulouse, France
- Université de Toulouse, IRIT, Toulouse, France
Abstract
Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises six subjects watching both images and short text descriptions of such images, as well as the conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders and even outperform them when decoding captions and mental imagery. Furthermore, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.
Reproduced under the paper's license (CC BY), from the paper cited above.
Repository
Its files are read in the Code ↔ Paper reader above, with 17 matches between paragraphs and lines of code.
mitjanikolaus/multimodal_decoding
57f7a1c4a5da1c96887ed79f0507cbd286c727fd, 17 April 2026Availability: 1 check, the latest on 29 September 2026: the link answers
- 29 September 2026: the link answers
56 files
- analyses/
cluster_analysis.py , Python, 312 lines - analyses/
decoding/ , Python, 229 lines, 1 matchridge_regression_decodin g.py - analyses/
decoding/ , Python, 56 linessearchlight/ combine_cluster_maps.py - analyses/
decoding/ , Python, 362 linessearchlight/ searchlight.py - analyses/
decoding/ , Python, 53 lines, 1 matchsearchlight/ searchlight_cluster_manu al_corrections.py - analyses/
decoding/ , Python, 675 linessearchlight/ searchlight_permutation_ testing.py - analyses/
decoding/ , Python, 167 linessearchlight/ searchlight_results_maps .py - analyses/
supplementary/ , Python, 96 linescalc_noise_ceilings.py - analyses/
supplementary/ , Python, 96 linesevent_file_analysis.py - analyses/
supplementary/ , Python, 339 linesmodeling_decoding.py - analyses/
supplementary/ , Python, 121 linest_value_simulations.py - analyses/
visualization/ , Python, 60 linescreate_subcortical_atlas .py - analyses/
visualization/ , Python, 83 linesplot_correlation_imagery _mod_agnostic_regions.py - analyses/
visualization/ , Python, 141 lines, 2 matchesplot_dataset_quality_sta ts.py - analyses/
visualization/ , Python, 520 linesplotting_utils.py - analyses/
visualization/ , Python, 415 linessearchlight_plot_method. py - analyses/
visualization/ , Python, 312 linessearchlight_plot_results .py - analyses/
visualization/ , Python, 59 linest_val_threshold.py - analyses/
visualization/ , Python, 85 linesview_decoding_results_fr eeview.py - analyses/
visualization/ , Python, 65 linesview_encoding_results_fr eeview.py - data.py, Python, 574 lines, 1 match
- eval.py, Python, 223 lines
- feature_extraction/
extract_base_lm_features , Python, 106 lines.py - feature_extraction/
extract_base_vision_feat , Python, 134 linesures.py - feature_extraction/
extract_blip_features.py , Python, 57 lines - feature_extraction/
extract_bridgetower_feat , Python, 56 lines, 1 matchures.py - feature_extraction/
extract_clip_features.py , Python, 38 lines - feature_extraction/
extract_dino_features.py , Python, 55 lines - feature_extraction/
extract_flava_features.p , Python, 66 linesy - feature_extraction/
extract_gabor_features.p , Python, 204 linesy - feature_extraction/
extract_imagebind_featur , Python, 42 lineses.py - feature_extraction/
extract_paligemma_featur , Python, 109 lineses.py - feature_extraction/
extract_siglip_features. , Python, 42 lines, 1 matchpy - feature_extraction/
extract_vilt_features.py , Python, 65 lines - feature_extraction/
extract_visualbert_featu , Python, 274 linesres.py - feature_extraction/
feat_extraction_utils.py , Python, 108 lines - feature_extraction/
transform_glow_features. , Python, 117 linespy - notebooks/
analysis_ranking.ipynb , Jupyter, 549 lines - notebooks/
analysis_stuff.ipynb , Jupyter, 392 lines - notebooks/
imagery_decoding.ipynb , Jupyter, 278 lines - notebooks/
intersession_alignment.i , Jupyter, 139 lines, 1 matchpynb - notebooks/
modality_agnostic_decodi , Jupyter, 819 lines, 3 matchesng.ipynb - notebooks/
notebook_utils.py , Python, 277 lines, 1 match - notebooks/
roi_based_decoding.ipynb , Jupyter, 155 lines - notebooks/
zero_shot_cross_modal_de , Jupyter, 292 lines, 1 matchcoding.ipynb - preprocessing/
create_gray_matter_masks , Python, 55 lines.py - preprocessing/
create_symlinks_beta_fil , Python, 71 lineses.py - preprocessing/
fmri_preprocessing.py , Python, 220 lines, 2 matches - preprocessing/
make_spm_design_job_mat. , Python, 332 lines, 2 matchespy - preprocessing/
recon_script.py , Python, 32 lines - preprocessing/
run_spm_glm.m , MATLAB, 39 lines - preprocessing/
transform_to_surface.py , Python, 93 lines - setup.py, Python, 12 lines
- utils.py, Python, 110 lines
- LICENSE, License, 21 lines
- README.md, Text, 133 lines
The paper's code and data availability statement is in the Data section.
Tracing map
Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.
What the map holds:
- 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
- 54 scripts, each with its path and the digest of its content;
- 17 matches between paragraphs of the paper and lines of the code (method lexical-v1);
- neither the text of the paper nor the code itself.
Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.
Data
Datasets cited
- openneuro:ds007272, at OpenNeuro; found in “Data availability”
Data availability
All fMRI data is available at https://
The following dataset was generated:
Nikolaus M, Mozafari M, Berry I, Asher N, Reddy L, VanRullen R. 2025. SemReps-8K. OpenNeuro.
Reproduced under the paper's license (CC BY), from the paper cited above.
Versions
The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.
Version 1, 29 September 2026: the first record
Recorded: type, language, journal, volume, pages, dates, 6 authors, 1 keyword, 8 MeSH terms, 2 funders, 119 references.
Cite
This paper
Nikolaus, M., Mozafari, M., Berry, I., Asher, N., Reddy, L., & VanRullen, R. (2026). Modality-agnostic decoding of vision and language from fMRI. eLife, 14, RP107933. https://
BibTeX
@article{nikolaus2026mod
author = {Nikolaus, Mitja and Mozafari, Milad and Berry, Isabelle and Asher, Nicholas and Reddy, Leila and VanRullen, Rufin},
title = {{Modality-agnostic decoding of vision and language from fMRI}},
journal = {eLife},
year = {2026},
month = apr,
volume = {14},
pages = {RP107933},
publisher = {eLife Sciences Publications, Ltd},
issn = {2050-084X},
doi = {10.7554/
url = {https://
pmid = {41995708},
pmcid = {PMC13090028}
}
RIS
TY - JOUR
AU - Nikolaus, Mitja
AU - Mozafari, Milad
AU - Berry, Isabelle
AU - Asher, Nicholas
AU - Reddy, Leila
AU - VanRullen, Rufin
TI - Modality-agnostic decoding of vision and language from fMRI
T2 - eLife
J2 - eLife
PY - 2026
DA - 2026/
VL - 14
SP - RP107933
SN - 2050-084X
PB - eLife Sciences Publications, Ltd
DO - 10.7554/
UR - https://
LA - en
ER -
CSL-JSON
{
"id": "10.7554/
"type": "article-journal",
"title": "Modality-agnostic decoding of vision and language from fMRI",
"container-title": "eLife",
"author": [
{
"family": "Nikolaus",
"given": "Mitja"
},
{
"family": "Mozafari",
"given": "Milad"
},
{
"family": "Berry",
"given": "Isabelle"
},
{
"family": "Asher",
"given": "Nicholas"
},
{
"family": "Reddy",
"given": "Leila"
},
{
"family": "VanRullen",
"given": "Rufin"
}
],
"container-title-short":
"volume": "14",
"page": "RP107933",
"DOI": "10.7554/
"PMID": "41995708",
"PMCID": "PMC13090028",
"ISSN": "2050-084X",
"publisher": "eLife Sciences Publications, Ltd",
"URL": "https://
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
17
]
]
}
}
The tracing map gets a citation of its own once an author has validated it and it has a DOI.
Similar papers
The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.
- [1] doi:10.1371/journal.pcbi.1014263 [code]
- MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.Journal: PLoS computational biologyIn common: Hugging Face Transformers, OpenCV, scikit-image, 9 other tools, fMRI, 11 references
- [2] doi:10.1038/s42003-025-08706-4 [code]
- Evidence for compositionality in fMRI visual representations via Brain Algebra.Journal: Communications biologyIn common: Hugging Face Transformers, Nilearn, OpenCV, 11 other tools, fMRI, cognitive, 5 references, author Rufin VanRullen
- [3] doi:10.1126/sciadv.adw1464 [code]
- Mind captioning: Evolving descriptive text of mental content from human brain activity.Journal: Science advancesIn common: Hugging Face Transformers, OpenCV, h5py, 9 other tools, cognitive, 11 references
- [4] doi:10.1038/s41597-026-07248-6 [code]
- A large-scale fMRI dataset for vision-language semantic association.Journal: Scientific dataIn common: Nilearn, FreeSurfer, OpenCV, 10 other tools, fMRI, 10 references
- [5] doi:10.1038/s41598-023-42891-8 [code]
- Natural scene reconstruction from fMRI signals using generative latent diffusion.Journal: Scientific reportsIn common: Hugging Face Transformers, OpenCV, scikit-image, 8 other tools, fMRI, 6 references, author Rufin VanRullen
- [6] doi:10.1038/s41467-024-49173-5 [code]
- Shared functional specialization in transformer-based language models and the human brain.Journal: Nature communicationsIn common: Hugging Face Transformers, Nilearn, FreeSurfer, 10 other tools, 7 references
- [7] doi:10.1016/j.isci.2024.110229 [code]
- Laminar dissociation of feedforward and feedback in high-level ventral visual cortex during imagery and perception.Journal: iScienceIn common: Hugging Face Transformers, Nipype, Nilearn, 11 other tools, cognitive, 4 references
- [8] doi:10.1038/s41562-025-02252-z [code]
- In silico discovery of representational relationships across visual cortex.Journal: Nature human behaviourIn common: Nilearn, FreeSurfer, SPM, 11 other tools, fMRI, 6 references
- [9] doi:10.1038/s42003-026-10169-0 [code]
- Shared representations in brains and models reveal a two-route cortical organization during scene perception.Journal: Communications biologyIn common: Hugging Face Transformers, h5py, Pillow, 9 other tools, cognitive, 7 references
- [10] doi:10.1038/s41467-024-50310-3 [code]
- Modeling short visual events through the BOLD moments video fMRI dataset and metadata.Journal: Nature communicationsIn common: Nilearn, FreeSurfer, Pillow, 8 other tools, fMRI, cognitive, 8 references
Contribute
The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.
Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.
Claim this paper
Correct its record
Say what each link of this record is, remove the ones that are not the paper's, add the ones that are missing. The correction becomes a new version of the record, in its Versions section.
Validate its tracing map
You validate the map as this page shows it: 1 repository of the authors' code, each at its verified commit and with its license, 54 scripts, and 17 matches between paragraphs and code (see the Code and Map sections). It then receives a DOI on Zenodo, with you (your ORCID iD) and OSCR as its creators; the code itself is not deposited.
The map's fingerprint: sha256:a3343c0d96f0c3ad…
Add the badge to the code's README
If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.
Markdown
<!-- oscr-badge:start -->
[](https://openscicode.org/badge/doi/10.7554/elife.107933)
<!-- oscr-badge:end -->reStructuredText
.. oscr-badge:start
.. image:: https://openscicode.org/badge/doi/10.7554/elife.107933.svg
:target: https://openscicode.org/badge/doi/10.7554/elife.107933
:alt: OSCR
.. oscr-badge:endHTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.7554/elife.107933"><img src="https://openscicode.org/badge/doi/10.7554/elife.107933.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.
Request its removal
To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).
Discussion, reproductions, activity
Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.
Report a mismatch between the code and the paper · Report a code error (signed in)
Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.
Report a reproduction failure (signed in)
Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.
Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.
