OSCR

Shared representations in brains and models reveal a two-route cortical organization during scene perception.

Code ↔ Paper

10 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 10 matches
  1. [1] § Results › Inter-subject shared geometry during scene viewing ↔ 5_notebooks/01_rsa_spatial_alignment.ipynb, lines 921–1016 · score 0.73 · power law, CI, language model alignment, bootstrap, model brain alignment, LOTC hub
  2. [2] § Results › Model-brain alignment reveals modality-specific shared geometry ↔ 5_notebooks/01_rsa_spatial_alignment.ipynb, lines 921–1016 · score 0.70 · power law, RSA spatial, model brain alignment, LOTC hubs, vision model alignment, overlap
  3. [3] § Results › Inter-subject shared geometry during scene viewing ↔ 3_alignment/2_rsa_nsd_subject_model_alignment.py, lines 1–64 · score 0.67 · deep neural networks, model RSA, fMRI, model layer, vector, activations
  4. [4] § Results › Inter-subject shared geometry during scene viewing ↔ 3_alignment/6_rsa_nsd_subject_group_subject_alignment.py, lines 1–56 · score 0.62 · representational dissimilarity matrices, shifted repetition, inter subject alignment, cortical, RDMs, hemisphere
  5. [5] § Results › Inter-subject shared geometry during scene viewing ↔ 3_alignment/7_rsa_other_subject_subject_alignment.py, lines 1–81 · score 0.59 · representational dissimilarity matrices, brain activity, inter subject RSA, RDMs, scene, alignment
  6. [6] § Methods › Alignment measure ↔ 3_alignment/7_rsa_other_subject_subject_alignment.py, lines 1–81 · score 0.55 · representational dissimilarity matrix, RSA score, RDM, brain, Alignment
  7. [7] § Results › Hierarchical correspondence between models and cortex ↔ 5_notebooks/03_rsa_model_hierarchy.ipynb, lines 534–575 · score 0.54 · Cortical surface map, peak alignment depths, normalized peak, vision model alignment, hierarchy, Ventral
  8. [8] § Methods › Model features extraction ↔ 4_other_alignments/7_untrained_models_alignment.py, lines 1–59 · score 0.54 · models encode, fMRI, training, computational, language, vision
  9. [9] § Results › A representational network linking ventromedial and lateral hubs ↔ 5_notebooks/03_rsa_model_hierarchy.ipynb, lines 437–525 · score 0.52 · peak alignment depth, LOTC hub, Ventral hub, FST, V4t, VMV1
  10. [10] § Results › Model-brain alignment reveals modality-specific shared geometry ↔ 2_model_extraction/1_extract_vision_features.py, lines 1–41 · score 0.52 · Vision Transformer, extracted layer, pretrained, vision models, activations, NSD

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 1,158 lines · 38 KB · MIT · 2 matches

  1. # %% [markdown]
  2. # # 1 - RSA Spatial Alignment
  3. #
  4. # Generate spatial RSA maps for inter‑subject and subject‑model alignment, including boxplots, cortical surface projections, scatter comparisons, and hub‐based contrasts.
  6. # %% [markdown]
  7. # ## 0. Imports & Setup
  8. # Load required libraries (path handling, data frames, plotting, stats), plus project utilities for power‐law fitting, parcel plotting, figure styling, and alignment processing. Initialize fonts.
  10. # %%
  11. # add .. to path for convergence module
  12. import sys
  13. from pathlib import Path
  14. sys.path.append(str(Path.cwd().parent))
  16. # %%
  17. from pathlib import Path
  19. import pandas as pd
  20. import matplotlib.lines as mlines
  21. import matplotlib.pyplot as plt
  22. import matplotlib.ticker as mticker
  23. import numpy as np
  24. import seaborn as sns
  25. from scipy.stats import ttest_rel
  26. from statsmodels.stats.multitest import multipletests
  28. # Functions reused in other parts of the project
  29. from convergence.power_law import fit_powerlaw_zero_with_bootstrap
  30. from convergence.plotting import plot_faverage_parcelation, add_area_labels
  31. from convergence.figures import (
  32. plot_boxplot_rois,
  33. add_pvalue_bracket,
  34. plot_cbar,
  35. plot_comparison,
  36. setup_matplotlib_fonts,
  37. )
  38. from convergence.figures_utils import add_cluster, process_intersubject_rois, proccess_alignment
  40. setup_matplotlib_fonts()
  42. # %% [markdown]
  43. # ## 1. Data Paths & Validation
  44. # Define paths to derivatives folder, metadata (HCP atlas, model info), RSA p‑value tables, and alignment parquet files. Create output directory and assert that all required inputs exist.
  46. # %%
  47. # Required filenames for the figures
  48. data_folder = Path("../derivatives")
  49. hcp_filename = data_folder / "metadata" / "hcp.csv"
  50. models_filename = data_folder / "metadata" / "models-info.csv"
  51. nsd_folder = data_folder / "nsd"
  52. pvalues_filename = nsd_folder / "rsa_pvalues_subject_language_vision.parquet"
  53. intersubject_shift1_joined_filename = nsd_folder / "rsa_subject_subject_alignment_shift_1_joined.parquet"
  54. intersubject_shift1_separated_filename = nsd_folder / "rsa_subject_subject_alignment_shift_1_separated.parquet"
  55. model_brain_joined_filename = nsd_folder / "rsa_subject_model_alignment_joined.parquet"
  56. model_brain_separated_filename = nsd_folder / "rsa_subject_model_alignment_separated.parquet"
  58. # Output folder for figures
  59. figure_folder = Path("../figures")
  60. output = figure_folder / "01_rsa_spatial_alignment"
  61. output.mkdir(exist_ok=True, parents=True)
  64. # %% [markdown]
  65. # ## 2. ROI RSA Boxplots
  66. # Visualize ROI‑wise RSA distributions with null distributions and statistical annotations.
  68. # %% [markdown]
  69. # ### 2.1 Inter‑subject — Top Areas
  70. # – Select the top 10 anatomical areas by mean alignment.
  71. # – Overlay group‑level null mean ± std.
  72. # – Draw per‑ROI boxplots of inter‑subject RSA and annotate paired t‑test p‑value brackets.
  74. # %%
  75. # Load the inter-subject alignment data
  76. df = pd.read_parquet(intersubject_shift1_joined_filename)
  78. # Process the data to get the top 10 ROIs (aggregate and filter) and select rois of top areas
  79. df_g = process_intersubject_rois(df, hcp_filename=hcp_filename, top=10)
  80. order = (
  81. df_g.sort_values(["area_id", "roi_order"])
  82. .drop_duplicates("roi")
  83. .drop_duplicates("name")
  84. .name.tolist()
  85. )
  86. order_roi = df_g.sort_values(["area_id", "roi_order"]).drop_duplicates("roi").roi.tolist()
  87. hue_order = df_g.sort_values("area_id").drop_duplicates("area_id").area.tolist()
  88. palette = df_g.sort_values("area_id").drop_duplicates("area_id").area_color.tolist()
  91. # Sort by order
  92. df_subject_pvalues = pd.read_parquet(pvalues_filename).query("comparison == 'intersubject'")
  93. df_subject_pvalues.name = df_subject_pvalues.name.replace("H", "Hipp")
  94. df_subject_pvalues = df_subject_pvalues.set_index("roi")
  95. df_subject_pvalues = df_subject_pvalues.loc[order_roi].reset_index()
  97. # The df_subject_pvalues is sorted in the same order that boxplots.
  98. # Plot the mean line given by null_mean column of df_subject_pvalues
  99. null_std = df_subject_pvalues.null_std.values
  100. null_mean = df_subject_pvalues.null_mean.values
  102. fig, ax = plt.subplots(1, 1, figsize=(20, 4), dpi=300)
  104. ax.fill_between(
  105. np.arange(len(order)), null_mean - null_std, null_mean + null_std, color="maroon", alpha=0.15
  106. )
  107. ax.plot(
  108. np.arange(len(order)),
  109. df_subject_pvalues.null_mean.values,
  110. color="maroon",
  111. linestyle="--",
  112. linewidth=1,
  113. )
  115. plot_boxplot_rois(
  116. df_g,
  117. ax,
  118. order,
  119. hue_order,
  120. palette,
  121. title="",
  122. legend=True,
  123. fontsize=10,
  124. legend_fontsize=13,
  125. legend_kwargs={"bbox_to_anchor": (1, 1), "title": ""},
  126. vmax=0.3,
  127. )
  128. ax.set_ylabel(r"Inter-subject Alignment (RSA)", fontsize=12)
  130. add_pvalue_bracket(
  131. ax=ax,
  132. x1=0,
  133. x2=3,
  134. y=0.25 - 0.011,
  135. y_text=0.268 - 0.011,
  136. height=0.008,
  137. text="Early Visual\nCortex",
  138. )
  139. add_pvalue_bracket(
  140. ax=ax, x1=8, x2=13, y=0.25 - 0.011, y_text=0.268 - 0.011, height=0.008, text="Ventral Hub"
  141. )
  142. add_pvalue_bracket(ax=ax, x1=30, x2=35, y=0.045, y_text=0.02, height=-0.008, text="LOTC Hub")
  145. # Set yaxis (RSA) in decimal format not percentage
  146. ax.yaxis.set_major_formatter(mticker.FormatStrFormatter("%.2f"))
  147. ax.set_ylim(-0.015, 0.30)
  148. ax.set_xlim(-0.8, 71.8)
  149. ax.text(
  150. 0,
  151. 0.008,
  152. r"Group-level null distribution (mean $\pm$ std)",
  153. color="maroon",
  154. fontsize=8,
  155. ha="left",
  156. )
  158. # Add p-values
  159. apa_significance = df_subject_pvalues.apa_star.values
  160. for i, (p, name) in enumerate(zip(apa_significance, order)):
  161. if p != "***":
  162. if p == "n.s.":
  163. ax.text(i, 0.012, p, fontsize=7, ha="center")
  164. else:
  165. ax.text(i, 0.01, p, fontsize=11, ha="center")
  167. # fig.savefig(
  168. # output / "01_rsa_intersubject_boxplot_selected_shift_1.pdf",
  169. # bbox_inches="tight",
  170. # transparent=True,
  171. # )
  173. # %% [markdown]
  174. # ### 2.2 Inter‑subject — All Areas
  175. # – Repeat full‑brain boxplot for every ROI.
  176. # – Adjust strip sizes and fonts for high‑density plotting.
  177. # – Annotate key area comparisons.
  179. # %%
  180. df = pd.read_parquet(intersubject_shift1_joined_filename)
  181. df_g = process_intersubject_rois(
  182. df, hcp_filename=hcp_filename, top=1000
  183. ) # If select more than max rois it will take all
  184. order = df_g.sort_values(["area_id", "roi_order"]).drop_duplicates("name").name.unique()
  185. order_roi = df_g.sort_values(["area_id", "roi_order"]).drop_duplicates("roi").roi.unique()
  186. hue_order = list(df_g.sort_values("area_id").drop_duplicates("area_id").area.tolist())
  187. palette = list(df_g.sort_values("area_id").drop_duplicates("area_id").area_color.tolist())
  188. models_info = pd.read_csv(models_filename)
  190. # Sort by order
  191. df_subject_pvalues = pd.read_parquet(pvalues_filename).query("comparison == 'intersubject'")
  192. df_subject_pvalues.name = df_subject_pvalues.name.replace("H", "Hipp")
  193. df_subject_pvalues = df_subject_pvalues.set_index("roi")
  194. #
  195. df_subject_pvalues = df_subject_pvalues.loc[order_roi].reset_index()
  198. fig, ax = plt.subplots(1, 1, figsize=(18, 4), dpi=300)
  200. null_mean = df_subject_pvalues.null_mean.values
  201. null_std = df_subject_pvalues.null_std.values
  203. ax.fill_between(
  204. np.arange(len(order)), null_mean - null_std, null_mean + null_std, color="maroon", alpha=0.15
  205. )
  206. ax.plot(
  207. np.arange(len(order)),
  208. df_subject_pvalues.null_mean.values,
  209. color="maroon",
  210. linestyle="--",
  211. linewidth=0.7,
  212. )
  214. plot_boxplot_rois(
  215. df_g,
  216. ax,
  217. order,
  218. hue_order,
  219. palette,
  220. title="",
  221. legend=True,
  222. fontsize=6,
  223. legend_fontsize=9,
  224. legend_kwargs={"bbox_to_anchor": (1, 1), "title": ""},
  225. vmax=0.3,
  226. strip_size=2,
  227. )
  228. ax.set_ylabel("Inter-subject Alignment (RSA)")
  229. ax.text(0, 0.008, "Null distribution\n(mean ± std)", color="maroon", fontsize=8, ha="left")
  231. # Set yaxis (RSA) in decimal format not percentage
  232. ax.yaxis.set_major_formatter(mticker.FormatStrFormatter("%.2f"))
  233. ax.set_title("Inter-subject RSA Alignment")
  235. apa_significance = df_subject_pvalues.apa_star.values
  236. max_observed_values = df_g.groupby("name").aggregate({"score": "max"}).loc[order].score.values
  238. for i, (p, name, max_observed) in enumerate(zip(apa_significance, order, max_observed_values)):
  239. if p != "***":
  240. if p == "n.s.":
  241. ax.text(i, max_observed + 0.01, p, fontsize=6, ha="center", rotation=90, va="bottom")
  242. else:
  243. ax.text(i, max_observed + 0.005, p, fontsize=6, ha="center", rotation=0, va="center")
  246. add_pvalue_bracket(
  247. ax=ax,
  248. x1=0,
  249. x2=3,
  250. y=0.25 - 0.006,
  251. y_text=0.268 - 0.011,
  252. height=0.006,
  253. text="Visual\nCortex",
  254. fontsize=7,
  255. )
  256. add_pvalue_bracket(
  257. ax=ax,
  258. x1=8,
  259. x2=13,
  260. y=0.25 - 0.006,
  261. y_text=0.268 - 0.011,
  262. height=0.006,
  263. text="Ventral Hub",
  264. fontsize=7,
  265. )
  266. add_pvalue_bracket(
  267. ax=ax,
  268. x1=38,
  269. x2=43,
  270. y=0.29 - 0.006 - 0.006,
  271. y_text=0.29 - 0.011 + 0.018 - 0.007,
  272. height=0.006,
  273. text="LOTC Hub",
  274. fontsize=7,
  275. )
  277. ax.set_ylim
  279. # fig.savefig(
  280. # output / "01_rsa_intersubject_boxplot_all_shift_1.pdf", bbox_inches="tight", transparent=True
  281. # )
  283. # %% [markdown]
  284. # ## Within subject all areas
  286. # %%
  287. df = pd.read_parquet(intersubject_shift1_joined_filename)
  288. df_g = process_intersubject_rois(
  289. df, hcp_filename=hcp_filename, top=1000, within_subject=True
  290. ) # If select more than max rois it will take all
  291. order = df_g.sort_values(["area_id", "roi_order"]).drop_duplicates("name").name.unique()
  292. order_roi = df_g.sort_values(["area_id", "roi_order"]).drop_duplicates("roi").roi.unique()
  293. hue_order = list(df_g.sort_values("area_id").drop_duplicates("area_id").area.tolist())
  294. palette = list(df_g.sort_values("area_id").drop_duplicates("area_id").area_color.tolist())
  295. models_info = pd.read_csv(models_filename)
  297. # Sort by order
  298. df_subject_pvalues = pd.read_parquet(pvalues_filename).query("comparison == 'intersubject'")
  299. df_subject_pvalues.name = df_subject_pvalues.name.replace("H", "Hipp")
  300. df_subject_pvalues = df_subject_pvalues.set_index("roi")
  301. #
  302. df_subject_pvalues = df_subject_pvalues.loc[order_roi].reset_index()
  305. fig, ax = plt.subplots(1, 1, figsize=(18, 4), dpi=300)
  307. null_mean = df_subject_pvalues.null_mean.values
  308. null_std = df_subject_pvalues.null_std.values
  310. ax.fill_between(
  311. np.arange(len(order)), null_mean - null_std, null_mean + null_std, color="maroon", alpha=0.15
  312. )
  313. ax.plot(
  314. np.arange(len(order)),
  315. df_subject_pvalues.null_mean.values,
  316. color="maroon",
  317. linestyle="--",
  318. linewidth=0.7,
  319. )
  321. plot_boxplot_rois(
  322. df_g,
  323. ax,
  324. order,
  325. hue_order,
  326. palette,
  327. title="",
  328. legend=True,
  329. fontsize=6,
  330. legend_fontsize=9,
  331. legend_kwargs={"bbox_to_anchor": (1, 1), "title": ""},
  332. vmax=0.55,
  333. strip_size=2,
  334. )
  335. ax.set_ylabel("Inter-subject Alignment (RSA)")
  336. ax.text(0, 0.008, "Null distribution\n(mean ± std)", color="maroon", fontsize=8, ha="left")
  338. # Set yaxis (RSA) in decimal format not percentage
  339. ax.yaxis.set_major_formatter(mticker.FormatStrFormatter("%.2f"))
  340. ax.set_title("Inter-subject RSA Alignment")
  342. apa_significance = df_subject_pvalues.apa_star.values
  343. max_observed_values = df_g.groupby("name").aggregate({"score": "max"}).loc[order].score.values
  345. for i, (p, name, max_observed) in enumerate(zip(apa_significance, order, max_observed_values)):
  346. if p != "***":
  347. if p == "n.s.":
  348. ax.text(i, max_observed + 0.01, p, fontsize=6, ha="center", rotation=90, va="bottom")
  349. else:
  350. ax.text(i, max_observed + 0.005, p, fontsize=6, ha="center", rotation=0, va="center")
  353. add_pvalue_bracket(
  354. ax=ax,
  355. x1=0,
  356. x2=3,
  357. y=0.25 - 0.006,
  358. y_text=0.268 - 0.011,
  359. height=0.006,
  360. text="Visual\nCortex",
  361. fontsize=7,
  362. )
  363. add_pvalue_bracket(
  364. ax=ax,
  365. x1=8,
  366. x2=13,
  367. y=0.25 - 0.006,
  368. y_text=0.268 - 0.011,
  369. height=0.006,
  370. text="Ventral Hub",
  371. fontsize=7,
  372. )
  373. add_pvalue_bracket(
  374. ax=ax,
  375. x1=38,
  376. x2=43,
  377. y=0.29 - 0.006 - 0.006,
  378. y_text=0.29 - 0.011 + 0.018 - 0.007,
  379. height=0.006,
  380. text="LOTC Hub",
  381. fontsize=7,
  382. )
  384. ax.set_ylim(-0.03, 0.53)
  385. ax.set_title("Within-subject RSA Alignment")
  386. ax.set_ylabel("Within-subject Alignment (RSA)")
  388. # fig.savefig(
  389. # output / "01_rsa_withinsubject_boxplot_all_shift_1.pdf", bbox_inches="tight", transparent=True
  390. # )
  392. # %%
  393. models_info = pd.read_csv(models_filename)
  395. df_comparison_subjects = proccess_alignment(
  396. models_filename=models_filename,
  397. models_alignment_filename=model_brain_joined_filename,
  398. subject_alignment_filename=intersubject_shift1_joined_filename,
  399. pvalues_filename=pvalues_filename,
  400. hcp_filename=hcp_filename,
  401. group_subject=False,
  402. )
  403. df = pd.read_parquet(intersubject_shift1_joined_filename)
  404. df_g_within = process_intersubject_rois(
  405. df, hcp_filename=hcp_filename, top=1000, within_subject=True
  406. ) # If select more than max rois it will take all
  407. df_g_within = df_g_within[["roi", "subject", "score"]].rename(columns={"score": "within_subject_rsa"})
  408. df_comparison_subjects = df_comparison_subjects.merge(df_g_within, on=["roi", "subject"], how="left")
  409. df = df_comparison_subjects[["roi", "subject", "intersubject_rsa", "within_subject_rsa", "vision_rsa", "language_rsa"]]
  411. hcp = pd.read_csv(hcp_filename)[["roi", "name", "area", "area_id", "area_color", "roi_order"]]
  412. df = df.merge(hcp, on="roi")
  414. # %%
  415. # Convert to long format for seaborn with roi, subject, name, area, measure (intersubject_rsa, within_subject_rsa, vision_rsa, language_rsa) and the score
  416. df_long = pd.melt(
  417. df,
  418. id_vars=["roi", "subject", "name", "area", "area_id", "area_color", "roi_order"],
  419. value_vars=["intersubject_rsa", "within_subject_rsa", "vision_rsa", "language_rsa"],
  420. var_name="measure",
  421. value_name="score",
  422. )
  424. df_subplot = df_long.query("area == 'Posterior Cingulate' and measure != 'language_rsa'").copy()
  425. order = df_subplot.query("measure == 'vision_rsa'").groupby("name").score.mean().reset_index().sort_values("score", ascending=False).name.tolist()
  427. hue_order = ["vision_rsa", "within_subject_rsa", "intersubject_rsa",]
  428. new_labels = {
  429. "vision_rsa": "Vision Model - Brain RSA",
  430. "within_subject_rsa": "Within-subject RSA",
  431. "intersubject_rsa": "Inter-subject RSA",
  432. }
  434. hue_order_new = [new_labels[h] for h in hue_order]
  435. df_subplot["measure"] = df_subplot["measure"].replace(new_labels)
  437. fig, (ax, ax2, ax3) = plt.subplots(1, 3, figsize=(4*4, 4))
  440. sns.boxplot(data=df_subplot, x="name", y="score", hue="measure", hue_order=hue_order_new, ax=ax,
  441. showfliers=False, order=order,
  442. )
  443. # Add an stripplot
  444. sns.stripplot(data=df_subplot, x="name", y="score", hue="measure", hue_order=hue_order_new,
  445. dodge=True, size=3, linewidth=0.8, ax=ax, legend=False, order=order,
  446. )
  448. # Remove legend title
  449. ax.legend_.set_title("")
  450. # Move legend to left center
  451. ax.legend_.set_bbox_to_anchor((1, 0.8))
  452. sns.despine(ax=ax)
  453. ax.set_xlabel("")
  454. ax.set_ylabel("RSA Score (Pearson's $\\rho$)")
  455. # Rotate xlabels
  456. # Set ticklabels for avoid warning
  457. ax.set_xticks(ax.get_xticks())
  458. ax.set_xticklabels(ax.get_xticklabels(), rotation=90, ha="center", fontsize=10);
  459. ax.set_title("Posterior Cingulate Cortex regions")
  461. palette = df.sort_values("area").drop_duplicates("area").set_index("area").area_color.to_dict()
  462. hue_order = df.sort_values("area").drop_duplicates("area").area.tolist()
  463. df_grouped = df.groupby(["roi", "name", "area"]).aggregate({"intersubject_rsa": "mean", "within_subject_rsa": "mean",
  464. "vision_rsa": "mean", "language_rsa": "mean"}).reset_index()
  467. for x_column, ax in zip(["intersubject_rsa", "within_subject_rsa"], [ax2, ax3]):
  469. df_grouped["greater"] = df_grouped["vision_rsa"] > df_grouped[x_column]
  471. sns.scatterplot(data=df_grouped, x=x_column, y="vision_rsa", hue="area", legend=False, palette=palette, hue_order=hue_order,
  472. style="greater", markers={True: "X", False: "o"}, ax=ax)
  474. # Plot x=y line
  475. max_val = max(df_grouped[x_column].max(), df_grouped["vision_rsa"].max())
  476. ax.plot([0, max_val], [0, max_val], color=(0.24, 0.24, 0.24), linestyle="--", zorder=-1000, lw=1)
  478. # Mark text of all regions of the posterior cingulate
  479. for i, row in df_grouped.query("name in ['DVT', 'POS1', 'ProS']").iterrows():
  480. ax.text(row[x_column], row["vision_rsa"] + 0.005, row["name"], fontsize=7,ha='center')
  484. xmin = df_grouped[x_column].min()
  486. # # Plot the powerlaw fit line
  487. x_fit, y_fit, y_lower, y_upper, r2, params = fit_powerlaw_zero_with_bootstrap(
  488. df_grouped[x_column] - xmin, df_grouped["vision_rsa"], n_boot=10000
  489. )
  490. print(r2)
  491. x_fit = x_fit + xmin
  493. ax.plot(
  494. x_fit, y_fit, color="maroon", label=f"$f(x)={params[0]:.2f}x^{{{params[1]:.2f}}}$", zorder=-10
  495. )
  496. ax.fill_between(x_fit, y_lower, y_upper, color="maroon", alpha=0.1, zorder=-20)
  497. x = np.linspace(0, 0.2, 100)
  498. ax.fill_between(x, x, 0.2, color='gray', alpha=0.1, label='y > x', zorder=-1000)
  501. ax.legend(loc="lower right")
  502. sns.despine(ax=ax)
  503. ax.set_ylabel("Vision Model - Brain RSA")
  504. ax.set_ylim(0, 0.18)
  506. ax2.set_xlabel("Inter-subject RSA")
  507. ax2.set_title("Inter-subject vs Vision Model alignment")
  508. ax3.set_xlabel("Within-subject RSA")
  509. ax3.set_title("Within-subject vs Vision Model alignment")
  512. #fig.savefig(figure_folder / "18_revision" / "posterior_cingulate_rsa_alignment.pdf", bbox_inches="tight", transparent=True)
  514. # %% [markdown]
  515. # ### 2.3 Subject‑Model — Vision Models
  516. # – Compute per‑ROI vision model vs. brain RSA.
  517. # – Overlay group null, plot boxplots, annotate p‑values, and save with/without legend.
  519. # %%
  520. # Load models info
  521. models_info = pd.read_csv(models_filename)
  523. df_comparison_subjects = proccess_alignment(
  524. models_filename=models_filename,
  525. models_alignment_filename=model_brain_joined_filename,
  526. subject_alignment_filename=intersubject_shift1_joined_filename,
  527. pvalues_filename=pvalues_filename,
  528. hcp_filename=hcp_filename,
  529. group_subject=False,
  530. )
  531. df_comparison_subjects.name = df_comparison_subjects.name.replace("H", "Hipp")
  532. df_comparison_subjects = df_comparison_subjects.sort_values(["area_id", "roi_order"]).reset_index(
  533. drop=True
  534. )
  535. order = (
  536. df_comparison_subjects.sort_values(["area_id", "roi_order"])
  537. .drop_duplicates("name")
  538. .name.unique()
  539. )
  540. order_roi = (
  541. df_comparison_subjects.sort_values(["area_id", "roi_order"]).drop_duplicates("roi").roi.unique()
  542. )
  543. hue_order = list(
  544. df_comparison_subjects.sort_values("area_id").drop_duplicates("area_id").area.tolist()
  545. )
  546. palette = list(
  547. df_comparison_subjects.sort_values("area_id").drop_duplicates("area_id").area_color.tolist()
  548. )
  551. null_mean = df_comparison_subjects.drop_duplicates("roi").vision_null_mean.values
  552. null_std = df_comparison_subjects.drop_duplicates("roi").vision_null_std.values
  554. # Make the plot
  555. fig, ax = plt.subplots(1, 1, figsize=(18, 4), dpi=300)
  556. ax.fill_between(
  557. np.arange(len(order)), null_mean - null_std, null_mean + null_std, color="maroon", alpha=0.15
  558. )
  559. ax.plot(np.arange(len(order)), null_mean, color="maroon", linestyle="--", linewidth=0.7)
  560. plot_boxplot_rois(
  561. df_comparison_subjects.rename(columns={"vision_rsa": "score"}),
  562. ax,
  563. order,
  564. hue_order,
  565. palette,
  566. title="",
  567. legend=True,
  568. fontsize=6,
  569. legend_fontsize=9,
  570. legend_kwargs={"bbox_to_anchor": (1, 1), "title": ""},
  571. vmax=0.23,
  572. strip_size=2,
  573. )
  574. ax.set_ylabel("Vision Model - Brain Alignment (RSA)")
  575. ax.text(
  576. 0,
  577. -0.006,
  578. "Group-level null distribution\n(mean ± std)",
  579. color="maroon",
  580. fontsize=8,
  581. ha="left",
  582. va="top",
  583. )
  585. # Plot the significance of the p-values
  586. apa_significance = df_comparison_subjects.drop_duplicates("roi").vision_apa_star
  587. max_observed_values = (
  588. df_comparison_subjects.groupby("name")
  589. .aggregate({"vision_rsa": "max"})
  590. .loc[order]
  591. .vision_rsa.values
  592. )
  594. for i, (p, name, max_observed) in enumerate(zip(apa_significance, order, max_observed_values)):
  595. if p != "***":
  596. # print(p, name, max_observed)
  597. if p == "n.s.":
  598. ax.text(i, max_observed + 0.01, p, fontsize=6, ha="center", rotation=90, va="bottom")
  599. else:
  600. ax.text(i, max_observed + 0.005, p, fontsize=6, ha="center", rotation=0, va="center")
  603. add_pvalue_bracket(
  604. ax=ax, x1=0, x2=3, y=0.20, y_text=0.21, height=0.005, text="Visual\nCortex", fontsize=7
  605. )
  606. add_pvalue_bracket(
  607. ax=ax, x1=8, x2=13, y=0.175, y_text=0.185, height=0.005, text="Ventral Hub", fontsize=7
  608. )
  609. add_pvalue_bracket(
  610. ax=ax, x1=38, x2=43, y=0.18, y_text=0.19, height=0.005, text="LOTC Hub", fontsize=7
  611. )
  613. ax.set_title("Vision Models - Brain RSA Alignment")
  615. ax.yaxis.set_major_formatter(mticker.FormatStrFormatter("%.2f"))
  616. ax.set_ylim(-0.043, 0.21)
  618. # Save with legend
  619. fig.savefig(output / "02_rsa_vision_boxplot_all.pdf", bbox_inches="tight", transparent=True)
  621. # Save without legend
  622. ax.get_legend().remove()
  623. fig.savefig(
  624. output / "03_rsa_vision_boxplot_all_no_legend.pdf", bbox_inches="tight", transparent=True
  625. )
  627. # %% [markdown]
  628. #
  629. # ### 2.4 Subject‑Model — Language Models
  630. # – Same as 2.3 but for language models, with separate null overlay and significance brackets.
  632. # %%
  633. models_info = pd.read_csv(models_filename)
  635. df_comparison_subjects = proccess_alignment(
  636. models_filename=models_filename,
  637. models_alignment_filename=model_brain_joined_filename,
  638. subject_alignment_filename=intersubject_shift1_joined_filename,
  639. pvalues_filename=pvalues_filename,
  640. hcp_filename=hcp_filename,
  641. group_subject=False,
  642. )
  643. df_comparison_subjects.name = df_comparison_subjects.name.replace("H", "Hipp")
  644. df_comparison_subjects = df_comparison_subjects.sort_values(["area_id", "roi_order"]).reset_index(
  645. drop=True
  646. )
  647. order = (
  648. df_comparison_subjects.sort_values(["area_id", "roi_order"])
  649. .drop_duplicates("name")
  650. .name.unique()
  651. )
  652. order_roi = (
  653. df_comparison_subjects.sort_values(["area_id", "roi_order"]).drop_duplicates("roi").roi.unique()
  654. )
  655. hue_order = list(
  656. df_comparison_subjects.sort_values("area_id").drop_duplicates("area_id").area.tolist()
  657. )
  658. palette = list(
  659. df_comparison_subjects.sort_values("area_id").drop_duplicates("area_id").area_color.tolist()
  660. )
  661. null_mean = df_comparison_subjects.drop_duplicates("roi").language_null_mean.values
  662. null_std = df_comparison_subjects.drop_duplicates("roi").language_null_std.values
  665. fig, ax = plt.subplots(1, 1, figsize=(18, 4), dpi=300)
  666. ax.fill_between(
  667. np.arange(len(order)), null_mean - null_std, null_mean + null_std, color="maroon", alpha=0.15
  668. )
  669. ax.plot(np.arange(len(order)), null_mean, color="maroon", linestyle="--", linewidth=0.7)
  671. plot_boxplot_rois(
  672. df_comparison_subjects.rename(columns={"language_rsa": "score"}),
  673. ax,
  674. order,
  675. hue_order,
  676. palette,
  677. title="",
  678. legend=True,
  679. fontsize=6,
  680. legend_fontsize=9,
  681. legend_kwargs={"bbox_to_anchor": (1, 1), "title": ""},
  682. vmax=0.23,
  683. strip_size=2,
  684. )
  685. ax.set_ylabel("Language Model - Brain Alignment (RSA)")
  686. ax.yaxis.set_major_formatter(mticker.FormatStrFormatter("%.2f"))
  689. apa_significance = df_comparison_subjects.drop_duplicates("roi").language_apa_star
  690. max_observed_values = (
  691. df_comparison_subjects.groupby("name")
  692. .aggregate({"language_rsa": "max"})
  693. .loc[order]
  694. .language_rsa.values
  695. )
  697. for i, (p, name, max_observed) in enumerate(zip(apa_significance, order, max_observed_values)):
  698. if p != "***":
  699. if p == "n.s.":
  700. ax.text(i, max_observed + 0.01, p, fontsize=6, ha="center", rotation=90, va="bottom")
  701. else:
  702. ax.text(i, max_observed + 0.005, p, fontsize=6, ha="center", rotation=0, va="center")
  705. add_pvalue_bracket(
  706. ax=ax, x1=0, x2=3, y=0.07, y_text=0.08, height=0.005, text="Visual\nCortex", fontsize=7
  707. )
  708. add_pvalue_bracket(
  709. ax=ax, x1=8, x2=13, y=0.1, y_text=0.11, height=0.005, text="Ventral\nHub", fontsize=7
  710. )
  711. add_pvalue_bracket(
  712. ax=ax, x1=38, x2=43, y=0.22, y_text=0.23, height=0.005, text="LOTC Hub", fontsize=7
  713. )
  715. ax.set_ylim(-0.08, 0.23)
  717. ax.set_title("Language Models - Brain RSA Alignment")
  718. fig.savefig(
  719. output / "04_rsa_language_boxplot_all.pdf", bbox_inches="tight", dpi=300, transparent=True
  720. )
  722. # Save without legend
  723. ax.get_legend().remove()
  724. fig.savefig(
  725. output / "05_rsa_language_boxplot_all_no_legend.pdf",
  726. bbox_inches="tight",
  727. dpi=300,
  728. transparent=True,
  729. )
  731. # %% [markdown]
  732. # ## 3. Group Average Cortical Maps
  733. # Project group‑mean RSA values onto the fsaverage surface for three modalities: inter‑subject, vision, and language.
  735. # %% [markdown]
  736. # ### 3.1 Surface Plotting
  737. # For each modality and hemisphere (lh/rh), render lateral, medial, ventral, and caudal views; save PNG/TIFF with and without area labels.
  739. # %%
  740. visual_cortex = ["V1", "V2", "V3", "V4"]
  741. ventral_cluster = ["VMV1", "VMV2", "VMV3", "PHA1", "PHA2", "PHA3"]
  742. dorsal_cluster = ["MT", "MST", "FST", "V4t", "TPOJ2", "TPOJ3"]
  743. visual_cortex_color = "gray"
  744. ventral_cluster_color = "#FE4A00"
  745. dorsal_cluster_color = "#2ca02c"
  747. # Load all the files and aggregate the data
  748. df_comparison_subjects = proccess_alignment(
  749. models_filename=models_filename,
  750. models_alignment_filename=model_brain_separated_filename,
  751. subject_alignment_filename=intersubject_shift1_separated_filename,
  752. pvalues_filename=pvalues_filename,
  753. hcp_filename=hcp_filename,
  754. group_subject=True,
  755. )
  757. # Load HCP and annotate hubs to add the areas in the map
  758. hcp_annotated = pd.read_csv(hcp_filename)
  759. hcp_annotated.loc[hcp_annotated.name.isin(ventral_cluster), "area_id"] = -1
  760. hcp_annotated.loc[hcp_annotated.name.isin(ventral_cluster), "area_color"] = ventral_cluster_color
  761. hcp_annotated.loc[hcp_annotated.name.isin(dorsal_cluster), "area_id"] = -2
  762. hcp_annotated.loc[hcp_annotated.name.isin(dorsal_cluster), "area_color"] = dorsal_cluster_color
  763. hcp_annotated.loc[hcp_annotated.name.isin(visual_cortex), "area_id"] = -3
  765. df = df_comparison_subjects.drop(columns=["name", "mne_name"]).merge(
  766. hcp_annotated[["roi", "name", "mne_name"]], on="roi"
  767. )
  770. modalities = {
  771. "language": {"column": "language_rsa", "pvalue_column": "language_p_value", "vmax": 0.15},
  772. "vision": {"column": "vision_rsa", "pvalue_column": "vision_p_value", "vmax": 0.15},
  773. "intersubject": {
  774. "column": "intersubject_rsa",
  775. "pvalue_column": "intersubject_p_value",
  776. "vmax": 0.2,
  777. },
  778. }
  780. faverage_folder = output / "06_faverage_spatial"
  781. faverage_folder.mkdir(exist_ok=True, parents=True)
  783. for modality in modalities.keys():
  784. params = modalities[modality]
  785. pvalue_column = params["pvalue_column"]
  786. vmax = params["vmax"]
  787. print(f"Processing modality: {modality} scale: ({-vmax}, 0, {vmax})")
  789. column = params["column"]
  790. significant = df_comparison_subjects.query(f"{pvalue_column} < 0.05").name.tolist()
  791. df_modality = df.query(f"name in @significant")
  792. for hemisphere in ["lh", "rh"]:
  793. brain = plot_faverage_parcelation(
  794. df_modality,
  795. cmap="RdBu_r",
  796. value_column=column,
  797. normalize=(-vmax, vmax),
  798. default_value=None,
  799. default_color="lightgray",
  800. hemisphere=hemisphere,
  801. size=(2 * 800, 2 * 600),
  802. )
  803. for view in ["lateral", "medial", "ventral", "caudal"]:
  804. brain.show_view(view)
  805. brain.save_image(
  806. faverage_folder / f"06_rsa_{modality}_{hemisphere}_{view}_{vmax:.2f}.png",
  807. mode="rgba",
  808. )
  809. # Save also as tigg
  810. brain.save_image(
  811. faverage_folder / f"06_rsa_{modality}_{hemisphere}_{view}_{vmax:.2f}.tiff",
  812. mode="rgba",
  813. )
  815. add_area_labels(brain, hcp_annotated, area_ids=[-1, -2, -3], hemispheres=[hemisphere])
  817. for view in ["lateral", "medial", "ventral", "caudal"]:
  818. brain.show_view(view)
  819. brain.save_image(
  820. faverage_folder / f"06_rsa_{modality}_{hemisphere}_{view}_{vmax:.2f}_annot.png",
  821. mode="rgba",
  822. )
  823. brain.save_image(
  824. faverage_folder / f"06_rsa_{modality}_{hemisphere}_{view}_{vmax:.2f}_annot.tiff",
  825. mode="rgba",
  826. )
  828. brain.close()
  830. # %% [markdown]
  831. # Inspect generated brain surface maps
  833. # %%
  834. hemisphere = "lh" # "lh", "rh"
  835. views = ["lateral", "medial", "ventral", "caudal"]
  837. fig, axes = plt.subplots(3, len(views), figsize=(12, 6))
  839. for j, modality in enumerate(["intersubject", "vision", "language"]):
  840. for i, view in enumerate(views):
  841. vmax_file = modalities[modality]["vmax"]
  842. img_filename = faverage_folder / f"06_rsa_{modality}_{hemisphere}_{view}_{vmax_file:.2f}_annot.png"
  843. img = plt.imread(img_filename)
  844. axes[j, i].axis("off")
  845. axes[j, i].imshow(img)
  846. axes[j, i].set_title(f"{hemisphere.upper()} {view} {modality}", fontsize=12)
  848. # %% [markdown]
  849. # ### 3.2 Colorbars
  850. # Generate horizontal and vertical colorbars for RSA scales (−vmax…+vmax) to accompany spatial maps.
  852. # %%
  853. eps = 0.00000001
  854. vmax = 0.20
  855. fig_cbar, ax_cbar = plot_cbar(
  856. cmap="RdBu_r",
  857. title=r"Inter-subject alignment (RSA Pearson's $\rho$)",
  858. vmin=-vmax - eps,
  859. vmax=vmax + eps,
  860. locator=0.04,
  861. horizontal=True,
  862. rotation=0,
  863. labelpad=5,
  864. figsize=(6, 0.25),
  865. percent=False,
  866. )
  867. ax_cbar.set_xlim(0, vmax + eps)
  869. fig_cbar.savefig(
  870. faverage_folder / f"06_colorbar_intersubject_{vmax:.2f}_BuRd.pdf",
  871. bbox_inches="tight",
  872. dpi=300,
  873. transparent=True,
  874. )
  876. # %%
  877. vmax = 0.15
  878. fig_cbar, ax_cbar = plot_cbar(
  879. cmap="RdBu_r",
  880. title=r"Models-brain alignment (RSA Pearsons's $\rho$)",
  881. vmin=-vmax - eps,
  882. vmax=vmax + eps,
  883. locator=0.03,
  884. horizontal=True,
  885. rotation=0,
  886. labelpad=5,
  887. figsize=(6, 0.25),
  888. percent=False,
  889. )
  890. ax_cbar.set_xlim(-0.06, vmax)
  891. fig_cbar.savefig(
  892. faverage_folder / f"06_colorbar_model_{vmax:.2f}_BuRd.pdf",
  893. bbox_inches="tight",
  894. dpi=300,
  895. transparent=True,
  896. )
  899. fig_cbar, ax_cbar = plot_cbar(
  900. cmap="RdBu_r",
  901. title=r"Models-brain alignment (RSA)",
  902. vmin=-vmax - eps,
  903. vmax=vmax + eps,
  904. locator=0.03,
  905. horizontal=False,
  906. rotation=90,
  907. labelpad=5,
  908. figsize=(0.25, 6),
  909. percent=False,
  910. )
  911. ax_cbar.set_ylim(-0.06, vmax)
  912. fig_cbar.savefig(
  913. faverage_folder / f"06_colorbar_model_{vmax:.2f}_BuRd_horizontal.pdf",
  914. bbox_inches="tight",
  915. dpi=300,
  916. transparent=True,
  917. )
  918. # Close vertical to improve the notebook readability
  919. plt.close(fig_cbar)
  921. # %% [markdown]
  922. # ## 4. Model–Brain Scatter Comparisons
  923. # Scatterplot inter‑subject RSA vs. model‑brain RSA (vision & language), fit a power‑law model with bootstrap confidence intervals, label outliers, and save vector graphics for manual annotation.
  925. # %%
  926. # Load all the files and aggregate the data
  927. df_comparison = proccess_alignment(
  928. models_filename=models_filename,
  929. models_alignment_filename=model_brain_joined_filename,
  930. subject_alignment_filename=intersubject_shift1_joined_filename,
  931. pvalues_filename=pvalues_filename,
  932. hcp_filename=hcp_filename,
  933. group_subject=True,
  934. )
  937. n_boot = 10000
  938. df_intersubject = pd.read_parquet(intersubject_shift1_joined_filename)
  939. top_areas = process_intersubject_rois(
  940. df_intersubject, hcp_filename=hcp_filename, top=10
  941. ).area.unique()
  942. palette = (
  943. df_comparison[["area", "area_color"]]
  944. .drop_duplicates()
  945. .set_index("area")
  946. .to_dict()["area_color"]
  947. )
  948. data = df_comparison.query("intersubject_p_value < 0.05").copy()
  949. data = add_cluster(data)
  950. data["top_area"] = data["area"].apply(lambda x: x in top_areas)
  953. fig1, ax1 = plt.subplots(1, 1, figsize=(6, 6)) # Vision models
  954. fig2, ax2 = plt.subplots(1, 1, figsize=(6, 6)) # Language models
  957. # Vision models
  958. plot_comparison(data=data, x="intersubject_rsa", y="vision_rsa", ax=ax1, palette=palette)
  960. x_fit, y_fit, y_lower, y_upper, r2, params = fit_powerlaw_zero_with_bootstrap(
  961. data["intersubject_rsa"], data["vision_rsa"], n_boot=n_boot
  962. )
  963. print(f"Vision Models - Brain Pseudo-R²: {r2:.3f} N Boostrap samples: {n_boot}")
  964. print(f"Power-law parameters: y={params[0]:.3f}x^{params[1]:.3f}")
  966. ax1.plot(
  967. x_fit, y_fit, color="maroon", label=f"$f(x)={params[0]:.2f}x^{{{params[1]:.2f}}}$", zorder=-10
  968. )
  969. ax1.fill_between(x_fit, y_lower, y_upper, color="maroon", alpha=0.1, label="95% CI", zorder=-20)
  970. ax1.legend(loc="lower right")
  973. # Language models
  974. plot_comparison(data=data, x="intersubject_rsa", y="language_rsa", ax=ax2, palette=palette)
  977. for i, row in data.iterrows():
  978. if row["vision_rsa"] > 0.04 or row["intersubject_rsa"] > 0.04 or row["language_rsa"] > 0.04:
  979. ax1.text(
  980. row["intersubject_rsa"],
  981. row["vision_rsa"],
  982. row["name"],
  983. ha="center",
  984. )
  985. ax2.text(row["intersubject_rsa"], row["language_rsa"], row["name"], ha="center")
  987. # Add labels, legend and some plot adjustments
  988. ax1.set_xlabel("Inter-subject Alignment (RSA)")
  989. ax1.set_ylabel("Vision Models - Brain Alignment (RSA)")
  990. ax2.set_xlabel("Inter-subject Alignment (RSA)")
  991. ax2.set_ylabel("Language Models - Brain Alignment (RSA)")
  992. triangle = mlines.Line2D(
  993. [], [], color="black", marker="^", linestyle="None", label="Early Visual Cortex"
  994. )
  995. square = mlines.Line2D([], [], color="black", marker="s", linestyle="None", label="Ventral Hub")
  996. diamond = mlines.Line2D([], [], color="black", marker="D", linestyle="None", label="LOTC Hub")
  997. circle = mlines.Line2D([], [], color="black", marker="o", linestyle="None", label="Other Regions")
  998. legend_handles = [triangle, square, diamond, circle]
  999. ax2.legend(handles=legend_handles, title="")
  1000. ax1.set_ylim(0, 0.175)
  1001. ax2.set_ylim(-0.05, 0.175)
  1002. ax1.set_xlim(-0.005, 0.225)
  1003. ax2.set_xlim(-0.005, 0.225)
  1005. # Add a major tick every 0.05, a minor tick every 0.025
  1006. for ax in (ax1, ax2):
  1007. ax.xaxis.set_major_locator(mticker.MultipleLocator(0.05))
  1008. ax.xaxis.set_minor_locator(mticker.MultipleLocator(0.025))
  1009. ax.yaxis.set_major_locator(mticker.MultipleLocator(0.05))
  1010. ax.yaxis.set_minor_locator(mticker.MultipleLocator(0.025))
  1012. # Save as svg to manually edit the text labels to avoid overlapping
  1013. fig1.savefig(output / "07_rsa_models_vision_comparison.svg", bbox_inches="tight", transparent=True)
  1014. fig2.savefig(
  1015. output / "07_rsa_models_language_comparison.svg", bbox_inches="tight", transparent=True
  1016. )
  1018. # %% [markdown]
  1019. # ## 5. Hub‑Based Modality Comparison
  1020. # Group alignments by predefined hubs (Visual Cortex, Ventral Hub, Dorsal Hub), plot boxplots comparing modalities (Inter‑subject, Vision, Language), conduct paired t‑tests with Bonferroni correction, and annotate significant differences.
  1022. # %%
  1023. df_comparison_subjects = proccess_alignment(
  1024. models_filename=models_filename,
  1025. models_alignment_filename=model_brain_joined_filename,
  1026. subject_alignment_filename=intersubject_shift1_joined_filename,
  1027. pvalues_filename=pvalues_filename,
  1028. hcp_filename=hcp_filename,
  1029. group_subject=False,
  1030. )
  1032. df_comparison_subjects = add_cluster(df_comparison_subjects)
  1033. df_comparison_subjects_grouped = (
  1034. df_comparison_subjects.groupby(["subject", "cluster"])
  1035. .aggregate({"intersubject_rsa": "mean", "vision_rsa": "mean", "language_rsa": "mean"})
  1036. .reset_index()
  1037. )
  1038. df_comparison_subjects_test = df_comparison_subjects_grouped.query("cluster != 'Other'").copy()
  1040. # Boxplot
  1041. fig, ax = plt.subplots(1, 1, figsize=(3.8, 6))
  1044. order = ["Visual\nCortex", "Ventral\nHub", "Dorsal\nHub"]
  1045. df_comparison_subjects_grouped.cluster = df_comparison_subjects_grouped.cluster.str.replace(
  1046. " ", "\n"
  1047. ).str.replace("Cluster", "Hub")
  1048. hue_names = {
  1049. "intersubject_rsa": "Inter-subject",
  1050. "vision_rsa": "Vision Models",
  1051. "language_rsa": "Language Models",
  1052. }
  1053. # hue_order = ["Inter-subject", "Vision Models", "Language Models"]
  1054. hue_order = ["Vision Models", "Language Models"]
  1055. # hue_order = ["Vision", "Language"]
  1056. df_comparison_subjects_grouped = df_comparison_subjects_grouped.rename(columns=hue_names)
  1057. df_comparison_subjects_grouped = df_comparison_subjects_grouped.melt(
  1058. id_vars=["subject", "cluster"], var_name="modality", value_name="score"
  1059. ).copy()
  1060. df_comparison_subjects_grouped
  1061. from cmap import Colormap
  1063. cm = Colormap("colorbrewer:Set2_3").to_matplotlib()([0.5, 0, 1])
  1064. cm = list(list(c) for c in cm)[:2]
  1065. sns.boxplot(
  1066. data=df_comparison_subjects_grouped,
  1067. x="cluster",
  1068. y="score",
  1069. hue="modality",
  1070. ax=ax,
  1071. order=order,
  1072. hue_order=hue_order,
  1073. palette=cm,
  1074. showfliers=False, # width=0.5,
  1075. )
  1076. # Plot individual points with jitter
  1077. sns.stripplot(
  1078. data=df_comparison_subjects_grouped,
  1079. x="cluster",
  1080. y="score",
  1081. hue="modality",
  1082. ax=ax,
  1083. order=order,
  1084. hue_order=hue_order,
  1085. palette=cm,
  1086. dodge=True,
  1087. alpha=1,
  1088. linewidth=0.5,
  1089. size=4,
  1090. jitter=True,
  1091. edgecolor=(0.24, 0.24, 0.24),
  1092. zorder=100,
  1093. legend=False,
  1094. )
  1096. ax.set_xlabel("")
  1097. ax.set_ylabel(r"Alignment (RSA Pearson's $\rho$)")
  1098. # Remove title from legend
  1099. ax.legend(title="", loc="upper left") # , bbox_to_anchor=(0, 0.3))
  1101. # xticks from each 0.05
  1102. ax.set_yticks(np.arange(-0.04, 0.21, 0.04))
  1104. sns.despine(ax=ax)
  1106. # Define dataframe
  1107. df_comparison_subjects_test
  1109. # Define comparisons explicitly
  1110. comparisons = [
  1111. ("Visual Cortex", "vision_rsa", "language_rsa"),
  1112. ("Ventral Cluster", "vision_rsa", "language_rsa"),
  1113. ("Dorsal Cluster", "vision_rsa", "language_rsa"),
  1114. # ("Dorsal Cluster", "language_rsa", "intersubject_rsa"),
  1115. # ("Visual Cortex", "vision_rsa", "intersubject_rsa"),
  1116. ]
  1118. results = []
  1119. for cluster, cond1, cond2 in comparisons:
  1120. subset = df_comparison_subjects_test.query("cluster == @cluster")
  1121. t_stat, p_value = ttest_rel(subset[cond1], subset[cond2])
  1123. results.append(
  1124. {
  1125. "cluster": cluster,
  1126. "comparison": f"{cond1} vs {cond2}",
  1127. "t_stat": t_stat,
  1128. "p_value": p_value,
  1129. "mean_diff": subset[cond1].mean() - subset[cond2].mean(),
  1130. }
  1131. )
  1133. # Convert to dataframe
  1134. results_df = pd.DataFrame(results)
  1136. # Correct p-values (FDR correction)
  1137. results_df["p_corrected"] = multipletests(results_df["p_value"], method="bonferroni")[
  1138. 1
  1139. ] # .round(5)
  1140. results_df["apa_star"] = results_df["p_corrected"].apply(
  1141. lambda x: "***" if x < 0.001 else ("**" if x < 0.01 else ("*" if x < 0.05 else "n.s."))
  1142. )
  1145. # Show results
  1146. display(results_df)
  1147. ax.set_ylim(-0.040, 0.22)
  1148. width = 0.135
  1149. add_pvalue_bracket(ax=ax, x1=-width, x2=width, y=0.175, height=0.0025, text="***")
  1150. add_pvalue_bracket(ax=ax, x1=1 - width, x2=1 + width, y=0.15, height=0.0025, text="***")
  1151. add_pvalue_bracket(ax=ax, x1=2 - width, x2=2 + width, y=0.19, height=0.0025, text="***")
  1153. ax.set_xticks([0, 1, 2])
  1154. ax.set_xticklabels(["Early Visual\nCortex", "Ventral\nHub", "LOTC\nHub"])
  1156. fig.savefig(
  1157. output / "08_rsa_modality_boxplot_comparison.pdf", bbox_inches="tight", transparent=True
  1158. )

01_rsa_spatial_alignment.ipynb at commit c1ebe1d, under MIT · at the source

Overview

Authors: Pablo Marcos-Manchón1,2, Lluís Fuentemilla1,2,3
ORCID iDs: Pablo Marcos-Manchón, Lluís Fuentemilla
  1. Department of Cognition, Development and Education Psychology, Faculty of Psychology, University of Barcelona,Barcelona, Spain
  2. Institute of Neurosciences, University of Barcelona,Barcelona, Spain
  3. Bellvitge Institute for Biomedical Research,Barcelona, Spain
Institutions: Universitat de Barcelona (Spain); Institut d'Investigació Biomédica de Bellvitge (Spain)
Journal: Communications biology, volume 9, issue 1, article 950
Dates: received 24 July 2025; accepted 22 April 2026; published online 7 May 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s42003-026-10169-0 · PMID 42098492 · PMCID PMC13365445 · OpenAlex W4416322499
Open access: gold, a free copy (OpenAlex)
Preprint: arxiv:2507.13941
Status: code verified
Categories: human (organism), cognitive (subfield)
Methods: Statistics, Connectivity, fMRI & imaging, Machine learning
Keywords: Neural encoding, Network models, Perception
MeSH: Brain*, Visual Cortex*, Visual Perception*, Adult, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Models, Neurological, Photic Stimulation, Visual Pathways, Young Adult (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: Ministry of Economy and Competitiveness | Agencia Estatal de Investigación (Spanish Agencia Estatal de Investigación) (PID2022-140426NB-I00); This work was supported by the Spanish Ministerio de Ciencia, Innovación y Universidades, which is part of Agencia Estatal de Investigación (AEI), through the project PID2022-140426NB-I00
Citations: not cited yet (Europe PMC); 100 references in the paper

Abstract

The brain transforms visual inputs into cortical representations that support diverse cognitive and behavioral goals. Characterizing how this information is organized and routed across the human brain is essential for understanding how we process complex visual scenes. Here, we applied representational similarity analysis to 7T fMRI data collected during natural scene viewing. We quantified representational geometry shared across individuals and compared it to hierarchical features from vision and language neural networks across model layers. By integrating these comparisons with representational connectivity between cortical regions, we identified two distinct processing routes: a ventromedial pathway specialized for scene layout and environmental context, and a lateral occipitotemporal pathway selective for animate content. Vision models aligned with shared structure in both routes, whereas language models corresponded primarily with the lateral pathway and showed negative alignment in early visual and ventral cortex. These findings refine classical visual-stream models by revealing a distributed cortical network with separable representational routes for context and animate content during scene perception.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 10 matches between paragraphs and lines of code.

huggingface.co/collections/pablomm

License: none: the authors keep all their rights
State: the link is dead, verified on 28 September 2026
Evidence: found in the paper
Software Heritage: not archived
Found in: “Data availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 28 September 2026: the link is dead
  • 28 September 2026: the link is dead
At the source: huggingface.co/collections/pablomm/convergent-transformations-6808f7de248fa9674acac588

memory-formation/shared-representations

License: MIT
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Commit: c1ebe1d4f1fd49e5f4f53b09f8c3d2a6acfb12e5, 20 March 2026
Languages: Python (68), Jupyter (25), Shell (2)
Size: 111 files, 95 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file, environment (pyproject.toml, requirements.txt), 25 notebooks
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: pandas (42 files), NumPy (38 files), PyTorch (32 files), Matplotlib (9 files), seaborn (6 files), NiBabel (4 files), SciPy (3 files), statsmodels (3 files), FSL (2 files), h5py (2 files), Pillow (2 files), scikit-learn (2 files), Plotly (1 file), Hugging Face Transformers (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers
  • 28 September 2026: the link answers
47 files
At the source: github.com/memory-formation/shared-representations

Zenodo 19581037

License: CC-BY-4.0
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: “Code availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: pandas (42 files), NumPy (38 files), PyTorch (32 files), Matplotlib (9 files), seaborn (6 files), NiBabel (4 files), SciPy (3 files), statsmodels (3 files), FSL (2 files), h5py (2 files), Pillow (2 files), scikit-learn (2 files), Plotly (1 file), Hugging Face Transformers (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers (HTTP 200)
  • 28 September 2026: the link answers (HTTP 200)
47 files
At the source:

Code availability

Code to reproduce analyses is available at github.com/memory-formation/shared-representations and has been archived at Zenodo 10.5281/zenodo.19581037.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 3 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 94 scripts, each with its path and the digest of its content;
  • 10 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data availability

Functional-MRI data and image stimuli were obtained from the Natural Scenes Dataset (accessed via https://registry.opendata.aws/nsd), BOLD5000 (10.18112/openneuro.ds001499.v1.3.0 and 10.1184/R1/c.5325683), and THINGS-fMRI (10.25452/figshare.plus.c.6161151.v1 and 10.17605/osf.io/jum2f). Image metadata were extracted from MS-COCO (https://cocodataset.org). Model checkpoints analyzed in this work are available on Hugging Face (https://huggingface.co/collections/pablomm/convergent-transformations-6808f7de248fa9674acac588). Precomputed derivative data generated from these raw sources are available in Figshare 10.6084/m9.figshare.30753239. All resources were publicly accessible in April 2026.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 28 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 2 authors, 3 keywords, 13 MeSH terms, 2 funders, 72 references.

Cite

This paper

Marcos-Manchón, P., & Fuentemilla, L. (2026). Shared representations in brains and models reveal a two-route cortical organization during scene perception. Communications biology, 9(1), 950. https://doi.org/10.1038/s42003-026-10169-0

BibTeX

@article{marcosmanchon2026shared,
author = {Marcos-Manchón, Pablo and Fuentemilla, Lluís},
title = {{Shared representations in brains and models reveal a two-route cortical organization during scene perception}},
journal = {Communications biology},
year = {2026},
month = may,
volume = {9},
number = {1},
pages = {950},
publisher = {Nature Publishing Group},
issn = {2399-3642},
doi = {10.1038/s42003-026-10169-0},
url = {https://doi.org/10.1038/s42003-026-10169-0},
pmid = {42098492},
pmcid = {PMC13365445}
}

RIS

TY - JOUR
AU - Marcos-Manchón, Pablo
AU - Fuentemilla, Lluís
TI - Shared representations in brains and models reveal a two-route cortical organization during scene perception
T2 - Communications biology
J2 - Commun Biol
PY - 2026
DA - 2026/05/07
VL - 9
IS - 1
SP - 950
SN - 2399-3642
PB - Nature Publishing Group
DO - 10.1038/s42003-026-10169-0
UR - https://doi.org/10.1038/s42003-026-10169-0
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s42003-026-10169-0",
"type": "article-journal",
"title": "Shared representations in brains and models reveal a two-route cortical organization during scene perception",
"container-title": "Communications biology",
"author": [
{
"family": "Marcos-Manchón",
"given": "Pablo"
},
{
"family": "Fuentemilla",
"given": "Lluís"
}
],
"container-title-short": "Commun Biol",
"volume": "9",
"issue": "1",
"page": "950",
"DOI": "10.1038/s42003-026-10169-0",
"PMID": "42098492",
"PMCID": "PMC13365445",
"ISSN": "2399-3642",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s42003-026-10169-0",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
7
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: Hugging Face Transformers, h5py, Pillow, 9 other tools, 14 references
[2] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: FSL, h5py, Pillow, 7 other tools, 14 references
[3] doi:10.1162/imag.a.1309 [code]
Probing the content of semantic representations in body-selective regions.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: h5py, Pillow, NiBabel, 8 other tools, 10 references
[4] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Hugging Face Transformers, h5py, Pillow, 9 other tools, cognitive, 7 references
[5] doi:10.1523/jneurosci.0038-26.2026 [code]
Multidimensional Feature Tuning in Category Selective Areas of Human Visual Cortex.
Journal: The Journal of neuroscience : the official journal of the Society for Neuroscience
In common: h5py, Pillow, NiBabel, 8 other tools, 8 references
[6] doi:10.1167/jov.26.5.7 [code]
Representations in vision and language converge in a shared, multidimensional space of perceived similarities.
Journal: Journal of vision
In common: h5py, Pillow, NiBabel, 8 other tools, cognitive, 7 references
[7] doi:10.1162/imag.a.1207 [code]
Investigating the temporal dynamics and modeling of mid-level feature representations in humans.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Pillow, statsmodels, PyTorch, 6 other tools, cognitive, 8 references
[8] doi:10.1016/j.isci.2026.117180 [code]
Developmental changes in similarity between neural representations of mental arithmetic and artificial neural networks.
Journal: iScience
In common: Hugging Face Transformers, h5py, NiBabel, 8 other tools, 5 references
[9] doi:10.1038/s41593-026-02285-1 [code]
Fixation duration on natural scenes is explained by memory encoding not processing demand.
Journal: Nature neuroscience
In common: h5py, Pillow, statsmodels, 7 other tools, cognitive, 5 references
[10] doi:10.1371/journal.pcbi.1014656 [code]
Contrastive learning to fine-tune feature extraction models for the visual cortex.
Journal: PLoS computational biology
In common: Pillow, NiBabel, PyTorch, 5 other tools, 7 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.