OSCR

Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training.

Code ↔ Paper

11 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 11 matches
  1. [1] § Methods › Simulation of monkey training using artificial neural networks › Base models ↔ analysis/paperFigures.py, lines 1514–1582 · score 0.80 · SimCLR, barlowTwins, MoCov2, ResNet18, VGG16, AlexNet
  2. [2] § Results › IT-like models best predict differences along untrained dimensions in IT ↔ analysis/paperFigures.py, lines 1692–1785 · score 0.72 · Horizontal position, Vertical position, mapped layers, rx, ry, rz
  3. [3] § Methods › Simulation of monkey training using artificial neural networks › Training strategies ↔ training/train_wandb_binaryTask.py, lines 382–522 · score 0.71 · cross entropy loss, learning rate scheduler, decay, epochs, validation, optimizing
  4. [4] § Methods › Simulation of monkey training using artificial neural networks › Training strategies ↔ training/train_wandb_v2.py, lines 18–145 · score 0.71 · cross entropy loss, learning rate scheduler, decay, epochs, validation, optimizing
  5. [5] § Methods › Neural and behavioral data analyses › Population-level object decoding ↔ analysis/categoryDecodes.py, lines 24–54 · score 0.61 · logistic regression, L2, stratified, fold, classifiers, splits
  6. [6] § Methods › Simulation of monkey training using artificial neural networks › Aligning base models to task-naïve IT responses ↔ analysis/paperFigures.py, lines 1216–1312 · score 0.59 · shifted log, 1–100, fit, accuracies, decodes, models
  7. [7] § Methods › Neural and behavioral data analyses › Neural recording quality metrics and pooling ↔ analysis/categoryDecodes.py, lines 638–740 · score 0.58 · spearman brown corrected, reliability criterion, split, repetition, stem, untrained
  8. [8] § Methods › Neural and behavioral data analyses › Behavioral metrics ↔ analysis/consistency.py, lines 15–81 · score 0.56 · Spearman Brown corrected, random splits, repetitions, reliability, behavioral, metric
  9. [9] § Results › Task-optimized ANNs reproduce IT’s task-training-induced differences ↔ analysis/paperFigures.py, lines 1788–1916 · score 0.55 · signal rotation, metrics matched, covariance, selectivity, models, training
  10. [10] § Methods › Simulation of monkey training using artificial neural networks › Aligning base models to task-naïve IT responses ↔ analysis/compute_modelFeatures.py, lines 79–195 · score 0.52 · brain score, mapped layer, cross, validated, predictivity, models
  11. [11] § Results › Task-optimized ANNs reproduce IT’s task-training-induced differences ↔ analysis/paperFigures.py, lines 1788–1916 · score 0.51 · Signal rotation, metrics matched, covariance, ratio, models, training

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 2,242 lines · 113 KB · MIT · 5 matches

  1. from pathlib import Path
  3. import joblib
  4. import matplotlib.pyplot as plt
  5. import numpy as np
  6. import pandas as pd
  7. import seaborn as sns
  8. from matplotlib_venn import venn3
  9. from pingouin import kruskal
  10. from scipy.optimize import curve_fit
  11. from scipy.stats import gaussian_kde
  12. from scipy.integrate import quad
  14. from analysis.categoryDecodes import load_ModelIT_categoryDecodes_allModels, load_MonkeyIT_Decodes_deltas
  15. from analysis.categoryDecodes import load_MonkeyIT_Decodes, get_ModelIT_categoryDecoding_UnitSiteScaling, \
  16. load_MonkeyIT_Decodes_permutation, load_MonkeyIT_subregion_Decodes, load_MonkeyIT_Decodes_novel500, \
  17. load_MonkeyIT_Decodes_novel500_HVMcontrol, load_MonkeyIT_Decodes_singleAnimal
  18. from analysis.consistency import get_consistency_modelIT_modelBehavior, get_consistency_behavior_ITneurons
  19. from analysis.orthogonalDecodes import load_ModelITorthogonalDecodes_selection, load_neuralITorthogonalDecodes,load_ModelLayer_orthogonalDecodes_selection
  20. from analysis.rsa import load_ModelRDMs_allModels, load_neuralRDMs_deltas
  21. from analysis.rsa import load_neuralRDMs, load_neuralRDMs_permutation, load_neuralRDMs_singleAnimal
  22. from analysis.selectivity import load_ModelSelectivity_allModels, load_neuralITselectivity_deltas
  23. from analysis.selectivity import load_neuralITselectivity, load_neuralITselectivity_permutation, load_neuralITselectivity_singleAnimal
  24. from analysis.trainingResults import loadTrainingResults
  25. from analysis.LFI import get_model_change_metrics
  27. from analysis.utils_behavioralData import load_MonkeyBehavior_individual, load_MonkeyBehavior_pooled, \
  28. getPerformanceThreshold, get_MonkeyBehavioralMetrics_draws, load_MonkeyTraining_dPrime, load_MonkeyTraining_individual
  29. from analysis.utils_neuralData import get_SiteReliability, load_MonkeyNeurons_pooled_rates, load_orthogonalFeatures
  30. from analysis.metrics import dPrime_monkey, computeDelta
  33. cm = 1 / 2.54
  35. pretraining_mapping = {'resnet18_v1': 'supervised',
  36. 'resnet34_v1': 'supervised',
  37. 'resnet50_v1': 'supervised',
  38. 'resnet50_MoCov2_200epochs': 'self-supervised',
  39. 'resnet50_barlowTwins_300epochs': 'self-supervised',
  40. 'resnet50_simclr_100epochs': 'self-supervised',
  41. 'resnet101_v1': 'supervised',
  42. 'resnet152_v1': 'supervised',
  43. 'alexnet': 'supervised',
  44. 'resnext101_32x8d_wsl': 'WSL-supervised',
  45. 'resnext101_32x16d_wsl': 'WSL-supervised',
  46. 'vgg16': 'supervised',
  47. 'vgg19': 'supervised'}
  49. def FigureS1_TrainTestSets(colors=None, figure_dir=None, source_file_dir=None):
  50. if figure_dir == None:
  51. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  53. if source_file_dir == None:
  54. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  56. train_df = load_orthogonalFeatures(split='train')
  58. train_df['category'] = train_df['obj'].map({'bear': 'bear','ELEPHANT_M': 'elephant','face0001': 'person',
  59. 'alfa155': 'car','breed_pug': 'dog','Apple_Fruit_obj': 'apple',
  60. '001': 'chair','f16': 'plane'})
  61. train_df['xpos'] = train_df['ty'] * -1
  62. train_df['ypos'] = train_df['tz']
  64. train_df['ecc'] = np.sqrt(train_df['xpos']**2 + train_df['ypos']**2)
  65. train_df['filename'] = 'im'+ train_df['legacy_mkturk_image_num'].astype(str)
  66. train_df = train_df.rename(columns={'s':'objSize'})
  67. train_df['split'] = 'train'
  69. test_df = load_orthogonalFeatures(split='test')
  70. test_df['split'] = 'test'
  72. df = pd.concat([train_df, test_df], ignore_index=True)
  74. title_mapping={'xpos': 'Vertical position',
  75. 'ypos': 'Horizontal position',
  76. 'objSize': 'Object size',
  77. 'ecc': 'Eccentricity',
  78. 'ryz': 'Rotation (rx)',
  79. 'rxz': 'Rotation (ry)',
  80. 'rxy': 'Rotation (rz)',}
  82. g, ax = plt.subplots(2,4, sharex=False, sharey=True, figsize=(15*cm,10*cm), squeeze=False)
  83. ax = ax.flatten()
  85. for i, var in enumerate(['xpos', 'ypos', 'objSize','ecc', 'ryz', 'rxz', 'rxy']):
  87. sns.histplot(data=df, hue='split', kde=True, x=var, bins=100,ax=ax[i], legend=False)
  88. ax[i].set_xlabel(f"{title_mapping[var]}")
  89. ax[i].set_ylabel(' ')
  91. ax[0].set_ylabel('# images')
  92. plt.tight_layout()
  93. g.savefig(figure_dir + 'FigureS1_datasets_v0.png', dpi=300)
  94. g.savefig(figure_dir + 'FigureS1_datasets_v0.pdf', dpi=300, transparent=True)
  96. df.to_csv(source_file_dir + 'FigureS1_datasets.csv', index=False)
  98. def FigureS2_trainingCurves(figure_dir=None, source_file_dir = None):
  100. if figure_dir == None:
  101. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  103. if source_file_dir == None:
  104. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  106. df_trials = load_MonkeyTraining_individual()
  108. num_trials_day = df_trials.groupby(['day', 'subject'], as_index=False)['trial'].nunique()
  110. g = sns.catplot(data=num_trials_day, x='day', hue='subject', y='trial', kind='bar',
  111. height=6 * cm, legend=False, aspect=1.6)
  112. g.set_xlabels('Training day')
  113. g.set_ylabels("# trials")
  114. g.savefig(figure_dir + 'FigureS2_LearningCurves_v2.png', dpi=300)
  115. g.savefig(figure_dir + 'FigureS2_LearningCurves_v2.pdf', dpi=300, transparent=True)
  117. num_trials_day.to_csv(source_file_dir + 'FigureS2_LearningCurves_v2.csv', index=False)
  119. # compute duration in number of days
  120. print(df_trials.groupby('subject')['date'].max() - df_trials.groupby('subject')['date'].min())
  122. df_dPrime = load_MonkeyTraining_dPrime()
  124. g = sns.relplot(data=df_dPrime, x= 'start_trial', y='dPrime', hue='subject', kind='line',
  125. height=6*cm, legend=False, aspect=1.2)
  126. g.set_xlabels('Trial')
  127. g.set_ylabels("Performance (d')")
  128. g.refline(x=0)
  129. g.refline(x=17400)
  130. g.savefig(figure_dir + 'FigureS2_LearningCurves_v0.png', dpi=300)
  131. g.savefig(figure_dir + 'FigureS2_LearningCurves_v0.pdf', dpi=300, transparent=True)
  133. df_dPrime.to_csv(source_file_dir + 'FigureS2_LearningCurves_v0.csv', index=False)
  135. g = sns.catplot(data=df_dPrime.loc[df_dPrime['start_trial'].isin([0, 17400])],
  136. x='start_trial', y='dPrime', hue='subject', kind='point',
  137. height=6 * cm, legend=False, aspect=0.8, dodge=0.2, join=False)
  138. g.set_ylabels("Performance (d')")
  139. g.set_xlabels("Training phase")
  140. g.set(ylim=[0, 4.5])
  141. g.set_xticklabels(['Early', 'Late'])
  143. g.savefig(figure_dir + 'FigureS2_LearningCurves_v1.png', dpi=300)
  144. g.savefig(figure_dir + 'FigureS2_LearningCurves_v1.pdf', dpi=300, transparent=True)
  146. df_dPrime.loc[df_dPrime['start_trial'].isin([0, 17400])].to_csv(source_file_dir + 'FigureS2_LearningCurves_v1.csv', index=False)
  148. def Figure1_monkeyPerformance(colors=None, figure_dir=None, data_dir=None, source_file_dir = None):
  149. if figure_dir == None:
  150. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  152. if source_file_dir == None:
  153. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  155. monkey_performance = load_MonkeyBehavior_individual(data_dir=data_dir)
  157. performance_subject = monkey_performance.groupby('subject').mean()
  158. performance_subject['trained?'] = 'yes'
  159. order = ['no', 'yes']
  160. g = sns.catplot(data=performance_subject, x='trained?', y='i1', hue='trained?', kind='point', errorbar=("ci", 95),
  161. order=order,
  162. palette={'no': colors[0], 'yes': sns.xkcd_palette(['raspberry'])[0]}, height=4.5 * cm, aspect=0.85,
  163. legend=False)
  164. g.refline(y=0)
  165. g.set_ylabels("Performance (d')")
  166. g.set(ylim=[-1, 3.5])
  167. g.savefig(figure_dir + 'Figure1_performance_i1.png', dpi=300)
  168. g.savefig(figure_dir + 'Figure1_performance_i1.pdf', dpi=300, transparent=True)
  170. performance_subject.to_csv(source_file_dir + 'Figure1_performance_i1.csv')
  172. def Figure1_trainingCurves(figure_dir = None, source_file_dir = None):
  173. if figure_dir == None:
  174. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  176. if source_file_dir == None:
  177. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  179. df_dPrime = load_MonkeyTraining_dPrime()
  181. df_dPrime_mean = df_dPrime.groupby(['class', 'start_trial'], as_index=False)['dPrime'].mean()
  183. g = sns.relplot(data=df_dPrime_mean.loc[df_dPrime_mean['start_trial'] <= 17400], x='start_trial', y='dPrime', kind='line',
  184. legend=False, height=4.5*cm, aspect=1.2, color=sns.xkcd_palette(['raspberry'])[0])
  185. g.set_xlabels('Trial')
  186. g.set_ylabels("Performance (d')")
  187. g.refline(y=0)
  188. g.savefig(figure_dir + 'Figure1_LearningCurves.png', dpi=300)
  189. g.savefig(figure_dir + 'Figure1_LearningCurves.pdf', dpi=300, transparent=True)
  191. df_dPrime_mean.loc[df_dPrime_mean['start_trial'] <= 17400].to_csv(source_file_dir + 'Figure1_LearningCurves.csv', index=False)
  194. def FigureS4_SiteReliability(figure_dir=None, source_file_dir=None, colors=None, recompute=False, criterion=0.3, n_sites_per_sub=53,
  195. draws=1000, version='v0',
  196. mode='empirical_conditioned_v2'):
  197. if figure_dir == None:
  198. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  200. if source_file_dir == None:
  201. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  203. if version in ['v1', 'v2']:
  204. rates = load_MonkeyNeurons_pooled_rates(subsampled_neurons=n_sites_per_sub, draws=draws,
  205. mode=mode, recompute=recompute, reliability_criterion=criterion)
  207. if version == 'v0':
  208. reliabilities = get_SiteReliability(recompute=recompute)
  209. df = []
  210. for state in ['trained', 'untrained']:
  211. subs = reliabilities[state].keys()
  212. for sub in subs:
  213. df.append(pd.DataFrame({'reliabilities': np.mean(reliabilities[state][sub], axis=1),
  214. 'state': state,
  215. 'subject': sub}))
  217. df = pd.concat(df, ignore_index=True)
  219. g = sns.displot(data=df, x='reliabilities', hue='state', palette={'trained': colors[1], 'untrained': colors[0]},
  220. height=5 * cm, aspect=1.2,
  221. bins=30)
  222. g.set_ylabels("# sites")
  223. g.refline(x=criterion)
  224. g.savefig(figure_dir + 'FigureS4_reliability.png', dpi=300)
  225. g.savefig(figure_dir + 'FigureS4_reliability.pdf', dpi=300, transparent=True)
  227. df.to_csv(source_file_dir + 'FigureS4_reliability.csv', index=False)
  230. elif version == 'v1':
  232. d = np.random.randint(draws)
  233. df = []
  234. for state in ['trained', 'untrained']:
  235. df.append(pd.DataFrame({'reliabilities': rates[f'reliability_{state}'][d],
  236. 'state': state,
  237. }))
  239. df = pd.concat(df, ignore_index=True)
  241. g = sns.displot(data=df, x='reliabilities', hue='state', palette={'trained': colors[1], 'untrained': colors[0]},
  242. height=5 * cm, aspect=1.2,
  243. bins=30)
  245. g.refline(x=criterion)
  246. g.savefig(figure_dir + 'FigureS4_reliability_v1.png', dpi=300)
  247. g.savefig(figure_dir + 'FigureS4_reliability_v1.pdf', dpi=300, transparent=True)
  251. elif version == 'v2':
  253. diff_dist = np.median(rates[f'reliability_trained'], axis=1) - np.median(rates[f'reliability_untrained'],
  254. axis=1)
  256. g = sns.displot(x=diff_dist,
  257. height=4.5 * cm, aspect=1.2,
  258. bins=30)
  260. g.refline(x=0, c='white')
  261. g.set_ylabels("# draws")
  262. g.set_xlabels("reliability differences\n(median in pools)")
  263. g.savefig(figure_dir + 'FigureS4_reliability_v2.png', dpi=300)
  264. g.savefig(figure_dir + 'FigureS4_reliability_v2.pdf', dpi=300, transparent=True)
  267. elif version == 'v3':
  268. df = []
  269. for hemi in ['LH', 'RH']:
  270. reliabilities = get_SiteReliability(recompute=recompute, area='IT', hemisphere=hemi)
  271. for state in ['trained', 'untrained']:
  272. subs = reliabilities[state].keys()
  273. for sub in subs:
  274. df.append(pd.DataFrame({'reliabilities': np.mean(reliabilities[state][sub], axis=1),
  275. 'state': state,
  276. 'hemisphere': hemi,
  277. 'subject': sub}))
  278. df = pd.concat(df, ignore_index=True)
  280. g = sns.displot(data=df.loc[df['state'] == 'untrained'], x='reliabilities', col='hemisphere', row='subject',
  281. height=3.5 * cm, aspect=1.2,
  282. bins=30, col_order=['LH', 'RH'], row_order=['monkeyT', 'monkeyC', 'monkeyS'])
  283. g.set_titles(' ')
  284. g.set(ylim=[0, 40])
  285. for condition in g.axes_dict:
  286. subject, hemi = condition[0], condition[1]
  287. site_yield = (df.loc[(df['state'] == 'untrained') & (df['subject'] == subject) & (df['hemisphere'] == hemi), 'reliabilities'] > criterion).sum()
  289. total = len(df.loc[(df['state'] == 'untrained') & (df['subject'] == subject) & (df['hemisphere'] == hemi), 'reliabilities'])
  290. g.axes_dict[condition].text(-0.2, 35, f"{site_yield}/{total}", size=8)
  292. g.set_ylabels("# sites")
  293. g.refline(x=criterion)
  294. g.savefig(figure_dir + 'FigureS3_reliability_naive_v3.png', dpi=300)
  295. g.savefig(figure_dir + 'FigureS3_reliability_naive_v3.pdf', dpi=300, transparent=True)
  297. df.loc[df['state'] == 'untrained'].to_csv(source_file_dir + 'FigureS3_reliability_naive_v3.csv', index=False)
  299. g = sns.displot(data=df.loc[df['state'] == 'trained'], x='reliabilities', col='hemisphere', row='subject',
  300. height=3.5 * cm, aspect=1.2,
  301. col_order=['LH', 'RH'], row_order=['monkeyN', 'monkeyM', 'monkeyB'],
  302. bins=30)
  304. g.set_titles(' ')
  305. g.set(ylim=[0, 40])
  306. for condition in g.axes_dict:
  307. subject, hemi = condition[0], condition[1]
  308. site_yield = (df.loc[(df['state'] == 'trained') & (df['subject'] == subject) & (
  309. df['hemisphere'] == hemi), 'reliabilities'] > criterion).sum()
  311. total = len(df.loc[(df['state'] == 'trained') & (df['subject'] == subject) & (
  312. df['hemisphere'] == hemi), 'reliabilities'])
  313. g.axes_dict[condition].text(-0.2, 35, f"{site_yield}/{total}", size=8)
  315. g.set_ylabels("# sites")
  316. g.refline(x=criterion)
  317. g.savefig(figure_dir + 'FigureS3_reliability_trained_v3.png', dpi=300)
  318. g.savefig(figure_dir + 'FigureS3_reliability_trained_v3.pdf', dpi=300, transparent=True)
  320. df.loc[df['state'] == 'trained'].to_csv(source_file_dir + 'FigureS3_reliability_trained_v3.csv', index=False)
  323. def Figure2_selectivity(figure_dir=None, source_file_dir=None, colors=None,
  324. recompute=False, num_sites=159,
  325. num_subs=3, draws=1000,
  326. mode='empirical_conditioned_v2',
  327. version='v0', n_jobs=-1):
  328. if figure_dir == None:
  329. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  331. if source_file_dir == None:
  332. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  334. if version == 'v0':
  335. neural_df = load_neuralITselectivity(recompute=recompute, draws=1, n_jobs=n_jobs, include='combinedCategories',
  336. mode='empirical_all')
  338. df_units = []
  339. for state in ['untrained', 'trained']:
  340. df_units.append(
  341. pd.DataFrame({'state': state, 'category_selectivity': np.array(neural_df[state]).flatten()}))
  342. df_units = pd.concat(df_units, ignore_index=True)
  343. print(df_units.groupby('state').size())
  345. g = sns.displot(data=df_units, x='category_selectivity', hue='state', stat='percent', common_norm=False,
  346. kde=True, palette=colors, height=5 * cm, aspect=1.25, legend=False)
  347. g.refline(x=0, ls='--', color='white')
  348. g.set(xticks=[0, 0.1, 0.2], yticks=[0, 10, 20])
  349. g.set_xlabels('Category selectivity')
  350. g.set_ylabels('% neural sites')
  351. g.savefig(figure_dir + 'Figure2_selectivity_v0.png', dpi=300)
  352. g.savefig(figure_dir + 'Figure2_selectivity_v0.pdf', dpi=300, transparent=True)
  355. df_units.index.name = 'unit'
  356. df_units.to_csv(source_file_dir + 'Figure2_selectivity_v0.csv')
  358. elif version == 'v1':
  359. neural_df_draws = load_neuralITselectivity(recompute=recompute, draws=draws, n_jobs=n_jobs,
  360. include='combinedCategories',
  361. mode=mode, neural_sites_perSubject=num_sites // num_subs)
  362. draws = len(neural_df_draws['trained'])
  363. df = []
  364. for state in ['untrained', 'trained']:
  365. for d in range(draws):
  366. df.append(
  367. {'state': state, 'draw': d, 'selectivity': np.median(neural_df_draws[f'{state}'][d])})
  369. df = pd.DataFrame(df)
  370. g = sns.catplot(data=df, x='state', y='selectivity', hue='state', kind='point', errorbar=("pi", 95),
  371. palette=colors, height=5 * cm, aspect=1,
  372. legend=False)
  373. g.set(ylim=[0.02, 0.06], yticks=[0.02, 0.04, 0.06])
  374. g.set_xticklabels(["naive", "trained"])
  375. g.set_xlabels(f'IT neuron pool\n({num_sites} sites)')
  376. g.set_ylabels("Selectivity")
  377. g.savefig(figure_dir + 'Figure2_selectivity_v1.png', dpi=300)
  378. g.savefig(figure_dir + 'Figure2_selectivity_v1.pdf', dpi=300, transparent=True)
  380. print('Medians:')
  381. print(df.groupby('state').median())
  382. print(
  383. f"{((df.groupby('state').median().loc['trained', 'selectivity'] / df.groupby('state').median().loc['untrained', 'selectivity']) - 1) * 100}% increase")
  385. print(
  386. f"CI (95%): {((df.loc[df['state'] == 'trained', 'selectivity'].reset_index() - df.loc[df['state'] == 'untrained', 'selectivity'].reset_index())['selectivity']).quantile(0.025)} - {((df.loc[df['state'] == 'trained', 'selectivity'].reset_index() - df.loc[df['state'] == 'untrained', 'selectivity'].reset_index())['selectivity']).quantile(0.975)}")
  389. df.to_csv(source_file_dir + 'Figure2_selectivity_v1.csv', index=False)
  392. def Figure2_rsa(figure_dir=None, source_file_dir=None,
  393. neural_sites_perSubject=53, draws=1000,
  394. colors=None, recompute=False,
  395. version='v0', mode='empirical_conditioned_v2'):
  396. if figure_dir == None:
  397. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  399. if source_file_dir == None:
  400. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  403. n_sites = np.unique(np.geomspace(1, neural_sites_perSubject, num=12, dtype='int'))
  404. print(n_sites)
  405. neural_df = load_neuralRDMs(recompute=recompute, draws=draws, mode=mode,
  406. neural_sites_perSubject=n_sites)
  408. if version == 'v0':
  410. df = []
  411. for state in ['untrained', 'trained']:
  412. for d in range(draws):
  413. df.append(
  414. {'state': state, 'draw': d,
  415. 'tau': np.mean(neural_df[f'{neural_sites_perSubject * 3}_sites'][f'{state}_category'][d])})
  416. df = pd.DataFrame(df)
  417. g = sns.catplot(data=df, x='state', y='tau', hue='state', kind='point', errorbar=("pi", 95),
  418. palette=colors, height=5 * cm, aspect=1,
  419. legend=False)
  420. g.set(ylim=[0.065, 0.13], yticks=[0.08, 0.1, 0.12])
  422. g.set_xlabels('IT neuron pool')
  423. g.set_ylabels("Object-level\ncorrelation (τ)")
  424. g.savefig(figure_dir + 'Figure2_rsa_v0.png', dpi=300)
  425. g.savefig(figure_dir + 'Figure2_rsa_v0.pdf', dpi=300, transparent=True)
  427. print('Means:')
  428. print(df.groupby('state').mean())
  430. print('SD:')
  431. print(df.groupby('state').std())
  433. print(
  434. f"{((df.groupby('state').median().loc['trained', 'tau'] / df.groupby('state').median().loc['untrained', 'tau']) - 1) * 100}% increase")
  436. print(
  437. f"CI (95%): {((df.loc[df['state'] == 'trained', 'tau'].reset_index() - df.loc[df['state'] == 'untrained', 'tau'].reset_index())['tau']).quantile(0.025)} - {((df.loc[df['state'] == 'trained', 'tau'].reset_index() - df.loc[df['state'] == 'untrained', 'tau'].reset_index())['tau']).quantile(0.975)}")
  438. print(
  439. f"Mean difference: {((df.loc[df['state'] == 'trained', 'tau'].reset_index() - df.loc[df['state'] == 'untrained', 'tau'].reset_index())['tau']).mean()}")
  440. print(
  441. f"SD (diff): {((df.loc[df['state'] == 'trained', 'tau'].reset_index() - df.loc[df['state'] == 'untrained', 'tau'].reset_index())['tau']).std()}")
  443. df.to_csv(source_file_dir + 'Figure2_rsa_v0.csv', index=False)
  446. elif version == 'v1':
  448. df = []
  449. for n in n_sites:
  450. for state in ['untrained', 'trained']:
  451. for d in range(draws):
  452. df.append(
  453. {'sites': 3 * n, 'state': state, 'draw': d,
  454. 'tau': np.mean(neural_df[f'{n * 3}_sites'][f'{state}_category'][d])})
  456. df = pd.DataFrame(df)
  458. g = sns.relplot(data=df, x='sites', y='tau', hue='state', kind='line', errorbar=("pi", 95),
  459. palette=colors, height=5 * cm, aspect=1.3,
  460. legend=False)
  461. g.set(xlim=[-5, 200], xticks=[0, 100, 200], ylim=[0, 0.15])
  463. g.set_ylabels("Object-level\ncorrelation (τ)")
  464. g.set_xlabels("IT neuron pool size")
  465. g.savefig(figure_dir + 'Figure2_rsa_v1.png', dpi=300)
  466. g.savefig(figure_dir + 'Figure2_rsa_v1.pdf', dpi=300, transparent=True)
  468. df.to_csv(source_file_dir + 'Figure2_rsa_v1.csv', index=False)
  471. def Figure2_decoding(figure_dir=None, source_file_dir=None, colors=None, version='v1', recompute=False,
  472. performance_threshold=None, num_sites=159, mode='empirical_conditioned_v2', draws=1000):
  473. if figure_dir == None:
  474. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  476. if source_file_dir == None:
  477. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  479. if version == 'v0':
  480. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False,
  481. recompute=recompute,
  482. mode=mode, draws=draws)
  484. df = []
  485. for state in ['untrained', 'trained']:
  486. for d in range(draws):
  487. df.append(
  488. {'state': state, 'draw': d, 'i1': np.mean(IT_neurons[f'{num_sites}_neurons'][state]['dprimes'][d])})
  490. df = pd.DataFrame(df)
  491. g = sns.catplot(data=df, x='state', y='i1', hue='state', kind='point', errorbar=("pi", 95),
  492. palette=colors, height=5 * cm, aspect=1,
  493. legend=False)
  495. g.set(ylim=[2.5, 3.5], yticks=[2.5, 3, 3.5])
  496. g.set_titles('Image-by-image\ndecoding')
  497. g.set_xticklabels(["naive", "trained"])
  498. g.set_ylabels("Performance (d')")
  499. g.set_xlabels("IT neuron pool\n(159 sites)")
  500. g.savefig(figure_dir + 'Figure2_decoding_v0.png', dpi=300)
  501. g.savefig(figure_dir + 'Figure2_decoding_v0.pdf', dpi=300, transparent=True)
  503. print('Means:')
  504. print(df.groupby('state').mean())
  506. print('SD:')
  507. print(df.groupby('state').std())
  509. print(
  510. f"{((df.groupby('state').median().loc['trained', 'i1'] / df.groupby('state').median().loc['untrained', 'i1']) - 1) * 100}% increase")
  512. print(
  513. f"CI (95%): {((df.loc[df['state'] == 'trained', 'i1'].reset_index() - df.loc[df['state'] == 'untrained', 'i1'].reset_index())['i1']).quantile(0.025)} - {((df.loc[df['state'] == 'trained', 'i1'].reset_index() - df.loc[df['state'] == 'untrained', 'i1'].reset_index())['i1']).quantile(0.975)}")
  515. print(
  516. f"Mean difference: {((df.loc[df['state'] == 'trained', 'i1'].reset_index() - df.loc[df['state'] == 'untrained', 'i1'].reset_index())['i1']).mean()}")
  517. print(
  518. f"SD (diff): {((df.loc[df['state'] == 'trained', 'i1'].reset_index() - df.loc[df['state'] == 'untrained', 'i1'].reset_index())['i1']).std()}")
  520. df.to_csv(source_file_dir + 'Figure2_decoding_v0.csv', index=False)
  523. elif version == 'v1':
  524. mean_performance = getPerformanceThreshold('test_d-prime', kind=('mean', None))
  525. n_sites = np.unique((np.geomspace(3, num_sites, num=12, dtype='int') // 3) * 3)
  526. dPrimes = load_MonkeyIT_Decodes(resultFile_stem='CategoryDecodes_ITneurons_manySizes_', num_neurons=n_sites,
  527. includeReliability=False,
  528. recompute=recompute, mode=mode, draws=draws)
  530. df = []
  531. for n in n_sites:
  532. for state in ['untrained', 'trained']:
  533. for d in range(draws):
  534. df.append(
  535. {'state': state, 'draw': d, 'sites': n,
  536. 'i1': np.mean(dPrimes[f'{n}_neurons'][state]['dprimes'][d])})
  538. df = pd.DataFrame(df)
  540. def func(x, a, b, c): # x-shifted log
  541. return a * np.log(x + b) + c
  543. fitting_df_trained = df[df['state'] == 'trained'].groupby('sites', as_index=False)['i1'].mean()
  544. fitting_df_untrained = df[df['state'] == 'untrained'].groupby('sites', as_index=False)['i1'].mean()
  546. popt_t, pcov_t = curve_fit(func, fitting_df_trained.sites.values, fitting_df_trained.i1.values)
  547. popt_u, pcov_u = curve_fit(func, fitting_df_untrained.sites.values, fitting_df_untrained.i1.values)
  549. fitting_df_trained['predicted_i1'] = func(fitting_df_trained.sites.values, *popt_t)
  550. fitting_df_untrained['predicted_i1'] = func(fitting_df_untrained.sites.values, *popt_u)
  552. extrapolation_df_trained = pd.DataFrame(
  553. {'sites': np.arange(1, 200), 'i1_predicted': func(np.arange(1, 200), *popt_t)})
  554. extrapolation_df_untrained = pd.DataFrame(
  555. {'sites': np.arange(1, 200), 'i1_predicted': func(np.arange(1, 200), *popt_u)})
  557. g = sns.relplot(data=df, x='sites', y='i1', hue='state', kind='line', errorbar=("pi", 95),
  558. palette=colors, height=5 * cm, aspect=1.2,
  559. legend=False)
  560. g.ax.plot(extrapolation_df_trained.sites.values, extrapolation_df_trained['i1_predicted'].values, ls=':',
  561. color=sns.xkcd_palette(['jade'])[0])
  562. g.ax.plot(extrapolation_df_untrained.sites.values, extrapolation_df_untrained['i1_predicted'].values, ls=':',
  563. color=sns.xkcd_palette(['slate grey'])[0])
  565. g.refline(y=mean_performance['min_value'], color='grey', zorder=1)
  566. g.map(plt.axhspan, ymin=performance_threshold['min_value'], ymax=performance_threshold['max_value'], zorder=0,
  567. color='grey', alpha=0.4)
  569. print(
  570. f"Untrained: {extrapolation_df_untrained.loc[extrapolation_df_untrained['i1_predicted'] >= performance_threshold['min_value'], 'sites'].min()}")
  571. print(
  572. f"Trained: {extrapolation_df_trained.loc[extrapolation_df_trained['i1_predicted'] >= performance_threshold['min_value'], 'sites'].min()}")
  574. g.set(xlim=[-5, 200])
  575. g.set_titles('Image-by-image\ndecoding')
  576. g.set_ylabels("Performance (d')")
  577. g.set_xlabels("IT neuron pool")
  578. g.savefig(figure_dir + 'Figure2_decoding_v1.png', dpi=300)
  579. g.savefig(figure_dir + 'Figure2_decoding_v1.pdf', dpi=300, transparent=True)
  581. df.to_csv(source_file_dir + 'Figure2_decoding_v1.csv', index=False)
  584. elif version == 'v2':
  586. monkey_behavior = get_MonkeyBehavioralMetrics_draws(recompute=recompute, draws=draws)
  587. monkey_behavior = monkey_behavior.rename(columns={'test_d-prime': 'delta_i1'})
  588. monkey_behavior.drop(columns=['test_acc_binary', 'test_d-prime_o1'], inplace=True)
  589. monkey_behavior['data'] = 'Behavior'
  591. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False,
  592. recompute=recompute,
  593. mode=mode, draws=draws)
  595. df = []
  597. for d in range(draws):
  598. df.append(
  599. {'data': 'IT', 'draw': d,
  600. 'delta_i1': np.mean(IT_neurons[f'{num_sites}_neurons']['trained']['dprimes'][d]) - np.mean(
  601. IT_neurons[f'{num_sites}_neurons']['untrained']['dprimes'][d])})
  603. df = pd.DataFrame(df)
  604. df = pd.concat([df, monkey_behavior])
  606. g = sns.catplot(data=df, x='data', y='delta_i1', kind='point', errorbar=("pi", 95),
  607. palette={'Behavior': sns.xkcd_palette(['raspberry'])[0],
  608. 'IT': sns.xkcd_palette(['royal blue'])[0]}, height=5 * cm, aspect=0.6, hue='data',
  609. legend=False, dodge=False)
  611. g.set(ylim=[0, 4], yticks=[0, 2, 4])
  612. g.set_xticklabels(rotation=90)
  613. g.set_ylabels("Δ performance (d')")
  614. g.savefig(figure_dir + 'Figure2_decoding_v2.png', dpi=300)
  615. g.savefig(figure_dir + 'Figure2_decoding_v2.pdf', dpi=300, transparent=True)
  617. df.to_csv(source_file_dir + 'Figure2_decoding_v2.csv', index=False)
  620. def FigureS5_decoding_reliability(reliabilities=[0.3, 0.5, 0.7], num_sites_all=[159, 147, 102],
  621. figure_dir=None, source_file_dir=None, colors=None, recompute=False,
  622. mode='empirical_conditioned_v2', draws=1000):
  624. if figure_dir == None:
  625. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  627. if source_file_dir == None:
  628. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  630. df = []
  631. for reliability, num_sites in zip(reliabilities, num_sites_all):
  632. if reliability == 0.3:
  633. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False,
  634. recompute=recompute,
  635. mode=mode, draws=draws)
  636. else:
  637. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False,
  638. recompute=recompute,
  639. mode=mode, draws=draws, reliability_criterion=reliability)
  641. for state in ['untrained', 'trained']:
  642. for d in range(draws):
  643. df.append(
  644. {'state': state, 'draw': d, 'i1': np.mean(IT_neurons[f'{num_sites}_neurons'][state]['dprimes'][d]),
  645. 'reliability criterion': reliability})
  647. df = pd.DataFrame(df)
  648. g = sns.catplot(data=df, x='state', y='i1', hue='state', kind='point', errorbar=("pi", 95),
  649. palette=colors, height=5 * cm, aspect=1, col= 'reliability criterion',
  650. legend=False)
  651. g.set_titles('{col_name}')
  652. g.set_xticklabels(["naive", "trained"])
  653. g.set_ylabels("Performance (d')")
  654. g.set_xlabels("IT neuron pool")
  655. g.savefig(figure_dir + 'FigureS5_decoding_reliabilityCriterion.png', dpi=300)
  656. g.savefig(figure_dir + 'FigureS5_decoding_reliabilityCriterion.pdf', dpi=300, transparent=True)
  658. df.to_csv(source_file_dir + 'FigureS5_decoding_reliabilityCriterion.csv', index=False)
  661. def FigureS6_singleSubject_analyses(figure_dir=None, source_file_dir=None, colors=None, recompute=False, reliability_criterion=0.3):
  663. if figure_dir == None:
  664. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  666. if source_file_dir == None:
  667. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  670. IT_decoding = load_MonkeyIT_Decodes_singleAnimal(recompute=recompute,reliability_criterion=reliability_criterion)
  671. IT_selectivity = load_neuralITselectivity_singleAnimal(recompute=recompute,reliability_criterion=reliability_criterion)
  672. IT_rsa = load_neuralRDMs_singleAnimal(recompute=recompute,reliability_criterion=reliability_criterion)
  674. df = []
  675. for state in ['untrained', 'trained']:
  676. subs = list(IT_decoding[state].keys())
  677. for sub in subs:
  679. df.append(
  680. {'state': state, 'subject': sub[-1:],
  681. 'i1': np.mean(IT_decoding[state][sub]['i1']),
  682. 'selectivity': np.median(IT_selectivity[state][sub]),
  683. 'tau': float(np.squeeze(IT_rsa[state][sub])),
  684. 'num_sites': np.mean(IT_decoding[state][sub]['n_sites'])})
  686. df = pd.DataFrame(df)
  689. g = sns.catplot(data=df, x='subject', y='i1', hue='state', kind='point',
  690. errorbar=("pi", 95),
  691. palette=colors, height=5 * cm, aspect=1,
  692. legend=False, join=False)
  694. g.set(ylim=[1, 4])
  695. g.set_ylabels("Performance (d')\n ")
  696. g.refline(y=df.loc[df['state'] == 'trained']['i1'].mean(), color=colors[1])
  697. g.refline(y=df.loc[df['state'] == 'untrained']['i1'].mean(), color=colors[0])
  698. g.set_xlabels("Subject")
  699. g.savefig(figure_dir + 'FigureS6_singleSubject_decoding_i1.png', dpi=300)
  700. g.savefig(figure_dir + 'FigureS6_singleSubject_decoding_i1.pdf', dpi=300, transparent=True)
  702. g = sns.catplot(data=df, x='subject', y='selectivity', hue='state', kind='point',
  703. errorbar=("pi", 95),
  704. palette=colors, height=5 * cm, aspect=1,
  705. legend=False, join=False)
  706. g.set(ylim=[0, 0.08])
  707. g.refline(y=df.loc[df['state'] == 'trained']['selectivity'].mean(), color=colors[1])
  708. g.refline(y=df.loc[df['state'] == 'untrained']['selectivity'].mean(), color=colors[0])
  709. g.set_ylabels("Selectivity")
  710. g.set_xlabels("Subject")
  711. g.savefig(figure_dir + 'FigureS6_singleSubject_selectivity.png', dpi=300)
  712. g.savefig(figure_dir + 'FigureS6_singleSubject_selectivity.pdf', dpi=300, transparent=True)
  716. g = sns.catplot(data=df, x='subject', y='tau', hue='state', kind='point',
  717. errorbar=("pi", 95),
  718. palette=colors, height=5 * cm, aspect=1,
  719. legend=False, join=False)
  720. g.set(ylim=[0.04, 0.12])
  721. g.set_ylabels("Correlation (τ)")
  722. g.set_xlabels("Subject")
  723. g.refline(y=df.loc[df['state'] == 'trained']['tau'].mean(), color=colors[1])
  724. g.refline(y=df.loc[df['state'] == 'untrained']['tau'].mean(), color=colors[0])
  725. g.savefig(figure_dir + 'FigureS6_singleSubject_rsa.png', dpi=300)
  726. g.savefig(figure_dir + 'FigureS6_singleSubject_rsa.pdf', dpi=300, transparent=True)
  728. df.to_csv(source_file_dir + 'FigureS6_singleSubject.csv', index=False)
  731. def FigureS7_selectivity(figure_dir=None, source_file_dir=None, recompute=False, n_jobs=-1,
  732. num_sites=159, mode='empirical_conditioned_v2', draws=200):
  733. if figure_dir == None:
  734. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  736. if source_file_dir == None:
  737. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  739. IT_neurons_null = load_neuralITselectivity_permutation(neural_sites_perSubject=num_sites // 3,
  740. include='combinedCategories',
  741. recompute=recompute, mode=mode, draws=draws, n_jobs=n_jobs)
  743. IT_neurons_observed = load_neuralITselectivity(recompute=False, n_jobs=n_jobs,
  744. include='combinedCategories',
  745. mode=mode, neural_sites_perSubject=num_sites // 3)
  747. deltas = pd.DataFrame()
  748. for permutation in IT_neurons_null.keys():
  749. if permutation == 'sites':
  750. pass
  751. else:
  752. deltas = pd.concat([deltas,
  753. pd.DataFrame({'permutation': str(permutation),
  754. 'delta': np.median(IT_neurons_null[permutation]['group1'],
  755. axis=1) - np.median(
  756. IT_neurons_null[permutation]['group2'], axis=1), 'kind': 'null'})],
  757. ignore_index=True)
  759. deltas = pd.concat([deltas, pd.DataFrame({'permutation': 'None',
  760. 'delta': np.median(IT_neurons_observed['trained'], axis=1)[
  761. :draws] - np.median(IT_neurons_observed['untrained'], axis=1)[
  762. :draws], 'kind': 'empirical'})], ignore_index=True)
  764. g = sns.displot(data=deltas[deltas['kind'] == 'null'], x='delta', hue='kind', palette='colorblind', height=5 * cm,
  765. aspect=1.2)
  766. g.refline(x=deltas.groupby('kind')['delta'].mean()['empirical'], c='red')
  768. g.set_xlabels("differences (Δ)")
  769. print(
  770. f"p-value (two-sided): {(deltas.groupby('kind')['delta'].mean().loc['empirical'] <= deltas.loc[deltas['kind'] == 'null', 'delta']).mean()}")
  771. g.savefig(figure_dir + 'FigureS7_permutation_selectivity.png', dpi=300)
  772. g.savefig(figure_dir + 'FigureS7_permutation_selectivity.pdf', dpi=300, transparent=True)
  774. deltas.to_csv(source_file_dir + 'FigureS7_permutation_selectivity.csv', index=False)
  777. def FigureS7_rsa(figure_dir=None, source_file_dir=None, recompute=False, subs=3,
  778. neural_sites_perSubject=53, mode='empirical_conditioned_v2', draws=200):
  779. if figure_dir == None:
  780. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  782. if source_file_dir == None:
  783. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  785. IT_neurons_null = load_neuralRDMs_permutation(neural_sites_perSubject=np.array([neural_sites_perSubject]),
  786. recompute=recompute,
  787. mode=mode, draws=draws)
  788. IT_neurons_observed = load_neuralRDMs(recompute=False, mode=mode,
  789. neural_sites_perSubject=np.array([neural_sites_perSubject]))
  791. num_sites = neural_sites_perSubject * subs
  792. deltas = pd.DataFrame()
  793. for permutation in IT_neurons_null[f'{num_sites}_sites'].keys():
  794. if permutation == 'sites':
  795. pass
  796. else:
  797. deltas = pd.concat([deltas,
  798. pd.DataFrame(
  799. {'delta': IT_neurons_null[f'{num_sites}_sites'][permutation]['group1_category']
  800. - IT_neurons_null[f'{num_sites}_sites'][permutation]['group2_category'],
  801. 'kind': 'null'})], ignore_index=True)
  803. deltas = pd.concat(
  804. [deltas, pd.DataFrame({'delta': IT_neurons_observed[f'{num_sites}_sites']['trained_category'][:draws] -
  805. IT_neurons_observed[f'{num_sites}_sites']['untrained_category'][:draws],
  806. 'kind': 'empirical'})], ignore_index=True)
  808. g = sns.displot(data=deltas[deltas['kind'] == 'null'], x='delta', hue='kind', palette='colorblind', height=5 * cm,
  809. aspect=1.2)
  810. g.refline(x=deltas.groupby('kind')['delta'].mean()['empirical'], c='red')
  812. g.set_xlabels("differences (Δ)")
  813. print(
  814. f"p-value (two-sided): {(deltas.groupby('kind')['delta'].mean().loc['empirical'] <= deltas.loc[deltas['kind'] == 'null', 'delta']).mean()}")
  815. g.savefig(figure_dir + 'FigureS7_permutation_rsa.png', dpi=300)
  816. g.savefig(figure_dir + 'FigureS7_permutation_rsa.pdf', dpi=300, transparent=True)
  818. deltas.to_csv(source_file_dir + 'FigureS7_permutation_rsa.csv', index=False)
  821. def FigureS7_decoding(figure_dir=None, source_file_dir=None,
  822. recompute=False,
  823. num_sites=159, mode='empirical_conditioned_v2', draws=200):
  824. if figure_dir == None:
  825. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  827. if source_file_dir == None:
  828. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  830. IT_neurons_null = load_MonkeyIT_Decodes_permutation(num_neurons=np.array([num_sites]),
  831. recompute=recompute, mode=mode, draws=draws)
  832. IT_neurons_observed = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False,
  833. recompute=False, mode=mode)
  835. deltas = pd.DataFrame()
  836. for permutation in IT_neurons_null[f'{num_sites}_neurons'].keys():
  837. if permutation == 'sites':
  838. pass
  839. else:
  840. deltas = pd.concat([deltas,
  841. pd.DataFrame({'delta': np.mean(
  842. IT_neurons_null[f'{num_sites}_neurons'][permutation]['group1']['dprimes'],
  843. axis=1) - np.mean(
  844. IT_neurons_null[f'{num_sites}_neurons'][permutation]['group2']['dprimes'], axis=1)
  845. , 'kind': 'null'})], ignore_index=True)
  847. deltas = pd.concat(
  848. [deltas, pd.DataFrame({'delta': np.mean(IT_neurons_observed[f'{num_sites}_neurons']['trained']['dprimes'],
  849. axis=1)[:draws] - np.mean(
  850. IT_neurons_observed[f'{num_sites}_neurons']['untrained']['dprimes'], axis=1)[:draws]
  851. , 'kind': 'empirical'})], ignore_index=True)
  853. g = sns.displot(data=deltas[deltas['kind'] == 'null'], x='delta', hue='kind', palette='colorblind', height=5 * cm,
  854. aspect=1.2)
  855. g.refline(x=deltas.groupby('kind')['delta'].mean()['empirical'], c='red')
  857. g.set_xlabels("differences (Δ)")
  859. g.savefig(figure_dir + 'FigureS7_permutation_decoding.png', dpi=300)
  860. g.savefig(figure_dir + 'FigureS7_permutation_decoding.pdf', dpi=300, transparent=True)
  862. deltas.to_csv(source_file_dir + 'FigureS7_permutation_decoding.csv', index=False)
  864. print(
  865. f"p-value (two-sided): {(deltas.groupby('kind')['delta'].mean().loc['empirical'] <= deltas.loc[deltas['kind'] == 'null', 'delta']).mean()}")
  867. def Figure3_decoding_anatomy(mode='empirical_conditioned_v2', draws=300, num_sites=159, recompute=False,
  868. figure_dir=None, source_file_dir=None, colors=None):
  870. if figure_dir == None:
  871. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  873. if source_file_dir == None:
  874. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  876. IT_decodes_AIT = load_MonkeyIT_subregion_Decodes('AIT', mode=mode, draws=draws, num_neurons=np.array([32]),
  877. includeReliability=False, recompute=recompute)
  878. IT_decodes_AIT['num_subs'] = 1
  879. IT_decodes_CIT = load_MonkeyIT_subregion_Decodes('CIT', mode=mode, draws=draws, num_neurons=np.array([19]),
  880. includeReliability=False, recompute=recompute)
  881. IT_decodes_CIT['num_subs'] = 3
  882. IT_decodes_PIT = load_MonkeyIT_subregion_Decodes('PIT', mode=mode, draws=draws, num_neurons=np.array([17]), includeReliability=False, recompute=recompute)
  883. IT_decodes_PIT['num_subs'] = 2
  885. df = []
  886. for area in ['AIT', 'CIT', 'PIT']:
  887. for state in ['untrained', 'trained']:
  888. for d in range(draws):
  889. num_sites = np.squeeze(eval(f'IT_decodes_{area}')['num_subs'] * eval(f'IT_decodes_{area}')['num_neurons'])
  890. df.append(
  891. {'area': area, 'sites': num_sites, 'subs':eval(f'IT_decodes_{area}')['num_subs'], 'state': state, 'draw': d, 'i1': np.mean(eval(f'IT_decodes_{area}')[f'{num_sites}_neurons'][state]['dprimes'][d])})
  893. df = pd.DataFrame(df)
  894. g = sns.catplot(data=df, col='area', x='state', y='i1', hue='state', kind='point', errorbar=("pi", 95),
  895. palette=colors, height=5 * cm, aspect=0.75,
  896. legend=False, sharey=False)
  898. g.set_titles('{col_name}')
  899. g.set_xticklabels(["naïve", "trained"])
  900. g.set_ylabels("Performance (d')")
  901. g.set_xlabels('IT neuron pool')
  904. g.savefig(figure_dir + 'Figure3_subregions_decoding_i1.png', dpi=300)
  905. g.savefig(figure_dir + 'Figure3_subregions_decoding_i1.pdf', dpi=300, transparent=True)
  907. df.to_csv(source_file_dir + 'Figure3_subregions_decoding_i1.csv', index=False)
  909. print('Means:')
  910. print(df.groupby(['area', 'state']).mean())
  912. print('SD:')
  913. print(df.groupby(['area', 'state']).std())
  914. for area in ['AIT', 'CIT', 'PIT']:
  915. print(area)
  916. print(
  917. f"{((df.loc[df['area'] == area].groupby(['state'])['i1'].mean()['trained'] / df.loc[df['area'] == area].groupby(['state'])['i1'].mean()['untrained']) - 1) * 100}% increase")
  919. df_dist = ((df.loc[(df['area'] == area) & (df['state'] == 'trained'), 'i1'].reset_index() - df.loc[
  920. (df['area'] == area) & (df['state'] == 'untrained'), 'i1'].reset_index())['i1'])
  921. print(
  922. f"CI (95%): {df_dist.quantile(0.025)} - {df_dist.quantile(0.975)}")
  923. print(
  924. f"p-val: {np.min([(df_dist < 0).mean(), (df_dist > 0).mean()]) * 2}")
  926. print(
  927. f"Mean diff: {((df.loc[(df['state'] == 'trained') & (df['area'] == area), 'i1'].reset_index() - df.loc[(df['state'] == 'untrained') & (df['area'] == area), 'i1'].reset_index())['i1']).mean()}")
  929. print(
  930. f"Mean SD: {((df.loc[(df['state'] == 'trained') & (df['area'] == area), 'i1'].reset_index() - df.loc[(df['state'] == 'untrained') & (df['area'] == area), 'i1'].reset_index())['i1']).std()}")
  933. IT_decodes_LH = load_MonkeyIT_subregion_Decodes('IT', hemisphere='LH', mode=mode, draws=draws,
  934. num_neurons=np.array([89]), # n_jobs=1,
  935. includeReliability=False, recompute=False)
  936. IT_decodes_LH['num_subs'] = 2
  937. area = 'LH'
  938. df_LH = []
  939. for state in ['untrained', 'trained']:
  940. for d in range(draws):
  941. num_sites = np.squeeze(
  942. eval(f'IT_decodes_{area}')['num_subs'] * eval(f'IT_decodes_{area}')['num_neurons'])
  943. df_LH.append(
  944. {'area': area, 'sites': num_sites, 'subs': eval(f'IT_decodes_{area}')['num_subs'], 'state': state,
  945. 'draw': d, 'i1': np.mean(eval(f'IT_decodes_{area}')[f'{num_sites}_neurons'][state]['dprimes'][d])})
  948. df_LH = pd.DataFrame(df_LH)
  949. g = sns.catplot(data=df_LH, col='area', x='state', y='i1', hue='state', kind='point',
  950. errorbar=("pi", 95),
  951. palette=colors, height=5 * cm, aspect=0.9,
  952. legend=False, sharey=False)
  954. g.set_titles('Left hemisphere')
  955. g.set_xticklabels(["naïve", "trained"])
  956. g.set_ylabels("Performance (d')")
  957. g.set_xlabels('IT neuron pool')
  958. g.savefig(figure_dir + 'Figure3_hemisphere-LH_decoding_i1.png', dpi=300)
  959. g.savefig(figure_dir + 'Figure3_hemisphere-LH_decoding_i1.pdf', dpi=300, transparent=True)
  961. df_LH.to_csv(source_file_dir + 'Figure3_hemisphere-LH_decoding_i1.csv', index=False)
  963. print('Means:')
  964. print(df_LH.groupby(['state'])['i1'].mean())
  966. print('SD:')
  967. print(df_LH.groupby(['state'])['i1'].std())
  969. print(f"{((df_LH.groupby(['state'])['i1'].mean()['trained'] / df_LH.groupby(['state'])['i1'].mean()['untrained']) - 1) * 100}% increase")
  971. df_dist = ((df_LH.loc[(df_LH['state'] == 'trained'), 'i1'].reset_index() - df_LH.loc[
  972. (df['state'] == 'untrained'), 'i1'].reset_index())['i1'])
  973. print(
  974. f"CI (95%): {df_dist.quantile(0.025)} - {df_dist.quantile(0.975)}")
  975. print(
  976. f"p-val: {np.min([(df_dist < 0).mean(), (df_dist > 0).mean()]) * 2}")
  978. print(
  979. f"Mean diff: {df_dist.mean()}")
  980. print(
  981. f"SD diff: {df_dist.std()}")
  983. n_sites = np.unique((np.geomspace(3, num_sites, num=12, dtype='int') // 3) * 3)
  984. dPrimes = load_MonkeyIT_Decodes(resultFile_stem='CategoryDecodes_ITneurons_manySizes_', num_neurons=n_sites,
  985. includeReliability=False,
  986. recompute=recompute, mode=mode, draws=draws)
  987. n_sites = dPrimes['num_neurons']
  988. df_multi = []
  989. for n in n_sites:
  990. for state in ['untrained', 'trained']:
  991. for d in range(draws):
  992. df_multi.append(
  993. {'state': state, 'draw': d, 'sites': n,
  994. 'i1': np.mean(dPrimes[f'{n}_neurons'][state]['dprimes'][d])})
  996. df_multi = pd.DataFrame(df_multi)
  998. def func(x, a, b, c): # x-shifted log
  999. return a * np.log(x + b) + c
  1001. fitting_df_trained = df_multi[df_multi['state'] == 'trained'].groupby('sites')['i1'].mean()
  1002. fitting_df_untrained = df_multi[df_multi['state'] == 'untrained'].groupby('sites')['i1'].mean()
  1004. popt_t, pcov_t = curve_fit(func, fitting_df_trained.index.values, fitting_df_trained.values)
  1005. popt_u, pcov_u = curve_fit(func, fitting_df_untrained.index.values, fitting_df_untrained.values)
  1007. fitting_df_trained['predicted_i1'] = func(fitting_df_trained.index.values, *popt_t)
  1008. fitting_df_untrained['predicted_i1'] = func(fitting_df_untrained.index.values, *popt_u)
  1010. extrapolation_df_trained = pd.DataFrame(
  1011. {'sites': np.arange(1, 200), 'i1_predicted': func(np.arange(1, 200), *popt_t)})
  1012. extrapolation_df_untrained = pd.DataFrame(
  1013. {'sites': np.arange(1, 200), 'i1_predicted': func(np.arange(1, 200), *popt_u)})
  1015. g = sns.relplot(data=df_multi, x='sites', y='i1', hue='state', kind='line', errorbar=("pi", 95),
  1016. palette=colors, height=5 * cm, aspect=1.2,
  1017. legend=False)
  1018. g.ax.plot(extrapolation_df_trained.sites.values, extrapolation_df_trained['i1_predicted'].values, ls=':',
  1019. color=sns.xkcd_palette(['jade'])[0])
  1020. g.ax.plot(extrapolation_df_untrained.sites.values, extrapolation_df_untrained['i1_predicted'].values, ls=':',
  1021. color=sns.xkcd_palette(['slate grey'])[0])
  1023. for area in ['AIT', 'CIT', 'PIT']:
  1024. num_sites = df.loc[(df['area'] == area), 'sites'].values[0]
  1025. for s, state in enumerate(['untrained', 'trained']):
  1026. g.ax.scatter(num_sites,df.loc[(df['area'] == area) & (df['state'] == state), 'i1'].mean(),
  1027. color=colors[s])
  1028. for s, state in enumerate(['untrained', 'trained']):
  1029. g.ax.scatter(df_LH['sites'].values[0], df_LH.loc[(df['state'] == state), 'i1'].mean(),
  1030. color=colors[s])
  1032. g.set(xlim=[-5, 200])
  1033. g.set_titles('Image-by-image\ndecoding')
  1034. g.set_ylabels("Performance (d')")
  1035. g.set_xlabels("IT neuron pool")
  1036. g.savefig(figure_dir + 'Figure3_anatomyScaling_decoding_i1.png', dpi=300)
  1037. g.savefig(figure_dir + 'Figure3_anatomyScaling_decoding_i1.pdf', dpi=300, transparent=True)
  1039. def Figure4_novel500_decoding(draws=1000, mode='empirical_conditioned_v2', figure_dir=None, source_file_dir=None,
  1040. colors=None, recompute=False, num_sites=55):
  1042. if figure_dir == None:
  1043. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1045. if source_file_dir == None:
  1046. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1048. n_sites = np.unique(np.geomspace(3, num_sites, num=12, dtype='int'))
  1050. IT_neurons_hvm = load_MonkeyIT_Decodes_novel500_HVMcontrol(num_neurons=n_sites, includeReliability=False,
  1051. recompute=recompute,
  1052. mode=mode, draws=draws)
  1054. IT_neurons_novel500 = load_MonkeyIT_Decodes_novel500(num_neurons=n_sites, includeReliability=False,
  1055. recompute=recompute,
  1056. mode=mode, draws=draws)
  1058. df = []
  1059. for state in ['untrained', 'trained']:
  1060. for site in n_sites:
  1061. for d in range(draws):
  1062. df.append(
  1063. {'dataset': 'Novel500', 'state': state, 'pool size': site, 'draw': d,
  1064. 'i1': np.mean(IT_neurons_novel500[f'{site}_neurons'][state]['dprimes'][d])})
  1065. df.append(
  1066. {'dataset': 'HVM', 'state': state, 'pool size': site, 'draw': d,
  1067. 'i1': np.mean(IT_neurons_hvm[f'{site}_neurons'][state]['dprimes'][d])})
  1069. df = pd.DataFrame(df)
  1071. # DIFFICULTY-CONTROLLED DATASETS
  1072. # sample images such that HVM has the same untrained difficulty
  1073. np.random.seed(2)
  1074. novel500_kde = gaussian_kde(np.mean(IT_neurons_novel500[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0))
  1075. hvm_kde = gaussian_kde(np.mean(IT_neurons_hvm[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0))
  1077. # Function to find the minimum of two KDE evaluations
  1078. shared_kde = lambda x: np.min([novel500_kde(x), hvm_kde(x)], axis=0)
  1079. overlap_area, _ = quad(shared_kde, -1, 4)
  1080. maximum_sample = int(500 * overlap_area)
  1082. # map this to the images
  1083. naive_img_hvm = pd.DataFrame({'dataset': 'HVM', 'img': np.arange(640),
  1084. 'i1': np.mean(IT_neurons_hvm[f'{num_sites}_neurons']['untrained']['dprimes'],
  1085. axis=0)})
  1086. naive_img_hvm['pdf'] = novel500_kde(naive_img_hvm['i1']) / hvm_kde(naive_img_hvm['i1'])
  1088. naive_img_novel500 = pd.DataFrame({'dataset': 'novel500', 'img': np.arange(500), 'i1': np.mean(
  1089. IT_neurons_novel500[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0)})
  1090. naive_img_novel500['pdf'] = hvm_kde(naive_img_novel500['i1']) / novel500_kde(naive_img_novel500['i1'])
  1092. hvm_mean = 0
  1093. novel500_mean = 1
  1094. while np.round(abs(hvm_mean - novel500_mean), 1) != 0:
  1095. sampled_hvm = naive_img_hvm.sample(maximum_sample, weights='pdf')
  1096. sampled_novel500 = naive_img_novel500.sample(maximum_sample, weights='pdf')
  1098. hvm_mean = sampled_hvm['i1'].mean()
  1099. novel500_mean = sampled_novel500['i1'].mean()
  1100. print(hvm_mean)
  1101. print(novel500_mean)
  1103. g, ax = plt.subplots(1, 2, figsize=(4.5 * cm * 0.9 * 2, 4.5 * cm), sharey=True, sharex=True)
  1104. sns.histplot(np.mean(IT_neurons_hvm[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0), ax=ax[0],
  1105. color=sns.xkcd_palette(['dull yellow'])[0], kde=True)
  1106. sns.histplot(np.mean(IT_neurons_novel500[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0), ax=ax[0],
  1107. color=sns.xkcd_palette(['pale lavender'])[0], kde=True)
  1108. sns.lineplot(x=np.arange(-1, 4, 0.1), y=shared_kde(np.arange(-1, 4, 0.1)) * 100,
  1109. color=sns.xkcd_palette(['blush'])[0], ax=ax[0])
  1111. ax[0].set_xlabel("Task-naive accuracy (d')")
  1112. ax[0].set_ylabel("# images")
  1114. sns.histplot(sampled_hvm['i1'], ax=ax[1], color=sns.xkcd_palette(['dull yellow'])[0])
  1115. sns.histplot(sampled_novel500['i1'], ax=ax[1], color=sns.xkcd_palette(['pale lavender'])[0])
  1116. sns.lineplot(x=np.arange(-1, 4, 0.1), y=shared_kde(np.arange(-1, 4, 0.1)) * 100,
  1117. color=sns.xkcd_palette(['blush'])[0], ax=ax[1])
  1118. ax[1].set_xlabel(" ")
  1119. plt.tight_layout()
  1120. g.savefig(figure_dir + 'Figure4_novel500_decoding_naiveDists.png', dpi=300)
  1121. g.savefig(figure_dir + 'Figure4_novel500_decoding_naiveDists.pdf', dpi=300, transparent=True)
  1124. # apply sampling to the whole dataset
  1125. df_matched = []
  1126. for state in ['untrained', 'trained']:
  1127. for d in range(draws):
  1128. df_matched.append(
  1129. {'dataset': 'Novel500', 'state': state, 'pool size': num_sites, 'draw': d,
  1130. 'i1': np.mean(
  1131. IT_neurons_novel500[f'{num_sites}_neurons'][state]['dprimes'][d, sampled_novel500['img'].values])})
  1132. df_matched.append(
  1133. {'dataset': 'HVM', 'state': state, 'pool size': num_sites, 'draw': d,
  1134. 'i1': np.mean(IT_neurons_hvm[f'{num_sites}_neurons'][state]['dprimes'][d, sampled_hvm['img'].values])})
  1136. df_matched = pd.DataFrame(df_matched)
  1138. g = sns.catplot(data=df_matched, x='state', y='i1', hue='state', kind='point',
  1139. errorbar=("pi", 95), col='dataset',
  1140. palette=colors, height=5 * cm, aspect=0.7,
  1141. legend=False)
  1142. g.set(ylim=[0.5, 2.5])
  1143. g.set_titles(' ')
  1144. g.set_xticklabels(["naïve", "trained"])
  1145. g.set_ylabels("Performance (d')")
  1146. g.set_xlabels("IT neuron pool")
  1147. g.refline(y=(sampled_hvm['i1'].mean() + sampled_novel500['i1'].mean()) / 2, color=sns.xkcd_palette(['blush'])[0], zorder=0)
  1148. g.savefig(figure_dir + 'Figure4_novel500_decoding_i1_naiveMatched.png', dpi=300)
  1149. g.savefig(figure_dir + 'Figure4_novel500_decoding_i1_naiveMatched.pdf', dpi=300, transparent=True)
  1151. df_matched.to_csv(source_file_dir + 'Figure4_novel500_decoding_i1_naiveMatched.csv', index=False)
  1153. print('Means:')
  1154. print(df_matched.loc[df_matched['pool size'] == num_sites].groupby(['dataset', 'state']).mean())
  1156. print('SD:')
  1157. print(df_matched.loc[df_matched['pool size'] == num_sites].groupby(['dataset', 'state']).std())
  1159. for dataset in ['HVM', 'Novel500']:
  1160. print(dataset)
  1161. print(
  1162. f"{((df_matched.loc[(df_matched['dataset'] == dataset) & (df_matched['pool size'] == num_sites)].groupby(['state'])['i1'].mean()['trained'] / df_matched.loc[(df_matched['dataset'] == dataset) & (df_matched['pool size'] == num_sites)].groupby(['state'])['i1'].mean()['untrained']) - 1) * 100}% increase")
  1164. df_dist = ((df_matched.loc[(df_matched['dataset'] == dataset) & (df_matched['pool size'] == num_sites) & (
  1165. df_matched['state'] == 'trained'), 'i1'].reset_index() - df_matched.loc[
  1166. (df_matched['dataset'] == dataset) & (df_matched['pool size'] == num_sites) & (
  1167. df_matched['state'] == 'untrained'), 'i1'].reset_index())['i1'])
  1168. print(
  1169. f"CI (95%): {df_dist.quantile(0.025)} - {df_dist.quantile(0.975)}")
  1170. print(
  1171. f"p-val: {np.min([(df_dist < 0).mean(), (df_dist > 0).mean()]) * 2}")
  1173. print(
  1174. f"Mean diff: {df_dist.mean()}")
  1175. print(
  1176. f"SD diff: {df_dist.std()}")
  1178. def FigureS9_BrainScoreLayerMapping(base_models, figure_dir=None, source_file_dir=None,
  1179. result_file=None
  1180. ):
  1181. if figure_dir == None:
  1182. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1183. if result_file == None:
  1184. result_file = f'{Path(__file__).parent.parent}/Results/meta/BrainScore_ITmappings.pkl'
  1185. if source_file_dir == None:
  1186. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1188. df_all = pd.read_pickle(result_file)
  1190. col_order = [' ']
  1191. col_order.extend(base_models)
  1193. g = sns.FacetGrid(data=df_all, col='baseModel', col_wrap=3, col_order=col_order, height=5 * cm, aspect=1.3,
  1194. sharex=False, sharey=False)
  1196. g.map(plt.errorbar, 'layer', 'scores_x', 'scores_y', color='grey', zorder=0)
  1198. for i, ax in enumerate(g.axes[1:]):
  1199. ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
  1200. xs, ys = ax.lines[-1].get_data()
  1202. ax.scatter(xs[np.argmax(ys)], np.max(ys), color=sns.xkcd_palette(['royal blue'])[0], s=20)
  1203. ax.axvline(xs[np.argmax(ys)], color=sns.xkcd_palette(['royal blue'])[0], ls='--')
  1204. ax.set_title(base_models[i])
  1206. g.set_xlabels('Model layer')
  1207. g.set_ylabels('IT encoding\nscore')
  1209. plt.subplots_adjust(hspace=1.75, wspace=0.2)
  1210. g.savefig(f'{figure_dir}FigureS9_ITMappings.png', dpi=300)
  1211. g.savefig(f'{figure_dir}FigureS9_ITMappings.pdf', transparent=True, dpi=300)
  1213. df_all.to_csv(source_file_dir + 'FigureS9_ITMappings.csv', index=False)
  1216. def FigureS10_UnitSiteScaling_IT(base_models, num_sites=159,
  1217. recompute=False,
  1218. reestimate_predictions=False,
  1219. draws=1000,
  1220. maximum_scaling=101,
  1221. resultFile=None,
  1222. figure_dir=None,
  1223. source_file_dir=None,
  1224. mode='empirical_conditioned_v2'):
  1225. if figure_dir == None:
  1226. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1228. if resultFile == None:
  1229. resultFile = f'{Path(__file__).parent.parent}/Results/meta/UnitSiteScaling.pkl'
  1231. if source_file_dir == None:
  1232. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1235. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False, mode=mode,
  1236. recompute=recompute, draws=draws)
  1238. dfs = get_ModelIT_categoryDecoding_UnitSiteScaling(base_models, reestimate_predictions=reestimate_predictions,
  1239. recompute=recompute, num_sites=num_sites)
  1241. target_value = IT_neurons[f'{num_sites}_neurons']['untrained']['dprimes'].mean()
  1242. dfs['siteScaling'] = dfs['units'] / num_sites
  1244. def func(x, a, b, c): # x-shifted log
  1245. return a * np.log(x + b) + c
  1247. fitting_df = dfs[dfs['siteScaling'] < maximum_scaling].groupby(['base_model', 'siteScaling']).mean()
  1248. sigma_df = dfs[dfs['siteScaling'] < maximum_scaling].groupby(['base_model', 'siteScaling']).std()
  1249. sigma_df.loc[sigma_df['i1'] == 0] = 1
  1251. g = sns.relplot(data=dfs, x='siteScaling', y='i1', kind='line', errorbar=("pi", 95),
  1252. color='seagreen', height=5 * cm, aspect=1.2, style='base_model',
  1253. dashes=False, col='base_model', col_wrap=3,
  1254. markers=['o' for _ in base_models], legend=False)
  1255. g.set_titles("{col_name}")
  1256. g.refline(y=target_value, color='grey', ls='-')
  1258. unit_scaling = []
  1259. for model, ax in zip(base_models, g.axes):
  1260. # print(model)
  1261. popt, pcov = curve_fit(func, fitting_df.loc[model].index.values, fitting_df.loc[model, 'i1'].values,
  1262. sigma=sigma_df.loc[model, 'i1'].values,
  1263. )
  1264. # print(popt)
  1265. fitting_df.loc[model, 'predicted_i1'] = func(fitting_df.loc[model].index.values, *popt)
  1267. interpolation_df = pd.DataFrame(
  1268. {'base_model': model, 'sitesScaling': np.arange(1, 100), 'i1_predicted': func(np.arange(1, 100), *popt)})
  1270. ax.plot(interpolation_df.sitesScaling.values, interpolation_df['i1_predicted'].values, ls=':', color='m')
  1272. required_sites = interpolation_df.loc[interpolation_df['i1_predicted'] >= target_value, 'sitesScaling'].min()
  1274. # Identify the adjacent actual datapoints
  1275. required_sites_empirical_upper = fitting_df.loc[model, 'i1'][
  1276. fitting_df.loc[model, 'i1'] >= target_value].index.min()
  1277. required_sites_empirical_lower = fitting_df.loc[model, 'i1'][
  1278. fitting_df.loc[model, 'i1'] <= target_value].index.max()
  1280. if pd.isna(required_sites_empirical_upper) | pd.isna(required_sites_empirical_lower):
  1281. required_sites_empirical_estimated = np.nan
  1282. else:
  1283. lower_val = fitting_df.loc[model].loc[required_sites_empirical_lower]['i1']
  1284. upper_val = fitting_df.loc[model].loc[required_sites_empirical_upper]['i1']
  1286. linearFit = np.polyfit([required_sites_empirical_lower, required_sites_empirical_upper],
  1287. [lower_val, upper_val], 1)
  1288. required_sites_empirical_estimated = (target_value - linearFit[1]) / linearFit[0]
  1290. # ax.axvline(x=required_sites_empirical_estimated, color='orange')
  1292. if required_sites != np.nan:
  1293. ax.axvline(x=required_sites, color='m')
  1294. print(f'{model}: {required_sites} / {required_sites_empirical_estimated}')
  1295. unit_scaling.append(
  1296. {'base_model': model, 'siteUnitScaling': required_sites, 'units': required_sites * num_sites})
  1298. g.set(xlim=[0, maximum_scaling])
  1299. g.set_ylabels("Accuracy (d')")
  1300. g.set_xlabels("Unit/site ratio")
  1301. g.savefig(figure_dir + 'FigureS10_siteUnitScaling.png', dpi=300)
  1302. g.savefig(figure_dir + 'FigureS10_siteUnitScaling.pdf', dpi=300, transparent=True)
  1304. dfs.to_csv(source_file_dir + 'FigureS10_siteUnitScaling.csv', index=False)
  1306. unit_scaling = pd.DataFrame(unit_scaling)
  1307. if recompute:
  1308. unit_scaling.to_pickle(resultFile)
  1312. return unit_scaling.loc[unit_scaling['siteUnitScaling'].isna() == False, 'base_model'].tolist()
  1314. def Figure6_performance(training_groups, base_models, model_suffix='best', metric='test_d-prime', recompute=False,
  1315. performance_threshold=None, figure_dir=None, source_file_dir=None,
  1316. version='v0', dataset='hvm_modelTest'):
  1317. if figure_dir == None:
  1318. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1319. if source_file_dir == None:
  1320. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1322. df = pd.DataFrame()
  1323. for f in training_groups:
  1324. trained_feature_dir = training_groups[f]['feature_dir'] + f'{dataset}/{model_suffix}/'
  1325. if (recompute == True) or (
  1326. not Path(f'{training_groups[f]["feature_dir"]}/TrainingMetrics.pkl').exists()):
  1327. ### This requires retraining. Wandb meta-files not included.
  1328. tmp = loadTrainingResults(base_models, training_groups[f]['feature_dir'],
  1329. project=training_groups[f]['project'], entity=training_groups[f]['entity'],
  1330. sweep_suffix=training_groups[f]['sweep_suffix'],
  1331. model_suffix=model_suffix, recompute=recompute)
  1332. ###
  1333. else:
  1334. tmp = joblib.load(
  1335. f'{training_groups[f]["feature_dir"]}/TrainingMetrics.pkl')
  1336. tmp['Training'] = f
  1337. df = pd.concat([df, tmp], ignore_index=True)
  1339. df = df.loc[df['base_model'].isin(base_models)]
  1341. if version == 'v0':
  1342. g = sns.FacetGrid(df, col="Training", sharex=True, sharey=False, col_wrap=3, aspect=1.25, height=1.7)
  1344. g.map(sns.histplot, metric, binwidth=0.2,
  1345. clip_on=False,
  1346. fill=True, alpha=0.5, linewidth=0.5, color='white')
  1348. g.figure.subplots_adjust(hspace=-0.5)
  1349. df['selected'] = False
  1350. if 'max_value' not in performance_threshold:
  1351. g.refline(x=performance_threshold['min_value'], color=sns.xkcd_palette(['raspberry'])[0], linestyle='--')
  1353. df.loc[df[metric] >= performance_threshold['min_value'], 'selected'] = True
  1354. else:
  1355. g.map(plt.axvspan, xmin=performance_threshold['min_value'], xmax=performance_threshold['max_value'],
  1356. zorder=0, color=sns.xkcd_palette(['raspberry'])[0],
  1357. alpha=0.2)
  1358. df.loc[(df[metric] >= performance_threshold['min_value']) & (
  1359. df[metric] <= performance_threshold['max_value']), 'selected'] = True
  1361. g.set_titles("")
  1363. g.set(xlabel="Performance (d')", ylabel="# models")
  1365. g.savefig(figure_dir + 'FigureS12_performance.png', dpi=300)
  1366. g.savefig(figure_dir + 'FigureS12_performance.pdf', dpi=300, transparent=True)
  1368. print(df.groupby('Training')['selected'].sum())
  1370. df.to_csv(source_file_dir + 'FigureS12_performance.csv', index=False)
  1373. elif version == 'v1':
  1375. g = sns.displot(data=df, x=metric,
  1376. color='white', height=3.75 * cm, aspect=1.25, clip_on=False, fill=True, alpha=0.5,
  1377. linewidth=0.5,
  1378. facet_kws={'sharex': False, 'sharey': False}, common_bins=False, bins=30)
  1380. g.figure.subplots_adjust(hspace=-0.5)
  1381. df['selected'] = False
  1382. if 'max_value' not in performance_threshold:
  1383. g.refline(x=performance_threshold['min_value'], color=sns.xkcd_palette(['raspberry'])[0], linestyle='--')
  1385. df.loc[df[metric] >= performance_threshold['min_value'], 'selected'] = True
  1386. else:
  1387. g.map(plt.axvspan, xmin=performance_threshold['min_value'], xmax=performance_threshold['max_value'],
  1388. zorder=0, color=sns.xkcd_palette(['raspberry'])[0], alpha=0.2,
  1389. )
  1390. df.loc[(df[metric] >= performance_threshold['min_value']) & (
  1391. df[metric] <= performance_threshold['max_value']), 'selected'] = True
  1393. g.set_titles("")
  1395. g.set(xlabel="Performance (d')", ylabel="# models")
  1396. g.savefig(figure_dir + 'Figure6_performance.png', dpi=300)
  1397. g.savefig(figure_dir + 'Figure6_performance.pdf', dpi=300, transparent=True)
  1399. print(df['selected'].sum())
  1400. df.to_csv(source_file_dir + 'Figure6_performance.csv', index=False)
  1402. return df
  1405. def Figure6_modelITComparison_means(base_models, training_groups, performance_threshold, model_draws=100,
  1406. num_sites=159, draws=1000, mode='empirical_conditioned_v2', recompute=False,
  1407. delta='proportion',
  1408. figure_dir=None,
  1409. meta_dir=None,
  1410. source_file_dir=None,
  1411. version='v0'
  1412. ):
  1413. if figure_dir == None:
  1414. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1416. if meta_dir == None:
  1417. meta_dir = f'{Path(__file__).parent.parent}/Results/meta/'
  1419. if source_file_dir == None:
  1420. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1422. df_selectivity = load_ModelSelectivity_allModels(base_models, training_groups, performance_threshold, delta=delta,
  1423. model_draws=model_draws, recompute=recompute)
  1424. df_rsa = load_ModelRDMs_allModels(base_models, training_groups, performance_threshold, delta=delta,
  1425. model_draws=model_draws, recompute=recompute)
  1426. df_decoding = load_ModelIT_categoryDecodes_allModels(base_models, training_groups, performance_threshold,
  1427. delta=delta,
  1428. model_draws=model_draws, recompute=recompute)
  1430. # Already computed in the neural analyses
  1431. neural_selectivity = load_neuralITselectivity_deltas(recompute=False, draws=draws, mode=mode, delta=delta)
  1432. neural_rsa = load_neuralRDMs_deltas(recompute=False, draws=draws, mode=mode, num_sites=num_sites, delta=delta)
  1433. neural_decoding = load_MonkeyIT_Decodes_deltas(num_sites=num_sites, includeReliability=False, recompute=False,
  1434. mode=mode, draws=draws, delta=delta)
  1436. df_combined = pd.DataFrame()
  1437. df_combined['delta'] = df_selectivity.groupby('model_ID')['delta (selectivity)'].median()
  1438. df_combined['metric'] = 'selectivity'
  1439. df_combined = df_combined.reset_index()
  1441. df_tmp = pd.DataFrame()
  1442. df_tmp['delta'] = df_rsa.groupby('model_ID')['delta (tau)'].mean()
  1443. df_tmp['metric'] = 'rsa'
  1444. df_tmp = df_tmp.reset_index()
  1445. df_combined = pd.concat([df_combined, df_tmp])
  1447. df_tmp = pd.DataFrame()
  1448. df_tmp['delta'] = df_decoding.groupby('model_ID')['delta (i1)'].mean()
  1449. df_tmp['metric'] = 'decoding'
  1450. df_tmp = df_tmp.reset_index()
  1451. df_combined = pd.concat([df_combined, df_tmp])
  1453. # Define subsets
  1454. set_decoding = set(df_combined.loc[(df_combined.loc[
  1455. df_combined['metric'] == 'decoding', 'delta'] >= np.quantile(
  1456. eval(f'neural_decoding'), 0.025)) & (
  1457. df_combined.loc[
  1458. df_combined['metric'] == 'decoding', 'delta'] <= np.quantile(
  1459. eval(f'neural_decoding'), 0.975)), 'model_ID'])
  1460. set_rsa = set(df_combined.loc[(df_combined.loc[df_combined['metric'] == 'rsa', 'delta'] >= np.quantile(
  1461. eval(f'neural_rsa'), 0.025)) & (
  1462. df_combined.loc[df_combined['metric'] == 'rsa', 'delta'] <= np.quantile(
  1463. eval(f'neural_rsa'), 0.975)), 'model_ID'])
  1464. set_selectivity = set(df_combined.loc[(df_combined.loc[
  1465. df_combined['metric'] == 'selectivity', 'delta'] >= np.quantile(
  1466. eval(f'neural_selectivity'), 0.025)) & (
  1467. df_combined.loc[df_combined[
  1468. 'metric'] == 'selectivity', 'delta'] <= np.quantile(
  1469. eval(f'neural_selectivity'), 0.975)), 'model_ID'])
  1471. models_pt3 = pd.DataFrame({'model_ID': df_combined.model_ID.unique()})
  1473. models_pt3['match_selectivity'] = models_pt3['model_ID'].isin(set_selectivity)
  1474. models_pt3['match_rsa'] = models_pt3['model_ID'].isin(set_rsa)
  1475. models_pt3['match_decoding'] = models_pt3['model_ID'].isin(set_decoding)
  1476. models_pt3['match_all'] = models_pt3['match_selectivity'] & models_pt3['match_rsa'] & models_pt3['match_decoding']
  1477. models_pt3['match_none'] = (models_pt3['match_selectivity'] == False) & (models_pt3['match_rsa'] == False) & (
  1478. models_pt3['match_decoding'] == False)
  1479. models_pt3['match_one'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(int) +
  1480. models_pt3['match_rsa'].astype(int)) == 1
  1481. models_pt3['match_two'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(
  1482. int) + models_pt3['match_rsa'].astype(int)) == 2
  1484. for feature in ['base_model', 'Training', 'base_model_type']:
  1485. models_pt3[feature] = models_pt3['model_ID'].map(dict(zip(df_decoding['model_ID'], df_decoding[feature])))
  1487. if recompute:
  1488. models_pt3.to_pickle(meta_dir + f'SelectedModels_pt3_means.pkl')
  1490. plotting_titles = {'rsa': 'Representational\nstrength',
  1491. 'selectivity': 'Selectivity',
  1492. 'decoding': 'Linear decodability'}
  1493. if version == 'v0':
  1494. g = sns.displot(data=df_combined, x='delta', row='metric',
  1495. color=sns.xkcd_palette(['raspberry'])[0], height=3.75 * cm, aspect=1.25, clip_on=False,
  1496. fill=True, alpha=0.2, linewidth=0.5,
  1497. facet_kws={'sharex': False, 'sharey': False}, common_bins=False, bins=30)
  1499. for title, ax in g.axes_dict.items():
  1500. print(title)
  1502. ax.axvspan(np.quantile(eval(f'neural_{title}'), 0.025),
  1503. np.quantile(eval(f'neural_{title}'), 0.975),
  1504. color=sns.xkcd_palette(['royal blue'])[0], zorder=0, alpha=0.2)
  1506. ax.set_title(plotting_titles[title], fontweight='bold', fontsize=10)
  1508. print(
  1509. f"# models falling below neural CI: {(df_combined.loc[df_combined['metric'] == title, 'delta'] < np.quantile(eval(f'neural_{title}'), 0.025)).sum()}")
  1511. print(
  1512. f"# models falling above neural CI: {(df_combined.loc[df_combined['metric'] == title, 'delta'] > np.quantile(eval(f'neural_{title}'), 0.975)).sum()}")
  1514. print(
  1515. f"# models within neural CI: {((df_combined.loc[df_combined['metric'] == title, 'delta'] >= np.quantile(eval(f'neural_{title}'), 0.025)) & (df_combined.loc[df_combined['metric'] == title, 'delta'] <= np.quantile(eval(f'neural_{title}'), 0.975))).sum()}")
  1517. g.set_ylabels('# models')
  1518. g.set_xlabels('Δ%')
  1520. g.savefig(figure_dir + f'Figure6_modelITComparison_means_v0.png', dpi=300)
  1521. g.savefig(figure_dir + f'Figure6_modelITComparison_means_v0.pdf', dpi=300, transparent=True)
  1523. df_combined.to_csv(source_file_dir + 'Figure6_modelITComparison_means_v0.csv', index=False)
  1525. elif version == 'v1':
  1526. # make a venn diagram
  1528. fig, ax = plt.subplots()
  1530. venn3([set_selectivity, set_decoding, set_rsa], ('Selectivity', 'Decoding', 'RSA'),
  1531. ax=ax)
  1533. fig.savefig(figure_dir + f'Figure6_modelITComparison_means_v1.png', dpi=300)
  1534. fig.savefig(figure_dir + f'Figure6_modelITComparison_means_v1.pdf', dpi=300, transparent=True)
  1536. elif version == 'v2':
  1537. df_plot = models_pt3.groupby('Training', as_index=False)['match_all'].mean()
  1538. df_plot['match_all_%'] = df_plot['match_all'] * 100
  1539. df_plot['dummy'] = 100
  1541. g = sns.catplot(data=df_plot, x='Training', y='match_all_%',
  1542. kind='bar', color=sns.xkcd_palette(['royal blue'])[0],
  1543. height=3.75 * cm, aspect=0.9, order=['FT', '2stepFT', 'binary'],
  1544. edgecolor=".5", linewidth=0.5,
  1545. )
  1546. g.map(sns.barplot, 'Training', 'dummy', order=['FT', '2stepFT', 'binary'], zorder=0, color='white',
  1547. linewidth=0.5, edgecolor=".5")
  1548. g.set_xticklabels(['Standard', 'Step-wise', 'Binary choice'], rotation=90)
  1549. g.set(ylim=[0, 100])
  1550. g.set_ylabels('% models')
  1551. g.savefig(figure_dir + f'FigureS12_modelITComparison_means_Training.png')
  1552. g.savefig(figure_dir + f'FigureS12_modelITComparison_means_Training.pdf', dpi=300, transparent=True)
  1554. df_plot.to_csv(source_file_dir + 'FigureS12_modelITComparison_means_Training.csv', index=False)
  1556. df_plot = models_pt3.groupby('base_model', as_index=False)['match_all'].mean()
  1557. df_plot['match_all_%'] = df_plot['match_all'] * 100
  1558. df_plot['dummy'] = 100
  1560. model_order = ['resnet18_v1', 'resnet34_v1', 'resnet50_v1', 'resnet101_v1', 'resnet152_v1',
  1561. 'alexnet', 'vgg16', 'vgg19', ' ',
  1562. 'resnet50_MoCov2_200epochs', 'resnet50_simclr_100epochs', 'resnet50_barlowTwins_300epochs',
  1563. ]
  1564. model_titles = ['18', '34', '50', '101', '152',
  1565. 'alexnet', 'vgg16', 'vgg19', ' ',
  1566. 'MoCov2', 'SimCLR', 'barlowTwins']
  1567. g = sns.catplot(data=df_plot, x='base_model', y='match_all_%',
  1568. kind='bar', color=sns.xkcd_palette(['royal blue'])[0],
  1569. height=3.75 * cm, aspect=2.4,
  1570. order=model_order,
  1571. edgecolor=".5", linewidth=0.5,
  1572. )
  1573. g.map(sns.barplot, 'base_model', 'dummy', order=model_order,
  1574. zorder=0, color='white',
  1575. linewidth=0.5, edgecolor=".5")
  1576. g.set_xticklabels(model_titles, rotation=90)
  1577. g.set(ylim=[0, 100])
  1578. g.set_ylabels('% models')
  1579. g.savefig(figure_dir + f'FigureS12_modelITComparison_means_baseModel.png')
  1580. g.savefig(figure_dir + f'FigureS12_modelITComparison_means_baseModel.pdf', dpi=300, transparent=True)
  1582. df_plot.to_csv(source_file_dir + 'FigureS12_modelITComparison_means_baseModel.csv', index=False)
  1585. def FigureS11_modelITComparison_controlLayers(layers, base_models, training_groups, performance_threshold, model_draws=100,
  1586. num_sites=159, draws=1000, mode='empirical_conditioned_v2', recompute=False,
  1587. delta='proportion', figure_dir=None, source_file_dir=None,):
  1589. if figure_dir == None:
  1590. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1591. if source_file_dir == None:
  1592. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1594. df_combined_all = []
  1595. for layer in layers:
  1596. print(layer)
  1597. area = f'Layer-{layer}'
  1599. df_selectivity = load_ModelSelectivity_allModels(base_models, training_groups, performance_threshold,
  1600. delta=delta, area=area, model_draws=model_draws,
  1601. recompute=recompute)
  1603. df_rsa = load_ModelRDMs_allModels(base_models, training_groups, performance_threshold, delta=delta, area=area,
  1604. model_draws=model_draws, recompute=recompute)
  1606. df_decoding = load_ModelIT_categoryDecodes_allModels(base_models, training_groups, performance_threshold,
  1607. delta=delta, area=area, model_draws=model_draws,
  1608. recompute=recompute)
  1610. df_combined = pd.DataFrame()
  1611. df_combined['delta'] = df_selectivity.groupby('model_ID')['delta (selectivity)'].median()
  1612. df_combined['metric'] = 'selectivity'
  1613. df_combined = df_combined.reset_index()
  1615. df_tmp = pd.DataFrame()
  1616. df_tmp['delta'] = df_rsa.groupby('model_ID')['delta (tau)'].mean()
  1617. df_tmp['metric'] = 'rsa'
  1618. df_tmp = df_tmp.reset_index()
  1619. df_combined = pd.concat([df_combined, df_tmp])
  1621. df_tmp = pd.DataFrame()
  1622. df_tmp['delta'] = df_decoding.groupby('model_ID')['delta (i1)'].mean()
  1623. df_tmp['metric'] = 'decoding'
  1624. df_tmp = df_tmp.reset_index()
  1625. df_combined = pd.concat([df_combined, df_tmp])
  1626. df_combined['area'] = area
  1627. df_combined['layer'] = layer
  1628. df_combined_all.append(df_combined)
  1630. df_combined_all = pd.concat(df_combined_all, ignore_index=True)
  1632. neural_selectivity = load_neuralITselectivity_deltas(recompute=False, draws=draws, mode=mode, delta=delta)
  1633. neural_decoding = load_MonkeyIT_Decodes_deltas(num_sites=num_sites, includeReliability=False, recompute=False,
  1634. mode=mode, draws=draws, delta=delta)
  1635. neural_rsa = load_neuralRDMs_deltas(recompute=False, draws=draws, mode=mode, num_sites=num_sites, delta=delta)
  1637. plotting_titles = {'rsa': 'Representational\nstrength',
  1638. 'selectivity': 'Selectivity',
  1639. 'decoding': 'Linear decodability'}
  1641. #model_IDs = df_combined_all.loc[df_combined_all['area'] == 'EarlyLayerControl', 'model_ID'].unique()
  1642. df_combined_plotting = df_combined_all.loc[
  1643. df_combined_all['layer'].isin(['layer2[0].relu', 'layer4[0].relu', 'avgpool'])]
  1644. df_combined_plotting.loc[df_combined_plotting['layer'] == 'layer4[0].relu', 'area'] = 'IT-mapped layer'
  1645. df_combined_plotting.loc[df_combined_plotting['layer'] == 'layer2[0].relu', 'area'] = 'Early control layer'
  1646. df_combined_plotting.loc[df_combined_plotting['layer'] == 'avgpool', 'area'] = 'Late control layer'
  1648. g = sns.displot(data=df_combined_plotting, x='delta', col='metric',
  1649. hue='area', height=5 * cm, aspect=1, clip_on=False, fill=True, alpha=0.2, linewidth=0.5,
  1650. facet_kws={'sharex': False, 'sharey': False}, common_bins=False, kde=True,
  1651. palette=sns.xkcd_palette(['tangerine', 'raspberry', 'pastel purple']))
  1653. for title, ax in g.axes_dict.items():
  1654. print(title)
  1655. ax.axvspan(np.quantile(eval(f'neural_{title}'), 0.025),
  1656. np.quantile(eval(f'neural_{title}'), 0.975),
  1657. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=0.3)
  1658. ax.set_title(plotting_titles[title])
  1659. g.set_ylabels('# models')
  1660. g.set_xlabels('Δ%')
  1661. #plt.show()
  1662. g.savefig(figure_dir + f'FigureS11_modelITComparison_controlLayers.png', dpi=300)
  1663. g.savefig(figure_dir + f'FigureS11_modelITComparison_controlLayers.pdf', dpi=300, transparent=True)
  1665. df_combined_plotting.to_csv(source_file_dir + 'FigureS11_modelITComparison_controlLayers.csv', index=False)
  1667. g = sns.catplot(data=df_combined_all, x='layer', y='delta', col='metric', kind='point', order=layers,
  1668. color=sns.xkcd_palette(['eggplant'])[0],
  1669. col_order=['selectivity', 'rsa', 'decoding'], errorbar=('pi', 95), height=5.5 * cm, aspect=1.3,
  1670. sharey=False)
  1671. for title, ax in g.axes_dict.items():
  1673. ax.axhspan(np.quantile(eval(f'neural_{title}'), 0.025),
  1674. np.quantile(eval(f'neural_{title}'), 0.975),
  1675. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=0.3)
  1676. ax.set_title(plotting_titles[title])
  1678. g.refline(x=layers.index('layer4[0].relu'), c=sns.xkcd_palette(['raspberry'])[0], ls='-', zorder=0)
  1679. g.refline(x=layers.index('layer2[0].relu'), c=sns.xkcd_palette(['tangerine'])[0], ls='-', zorder=0)
  1680. g.refline(x=layers.index('avgpool'), c=sns.xkcd_palette(['pastel purple'])[0], ls='-', zorder=0)
  1681. g.refline(y=0)
  1682. g.set_xticklabels([])
  1683. g.set_ylabels('Δ%')
  1684. g.set_xlabels('Model layer')
  1686. g.savefig(figure_dir + f'FigureS11_modelITComparison_controlLayers_v1.png', dpi=300)
  1687. g.savefig(figure_dir + f'FigureS11_modelITComparison_controlLayers_v1.pdf', dpi=300, transparent=True)
  1689. df_combined_all.to_csv(
  1690. source_file_dir + 'FigureS11_modelITComparison_controlLayers_v1.csv', index=False)
  1692. def FigureS11_modelITcomparison_controlLayers_orthogonalDecoding(training_groups, layers=None, seed=6,
  1693. recompute=False, metric='corrs', model_draws=100, draws=1000,
  1694. delta='proportion', mode='empirical_conditioned',reestimate_predictions=False,
  1695. figure_dir=None, meta_dir=None, source_file_dir=None):
  1697. if figure_dir == None:
  1698. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1700. if meta_dir == None:
  1701. meta_dir = f'{Path(__file__).parent.parent}/Results/meta/'
  1703. if source_file_dir == None:
  1704. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1706. if layers == None:
  1707. layers = ['layer2[0].relu', 'layer4[0].relu', 'avgpool']
  1709. np.random.seed(seed)
  1710. models_pt3 = pd.read_pickle(meta_dir + f'SelectedModels_pt3_means.pkl')
  1711. models_pt3['match_one'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(int) + models_pt3['match_rsa'].astype(int)) == 1
  1712. models_pt3['match_two'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(
  1713. int) + models_pt3['match_rsa'].astype(int)) == 2
  1714. # Only include IT-like models and vary the layer...
  1715. models_pt3 = models_pt3.loc[(models_pt3['base_model'] == 'resnet152_v1') & (models_pt3['match_all'])]
  1716. df = []
  1717. for layer in layers:
  1718. print(layer)
  1719. area = f'Layer-{layer}'
  1721. df.append(load_ModelLayer_orthogonalDecodes_selection(models_pt3, training_groups,area=area, draws=model_draws, recompute=recompute,
  1722. reestimate_predictions=reestimate_predictions))
  1723. df[-1]['layer'] = layer
  1725. df = pd.concat(df, ignore_index=True)
  1726. df.loc[df['layer'] == 'layer4[0].relu', 'area'] = 'IT-mapped layer'
  1727. df.loc[df['layer'] == 'layer2[0].relu', 'area'] = 'Early control layer'
  1728. df.loc[df['layer'] == 'avgpool', 'area'] = 'Late control layer'
  1730. neural_df = load_neuralITorthogonalDecodes(recompute=recompute, draws=draws, mode=mode, metric=metric)
  1731. neural_df = neural_df.reset_index()
  1732. neural_df['delta'] = computeDelta(neural_df['corrs', 'untrained'], neural_df['corrs', 'trained'], kind=delta)
  1735. neural_df_plot = neural_df.copy()
  1736. neural_df_plot.columns = ["_".join(a) for a in neural_df_plot.columns.to_flat_index()]
  1737. neural_df_plot = neural_df_plot.melt(value_vars=['corrs_trained', 'corrs_untrained'], id_vars=['draws_', 'targetFeature_'])
  1739. df['delta'] = computeDelta(df['corrs_pre'], df['corrs_post'])
  1741. df['# matched IT-category metrics'] = None
  1743. df.loc[df['model_ID'].isin(models_pt3.loc[models_pt3['match_all'], 'model_ID']), '# matched IT-category metrics'] = '3/3'
  1744. df.loc[df['model_ID'].isin(models_pt3.loc[models_pt3['match_one'], 'model_ID']), '# matched IT-category metrics'] = '1/3'
  1745. df.loc[df['model_ID'].isin(models_pt3.loc[models_pt3['match_two'], 'model_ID']), '# matched IT-category metrics'] = '2/3'
  1746. df.loc[df['model_ID'].isin(models_pt3.loc[models_pt3['match_none'], 'model_ID']), '# matched IT-category metrics'] = '0/3'
  1748. df_means = df.groupby(['model_ID', 'targetFeature', 'base_model', 'base_model_type', '# matched IT-category metrics', 'area'], as_index=False)['delta'].mean()
  1749. df_means['targetFeature'] = df_means['targetFeature'].str.split('_', expand=True)[2]
  1751. col_order = ['xpos', 'ypos', 'objSize', 'ecc', 'ryz', 'rxz', 'rxy']
  1752. title_mapping = {'xpos': 'Vertical position',
  1753. 'ypos': 'Horizontal position',
  1754. 'objSize': 'Object size',
  1755. 'ecc': 'Eccentricity',
  1756. 'rxz': 'Rotation (ry)',
  1757. 'ryz': 'Rotation (rx)',
  1758. 'rxy': 'Rotation (rz)'}
  1760. colors = sns.xkcd_palette(['tangerine', 'raspberry', 'pastel purple'])
  1762. g = sns.catplot(df_means, col='targetFeature', height=4.5 * cm, aspect=0.8,
  1763. y='delta', x='area', kind='strip', order=['Early control layer', 'IT-mapped layer', 'Late control layer'],
  1764. palette=colors,
  1765. sharey=False, sharex=False, col_order=col_order, alpha=0.5, zorder=1)
  1767. g.refline(y=0)
  1768. for title, ax in g.axes_dict.items():
  1770. ax.axhspan(np.quantile(neural_df.loc[neural_df['targetFeature'] == title, 'delta'], 0.025),
  1771. np.quantile(neural_df.loc[neural_df['targetFeature'] == title, 'delta'], 0.975),
  1772. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=.4)
  1773. ax.set_title(title_mapping[title], fontsize=10)
  1776. g.set_ylabels('%Δ performance')
  1777. g.set_xlabels('Layer')
  1778. g.set_xticklabels('')
  1779. plt.tight_layout()
  1780. #plt.show()
  1781. g.savefig(figure_dir + f'FigureS11_orthogonalDecodes_change_controlLayers.pdf', dpi=300, transparent=True)
  1782. g.savefig(figure_dir + f'FigureS11_orthogonalDecodes_change_controlLayers.png', dpi=300)
  1784. df_means.to_csv(
  1785. source_file_dir + 'FigureS11_orthogonalDecodes_change_controlLayers.csv', index=False)
  1788. def Figure7_LFI_models(training_groups, recompute=False, meta_dir = None, figure_dir= None, source_file_dir=None, max_dim = 250):
  1790. if figure_dir == None:
  1791. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1793. if meta_dir == None:
  1794. meta_dir = f'{Path(__file__).parent.parent}/Results/meta/'
  1796. if source_file_dir == None:
  1797. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1799. models_pt3 = pd.read_pickle(meta_dir + f'SelectedModels_pt3_means.pkl')
  1800. models_pt3['# metrics matched'] = models_pt3['match_selectivity'].astype(int) + models_pt3['match_rsa'].astype(
  1801. int) + models_pt3['match_decoding'].astype(int)
  1803. df_model = get_model_change_metrics(models_pt3, training_groups, recompute=recompute)
  1804. categories = np.arange(8)
  1806. plot_df = []
  1807. pca_df = []
  1809. for model_ID in df_model:
  1810. for c in categories:
  1812. plot_df.append({'model_ID': model_ID,
  1813. '# metrics matched': models_pt3.loc[models_pt3['model_ID'] == model_ID, '# metrics matched'].item(),
  1814. 'base_model': df_model[model_ID]['base_model'],
  1815. 'training': df_model[model_ID]['Training'],
  1816. 'category': c,
  1817. 'state': 'naive',
  1818. 'N_units': int(df_model[model_ID][c]['N_units']),
  1819. 'Signal strength': df_model[model_ID][c]['df_abs_pre'],
  1820. 'Response variance': df_model[model_ID][c]['variance_median_pre'],
  1821. 'Noise correlations': df_model[model_ID][c]['NoiseCorrelation_median_pre'],
  1822. 'Fano Factor': df_model[model_ID][c]['FanoFactor_median_pre'],
  1823. 'aLFI': float(df_model[model_ID][c]['LFI_pre']),
  1825. })
  1827. pca_df.append(pd.DataFrame({'model_ID': model_ID,
  1828. '# metrics matched': models_pt3.loc[
  1829. models_pt3['model_ID'] == model_ID, '# metrics matched'].item(),
  1830. 'base_model': df_model[model_ID]['base_model'],
  1831. 'training': df_model[model_ID]['Training'],
  1832. 'category': c,
  1833. 'N_units': int(df_model[model_ID][c]['N_units']),
  1834. 'N_components': np.arange(len(df_model[model_ID][c]['PCA_rotation']))[:max_dim],
  1835. 'PCA rotation': np.array(df_model[model_ID][c]['PCA_rotation'])[:max_dim],
  1836. 'PC absolute (naive)': df_model[model_ID][c]['PC_abs_pre'][:max_dim],
  1837. 'PC relative (naive)': df_model[model_ID][c]['PC_abs_pre'][:max_dim]/ np.array(df_model[model_ID][c]['PC_abs_pre']).sum(),
  1838. 'PC absolute (trained)': df_model[model_ID][c]['PC_abs_post'][:max_dim],
  1839. 'PC relative (trained)': df_model[model_ID][c]['PC_abs_post'][:max_dim] / np.array(
  1840. df_model[model_ID][c]['PC_abs_post'][:max_dim]).sum(),
  1842. 'PC ratio': (df_model[model_ID][c]['PC_abs_post'][:max_dim] / np.array(
  1843. df_model[model_ID][c]['PC_abs_post'][:max_dim]).sum())/(df_model[model_ID][c]['PC_abs_pre'][:max_dim]/ np.array(df_model[model_ID][c]['PC_abs_pre'][:max_dim]).sum())
  1844. }))
  1846. plot_df.append({'model_ID': model_ID,
  1847. '# metrics matched': models_pt3.loc[
  1848. models_pt3['model_ID'] == model_ID, '# metrics matched'].item(),
  1849. 'base_model': df_model[model_ID]['base_model'],
  1850. 'training': df_model[model_ID]['Training'],
  1851. 'category': c,
  1852. 'state': 'trained',
  1853. 'N_units': int(df_model[model_ID][c]['N_units']),
  1854. 'Signal strength': df_model[model_ID][c]['df_abs_post'],
  1855. 'Response variance': df_model[model_ID][c]['variance_median_post'],
  1856. 'Noise correlations': df_model[model_ID][c]['NoiseCorrelation_median_post'],
  1857. 'Fano Factor': df_model[model_ID][c]['FanoFactor_median_post'],
  1858. 'aLFI': float(df_model[model_ID][c]['LFI_post']),
  1859. 'rotation_pre_post': float(df_model[model_ID][c]['rotation_pre_post']),
  1860. 'gain_LFI': np.log10(df_model[model_ID][c]['LFI_post']) - np.log10(df_model[model_ID][c]['LFI_pre']),
  1861. 'ratio_Signal strength': float(df_model[model_ID][c]['df_abs_post']) / float(
  1862. df_model[model_ID][c]['df_abs_pre']),
  1863. 'ratio_Response variance': float(df_model[model_ID][c]['variance_median_post']) / float(
  1864. df_model[model_ID][c]['variance_median_pre']),
  1865. 'ratio_FanoFactor': df_model[model_ID][c]['FanoFactor_median_post'] / df_model[model_ID][c]['FanoFactor_median_pre'],
  1866. 'ratio_NC': df_model[model_ID][c]['NoiseCorrelation_median_post'] / df_model[model_ID][c][
  1867. 'NoiseCorrelation_median_pre']
  1869. })
  1871. plot_df = pd.DataFrame(plot_df)
  1872. pca_df = pd.concat(pca_df, ignore_index=True)
  1874. plot_df_mean = plot_df.groupby(['model_ID', 'state', 'base_model', 'training'], as_index=False).mean()
  1875. palette = sns.color_palette(['#e9d2de', '#c0bcde', '#5369B0', '#4a3f99'])
  1876. # Plot signal separation
  1877. g = sns.catplot(data=plot_df_mean, x='# metrics matched', y='ratio_Signal strength',
  1878. aspect=0.9, height=6 * cm, kind='point', legend=False,
  1879. hue='# metrics matched', palette=palette)
  1880. g.set_xticklabels([0, 1, 2, 3])
  1881. g.set_ylabels('Signal strength ratio\n(trained/naive)')
  1882. g.refline(y=1)
  1883. g.savefig(figure_dir + f'Figure7_InfoMetrics_Signal.png', dpi=300)
  1884. g.savefig(figure_dir + f'Figure7_InfoMetrics_Signal.pdf', dpi=300, transparent=True)
  1886. plot_df_mean.to_csv(
  1887. source_file_dir + 'Figure7_InfoMetrics.csv', index=False)
  1889. g = sns.catplot(data=plot_df_mean, x='# metrics matched', y='ratio_Response variance',
  1890. aspect=0.9, height=6 * cm, kind='point', legend=False,
  1891. hue='# metrics matched', palette=palette)
  1892. g.set_xticklabels([0, 1, 2, 3])
  1893. g.set_ylabels('Response variance\nratio (trained/naive)')
  1894. g.refline(y=1)
  1895. g.savefig(figure_dir + f'Figure7_InfoMetrics_Var.png', dpi=300)
  1896. g.savefig(figure_dir + f'Figure7_InfoMetrics_Var.pdf', dpi=300, transparent=True)
  1898. g = sns.catplot(data=plot_df_mean, x='# metrics matched', y='rotation_pre_post',
  1899. aspect=0.9, height=6 * cm, kind='point', legend=False,
  1900. hue='# metrics matched', palette=palette)
  1901. g.set_xticklabels([0, 1, 2, 3])
  1902. g.refline(y=0)
  1903. g.set_ylabels('Signal rotation angle\n(trained/naive)')
  1904. g.savefig(figure_dir + f'Figure7_InfoMetrics_Rot.png', dpi=300)
  1905. g.savefig(figure_dir + f'Figure7_InfoMetrics_Rot.pdf', dpi=300, transparent=True)
  1907. g = sns.relplot(data=pca_df, x='N_components', y='PCA rotation', kind='line',
  1908. hue='# metrics matched', palette=palette, height=6 * cm, aspect=0.9, legend=False)
  1909. g.set(xlim=[0, 25])
  1910. g.set_xlabels('Principal components')
  1911. g.set_ylabels('Covariance rotation\nangle (trained/naive)')
  1912. g.savefig(figure_dir + f'Figure7_InfoMetrics_PCsRot.png', dpi=300)
  1913. g.savefig(figure_dir + f'Figure7_InfoMetrics_PCsRot.pdf', dpi=300, transparent=True)
  1915. plot_df_mean.to_csv(
  1916. source_file_dir + 'Figure7_InfoMetrics_PCsRot.csv', index=False)
  1919. def Figure8_orthogonalDecoding_means(training_groups,
  1920. recompute=False, metric='corrs', model_draws=100, draws=1000,
  1921. delta='proportion', mode='empirical_conditioned_v2',
  1922. reestimate_predictions=False,
  1923. figure_dir=None,
  1924. meta_dir=None,
  1925. source_file_dir=None,
  1926. version='v0'):
  1927. if figure_dir == None:
  1928. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  1930. if meta_dir == None:
  1931. meta_dir = f'{Path(__file__).parent.parent}/Results/meta/'
  1933. if source_file_dir == None:
  1934. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  1936. models_pt3 = pd.read_pickle(meta_dir + f'SelectedModels_pt3_means.pkl')
  1937. models_pt3['match_one'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(int) +
  1938. models_pt3['match_rsa'].astype(int)) == 1
  1939. models_pt3['match_two'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(
  1940. int) + models_pt3['match_rsa'].astype(int)) == 2
  1942. df = load_ModelITorthogonalDecodes_selection(models_pt3, training_groups, draws=model_draws, recompute=recompute,
  1943. reestimate_predictions=reestimate_predictions)
  1945. neural_df = load_neuralITorthogonalDecodes(recompute=recompute, draws=draws, mode=mode, metric=metric)
  1946. neural_df = neural_df.reset_index()
  1947. neural_df['delta'] = computeDelta(neural_df['corrs', 'untrained'], neural_df['corrs', 'trained'], kind=delta)
  1949. if version == 'v0':
  1950. print('Medians:')
  1951. print(neural_df.groupby(['targetFeature']).median())
  1952. for target in neural_df['targetFeature'].unique():
  1953. print(target)
  1954. print(
  1955. f"{((neural_df.loc[neural_df['targetFeature'] == target, 'corrs']['trained'].mean() / neural_df.loc[neural_df['targetFeature'] == target, 'corrs']['untrained'].mean()) - 1) * 100}% increase")
  1957. difference_dist = neural_df.loc[neural_df['targetFeature'] == target, 'corrs']['trained'] - \
  1958. neural_df.loc[neural_df['targetFeature'] == target, 'corrs']['untrained']
  1960. lower_CI = difference_dist.quantile(0.025)
  1961. upper_CI = difference_dist.quantile(0.975)
  1962. p_val = np.min([(difference_dist < 0).mean(), (difference_dist > 0).mean()]) * 2
  1963. print(f'mean diff {difference_dist.mean()}')
  1964. print(f'mean SD {difference_dist.std()}')
  1966. print(
  1967. f"CI (95%): {lower_CI} - {upper_CI}")
  1968. print(f"p-val: {p_val}")
  1970. neural_df_plot = neural_df.copy()
  1971. neural_df_plot.columns = ["_".join(a) for a in neural_df_plot.columns.to_flat_index()]
  1972. neural_df_plot = neural_df_plot.melt(value_vars=['corrs_trained', 'corrs_untrained'],
  1973. id_vars=['draws_', 'targetFeature_'])
  1975. g = sns.catplot(data=neural_df_plot, x='targetFeature_', y='value', hue='variable', kind='point',
  1976. errorbar=("pi", 95),
  1977. palette={'corrs_untrained': sns.xkcd_palette(['slate grey'])[0],
  1978. 'corrs_trained': sns.xkcd_palette(['jade'])[0]}, height=4.5 * cm, aspect=2,
  1980. legend=False, order=['xpos', 'ypos', 'ecc', 'objSize', 'ryz', 'rxz', 'rxy'], join=False,
  1981. dodge=0.25)
  1983. g.refline(y=0)
  1984. g.set_xticklabels(["x-pos.", "y-pos.", 'ecc.', 'size', 'rx', 'ry', 'rz'])
  1985. g.set_ylabels("Correlation")
  1986. g.set_xlabels("Category-orthogonal property")
  1987. g.savefig(figure_dir + 'FigureS13_orthogonal_neuralDecodes.png', dpi=300)
  1988. g.savefig(figure_dir + 'FigureS13_orthogonal_neuralDecodes.pdf', dpi=300, transparent=True)
  1990. neural_df_plot.to_csv(
  1991. source_file_dir + 'FigureS13_orthogonal_neuralDecodes.csv', index=False)
  1993. elif version == 'v1':
  1994. df['delta'] = computeDelta(df['corrs_pre'], df['corrs_post'])
  1996. df['# matched IT-category metrics'] = None
  1997. df.loc[df['model_ID'].isin(
  1998. models_pt3.loc[models_pt3['match_all'], 'model_ID']), '# matched IT-category metrics'] = '3/3'
  1999. df.loc[df['model_ID'].isin(
  2000. models_pt3.loc[models_pt3['match_one'], 'model_ID']), '# matched IT-category metrics'] = '1/3'
  2001. df.loc[df['model_ID'].isin(
  2002. models_pt3.loc[models_pt3['match_two'], 'model_ID']), '# matched IT-category metrics'] = '2/3'
  2003. df.loc[df['model_ID'].isin(
  2004. models_pt3.loc[models_pt3['match_none'], 'model_ID']), '# matched IT-category metrics'] = '0/3'
  2006. df_means = df.groupby(
  2007. ['model_ID', 'targetFeature', 'base_model', 'base_model_type', '# matched IT-category metrics'],
  2008. as_index=False).mean()
  2009. df_means['targetFeature'] = df_means['targetFeature'].str[14:]
  2011. col_order = ['xpos', 'ypos', 'objSize', 'ecc', 'ryz', 'rxz', 'rxy']
  2012. title_mapping = {'xpos': 'Vertical position',
  2013. 'ypos': 'Horizontal position',
  2014. 'objSize': 'Object size',
  2015. 'ecc': 'Eccentricity',
  2016. 'rxz': 'Rotation (ry)',
  2017. 'ryz': 'Rotation (rx)',
  2018. 'rxy': 'Rotation (rz)'}
  2020. g = sns.catplot(df_means, col='targetFeature', col_wrap=4, height=4 * cm, aspect=1,
  2021. y='delta', x='# matched IT-category metrics', kind='strip', order=['0/3', '1/3', '2/3', '3/3'],
  2022. color=sns.xkcd_palette(['cerulean'])[0],
  2023. sharey=False, sharex=False, col_order=col_order, alpha=0.1, zorder=1)
  2025. g.refline(y=0)
  2026. for title, ax in g.axes_dict.items():
  2027. ax.axhspan(np.quantile(neural_df.loc[neural_df['targetFeature'] == title, 'delta'], 0.025),
  2028. np.quantile(neural_df.loc[neural_df['targetFeature'] == title, 'delta'], 0.975),
  2029. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=.5)
  2030. ax.set_title(title_mapping[title], fontsize=10)
  2032. sns.pointplot(
  2033. data=df_means.loc[df_means['targetFeature'] == title], x='# matched IT-category metrics', y='delta',
  2034. join=False,
  2035. order=['0/3', '1/3', '2/3', '3/3'],
  2036. errorbar=None, estimator=np.median, color=sns.xkcd_palette(['cerulean'])[0],
  2037. markers="_", scale=2, ax=ax,
  2038. )
  2039. g.set_ylabels('%Δ accuracy')
  2040. g.set_xlabels('# IT-aligned\ncategory metrics')
  2041. g.savefig(figure_dir + f'Figure8_orthogonalDecodes.pdf', dpi=300, transparent=True)
  2042. g.savefig(figure_dir + f'Figure8_orthogonalDecodes.png', dpi=300)
  2044. df_means.to_csv(
  2045. source_file_dir + 'Figure8_orthogonalDecodes.csv', index=False)
  2047. k = df_means['# matched IT-category metrics'].nunique() # four model groups
  2048. n = df_means['model_ID'].nunique()
  2049. for target in col_order:
  2050. print(target)
  2051. stats = kruskal(data=df_means.loc[df_means['targetFeature'] == target], dv='delta',
  2052. between='# matched IT-category metrics')
  2053. print(stats)
  2054. epsilon_sq = (stats['H'].item() - k + 1) / (n - k)
  2056. print('Effect size (epsilon_sq):', epsilon_sq)
  2059. def Figure8_performanceStripes(figure_dir=None,
  2060. recompute=False, num_sites=159,
  2061. ):
  2062. if figure_dir == None:
  2063. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  2065. IT_neurons = load_MonkeyIT_Decodes(num_neurons=np.array([num_sites]), includeReliability=False, recompute=recompute)
  2067. IT_image_trained = np.mean(IT_neurons[f'{num_sites}_neurons']['trained']['dprimes'], axis=0)
  2069. g = sns.FacetGrid(data=pd.DataFrame(np.zeros((2, 1))), height=10 * cm, aspect=0.3)
  2070. sns.heatmap(IT_image_trained[:, None], cmap='Spectral', ax=g.ax, cbar=False,
  2071. vmin=np.percentile(IT_image_trained, 5), vmax=np.percentile(IT_image_trained, 95))
  2073. g.ax.set(xlabel="", ylabel="", yticks=np.arange(1, 8) * 80, xticks=[])
  2075. plt.savefig(figure_dir + 'Figure6_performanceStripes_ITtrained.png', dpi=300)
  2076. plt.savefig(figure_dir + 'Figure6_performanceStripes_ITtrained.pdf', dpi=300, transparent=True)
  2078. IT_image_naive = np.mean(IT_neurons[f'{num_sites}_neurons']['untrained']['dprimes'], axis=0)
  2080. g = sns.FacetGrid(data=pd.DataFrame(np.zeros((2, 1))), height=10 * cm, aspect=0.3)
  2081. sns.heatmap(IT_image_naive[:, None], cmap='Spectral', ax=g.ax, cbar=False,
  2082. vmin=np.percentile(IT_image_naive, 5), vmax=np.percentile(IT_image_naive, 95))
  2084. g.ax.set(xlabel="", ylabel="", yticks=np.arange(1, 8) * 80, xticks=[])
  2086. plt.savefig(figure_dir + 'Figure6_performanceStripes_ITnaive.png', dpi=300)
  2087. plt.savefig(figure_dir + 'Figure6_performanceStripes_ITnaive.pdf', dpi=300, transparent=True)
  2089. pooled_behavior = load_MonkeyBehavior_pooled()
  2090. dPrimes, _ = dPrime_monkey(pooled_behavior)
  2092. g = sns.FacetGrid(data=pd.DataFrame(np.zeros((2, 1))), height=10 * cm, aspect=0.3)
  2093. sns.heatmap(dPrimes['i1'][:, None], cmap='Spectral', ax=g.ax, cbar=False,
  2094. vmin=np.percentile(dPrimes['i1'], 5), vmax=np.percentile(dPrimes['i1'], 95))
  2096. g.ax.set(xlabel="", ylabel="", yticks=np.arange(1, 8) * 80, xticks=[])
  2097. plt.savefig(figure_dir + 'Figure6_performanceStripes_behavior.png', dpi=300)
  2098. plt.savefig(figure_dir + 'Figure6_performanceStripes_behavior.pdf', dpi=300, transparent=True)
  2101. def Figure8_ITbehavioralConsistency_means(training_groups, seed=6,
  2102. recompute=False, metric='i1', model_draws=100, draws=1000, num_sites=159,
  2103. delta='proportion', mode='empirical_conditioned_v2',
  2104. source_file_dir=None,
  2105. figure_dir=None,
  2106. meta_dir=None, version='v1'):
  2107. if figure_dir == None:
  2108. figure_dir = f'{Path(__file__).parent.parent}/Figures/'
  2109. if meta_dir == None:
  2110. meta_dir = f'{Path(__file__).parent.parent}/Results/meta/'
  2111. if source_file_dir == None:
  2112. source_file_dir = f'{Path(__file__).parent.parent}/SourceDataFiles/'
  2114. np.random.seed(seed)
  2115. models_pt3 = pd.read_pickle(meta_dir + f'SelectedModels_pt3_means.pkl')
  2116. models_pt3['match_one'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(int) +
  2117. models_pt3['match_rsa'].astype(int)) == 1
  2118. models_pt3['match_two'] = (models_pt3['match_decoding'].astype(int) + models_pt3['match_selectivity'].astype(
  2119. int) + models_pt3['match_rsa'].astype(int)) == 2
  2121. neural_df = get_consistency_behavior_ITneurons(consistency_metric=metric, recompute=recompute, num_sites=num_sites,
  2122. mode=mode, draws=draws)
  2124. neural_df = neural_df.reset_index()
  2125. neural_df['delta'] = computeDelta(neural_df['consistency_i1_untrained'], neural_df['consistency_i1_trained'],
  2126. kind=delta)
  2127. if version == 'v0':
  2128. print('Medians:')
  2129. print(neural_df.median())
  2131. print('Means:')
  2132. print(neural_df.mean())
  2134. print('SD:')
  2135. print(neural_df.std())
  2137. print(
  2138. f"{((neural_df['consistency_i1_trained'].mean() / neural_df['consistency_i1_untrained'].mean()) - 1) * 100}% increase")
  2140. difference_dist = neural_df['consistency_i1_trained'] - \
  2141. neural_df['consistency_i1_untrained']
  2143. lower_CI = difference_dist.quantile(0.025)
  2144. upper_CI = difference_dist.quantile(0.975)
  2145. p_val = np.min([(difference_dist < 0).mean(), (difference_dist > 0).mean()]) * 2
  2147. print(
  2148. f"Behavioral consistency CI (95%): {lower_CI} - {upper_CI}")
  2149. print(f"p-val: {p_val}")
  2151. neural_df_plot = neural_df.copy()
  2152. neural_df_plot = neural_df_plot.melt(value_vars=['consistency_i1_untrained', 'consistency_i1_trained'],
  2153. id_vars='draw')
  2155. g = sns.catplot(data=neural_df_plot, x='variable', y='value', hue='variable', kind='point',
  2156. errorbar=("pi", 95),
  2157. palette={'consistency_i1_untrained': sns.xkcd_palette(['slate grey'])[0],
  2158. 'consistency_i1_trained': sns.xkcd_palette(['jade'])[0]}, height=4.5 * cm, aspect=1.05,
  2159. legend=False, join=False, dodge=0.25)
  2161. g.refline(y=0)
  2162. g.set_ylabels("IT-behavior\nconsistency")
  2163. g.set_xlabels("IT neuron pools")
  2164. g.set_xticklabels(['naïve', 'trained'])
  2166. g.savefig(figure_dir + 'FigureS13_consistency_neuralDecodes.png', dpi=300)
  2167. g.savefig(figure_dir + 'FigureS13_consistency_neuralDecodes.pdf', dpi=300, transparent=True)
  2169. neural_df_plot.to_csv(
  2170. source_file_dir + 'FigureS13_consistency_neuralDecodes.csv', index=False)
  2173. elif version == 'v1':
  2174. df = get_consistency_modelIT_modelBehavior(models_pt3, training_groups, recompute=recompute)
  2176. df['delta'] = computeDelta(df['consistency_i1_pre'], df['consistency_i1_post'])
  2178. df['Training'] = df['model_ID'].map(dict(zip(models_pt3['model_ID'], models_pt3['Training'])))
  2179. df['# matched IT-category metrics'] = None
  2180. df.loc[df['model_ID'].isin(
  2181. models_pt3.loc[models_pt3['match_all'], 'model_ID']), '# matched IT-category metrics'] = '3/3'
  2182. df.loc[df['model_ID'].isin(
  2183. models_pt3.loc[models_pt3['match_one'], 'model_ID']), '# matched IT-category metrics'] = '1/3'
  2184. df.loc[df['model_ID'].isin(
  2185. models_pt3.loc[models_pt3['match_two'], 'model_ID']), '# matched IT-category metrics'] = '2/3'
  2187. df.loc[df['model_ID'].isin(
  2188. models_pt3.loc[models_pt3['match_none'], 'model_ID']), '# matched IT-category metrics'] = '0/3'
  2190. df_means = df.groupby(['model_ID', '# matched IT-category metrics', 'Training'], as_index=False).mean()
  2192. print(kruskal(data=df_means, dv='delta', between='# matched IT-category metrics'))
  2194. g = sns.catplot(df_means, height=4 * cm, aspect=1.15,
  2195. y='delta', x='# matched IT-category metrics', kind='strip', order=['0/3', '1/3', '2/3', '3/3'],
  2196. color=sns.xkcd_palette(['cerulean'])[0],
  2197. sharey=False, sharex=False, zorder=1, alpha=0.1, dodge=True)
  2198. g.refline(y=0)
  2200. g.ax.axhspan(np.quantile(neural_df['delta'], 0.025),
  2201. np.quantile(neural_df['delta'], 0.975),
  2202. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=.5)
  2204. sns.pointplot(
  2205. data=df_means, x='# matched IT-category metrics', y='delta', join=False,
  2206. order=['0/3', '1/3', '2/3', '3/3'],
  2207. errorbar=None, estimator=np.median, color=sns.xkcd_palette(['cerulean'])[0],
  2208. markers="_", scale=2, ax=g.ax,
  2209. )
  2210. g.set_ylabels('%Δ consistency')
  2211. g.set_xlabels('# IT-aligned\ncategory metrics')
  2213. g.savefig(figure_dir + f'Figure8_behavioralConsistency_v1.pdf', dpi=300, transparent=True)
  2214. g.savefig(figure_dir + f'Figure8_behavioralConsistency_v1.png', dpi=300)
  2216. df_means.to_csv(
  2217. source_file_dir + 'Figure8_behavioralConsistency_v1.csv', index=False)
  2219. g = sns.catplot(df_means, height=4 * cm, aspect=1.15,
  2220. y='delta', x='# matched IT-category metrics', kind='strip', order=['0/3', '1/3', '2/3', '3/3'],
  2221. color=sns.xkcd_palette(['cerulean'])[0],
  2222. sharey=False, sharex=False, zorder=1, alpha=0.1,
  2223. dodge=True)
  2224. g.refline(y=0)
  2226. g.ax.axhspan(np.quantile(neural_df['delta'], 0.025),
  2227. np.quantile(neural_df['delta'], 0.975),
  2228. color=sns.xkcd_palette(['grey'])[0], zorder=0, alpha=.5)
  2230. sns.pointplot(
  2231. data=df_means, x='# matched IT-category metrics', y='delta', join=False,
  2232. order=['0/3', '1/3', '2/3', '3/3'],
  2234. errorbar=None, estimator=np.median, color=sns.xkcd_palette(['cerulean'])[0],
  2235. markers="_", scale=2, ax=g.ax,
  2236. )
  2237. g.set_ylabels('%Δ consistency')
  2238. g.set_xlabels('# IT-aligned\ncategory metrics')
  2239. g.set(ylim=[-20, 105])
  2240. g.savefig(figure_dir + f'Figure8_behavioralConsistency_v1_zoom.pdf', dpi=300, transparent=True)
  2241. g.savefig(figure_dir + f'Figure8_behavioralConsistency_v1_zoom.png', dpi=300)

paperFigures.py at commit 85fe071, under MIT · at the source

Overview

Authors: Lynn K A Sörensen1,2,3,4, James J DiCarlo1,2,3, Kohitij Kar4,5
ORCID iDs: Lynn K A Sörensen, James J DiCarlo, Kohitij Kar
  1. McGovern Institute for Brain Research, Dept. of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, USA
  2. Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, USA
  3. MIT Quest for Intelligence, Cambridge, USA
  4. Centre for Integrative and Applied Neuroscience, York University, Toronto, Canada
  5. Department of Biology, Centre for Vision Research, York University, Toronto, Canada
Institutions: McGovern Institute for Brain Research (United States); York University (Canada); Massachusetts Institute of Technology (United States)
Journal: Nature communications, volume 17, issue 1, article 8434
Dates: received 14 July 2025; accepted 12 June 2026; published online 8 July 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s41467-026-74816-0 · PMID 42420264 · PMCID PMC13478300 · OpenAlex W7167740178
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: non-human primate (organism)
Methods: Spectral & time-frequency, Connectivity, Statistics, Smoothing, state filtering, decompositions, Machine learning, Preprocessing, Single-unit activity, calcium imaging, Physiology & signal measures
Keywords: Object vision, Cortex
MeSH: Learning*, Neuronal Plasticity*, Temporal Lobe*, Animals, Macaca mulatta, Male, Models, Neurological, Neural Networks, Computer, Neurons, Pattern Recognition, Visual, Photic Stimulation, Visual Pathways (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: National Science Foundation (NSF) (2124136, CCF-1231216); Simons Foundation (AN-NC-GB-Culmination-00002986-04, SFARI, 967073); Canada Research Chairs (CRC-2021-00326); Canada First Research Excellence Fund (VISTA Program); Deutsche Forschungsgemeinschaft (German Research Foundation) (547591872); United States Department of Defense | United States Navy | Office of Naval Research (ONR) (N00014-21-1-2801, N00014-20-1-2589); Canadian Network for Research and Innovation in Machining Technology, Natural Sciences and Engineering Research Council of Canada (RGPIN 2024-06223)
Citations: cited by 1 paper (Europe PMC); 131 references in the paper

Abstract

How does the primate brain coordinate plasticity when learning to discriminate new objects? We measured consequences of object learning on inferior temporal (IT) cortex, a key waypoint supporting object recognition in the ventral visual stream, in male macaques. Neural activity in task-trained monkeys’ IT showed increased object selectivity, enhanced linear separability across objects, and more object-invariant representations compared to task-naïve monkeys. To model these differences, we developed a computational framework using anatomically-mapped artificial neural network (ANN) models of the ventral stream with various learning algorithms. Simulations revealed that gradient-based, performance-optimizing updates of ANN internal representations accurately approximated observed IT cortex changes. These models predict novel training-induced phenomena in the IT cortex, including changes independent of object identity and IT’s alignment with behavior. This convergence between empirical measurements and model predictions suggests ventral stream plasticity follows task optimization principles well-approximated by gradient descent, enabling accurate predictions about visual plasticity and generalization to test images.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 11 matches between paragraphs and lines of code.

dicarlolab/mkturk

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 4bdfaca7d1caa02115fb16c97b158dca1e620595, 1 September 2018
Languages: JavaScript (14)
Size: 69 files, 14 scripts
Software Heritage: archived
Found in: the text, “Object training and active binary object discrim”
Holds: README
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
15 files
At the source: github.com/dicarlolab/mkturk

OSF dcgze

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Languages: Python (19)
Size: 170 files, 19 scripts
Software Heritage: not checked
Found in: “Code availability”
Holds: README, license file
Not found: CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (14 files), pandas (13 files), scikit-learn (7 files), PyTorch (6 files), SciPy (5 files), h5py (1 file), Matplotlib (1 file), Pingouin (1 file), seaborn (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers (HTTP 200)
  • 27 September 2026: the link answers (HTTP 200)
21 files
At the source: osf.io/dcgze/?view_only=6dfe548c7ba24238932d247e65523053

lynnsoerensen/ObjectTraining_IT_ANNs

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 85fe071926ac2f44f8ac27428a33ca4a4fd1d4c4, 1 May 2026
Languages: Python (23)
Size: 2,132 files, 23 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file
Not found: CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (17 files), pandas (13 files), PyTorch (10 files), scikit-learn (8 files), SciPy (5 files), h5py (1 file), Matplotlib (1 file), Pingouin (1 file), seaborn (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
25 files
At the source: github.com/lynnsoerensen/ObjectTraining_IT_ANNs

Zenodo 19955396

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: “Code availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 27 September 2026: the link answers (HTTP 200)
  • 27 September 2026: the link answers (HTTP 200)
At the source:

Code availability

All code can be accessed on the Open Science Framework repository (https://osf.io/dcgze/ (https://osf.io/dcgze/?view_only=6dfe548c7ba24238932d247e65523053)), on GitHub (https://github.com/lynnsoerensen/ObjectTraining_IT_ANNs) and on Zenodo (10.5281/zenodo.19955396).

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 4 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 56 scripts, each with its path and the digest of its content;
  • 11 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability

All data has been deposited in the following Open Science Framework repository (https://osf.io/dcgze/ (https://osf.io/dcgze/?view_only=6dfe548c7ba24238932d247e65523053)). Source data are provided in this paper.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 3 authors, 2 keywords, 12 MeSH terms, 7 funders, 116 references.

Cite

This paper

Sörensen, L. K. A., DiCarlo, J. J., & Kar, K. (2026). Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training. Nature communications, 17(1), 8434. https://doi.org/10.1038/s41467-026-74816-0

BibTeX

@article{sorensen2026hierarchical,
author = {Sörensen, Lynn K A and DiCarlo, James J and Kar, Kohitij},
title = {{Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training}},
journal = {Nature communications},
year = {2026},
month = jul,
volume = {17},
number = {1},
pages = {8434},
publisher = {Nature Publishing Group},
issn = {2041-1723},
doi = {10.1038/s41467-026-74816-0},
url = {https://doi.org/10.1038/s41467-026-74816-0},
pmid = {42420264},
pmcid = {PMC13478300}
}

RIS

TY - JOUR
AU - Sörensen, Lynn K A
AU - DiCarlo, James J
AU - Kar, Kohitij
TI - Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training
T2 - Nature communications
J2 - Nat Commun
PY - 2026
DA - 2026/07/08
VL - 17
IS - 1
SP - 8434
SN - 2041-1723
PB - Nature Publishing Group
DO - 10.1038/s41467-026-74816-0
UR - https://doi.org/10.1038/s41467-026-74816-0
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41467-026-74816-0",
"type": "article-journal",
"title": "Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training",
"container-title": "Nature communications",
"author": [
{
"family": "Sörensen",
"given": "Lynn K A"
},
{
"family": "DiCarlo",
"given": "James J"
},
{
"family": "Kar",
"given": "Kohitij"
}
],
"container-title-short": "Nat Commun",
"volume": "17",
"issue": "1",
"page": "8434",
"DOI": "10.1038/s41467-026-74816-0",
"PMID": "42420264",
"PMCID": "PMC13478300",
"ISSN": "2041-1723",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41467-026-74816-0",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
8
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41467-026-76106-1 [code]
Facial expression discrimination emerges from partially overlapping neural subspaces of detection and identity.
Journal: Nature communications
In common: h5py, seaborn, scikit-learn, 4 other tools, non-human primate, 12 references, author Kohitij Kar
[2] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: h5py, PyTorch, seaborn, 5 other tools, 6 references
[3] doi:10.1038/s41593-026-02207-1 [code]
Neuronal tuning aligns dynamically with object and texture manifolds across the visual hierarchy.
Journal: Nature neuroscience
In common: PyTorch, seaborn, scikit-learn, 4 other tools, non-human primate, 6 references
[4] doi:10.1038/s41467-026-72146-9 [code]
Modeling attention and binding in the brain through bidirectional recurrent gating.
Journal: Nature communications
In common: PyTorch, seaborn, SciPy, 2 other tools, 8 references
[5] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: h5py, PyTorch, seaborn, 5 other tools, 6 references
[6] doi:10.1038/s41598-026-43946-2
Human lateral occipital complex is invariant for position and size transformations at the single-neuron level.
Journal: Scientific reports
In common: 8 references
[7] doi:10.1371/journal.pone.0347992 [code]
Rotation-tolerant representations elucidate the time-course of high-level object processing.
Journal: PloS one
In common: pandas, NumPy, 7 references
[8] doi:10.1038/s41562-026-02414-7 [code]
Optimized feature gains explain and predict successes and failures of human selective listening.
Journal: Nature human behaviour
In common: Pingouin, h5py, PyTorch, 6 other tools, 2 references
[9] doi:10.1038/s41467-026-74460-8 [code]
Spike-based alignment learning solves the weight transport problem.
Journal: Nature communications
In common: PyTorch, seaborn, scikit-learn, 4 other tools, 4 references
[10] doi:10.7554/elife.105968 [code]
Modeling the hallucinatory effects of classical psychedelics in terms of replay-dependent plasticity mechanisms.
Journal: eLife
In common: PyTorch, scikit-learn, Matplotlib, 1 other tool, 5 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.