OSCR

A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § RESULTS › TMS Effects on Semantic Processing Are Still Present After Accounting for Word Length and Frequency ↔ plot01_gen_all_barplots.py, lines 1049–1104 · score 0.61 · semantic relatedness, sound feature, semantic priming, residualization, social
  2. [2] § MATERIALS AND METHODS › Models › Statistical significance testing, correction for multiple comparisons, and effect sizes ↔ utils/plot_utils.py, lines 57–70 · score 0.52 · cognitive neuroscience, literature, mid

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,558 lines · 78 KB · no license · 1 match

  1. import matplotlib
  2. import mne
  3. import numpy
  4. import os
  5. import pingouin
  6. import random
  7. import re
  8. import scipy
  10. from matplotlib import colormaps, font_manager, pyplot
  11. from mne import stats
  12. from scipy import stats
  13. from tqdm import tqdm
  15. from utils.plot_utils import font_setup, perm_against_zero, permutation_two_samples, read_effsizes, set_colors
  17. ### effect_sizes
  18. lo_e, mid_e, hi_e = read_effsizes(mode='cog_neuro')
  19. print(lo_e)
  21. font_folder = '../../fonts'
  22. font_setup(font_folder)
  24. for mode in [
  25. 'residualize',
  26. 'bootstrap',
  27. ]:
  28. for corpus in [
  29. #'opensubs',
  30. 'wac',
  31. #'cc100',
  32. ]:
  34. results = dict()
  36. for root, direc, fz in os.walk(
  37. os.path.join(
  38. 'test_results',
  39. )):
  40. for f in fz:
  41. parts = root.split('/')
  42. approach = parts[1]
  43. if approach != 'correlation':
  44. continue
  45. stat_approach = parts[2]
  46. if stat_approach != mode:
  47. continue
  48. evaluation = parts[3]
  49. if evaluation != 'spearman':
  50. continue
  51. lang = parts[4]
  52. n = '100000'
  53. if 'gpt2-small_surprisal' in root:
  54. model = 'GPT2\nsurprisal'
  55. pass
  56. elif corpus in root and n in root:
  57. model = '{}\nPPMI'.format(corpus)
  58. pass
  59. elif corpus in root and 'abs-prob' in root:
  60. if 'one-' in root:
  61. model = 'Word\nfrequency\n(neg.\nfirst w.)'
  62. elif 'two-' in root:
  63. model = 'Word\nfrequency\n(neg.\nsecond w.)'
  64. elif 'visual-' in root:
  65. model = 'Word\nfrequency\n(neg.\nvisual)'
  66. elif 'produced-' in root:
  67. model = 'Word\nfrequency\n(neg.\nuttered)'
  68. elif 'overall-' in root:
  69. model = 'Word\nfrequency\n(neg. sum)'
  70. else:
  71. raise RuntimeError()
  72. continue
  73. #print(root)
  74. #print(model.replace('\n', ' '))
  75. pass
  76. elif 'word-length' in root:
  77. if 'one-' in root:
  78. model = 'Word\nlength\n(first w.)'
  79. elif 'two-' in root:
  80. model = 'Word\nlength\n(second w.)'
  81. elif 'visual-' in root:
  82. model = 'Word\nlength\n(visual)'
  83. elif 'produced-' in root:
  84. model = 'Word\nlength\n(uttered)'
  85. elif 'overall-' in root:
  86. model = 'Word\nlength\n(sum)'
  87. else:
  88. continue
  89. pass
  90. else:
  91. print(f)
  92. continue
  93. with open(os.path.join(root, f)) as i:
  94. for l in i:
  95. line = l.strip().split('\t')
  96. lang = line[0]
  97. if lang not in results.keys():
  98. results[lang] = dict()
  99. old_model = line[1]
  100. all_task = line[2]
  101. ### modality
  102. assert all_task[:3] == '{}_'.format(lang)
  103. task = all_task[3:].split('#')[0].split('_')[0]
  104. if 'sem' in task or 'pmtg' in task:
  105. splitter = '-'
  106. else:
  107. splitter = '_'
  108. if 'sound' in task:
  109. cond_idx = -2
  110. else:
  111. cond_idx = -1
  112. case = all_task.split('#')[-1].split(splitter)[0]
  113. cond = all_task.split('#')[-1].split(splitter)[cond_idx]
  114. cond = '{}{}'.format(cond[0].lower(), cond[1:])
  115. if cond == 'cedx':
  116. cond = 'rCereb'
  117. if cond == 'cz':
  118. cond = 'vertex'
  119. if cond not in ['sham', 'vertex']:
  120. cond = 'TMS\n{}'.format(cond)
  121. if 'distr-learn' in all_task:
  122. pass
  123. elif 'pmtg-prod' in all_task:
  124. if '-but-' in all_task:
  125. continue
  126. pass
  127. elif 'sem-phon' in all_task:
  128. pass
  129. elif 'sound-act' in all_task:
  130. if 'together-pos-all' not in all_task:
  131. continue
  132. if 'all_all' in all_task:
  133. continue
  134. if 'detailed' in all_task:
  135. continue
  136. case = '{}-{}'.format(case, all_task.split('_')[-1])
  137. pass
  138. elif 'social' in all_task:
  139. if 'prime-cat' not in all_task:
  140. continue
  141. if 'cong' in all_task:
  142. continue
  143. pass
  144. else:
  145. continue
  146. if 'freq' in model:
  147. print(f)
  148. if task not in results[lang].keys():
  149. results[lang][task] = dict()
  150. if case not in results[lang][task].keys():
  151. results[lang][task][case] = dict()
  152. if cond not in results[lang][task][case].keys():
  153. results[lang][task][case][cond] = dict()
  154. non_nan_res = [v if v!='nan' else 0. for v in line[3:]]
  155. res = numpy.array(non_nan_res, dtype=numpy.float32)
  156. results[lang][task][case][cond][model] = res[:1000]
  157. colors = {
  158. '{}\nPPMI'.format(corpus) : ('navy', 'royalblue', 'lightsteelblue',),
  159. 'GPT2\nvectors' : ('lightskyblue', 'lightblue', 'paleturquoise'),
  160. '{}\nsurprisal'.format(corpus) : ('mediumorchid', 'thistle', 'plum'),
  161. 'GPT2\nsurprisal' : ('mediumvioletred', 'pink', 'palevioletred'),
  162. 'Word\nfrequency' : ('sienna', 'peru', 'sandybrown'),
  163. 'Word\nlength' : ('gray', 'darkgray', 'lightgray'),
  164. }
  167. out_f = os.path.join('plots', '00', mode, corpus)
  168. os.makedirs(out_f, exist_ok=True)
  170. for lang, l_results in results.items():
  171. for task, t_results in l_results.items():
  172. for case, c_results in t_results.items():
  173. ### getting ready to write things down...
  174. lines = list()
  175. gen_line = [mode, corpus, lang, task, case]
  176. curr_fold = os.path.join(out_f, lang, task, case)
  177. os.makedirs(curr_fold, exist_ok=True)
  178. conds = sorted(c_results.keys(), reverse=True)
  179. models = set([m for _ in c_results.values() for m in _.keys()])
  180. no_tms_cond = [c for c in conds if 'ver' in c or 'sh' in c][0]
  181. models = set([m for _ in c_results.values() for m in _.keys()])
  182. print(models)
  183. best_model = sorted(
  184. [(c_results[no_tms_cond][m], m) for m in models if 'length' not in m and 'frequency' not in m],
  185. key=lambda item : numpy.average(item[0]),
  186. reverse=True,
  187. )[0][1]
  188. sorted_models = [
  189. '{}\nPPMI'.format(corpus),
  190. ] +\
  191. ['GPT2\nsurprisal',] +\
  192. [m for m in models if 'freq' in m and ('visual' in m or 'first' in m)] +\
  193. [m for m in models if 'freq' in m and ('utter' in m or 'second' in m)] +\
  194. [m for m in models if 'freq' in m and ('sum' in m)] +\
  195. [m for m in models if 'ength' in m and ('visual' in m or 'first' in m)] +\
  196. [m for m in models if 'ength' in m and ('utter' in m or 'second' in m)] +\
  197. [m for m in models if 'ength' in m and ('sum' in m)]
  198. print(sorted_models)
  199. try:
  200. if 'together' not in case:
  201. assert len(sorted_models) > 6
  202. else:
  203. assert len(sorted_models) == 4
  205. except AssertionError:
  206. print(case)
  207. print(models)
  208. continue
  209. xs = list(range(len(sorted_models)))
  210. if len(conds) == 2:
  211. corrections = list(numpy.linspace(-.33, .33, len(conds)))
  212. txt_corrections = list(numpy.linspace(-.4, .4, len(conds)))
  213. m_sc = 2000
  214. t_s = 20
  215. else:
  216. corrections = list(numpy.linspace(-.5, .5, len(conds)))
  217. txt_corrections = list(numpy.linspace(-.55, .55, len(conds)))
  218. m_sc = 1400
  219. t_s = 15
  220. if 'together' in case:
  221. figsize=(15, 10)
  222. else:
  223. figsize=(20, 10)
  224. fig, ax = pyplot.subplots(constrained_layout=True, figsize=figsize)
  225. x_shift = 0
  226. xticks = list()
  227. counter = -1
  228. ps = list()
  229. for m_i, m in enumerate(sorted_models):
  230. if 'Resp' in m:
  231. gen_avg = numpy.average([v for _ in c_results.values() for __ in _.values() for v in __])
  232. gen_std = numpy.std([v for _ in c_results.values() for __ in _.values() for v in __])
  233. counter += 1
  234. for c_i, c in enumerate(conds):
  235. spec_line = [v for v in gen_line]
  236. spec_line.extend([m, c])
  237. if 'length' in m:
  238. color=colors['Word\nlength'][c_i]
  239. elif 'freq' in m:
  240. color=colors['Word\nfrequency'][c_i]
  241. else:
  242. color=colors[m][c_i]
  243. xticks.append((counter, m))
  245. if len(conds) == 2:
  246. w = 0.6
  247. else:
  248. w = 0.45
  249. if c_i == 0:
  250. comps = list()
  251. for other_i, other in enumerate(conds[1:]):
  252. two = c_results[other][m]
  253. t_val, p_val, fake_distr, ci = permutation_two_samples(c_results[c][m], two)
  254. print([m, c, other, p_val, t_val])
  255. ps.append((m, (c, other), p_val, t_val))
  256. comps.append('{}_{}_{}_{}_{}_{}@{}'.format(c, other, ci[0], ci[1], p_val, t_val, ','.join([str(v) for v in fake_distr])))
  257. spec_line.append('#'.join(comps))
  258. else:
  259. spec_line.append('na')
  260. ### simple p-value
  261. t, p, ci = perm_against_zero(c_results[c][m])
  262. #print(p)
  263. ps.append((m, c, p))
  264. spec_line.extend([p, t, ci])
  265. ### bar
  266. ax.bar(
  267. m_i+corrections[c_i]+x_shift,
  268. numpy.average(c_results[c][m]),
  269. width=w,
  270. color=color,
  271. edgecolor='gray',
  272. zorder=2.
  273. )
  274. ax.errorbar(
  275. m_i+corrections[c_i]+x_shift,
  276. numpy.average(c_results[c][m]),
  277. yerr=numpy.std(c_results[c][m]),
  278. color='black',
  279. capsize=5,
  280. zorder=3.
  281. )
  282. spec_line.extend([m_i+corrections[c_i]+x_shift, m_i+txt_corrections[c_i]+x_shift])
  283. spec_line.append(numpy.average(c_results[c][m]))
  284. spec_line.append(','.join([str(val) for val in c_results[c][m]]))
  285. lines.append(spec_line)
  286. if len(c_results[c][m]) == 1000:
  287. alpha = 0.2
  288. elif len(c_results[c][m]) == 10000:
  289. alpha = 0.02
  290. ax.scatter(
  291. [m_i+corrections[c_i]+x_shift+(random.randrange(-m_sc, m_sc)*0.0001) for rand in range(len(c_results[c][m]))],
  292. c_results[c][m],
  293. color=color,
  294. edgecolor='white',
  295. alpha=alpha,
  296. zorder=2.5
  297. )
  298. ax.text(
  299. m_i+txt_corrections[c_i]+x_shift,
  300. -.05,
  301. s=c,
  302. fontsize=t_s,
  303. ha='center',
  304. va='center',
  305. )
  306. x_shift += 1
  307. counter += 1
  308. if m_i in [0, 1, 4]:
  309. x_shift += 1
  310. counter += 1
  311. ax.set_ylim(bottom=-.08, top=.38)
  312. ax.hlines(xmin=-.8, xmax=len(sorted_models)+x_shift-2.2, color='black', y=0)
  313. ax.hlines(xmin=-.8, xmax=len(sorted_models)+x_shift-2.2, color='silver',alpha=0.5,linestyle='dashed', y=[y*0.01 for y in range(-5, 35, 5)], zorder=1)
  314. pyplot.ylabel('Spearman correlation (RSA RT-model)', fontsize=23)
  315. pyplot.xticks(
  316. [x[0] for x in xticks],
  317. [x[1] for x in xticks],
  318. fontsize=25,
  319. fontweight='bold')
  320. print(curr_fold)
  321. pyplot.savefig(os.path.join(curr_fold, '{}_00.jpg'.format(case)), dpi=300)
  322. ### writing to file
  323. with open(os.path.join(curr_fold, '{}_00.txt'.format(case)), 'w') as o:
  324. o.write('mode\tcorpus\tlang\ttask\tcase\tmodel\tcondition\t')
  325. o.write('comparisons\t')
  326. o.write('p_val_raw\tt_val\tci_constant\tx_bar\tx_label\tperms_avg\tperms\n')
  327. for lin in lines:
  328. for li in lin:
  329. o.write('{}\t'.format(str(li).replace('\n', '$')))
  330. o.write('\n')
  331. ### running correction for multiple comparisons
  332. corr_f = os.path.join('plots', 'stats_after_correction')
  333. os.makedirs(corr_f, exist_ok=True)
  334. for corpus in [
  335. 'wac',
  336. ]:
  337. final_corr = list()
  338. final_to_be_corr = list()
  339. results = dict()
  340. fakes = list()
  341. for approach in [
  342. 'residualize',
  343. 'bootstrap',
  344. ]:
  345. fold = os.path.join('plots', '00', approach, corpus)
  346. for lang in ['de', 'it']:
  347. for root, direc, fz in os.walk(os.path.join(fold, lang)):
  348. for f in fz:
  349. if 'txt' in f:
  350. assert '00' in f
  351. with open(os.path.join(root, f)) as i:
  352. for l_i, l in enumerate(i):
  353. line = l.strip().split('\t')
  354. if l_i == 0:
  355. header = line.copy()
  356. continue
  357. task = line[header.index('task')]
  358. case = line[header.index('case')]
  359. if 'sound' in task:
  360. full_task = '{}_{}_{}'.format(approach, task, case[0])
  361. elif 'social' in task:
  362. full_task = '{}_{}_{}'.format(approach, task, case.split('-')[0])
  363. else:
  364. full_task = '{}_{}'.format(approach, task)
  365. if full_task not in results.keys():
  366. results[full_task] = list()
  367. cond = line[header.index('condition')]
  368. model = line[header.index('model')]
  369. p_raw = float(line[header.index('p_val_raw')])
  370. t_val = float(line[header.index('t_val')])
  371. bar_y = float(line[header.index('perms_avg')])
  372. bar_x = float(line[header.index('x_bar')])
  373. ci_constant = float(line[header.index('ci_constant')])
  374. label_x = float(line[header.index('x_label')])
  375. perms = [float(val) for val in line[header.index('perms')].split(',')]
  376. assert len(perms) in [1000, 10000]
  377. comparisons = [val.split('@')[0].split('_') for val in line[header.index('comparisons')].split('#')]
  378. curr_dict = {
  379. 'cond' : cond,
  380. 'case' : case,
  381. 'model' : model,
  382. 'p_raw' : p_raw,
  383. 't_val' : t_val,
  384. 'bar_y' : bar_y,
  385. 'bar_x' : bar_x,
  386. 'label_x' : label_x,
  387. 'perms' : perms,
  388. 'ci_constant' : ci_constant,
  389. 'comparisons' : comparisons,
  390. }
  391. results[full_task].append(curr_dict)
  392. print(results.keys())
  393. ### correcting p-values only once
  394. to_be_corr = list()
  395. counter = 0
  396. for k, v in results.items():
  397. for case in v:
  398. to_be_corr.append(
  399. (
  400. (k,
  401. case['case'],
  402. case['cond'],
  403. case['model'],
  404. numpy.mean(case['perms']),
  405. case['ci_constant'],
  406. ),
  407. case['t_val'],
  408. case['p_raw']
  409. )
  410. )
  411. for cmpr in case['comparisons']:
  412. #print(cmpr)
  413. if cmpr != ['na']:
  414. counter += 1
  415. to_be_corr.append(
  416. (
  417. (k,
  418. case['case'],
  419. case['cond'],
  420. case['model'],
  421. cmpr[:4]
  422. ),
  423. ### t-val
  424. float(cmpr[5]),
  425. ### p-val
  426. float(cmpr[4])
  427. )
  428. )
  429. print(counter)
  430. ### baseline
  431. corr = mne.stats.fdr_correction([v[2] for v in to_be_corr])[1]
  432. final_corr.extend(corr)
  433. final_to_be_corr.extend(to_be_corr)
  434. with open(os.path.join(corr_f, '{}_corrected_baseline_p-vals.tsv'.format(corpus)), 'w') as o:
  435. o.write('approach\tdataset\tcase\tcondition\tmodel\tci_min\tci_max\tperms_avg\tt_value\traw_p\tfdr_corrected_p\n')
  436. for d, cp in zip(final_to_be_corr, final_corr):
  437. if len(d[0]) == 6:
  438. approach = d[0][0].split('_')[0]
  439. task = d[0][0].split('_')[1]
  440. case = d[0][1]
  441. cond = d[0][2]
  442. model = d[0][3]
  443. avg = d[0][4]
  444. ci_constant = d[0][5]
  445. ci_min = float(avg)-float(ci_constant)
  446. ci_max = float(avg)+float(ci_constant)
  447. raw_p = d[2]
  448. t = d[1]
  449. o.write('{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\n'.format(approach, task, case, cond, model, ci_min, ci_max, avg, t, raw_p, cp))
  450. with open(os.path.join(corr_f, '{}_corrected_comparisons_p-vals.tsv'.format(corpus)), 'w') as o:
  451. o.write('approach\tdataset\tcase\tcondition_one\tcondition_two\tmodel\tci_min\tci_max\tt_value\traw_p\tfdr_corrected_p\n')
  452. for d, cp in zip(final_to_be_corr, final_corr):
  453. if len(d[0]) == 5:
  454. #print(d)
  455. approach = d[0][0].split('_')[0]
  456. task = d[0][0].split('_')[1]
  457. case = d[0][1]
  458. model = d[0][3]
  459. cond_one = d[0][4][0]
  460. cond_two = d[0][4][1]
  461. ci_min = d[0][4][2]
  462. ci_max = d[0][4][3]
  463. raw_p = d[2]
  464. t = d[1]
  465. o.write('{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\t{}\n'.format(
  466. approach,
  467. task,
  468. case,
  469. cond_one,
  470. cond_two,
  471. model,
  472. ci_min,
  473. ci_max,
  474. t,
  475. raw_p,
  476. cp)
  477. )
  480. font_folder = '../../fonts'
  481. font_setup(font_folder)
  483. for model_sel in [
  484. 'ppmi',
  485. 'both',
  486. ]:
  487. for mode in [
  488. 'residualize',
  489. 'bootstrap',
  490. ]:
  491. for corpus in [
  492. 'wac',
  493. ]:
  494. baseline = list()
  495. with open(os.path.join(corr_f, '{}_corrected_baseline_p-vals.tsv'.format(corpus))) as i:
  496. for l_i, l in enumerate(i):
  497. line = l.strip().split('\t')
  498. if l_i == 0:
  499. continue
  500. baseline.append(line)
  502. comparisons = list()
  503. with open(os.path.join(corr_f, '{}_corrected_comparisons_p-vals.tsv'.format(corpus))) as i:
  504. for l_i, l in enumerate(i):
  505. line = l.strip().split('\t')
  506. if l_i == 0:
  507. continue
  508. comparisons.append(line)
  509. results = dict()
  511. for root, direc, fz in os.walk(
  512. os.path.join(
  513. 'test_results',
  514. )):
  515. for f in fz:
  516. parts = root.split('/')
  517. approach = parts[1]
  518. if approach != 'correlation':
  519. continue
  520. stat_approach = parts[2]
  521. if stat_approach != mode:
  522. continue
  523. evaluation = parts[3]
  524. if evaluation != 'spearman':
  525. continue
  526. lang = parts[4]
  527. n = '100000'
  528. if 'gpt2-small_surprisal' in root:
  529. model = 'GPT2\nsurprisal'
  530. pass
  531. elif corpus in root and n in root:
  532. model = '{}\nPPMI'.format(corpus)
  533. pass
  534. elif corpus in root and 'abs-prob' in root:
  535. if 'one-' in root:
  536. model = 'Word\nfrequency\n(neg.\nfirst w.)'
  537. elif 'two-' in root:
  538. model = 'Word\nfrequency\n(neg.\nsecond w.)'
  539. elif 'visual-' in root:
  540. model = 'Word\nfrequency\n(neg.\nvisual)'
  541. elif 'produced-' in root:
  542. model = 'Word\nfrequency\n(neg.\nuttered)'
  543. elif 'overall-' in root:
  544. model = 'Word\nfrequency\n(neg. sum)'
  545. else:
  546. continue
  547. print(root)
  548. print(model.replace('\n', ' '))
  549. pass
  550. elif 'word-length' in root:
  551. if 'one-' in root:
  552. model = 'Word\nlength\n(first w.)'
  553. elif 'two-' in root:
  554. model = 'Word\nlength\n(second w.)'
  555. elif 'visual-' in root:
  556. model = 'Word\nlength\n(visual)'
  557. elif 'produced-' in root:
  558. model = 'Word\nlength\n(uttered)'
  559. elif 'overall-' in root:
  560. model = 'Word\nlength\n(sum)'
  561. else:
  562. continue
  563. pass
  564. else:
  565. continue
  566. with open(os.path.join(root, f)) as i:
  567. for l in i:
  568. line = l.strip().split('\t')
  569. lang = line[0]
  570. if lang not in results.keys():
  571. results[lang] = dict()
  572. old_model = line[1]
  573. all_task = line[2]
  574. ### modality
  575. assert all_task[:3] == '{}_'.format(lang)
  576. task = all_task[3:].split('#')[0].split('_')[0]
  577. if 'sem' in task or 'pmtg' in task:
  578. splitter = '-'
  579. else:
  580. splitter = '_'
  581. if 'sound' in task:
  582. cond_idx = -2
  583. else:
  584. cond_idx = -1
  585. case = all_task.split('#')[-1].split(splitter)[0]
  586. cond = all_task.split('#')[-1].split(splitter)[cond_idx]
  587. cond = '{}{}'.format(cond[0].lower(), cond[1:])
  588. if cond == 'cedx':
  589. cond = 'rCereb'
  590. if cond == 'cz':
  591. cond = 'vertex'
  592. if cond not in ['sham', 'vertex']:
  593. cond = 'TMS\n{}'.format(cond)
  594. if 'distr-learn' in all_task:
  595. pass
  596. elif 'pmtg-prod' in all_task:
  597. if '-but-' in all_task:
  598. continue
  599. pass
  600. elif 'sem-phon' in all_task:
  601. pass
  602. elif 'sound-act' in all_task:
  603. if 'together-pos-all' not in all_task:
  604. continue
  605. if 'all_all' in all_task:
  606. continue
  607. if 'detailed' in all_task:
  608. continue
  609. case = '{}-{}'.format(case, all_task.split('_')[-1])
  610. pass
  611. elif 'social' in all_task:
  612. if 'prime-cat' not in all_task:
  613. continue
  614. if 'cong' in all_task:
  615. continue
  616. pass
  617. else:
  618. continue
  619. if task not in results[lang].keys():
  620. results[lang][task] = dict()
  621. if case not in results[lang][task].keys():
  622. results[lang][task][case] = dict()
  623. if cond not in results[lang][task][case].keys():
  624. results[lang][task][case][cond] = dict()
  625. non_nan_res = [v if v!='nan' else 0. for v in line[3:]]
  626. res = numpy.array(non_nan_res, dtype=numpy.float32)
  627. results[lang][task][case][cond][model] = res[:1000]
  628. colors = {
  629. '{}\nPPMI'.format(corpus) : ('seagreen', 'mediumaquamarine', 'mediumseagreen'),
  630. 'GPT2\nvectors' : ('lightskyblue', 'lightblue', 'paleturquoise'),
  631. '{}\nsurprisal'.format(corpus) : ('mediumorchid', 'thistle', 'plum'),
  632. 'GPT2\nsurprisal' : ('mediumvioletred', 'pink', 'palevioletred'),
  633. 'Word\nfrequency' : ('sienna', 'peru', 'sandybrown'),
  634. 'Word\nlength' : ('gray', 'darkgray', 'lightgray'),
  635. }
  637. out_f = os.path.join('plots', 'corrected', model_sel, mode, corpus)
  638. os.makedirs(out_f, exist_ok=True)
  640. for lang, l_results in results.items():
  641. for task, t_results in l_results.items():
  642. for case, c_results in t_results.items():
  643. ### getting ready to write things down...
  644. lines = list()
  645. gen_line = [mode, corpus, lang, task, case]
  646. curr_fold = os.path.join(out_f, lang, task, case)
  647. os.makedirs(curr_fold, exist_ok=True)
  648. conds = sorted(c_results.keys(), reverse=True)
  649. models = set([m for _ in c_results.values() for m in _.keys()])
  650. print(models)
  651. no_tms_cond = [c for c in conds if 'ver' in c or 'sh' in c][0]
  652. models = set([m for _ in c_results.values() for m in _.keys()])
  653. best_model = sorted(
  654. [(c_results[no_tms_cond][m], m) for m in models if 'length' not in m and 'frequency' not in m],
  655. key=lambda item : numpy.average(item[0]),
  656. reverse=True,
  657. )[0][1]
  658. print('best: {}'.format(best_model))
  659. if model_sel == 'ppmi':
  660. sorted_models = ['{}\nPPMI'.format(corpus)] +\
  661. [m for m in models if 'freq' in m and 'sum' in m] +\
  662. [m for m in models if 'ength' in m and 'sum' in m]
  663. elif model_sel == 'both':
  664. sorted_models = [
  665. '{}\nPPMI'.format(corpus),
  666. ] +\
  667. ['GPT2\nsurprisal',] +\
  668. [m for m in models if 'freq' in m and ('visual' in m or 'first' in m)] +\
  669. [m for m in models if 'freq' in m and ('utter' in m or 'second' in m)] +\
  670. [m for m in models if 'freq' in m and ('sum' in m)] +\
  671. [m for m in models if 'ength' in m and ('visual' in m or 'first' in m)] +\
  672. [m for m in models if 'ength' in m and ('utter' in m or 'second' in m)] +\
  673. [m for m in models if 'ength' in m and ('sum' in m)]
  674. try:
  675. assert len(sorted_models) in [3, 4, 6, 8]
  676. except AssertionError:
  677. print(case)
  678. print(models)
  679. continue
  680. xs = list(range(len(sorted_models)))
  681. if len(conds) == 2:
  682. corrections = list(numpy.linspace(-.33, .33, len(conds)))
  683. txt_corrections = list(numpy.linspace(-.4, .4, len(conds)))
  684. m_sc = 2000
  685. t_s = 20
  686. else:
  687. corrections = list(numpy.linspace(-.5, .5, len(conds)))
  688. txt_corrections = list(numpy.linspace(-.55, .55, len(conds)))
  689. m_sc = 1400
  690. t_s = 15
  691. fig, ax = pyplot.subplots(constrained_layout=True, figsize=(20, 10))
  692. x_shift = 0
  693. xticks = list()
  694. counter = -1
  695. ps = list()
  696. for m_i, m in enumerate(sorted_models):
  697. if 'Resp' in m:
  698. gen_avg = numpy.average([v for _ in c_results.values() for __ in _.values() for v in __])
  699. gen_std = numpy.std([v for _ in c_results.values() for __ in _.values() for v in __])
  700. counter += 1
  701. for c_i, c in enumerate(conds):
  702. spec_line = [v for v in gen_line]
  703. spec_line.extend([m, c])
  704. if 'length' in m:
  705. color=colors['Word\nlength'][c_i]
  706. elif 'freq' in m:
  707. color=colors['Word\nfrequency'][c_i]
  708. else:
  709. color=colors[m][c_i]
  710. xticks.append((counter, m))
  712. if len(conds) == 2:
  713. w = 0.6
  714. else:
  715. w = 0.45
  716. if c_i == 0:
  717. comps = list()
  718. for other_i, other in enumerate(conds[1:]):
  719. two = c_results[other][m]
  720. ### collecting corrected pvalue
  721. for line in comparisons:
  722. if line[0] != mode:
  723. continue
  724. if line[1] != task:
  725. continue
  726. if line[2] != case:
  727. continue
  728. if line[3] != c.replace('\n', '$'):
  729. continue
  730. if line[4] != other.replace('\n', '$'):
  731. continue
  732. if line[5] != m.replace('\n', '$'):
  733. continue
  734. print(line)
  735. t_val = float(line[-3])
  736. raw_p = float(line[-2])
  737. p_val = float(line[-1])
  738. print(line)
  739. print('comp {}'.format(comparisons))
  740. ps.append((m, (c, other), p_val, t_val))
  741. fake_distr = ['na']
  742. comps.append('{}_{}_{}_{}_{}@{}'.format(c, other, raw_p, p_val, t_val, ','.join([str(v) for v in fake_distr])))
  743. spec_line.append('#'.join(comps))
  744. else:
  745. spec_line.append('na')
  746. for line in baseline:
  747. if line[0] != mode:
  748. continue
  749. if line[1] != task:
  750. continue
  751. if line[2] != case:
  752. continue
  753. if line[3] != c.replace('\n', '$'):
  754. continue
  755. if line[4] != m.replace('\n', '$'):
  756. continue
  757. t = float(line[-3])
  758. raw_p = float(line[-2])
  759. p = float(line[-1])
  760. ps.append((m, c, p, t))
  761. spec_line.extend([raw_p, p, t])
  762. ### bar
  763. ax.bar(
  764. m_i+corrections[c_i]+x_shift,
  765. numpy.average(c_results[c][m]),
  766. width=w,
  767. color=color,
  768. edgecolor='gray',
  769. zorder=2.
  770. )
  771. ax.bar(
  772. m_i+corrections[c_i]+x_shift,
  773. numpy.average(c_results[c][m]),
  774. width=w,
  775. fill=False,
  776. edgecolor='dimgray',
  777. zorder=3.
  778. )
  779. ax.errorbar(
  780. m_i+corrections[c_i]+x_shift,
  781. numpy.average(c_results[c][m]),
  782. yerr=numpy.std(c_results[c][m]),
  783. color='dimgray',
  784. capsize=5,
  785. zorder=3.
  786. )
  787. spec_line.extend([m_i+corrections[c_i]+x_shift, m_i+txt_corrections[c_i]+x_shift])
  788. spec_line.append(numpy.average(c_results[c][m]))
  789. spec_line.append(','.join([str(val) for val in c_results[c][m]]))
  790. spec_line.append(c_i)
  791. lines.append(spec_line)
  792. if len(c_results[c][m]) == 1000:
  793. alpha = 0.2
  794. elif len(c_results[c][m]) == 10000:
  795. alpha = 0.02
  796. ax.scatter(
  797. [m_i+corrections[c_i]+x_shift+(random.randrange(-m_sc, m_sc)*0.0001) for rand in range(len(c_results[c][m]))],
  798. c_results[c][m],
  799. color=color,
  800. edgecolor='white',
  801. alpha=alpha,
  802. zorder=2.5
  803. )
  804. ax.text(
  805. m_i+txt_corrections[c_i]+x_shift,
  806. -.08,
  807. s=c,
  808. fontsize=t_s,
  809. ha='center',
  810. va='center',
  811. )
  812. x_shift += 1
  813. counter += 1
  814. if m_i in [11, 33]:
  815. x_shift += 1
  816. counter += 1
  817. nonshaded = list()
  818. ### absolute p-values
  819. corr_ps = mne.stats.fdr_correction([v[2] for v in ps])[1]
  820. corr_ps = [(ps[i][0], ps[i][1], ps[i][2], ps[i][3]) for i, p in enumerate(corr_ps) if type(ps[i][1]!=tuple)]
  821. x_shift = 0
  822. counter = -1
  823. for m_i, m in enumerate(sorted_models):
  824. counter += 1
  825. for c_i, c in enumerate(conds):
  826. for pm, pc, pp, t in corr_ps:
  827. if pm == m and pc == c:
  828. if abs(t)>lo_e:
  829. print('FOTTUTO D {}'.format(t))
  830. if pp < 0.005 and abs(t)>lo_e:
  831. print(pp)
  832. ax.scatter(
  833. m_i+corrections[c_i]+x_shift-.075,
  834. 0.01,
  835. color='black',
  836. edgecolor='white',
  837. zorder=3.,
  838. marker='*',
  839. s=300
  840. )
  841. ax.scatter(
  842. m_i+corrections[c_i]+x_shift+.075,
  843. 0.01,
  844. color='black',
  845. edgecolor='white',
  846. zorder=3.,
  847. marker='*',
  848. s=300
  849. )
  850. elif (str(pp)[:4] == '0.05' or pp<0.05) and abs(t)>lo_e:
  851. print(pp)
  852. ax.scatter(
  853. m_i+corrections[c_i]+x_shift,
  854. 0.01,
  855. color='black',
  856. edgecolor='white',
  857. zorder=3.,
  858. marker='*',
  859. s=400
  860. )
  861. x_shift += 1
  862. counter += 1
  863. if m_i in [11, 33]:
  864. x_shift += 1
  865. counter += 1
  866. print('nonshaded {}'.format(nonshaded))
  867. ### relative p-values
  868. corr_ps = mne.stats.fdr_correction([v[2] for v in ps])[1]
  869. corr_ps = [(ps[i][0], ps[i][1], ps[i][2], ps[i][3]) for i, p in enumerate(corr_ps) if type(ps[i][1])==tuple]
  870. x_shift = 0
  871. counter = -1
  872. for m_i, m in enumerate(sorted_models):
  873. counter += 1
  874. for c_i, c in enumerate(conds):
  875. for pm, pc, pp, t in corr_ps:
  876. alpha = 1.
  877. if pm == m and pc[0] == c:
  878. for other_i, other in enumerate(conds[1:]):
  879. if pc[1] == other:
  880. if 'length' in m:
  881. color=colors['Word\nlength'][other_i+1]
  882. elif 'freq' in m:
  883. color=colors['Word\nfrequency'][other_i+1]
  884. else:
  885. color=colors[m][c_i]
  886. if pp < 0.05 and abs(t)>lo_e:
  887. ax.vlines(
  888. ymin=0.41-(other_i*0.03),
  889. ymax=0.41-(other_i*0.03)-0.01,
  890. x=m_i+corrections[c_i]+x_shift,
  891. color=color,
  892. alpha=alpha,
  893. zorder=3.,
  894. linewidth=5.
  895. )
  896. ax.vlines(
  897. ymin=0.41-(other_i*0.03),
  898. ymax=0.41-(other_i*0.03)-0.01,
  899. x=m_i+corrections[c_i+1+other_i]+x_shift,
  900. color=color,
  901. alpha=alpha,
  902. zorder=3.,
  903. linewidth=5.
  904. )
  905. ax.hlines(
  906. xmin=m_i+corrections[c_i]+x_shift-0.025,
  907. xmax=m_i+corrections[c_i+1+other_i]+x_shift+0.03,
  908. y=0.41-(other_i*0.03),
  909. color=color,
  910. zorder=3.,
  911. alpha=alpha,
  912. linewidth=5.
  913. )
  914. xmin=m_i+corrections[c_i]+x_shift
  915. xmax=m_i+corrections[c_i+1+other_i]+x_shift
  916. middle = xmin + ((xmax-xmin)*.5)
  917. if pp < 0.005 and abs(t)>lo_e:
  918. if abs(t) > hi_e:
  919. ax.text(
  920. middle-0.275,
  921. y=0.42-(other_i*0.03),
  922. s='L',
  923. fontweight='bold',
  924. fontsize=15,
  925. color=color,
  926. ha='center',
  927. va='center',
  928. )
  929. elif abs(t) > mid_e:
  930. ax.text(
  931. middle-0.275,
  932. y=0.42-(other_i*0.03),
  933. s='M',
  934. fontweight='bold',
  935. fontsize=15,
  936. ha='center',
  937. va='center',
  938. color=color,
  939. )
  940. elif abs(t) > lo_e:
  941. ax.text(
  942. middle-0.275,
  943. y=0.42-(other_i*0.03),
  944. s='S',
  945. fontweight='bold',
  946. fontsize=15,
  947. ha='center',
  948. va='center',
  949. color=color,
  950. )
  951. print(pp)
  952. ax.scatter(
  953. middle-0.075,
  954. y=0.42-(other_i*0.03),
  955. color=color,
  956. edgecolor='gray',
  957. zorder=3.,
  958. marker='*',
  959. alpha=alpha,
  960. s=300
  961. )
  962. ax.scatter(
  963. middle+0.075,
  964. y=0.42-(other_i*0.03),
  965. color=color,
  966. edgecolor='gray',
  967. alpha=alpha,
  968. zorder=3.,
  969. marker='*',
  970. s=300
  971. )
  972. elif pp < 0.05 and abs(t)>lo_e:
  973. if abs(t) > hi_e:
  974. ax.text(
  975. middle-0.15,
  976. y=0.42-(other_i*0.03),
  977. s='L',
  978. fontweight='bold',
  979. fontsize=15,
  980. ha='center',
  981. va='center',
  982. color=color,
  983. )
  984. elif abs(t) > mid_e:
  985. ax.text(
  986. middle-0.15,
  987. y=0.42-(other_i*0.03),
  988. s='M',
  989. color=color,
  990. fontweight='bold',
  991. fontsize=15,
  992. ha='center',
  993. va='center',
  994. )
  995. elif abs(t) > lo_e:
  996. ax.text(
  997. middle-0.15,
  998. y=0.42-(other_i*0.03),
  999. s='S',
  1000. color=color,
  1001. fontweight='bold',
  1002. fontsize=15,
  1003. ha='center',
  1004. va='center',
  1005. )
  1006. print(pp)
  1007. ax.scatter(
  1008. middle,
  1009. y=0.42-(other_i*0.03),
  1010. color=color,
  1011. edgecolor='gray',
  1012. alpha=alpha,
  1013. zorder=3.,
  1014. marker='*',
  1015. s=300
  1016. )
  1017. x_shift += 1
  1018. counter += 1
  1019. if m_i in [11, 33]:
  1020. x_shift += 1
  1021. counter += 1
  1022. ax.set_ylim(bottom=-.1, top=.43)
  1023. ax.hlines(xmin=-.8, xmax=len(sorted_models)+x_shift-1.2, color='black', y=0)
  1024. ax.hlines(xmin=-.8, xmax=len(sorted_models)+x_shift-1.2, color='silver',alpha=0.5,linestyle='dashed', y=[y*0.01 for y in range(-5, 40, 5)], zorder=1)
  1025. pyplot.ylabel('Spearman correlation (RT-model)', fontsize=23)
  1026. pyplot.xticks(
  1027. [x[0] for x in xticks],
  1028. [x[1] for x in xticks],
  1029. fontsize=25,
  1030. fontweight='bold')
  1031. print(curr_fold)
  1032. pyplot.savefig(os.path.join(curr_fold, '{}.jpg'.format(case)), dpi=300)
  1033. pyplot.savefig(os.path.join(curr_fold, '{}.svg'.format(case)),)
  1034. pyplot.clf()
  1035. pyplot.close()
  1036. ### writing to file
  1037. with open(os.path.join(curr_fold, '{}.txt'.format(case)), 'w') as o:
  1038. o.write('mode\tcorpus\tlang\ttask\tcase\tmodel\tcondition\t')
  1039. o.write('comparisons\t')
  1040. o.write('raw_p\tfdr_corrected_p_val\tt_val\tx_bar\tx_label\tperms_avg\tperms\trel_idx\n')
  1041. for lin in lines:
  1042. for li in lin:
  1043. o.write('{}\t'.format(str(li).replace('\n', '$')))
  1044. o.write('\n')
  1046. font_folder = '../../fonts'
  1047. font_setup(font_folder)
  1049. plot_info = {
  1050. 'bootstrap' : [
  1051. {
  1052. 'figsize' : (16, 10),
  1053. 'correction' : (.7, 1.),
  1054. 'jump' : 6.5,
  1055. 'font_size' : 16,
  1056. 'title_size' : 25,
  1057. 'width' : (0.4, 0.4),
  1058. 'xticks' : [2., 7.5],
  1059. 'xlabels' : ['Picture naming w/\ninterference', 'Semantic production'],
  1060. },
  1061. {
  1062. 'figsize' : (16, 10),
  1063. 'correction' : (.7, .7, .7),
  1064. 'jump' : 6,
  1065. 'font_size' : 15,
  1066. 'title_size' : 25,
  1067. 'width' : (0.4, 0.4, 0.4),
  1068. 'xticks' : [1.5, 7.5, 13.5],
  1069. 'xlabels' : ['Semantic relatedness\njudgement', 'Action feature\njudgement', 'Sound feature\njudgement'],
  1070. },
  1071. #{
  1072. #'figsize' : (8, 10),
  1073. #'correction' : (.7,),
  1074. #'jump' : 0.,
  1075. #'font_size' : 16,
  1076. #'title_size' : 25,
  1077. #'width' : (0.4,),
  1078. #'xticks' : [1.5],
  1079. #'xlabels' : ['Semantic relatedness\njudgement'],
  1080. #},
  1081. {
  1082. 'figsize' : (16, 10),
  1083. 'correction' : (1., 1.),
  1084. 'jump' : 8,
  1085. 'font_size' : 15,
  1086. 'title_size' : 25,
  1087. 'width' : (0.4, 0.4),
  1088. 'xticks' : [2., 10.],
  1089. 'xlabels' : ['Semantic priming\n(quantity)', 'Semantic priming\n(social)'],
  1090. },
  1091. ],
  1092. 'residualize' : [
  1093. {
  1094. 'figsize' : (16, 10),
  1095. 'correction' : (.75, .9, .75, .75, .75, .9, .9),
  1096. 'jump' : 2.,
  1097. 'font_size' : 15,
  1098. 'title_size' : 20,
  1099. 'width' : (0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4),
  1100. 'xticks' : [0., 2., 4., 6., 8., 10., 12.],
  1101. 'xlabels' : ['Picture\nnaming w/\ninterference', 'Semantic\nproduction','Semantic\nrelatedness\njudgement', 'Action\nfeature\njudgement', 'Sound\nfeature\njudgement', 'Semantic\npriming\n(quantity)', 'Semantic\npriming\n(social)'],
  1102. },
  1103. ],
  1104. }
  1106. corpus = 'wac'
  1107. plots = {
  1108. 'bootstrap' : [
  1109. ('pmtg-production','sem-phon'),
  1110. ('distr-learn', 'sound-action_A', 'sound-action_S'),
  1111. ('social-quantity_quantity', 'social-quantity_social'),
  1112. ],
  1113. 'residualize' : [
  1114. ('pmtg-production', 'sem-phon',
  1115. 'distr-learn',
  1116. 'sound-action_A', 'sound-action_S',
  1117. 'social-quantity_quantity', 'social-quantity_social',
  1118. ),
  1119. ],
  1120. }
  1122. results = dict()
  1123. fold = os.path.join('plots', 'corrected', 'ppmi')
  1124. for root, direc, fz in os.walk(fold):
  1125. for f in fz:
  1126. if 'txt' not in f:
  1127. continue
  1128. splt_root = root.split('/')
  1129. if 'wac' not in splt_root:
  1130. continue
  1131. dataset = splt_root[-2]
  1132. if 'sound' in dataset:
  1133. full_task = '{}_{}'.format(dataset, splt_root[-1][0])
  1134. elif 'social' in dataset:
  1135. full_task = '{}_{}'.format(dataset, splt_root[-1].split('-')[0])
  1136. else:
  1137. full_task = '{}'.format(dataset)
  1138. approach = splt_root[3]
  1139. if approach not in results.keys():
  1140. results[approach] = dict()
  1141. if full_task not in results[approach].keys():
  1142. results[approach][full_task] = list()
  1143. with open(os.path.join(root, f)) as i:
  1144. for l_i, l in enumerate(i):
  1145. line = l.strip().split('\t')
  1146. if l_i == 0:
  1147. header = line.copy()
  1148. #print(header)
  1149. continue
  1150. ### just checking
  1151. task = line[header.index('task')]
  1152. case = line[header.index('case')]
  1153. if 'sound' in task:
  1154. new_full_task = '{}_{}'.format(task, case[0])
  1155. elif 'social' in task:
  1156. new_full_task = '{}_{}'.format(task, case.split('-')[0])
  1157. else:
  1158. new_full_task = '{}'.format(task)
  1159. assert new_full_task == full_task
  1160. ### collecting results
  1161. cond = line[header.index('condition')]
  1162. model = line[header.index('model')]
  1163. p_raw = float(line[header.index('raw_p')])
  1164. p_fdr = float(line[header.index('fdr_corrected_p_val')])
  1165. t_val = float(line[header.index('t_val')])
  1166. bar_y = float(line[header.index('perms_avg')])
  1167. bar_x = float(line[header.index('x_bar')])
  1168. rel_idx = int(line[header.index('rel_idx')])
  1169. label_x = float(line[header.index('x_label')])
  1170. perms = [float(val) for val in line[header.index('perms')].split(',')]
  1171. assert len(perms) in [1000, 10000]
  1172. #print([model, cond])
  1173. comparisons = [val.split('@')[0].split('_') for val in line[header.index('comparisons')].split('#')]
  1174. curr_dict = {
  1175. 'cond' : cond,
  1176. 'model' : model,
  1177. 'p_raw' : p_raw,
  1178. 'p_fdr' : p_fdr,
  1179. 't_val' : t_val,
  1180. 'bar_y' : bar_y,
  1181. 'bar_x' : bar_x,
  1182. 'label_x' : label_x,
  1183. 'rel_idx' : rel_idx,
  1184. 'perms' : perms,
  1185. 'comparisons' : comparisons,
  1186. }
  1187. print(os.path.join(root, f))
  1188. results[approach][full_task].append(curr_dict)
  1189. ### just checking...
  1190. for a, a_r in results.items():
  1191. for t, t_r in a_r.items():
  1192. assert len(t_r) in [6, 9]
  1193. out_f = os.path.join('plots', 'main_text')
  1194. os.makedirs(out_f, exist_ok=True)
  1195. for plot, cases in plots.items():
  1196. colors = set_colors(plot)
  1197. plot_n = 0
  1198. for datasets in cases:
  1199. print(datasets)
  1200. plot_n += 1
  1201. figsize = plot_info[plot][plot_n-1]['figsize']
  1202. out = os.path.join(out_f, 'R{:02}_{}.jpg'.format(plot_n, plot))
  1203. #figsize=(13, 10)
  1204. fig, ax = pyplot.subplots(figsize=figsize, constrained_layout=True)
  1205. ax.set_ylim(bottom=-.09, top=.42)
  1206. ax.spines['bottom'].set_visible(False)
  1207. ax.spines['right'].set_visible(False)
  1208. ax.spines['top'].set_visible(False)
  1209. pyplot.ylabel('Spearman correlation (RT-model)', fontsize=23)
  1210. pyplot.yticks(fontsize=20)
  1211. lgnds = list()
  1212. comps = list()
  1213. mapper = dict()
  1214. xs = list()
  1215. for start_i, dataset in enumerate(datasets):
  1216. all_data = results[plot][dataset]
  1217. w = plot_info[plot][plot_n-1]['width'][start_i]
  1218. jump = plot_info[plot][plot_n-1]['jump']
  1219. corr = plot_info[plot][plot_n-1]['correction'][start_i]
  1220. t_s = plot_info[plot][plot_n-1]['font_size']
  1221. title_size = plot_info[plot][plot_n-1]['title_size']
  1222. m_sc = 1000
  1223. start = start_i * jump
  1224. for data in all_data:
  1225. label = data['cond']
  1226. model = data['model']
  1227. if plot == 'residualize' and ('length' in model or 'frequency' in model):
  1228. continue
  1229. p_abs = data['p_fdr']
  1230. t_abs = data['t_val']
  1231. perms = data['perms']
  1232. rel_idx = data['rel_idx']
  1233. color = colors[model][rel_idx]
  1234. if rel_idx == 0:
  1235. if plot == 'residualize':
  1236. label_mapper = {
  1237. 'GPT2$surprisal' : 'GPT2 surprisal',
  1238. 'wac$PPMI' : 'Semantic dissimilarity (beyond word length and freq.)',
  1239. 'Word$frequency$(neg. sum)' : 'Word frequency (neg.)',
  1240. 'Word$length$(sum)': 'Word length',
  1241. }
  1242. else:
  1243. label_mapper = {
  1244. 'GPT2$surprisal' : 'GPT2 surprisal',
  1245. 'wac$PPMI' : 'Semantic dissimilarity',
  1246. #'Word$frequency' : 'Word frequency (neg.)',
  1247. #'Word$length': 'Word length',
  1248. 'Word$frequency$(neg. sum)' : 'Word frequency (neg.)',
  1249. 'Word$length$(sum)': 'Word length',
  1250. }
  1251. if 'distr' not in dataset:
  1252. if start_i in [0, 2] and ('length' in model or 'frequency' in model):
  1253. pass
  1254. else:
  1255. legend_label = label_mapper[model]
  1256. if legend_label not in lgnds:
  1257. lgnds.append(legend_label)
  1258. lgnd_w = 2.
  1259. ax.bar(0, 0, color=color, label=legend_label)
  1260. else:
  1261. legend_label = label_mapper[model]
  1262. if legend_label not in lgnds:
  1263. lgnds.append(legend_label)
  1264. lgnd_w = 0.6
  1265. ax.bar(0, 0, color=color, label=legend_label)
  1266. x_bar = data['bar_x']*corr
  1267. xs.append(x_bar+start)
  1268. x_label = data['label_x']*corr
  1269. mapper[(label, model, dataset)] = (x_bar+start, rel_idx)
  1270. for cmpr in data['comparisons']:
  1271. if cmpr != ['na']:
  1272. print(cmpr)
  1273. p_comp = float(cmpr[-2])
  1274. t_comp = float(cmpr[-1])
  1275. if p_comp < 0.05 and abs(t_comp)>lo_e:
  1276. comps.append([
  1277. label,
  1278. model,
  1279. dataset,
  1280. cmpr[0],
  1281. cmpr[1],
  1282. p_comp,
  1283. t_comp,
  1284. ])
  1285. ax.bar(
  1286. x_bar+(start),
  1287. numpy.average(perms),
  1288. width=w,
  1289. color=color,
  1290. edgecolor='gray',
  1291. zorder=1.5
  1292. )
  1293. ax.bar(
  1294. x_bar+(start),
  1295. numpy.average(perms),
  1296. width=w,
  1297. fill=False,
  1298. edgecolor='dimgray',
  1299. zorder=2.5
  1300. )
  1301. ax.errorbar(
  1302. x_bar+(start),
  1303. numpy.average(perms),
  1304. yerr=numpy.std(perms),
  1305. color='dimgray',
  1306. capsize=5,
  1307. zorder=3.
  1308. )
  1309. ax.scatter(
  1310. [x_bar+start+(random.randrange(-m_sc, m_sc)*0.0001) for rand in range(len(perms))],
  1311. perms,
  1312. color=color,
  1313. edgecolor='white',
  1314. alpha=0.2,
  1315. zorder=2.
  1316. )
  1317. if rel_idx == 0:
  1318. ax.text(
  1319. x_label+start,
  1320. -.02,
  1321. s=label.replace('$', '\n'),
  1322. fontsize=t_s,
  1323. #fontweight='bold',
  1324. ha='center',
  1325. va='center',
  1326. )
  1327. ### absolute p-value
  1328. if p_abs < 0.005 and abs(t_abs)>lo_e:
  1329. '''
  1330. if abs(t_abs) > 0.8:
  1331. ax.text(
  1332. x_bar+start,
  1333. 0.035,
  1334. s='H',
  1335. fontweight='bold',
  1336. fontsize=15,
  1337. color='black',
  1338. ha='center',
  1339. va='center',
  1340. )
  1341. elif abs(t_abs) > 0.5:
  1342. ax.text(
  1343. x_bar+start,
  1344. 0.035,
  1345. s='M',
  1346. fontweight='bold',
  1347. fontsize=15,
  1348. ha='center',
  1349. va='center',
  1350. color='black',
  1351. )
  1352. '''
  1353. ax.scatter(
  1354. x_bar+start-0.08,
  1355. 0.02,
  1356. color='black',
  1357. edgecolor='white',
  1358. zorder=3.,
  1359. marker='*',
  1360. s=200
  1361. )
  1362. ax.scatter(
  1363. x_bar+start+0.08,
  1364. 0.02,
  1365. color='black',
  1366. edgecolor='white',
  1367. zorder=3.,
  1368. marker='*',
  1369. s=200
  1370. )
  1371. elif (str(p_abs)[:4] == '0.05' or p_abs<0.05) and abs(t_abs)>lo_e:
  1372. '''
  1373. if abs(t_abs) > 0.8:
  1374. ax.text(
  1375. x_bar+start,
  1376. 0.035,
  1377. s='H',
  1378. fontweight='bold',
  1379. fontsize=15,
  1380. color='black',
  1381. ha='center',
  1382. va='center',
  1383. )
  1384. elif abs(t_abs) > 0.5:
  1385. ax.text(
  1386. x_bar+start,
  1387. 0.035,
  1388. s='M',
  1389. fontweight='bold',
  1390. fontsize=15,
  1391. ha='center',
  1392. va='center',
  1393. color='black',
  1394. )
  1395. '''
  1396. ax.scatter(
  1397. x_bar+start,
  1398. 0.02,
  1399. color='black',
  1400. edgecolor='white',
  1401. zorder=3.,
  1402. marker='*',
  1403. s=200
  1404. )
  1405. ### comparative p-value
  1406. for l, m, d, start, end, p, t in comps:
  1407. st, st_idx = mapper[(start, m, d)]
  1408. nd, nd_idx = mapper[(end, m, d)]
  1409. ax.vlines(
  1410. ymin=0.39-(nd_idx*0.03),
  1411. ymax=0.39-(nd_idx*0.03)-0.01,
  1412. x=st,
  1413. color=colors[m][nd_idx],
  1414. #alpha=alpha,
  1415. zorder=3.,
  1416. linewidth=5.
  1417. )
  1418. ax.vlines(
  1419. ymin=0.39-(nd_idx*0.03),
  1420. ymax=0.39-(nd_idx*0.03)-0.01,
  1421. x=nd,
  1422. color=colors[m][nd_idx],
  1423. #alpha=alpha,
  1424. zorder=3.,
  1425. linewidth=5.
  1426. )
  1427. ax.hlines(
  1428. xmin=st-0.025,
  1429. xmax=nd+0.03,
  1430. y=0.39-(nd_idx*0.03),
  1431. color=colors[m][nd_idx],
  1432. zorder=3.,
  1433. #alpha=alpha,
  1434. linewidth=5.
  1435. )
  1436. if p < 0.005:
  1437. if abs(t) > hi_e:
  1438. ax.text(
  1439. st+((nd-st)*.5)-0.28,
  1440. .3975-(nd_idx*0.03),
  1441. s='L',
  1442. fontweight='bold',
  1443. fontsize=15,
  1444. color=colors[m][0],
  1445. ha='center',
  1446. va='center',
  1447. )
  1448. elif abs(t) > mid_e:
  1449. ax.text(
  1450. st+((nd-st)*.5)-0.28,
  1451. .3975-(nd_idx*0.03),
  1452. s='M',
  1453. fontweight='bold',
  1454. fontsize=15,
  1455. ha='center',
  1456. va='center',
  1457. color=colors[m][0],
  1458. )
  1459. elif abs(t) > lo_e:
  1460. ax.text(
  1461. st+((nd-st)*.5)-0.28,
  1462. .3975-(nd_idx*0.03),
  1463. s='S',
  1464. fontweight='bold',
  1465. fontsize=15,
  1466. ha='center',
  1467. va='center',
  1468. color=colors[m][0],
  1469. )
  1470. ax.scatter(
  1471. st+((nd-st)*.5)-0.08,
  1472. .3975-(nd_idx*0.03),
  1473. color=colors[m][0],
  1474. edgecolor='gray',
  1475. zorder=3.,
  1476. marker='*',
  1477. s=200
  1478. )
  1479. ax.scatter(
  1480. st+((nd-st)*.5)+0.08,
  1481. .3975-(nd_idx*0.03),
  1482. color=colors[m][0],
  1483. edgecolor='gray',
  1484. zorder=3.,
  1485. marker='*',
  1486. s=200
  1487. )
  1488. elif p < 0.05:
  1489. if abs(t) > hi_e:
  1490. ax.text(
  1491. st+((nd-st)*.5)-0.2,
  1492. .3975-(nd_idx*0.03),
  1493. s='L',
  1494. fontweight='bold',
  1495. fontsize=15,
  1496. color=colors[m][0],
  1497. ha='center',
  1498. va='center',
  1499. )
  1500. elif abs(t) > mid_e:
  1501. ax.text(
  1502. st+((nd-st)*.5)-0.2,
  1503. .3975-(nd_idx*0.03),
  1504. s='M',
  1505. fontweight='bold',
  1506. fontsize=15,
  1507. ha='center',
  1508. va='center',
  1509. color=colors[m][0],
  1510. )
  1511. elif abs(t) > lo_e:
  1512. ax.text(
  1513. st+((nd-st)*.5)-0.2,
  1514. .3975-(nd_idx*0.03),
  1515. s='S',
  1516. fontweight='bold',
  1517. fontsize=15,
  1518. ha='center',
  1519. va='center',
  1520. color=colors[m][0],
  1521. )
  1522. ax.scatter(
  1523. st+((nd-st)*.5),
  1524. .3975-(nd_idx*0.03),
  1525. color=colors[m][0],
  1526. edgecolor='gray',
  1527. zorder=3.,
  1528. marker='*',
  1529. s=200
  1530. )
  1531. ax.hlines(
  1532. xmin=min(xs),
  1533. xmax=max(xs),
  1534. color='black',
  1535. y=0,
  1536. )
  1537. ax.hlines(
  1538. xmin=min(xs),
  1539. xmax=max(xs),
  1540. color='silver',
  1541. alpha=0.5,
  1542. linestyle='dashed',
  1543. y=[y*0.01 for y in range(-5, 40, 5)],
  1544. zorder=1,
  1545. )
  1546. ax.legend(ncol=4, loc=9, columnspacing=lgnd_w, fontsize=17)
  1547. ax.set_xticks(
  1548. plot_info[plot][plot_n-1]['xticks'],
  1549. plot_info[plot][plot_n-1]['xlabels'],
  1550. fontsize = title_size,
  1551. fontweight = 'bold',
  1552. linespacing=1.5,
  1553. #fontstyle='italic',
  1554. )
  1555. ax.tick_params(axis='x',pad=30)
  1556. pyplot.savefig(out, dpi=600)
  1557. pyplot.clf()
  1558. pyplot.close()

plot01_gen_all_barplots.py, no license · at the source

Overview

Authors: Andrea Bruera1, Gesa Hartwigsen1,2
ORCID iDs: Andrea Bruera, Gesa Hartwigsen
  1. Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
  2. Cognitive and Biological Psychology, Wilhelm Wundt Institute for Psychology, Leipzig University, Leipzig, Germany
Institutions: Max Planck Institute for Human Cognitive and Brain Sciences (Germany); Leipzig University (Germany)
Journal: Neurobiology of language (Cambridge, Mass.), volume 7, article NOL.a.244
Dates: received 4 July 2025; accepted 5 February 2026; published online 5 May 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1162/nol.a.244 · PMID 42137738 · PMCID PMC13171204 · OpenAlex W7128715304
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: other (modality)
Methods: Connectivity, Statistics, Machine learning
Keywords: brain stimulation, causality, language models, semantics, semantic models, semantic similarity, surprisal, transcranial magnetic stimulation (TMS)
Topic: Transcranial Magnetic Stimulation Studies (Neurology, Neuroscience), according to OpenAlex
Funding: EC | EU Framework Programme for Research and Innovation H2020 | H2020 Priority Excellent Science | H2020 European Research Council (H2020 Excellent Science - European Research Council) (ERC-COG-2021-101043747); Deutsche Forschungsgemeinschaft (DFG) (HA 6314/4-2, HA 6314/10-1)
Citations: not cited yet (Europe PMC); 167 references in the paper

Abstract

Noninvasive brain stimulation studies on semantic cognition hold the promise of revealing the functional relevance of brain areas through causal intervention. A primary challenge, however, is that findings are often interpreted through binary distinctions between sets of stimuli (e.g., related/unrelated words, same/different semantic category). This approach ignores the analysis of individual words, which mirrors every day language use and is crucial for understanding semantic cognition. In this work, we used semantic similarity, as measured by a language model, to investigate how transcranial magnetic stimulation (TMS) effects on semantic cognition unfold at the level of individual words. We reanalyzed five publicly available TMS data sets, covering multiple stimulation sites and lexical semantics tasks. We propose a simple methodology that can straightforwardly be applied to any TMS experiment on semantic cognition and showcase its potential to generate new insights. We modeled trial-level response times using the language model and computed the correlation between the two. We also repeated the analyses for two lower-level variables (word frequency and length). Importantly, for each data set, we compared correlations for effective and control (sham or vertex) stimulation conditions. We found that, for the language model, correlation was almost always significantly different depending on the type of stimulation (effective or control). Our results provide evidence that the stimulation effect interacts with the meaning of individual words. However, a similar pattern emerged in some cases for word frequency and length, suggesting that the effects of TMS on cognition can be widespread, well beyond their intended functional target. Collectively, our results demonstrate that language models provide new insight into the impact of neurostimulation on semantic processing, complementing standard measures.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

OSF r3pwk

License: none: the authors keep all their rights
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Languages: Python (17)
Size: 35 files, 17 scripts
Software Heritage: not checked
Found in: “DATA AND CODE AVAILABILITY STATEMENT”
Holds: environment (requirements.txt), tests
Not found: README, license file, CITATION.cff, continuous integration, documentation
Tools: NumPy (11 files), Matplotlib (4 files), SciPy (4 files), Pingouin (2 files), MNE-Python (1 file), PyTorch (1 file), scikit-learn (1 file), Hugging Face Transformers (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers (HTTP 200)
  • 28 September 2026: the link answers (HTTP 200)
17 files
At the source: osf.io/r3pwk

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 17 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data and code availability statement

All the code, models, and (previously published) experimental data are available on the Open Science Framework website: https://osf.io/r3pwk.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 28 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 2 authors, 8 keywords, 2 funders, 162 references.

Cite

This paper

Bruera, A., & Hartwigsen, G. (2026). A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models. Neurobiology of language (Cambridge, Mass.), 7, NOL.a.244. https://doi.org/10.1162/nol.a.244

BibTeX

@article{bruera2026novel,
author = {Bruera, Andrea and Hartwigsen, Gesa},
title = {{A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models}},
journal = {Neurobiology of language (Cambridge, Mass.)},
year = {2026},
month = may,
volume = {7},
pages = {NOL.a.244},
publisher = {MIT Press},
issn = {2641-4368},
doi = {10.1162/nol.a.244},
url = {https://doi.org/10.1162/nol.a.244},
pmid = {42137738},
pmcid = {PMC13171204}
}

RIS

TY - JOUR
AU - Bruera, Andrea
AU - Hartwigsen, Gesa
TI - A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models
T2 - Neurobiology of language (Cambridge, Mass.)
J2 - Neurobiol Lang (Camb)
PY - 2026
DA - 2026/05/05
VL - 7
SP - NOL.a.244
SN - 2641-4368
PB - MIT Press
DO - 10.1162/nol.a.244
UR - https://doi.org/10.1162/nol.a.244
LA - en
ER -

CSL-JSON

{
"id": "10.1162/nol.a.244",
"type": "article-journal",
"title": "A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models",
"container-title": "Neurobiology of language (Cambridge, Mass.)",
"author": [
{
"family": "Bruera",
"given": "Andrea"
},
{
"family": "Hartwigsen",
"given": "Gesa"
}
],
"container-title-short": "Neurobiol Lang (Camb)",
"volume": "7",
"page": "NOL.a.244",
"DOI": "10.1162/nol.a.244",
"PMID": "42137738",
"PMCID": "PMC13171204",
"ISSN": "2641-4368",
"publisher": "MIT Press",
"URL": "https://doi.org/10.1162/nol.a.244",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
5
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41467-026-75841-9 [code]
Model-based semantic distance reveals adaptive coordination of distinct cognitive systems in flexible knowledge retrieval.
Journal: Nature communications
In common: Hugging Face Transformers, PyTorch, NumPy, 11 references
[2] doi:10.3758/s13415-026-01439-9
Stimulation success!? Improved response inhibition performance after prefrontal single-site and condition-and-perturb transcranial magnetic stimulation.
Journal: Cognitive, affective & behavioral neuroscience
In common: other, 7 references, author Gesa Hartwigsen
[3] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Hugging Face Transformers, PyTorch, scikit-learn, 3 other tools, 8 references
[4] doi:10.1038/s41467-026-72253-7 [code]
Spurious alignment between large language models and brains can emerge from non-robust methods and overlooked confounds.
Journal: Nature communications
In common: Hugging Face Transformers, PyTorch, scikit-learn, 3 other tools, 5 references
[5] doi:10.7554/elife.106543 [code]
Stimulus dependencies-rather than next-word prediction-can explain pre-onset brain encoding in naturalistic listening designs.
Journal: eLife
In common: Hugging Face Transformers, MNE-Python, PyTorch, 4 other tools, 3 references
[6] doi:10.1162/imag.a.1227 [code]
Large language models reveal the neural tracking of linguistic context in attended and unattended multi-talker speech.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Hugging Face Transformers, MNE-Python, PyTorch, 4 other tools, 3 references
[7] doi:10.1038/s41586-026-10691-5 [code]
Mapping the neuronal building blocks of human language with language models.
Journal: Nature
In common: scikit-learn, SciPy, Matplotlib, 1 other tool, 6 references
[8] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: Hugging Face Transformers, PyTorch, scikit-learn, 3 other tools, 5 references
[9] doi:10.1038/s41598-026-41532-0 [code]
Prediction, syntax and semantic grounding in the brain and large language models.
Journal: Scientific reports
In common: Hugging Face Transformers, MNE-Python, PyTorch, 4 other tools, 2 references
[10] doi:10.7554/elife.109400 [code]
High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework.
Journal: eLife
In common: Hugging Face Transformers, PyTorch, SciPy, 2 other tools, 4 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.