OSCR

Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play.

Code ↔ Paper

3 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 3 matches
  1. [1] § Materials and Methods › Dataset › Shinobi videogame ↔ stage-2-genframes-bw.py, lines 116–173 · score 0.65 · Ninja Master, Shinobi III, gym, retro, frames
  2. [2] § Materials and Methods › Dataset ↔ stage-2-genframes-bw.py, lines 75–113 · score 0.55 · Ninja Master, Shinobi III
  3. [3] § Materials and Methods › Dataset › Data preprocessing ↔ stage-3-parcel-confounds9.py, lines 9–67 · score 0.55 · preprocessed, confounds, fMRIprep, FWHM, atlas, nilearn

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 240 lines · 6.8 KB · no license · 2 matches

  1. #!/usr/bin/env python
  2. # coding: utf-8
  4. # In[1]:
  7. import argparse
  8. from gym import wrappers
  9. import numpy as np
  10. import os
  11. import pickle
  12. import retro
  13. import sys
  14. import time
  15. import warnings
  16. from os import listdir
  17. from os.path import isfile, join
  18. import matplotlib.pyplot as plt
  19. import h5py
  20. import pandas as pd
  21. from PIL import Image, ImageOps
  24. # In[2]:
  25. ## HYPERPARAMETER CELL
  26. def main(args):
  28. subject= int(args[1])
  30. frame_skip = 5
  31. sequence_length = int(90/frame_skip)
  32. df_file_name_stage1 = '../temp_files/sub-'+str(subject)+'-stage-1-df.pkl'
  33. h5_store_name = '../data/bw-sub-'+str(subject)+'-len-'+str(sequence_length)+'.h5'
  34. runshapes_store_name = '../temp_files/sub-'+str(subject)+'-stage-2-runshapes-'+str(sequence_length)+'.pkl'
  35. bk2dir = '../data/shinobi.fmriprep/sourcedata/shinobi/'
  36. output_image_shape = (1,50,100)
  38. print('subject:',subject)
  40. def process_state(obs):
  41. obs = obs[50:-10,:,:] #crop
  42. obs = Image.fromarray(obs.astype(np.uint8))
  43. obs = ImageOps.grayscale(obs)
  44. obs = np.array(obs.resize((output_image_shape[2], output_image_shape[1])))
  45. obs = obs/255 # normalize
  46. obs = np.expand_dims(obs,2)
  47. obs = obs.transpose(2,0,1) # HWC to CHW
  48. return obs
  50. def process_action(action):
  51. '''
  52. Keys = [attack , special,?, ?, up, down, left, right, jump,?,?,?]
  53. '''
  55. # converting to 6 actions
  56. new_action = [0,0,0,0,0,0]
  58. if action[0]==True:
  59. new_action[0]=1
  60. if action[4]==True:
  61. new_action[1]=1
  62. if action[5]==True:
  63. new_action[2]=1
  64. if action[6]==True:
  65. new_action[3]=1
  66. if action[7]==True:
  67. new_action[4]=1
  68. if action[8]==True:
  69. new_action[5]=1
  71. new_action = int("".join(str(x) for x in new_action), 2)
  72. return new_action
  75. # In[3]:
  78. max_episode_length = 30000
  80. gym_folder = '/project/rrg-pbellec/ani686/env/lib/python3.6/site-packages/retro/data/stable/ShinobiIIIReturnOfTheNinjaMaster-Genesis/'
  81. # In[5]:
  84. allstates = np.empty((0,sequence_length,1, output_image_shape[1], output_image_shape[2]),dtype=np.float32)
  85. allactions = np.empty((0,sequence_length,1),dtype=np.float32)
  86. allsections = np.empty((0,1),dtype=np.float32)
  87. allruns = np.empty((0,1),dtype=np.float32)
  88. allreps = np.empty((0,1),dtype=np.float32)
  89. allrunnumber = np.empty((0,1),dtype=np.float32)
  93. hf = h5py.File(h5_store_name,'a')
  94. hf.create_dataset('state', data=allstates,chunks=True, maxshape=(None,sequence_length,1,output_image_shape[1], output_image_shape[2]))
  95. hf.create_dataset('action', data=allactions,chunks=True, maxshape=(None,sequence_length,1))
  96. hf.create_dataset('session', data=allsections,chunks=True, maxshape=(None,1))
  97. hf.create_dataset('run', data=allruns,chunks=True, maxshape=(None,1))
  98. hf.create_dataset('rep', data=allreps,chunks=True, maxshape=(None,1))
  99. hf.create_dataset('runnumber', data=allrunnumber,chunks=True, maxshape=(None,1))
  102. # In[6]:
  104. df = pd.read_pickle(df_file_name_stage1)
  107. # In[7]:
  110. all_files=[]
  111. for i in range(len(df)):
  112. t = bk2dir+df['bk2'][i]
  113. all_files.append(t)
  116. # In[ ]:
  119. cf=0
  120. all_shapes = []
  121. for f in all_files[:]:
  125. sess = df['session'][cf]
  126. run = df['run'][cf]
  127. rep = df['rep'][cf]
  128. level = df['level'][cf]
  129. runnumber = df['run_number'][cf]
  131. print(cf)
  132. cf+=1
  133. filename = f
  136. file = filename.replace('.bk2', '')
  137. key_log = retro.Movie(filename)
  140. frames=[]
  141. actions=[]
  142. times=[]
  144. if level==1:
  145. env = retro.make('ShinobiIIIReturnOfTheNinjaMaster-Genesis', state='Level1-0',scenario = gym_folder+'scenario1-0.json')
  146. if level==4:
  147. env = retro.make('ShinobiIIIReturnOfTheNinjaMaster-Genesis', state='Level4-1',scenario = gym_folder+'scenario4-1.json')
  148. if level==5:
  149. env = retro.make('ShinobiIIIReturnOfTheNinjaMaster-Genesis', state='Level5-0',scenario = gym_folder+'scenario5-0.json')
  151. state = env.reset()
  152. start_action=False
  154. for i in range(max_episode_length):
  156. if '.bk2' in filename:
  157. key_log.step()
  158. action = [key_log.get_key(i, 0) for i in range(env.num_buttons)]
  161. prev_state = process_state(state)
  162. dec_action = process_action(action)
  164. state, _, done, info = env.step( action )
  166. if True:
  167. if i%frame_skip == 0:
  168. frames.append(prev_state)
  169. actions.append(dec_action)
  172. if done:
  173. break
  179. if True:
  180. frames = np.float32(np.array(frames))
  181. actions_arr = np.float32(np.array(actions))
  182. actions_arr = actions_arr.reshape((actions_arr.shape[0],1))
  184. num_batches = int(frames.shape[0]/sequence_length)
  185. final_index = num_batches*sequence_length
  187. frames = frames[:final_index]
  188. frames = frames.reshape((-1,sequence_length,frames.shape[1],frames.shape[2],frames.shape[3]))
  189. actions_arr = actions_arr[:final_index]
  190. actions_arr = actions_arr.reshape((-1,sequence_length,1))
  192. sess_arr = np.array([sess]*actions_arr.shape[0]).reshape(-1,1)
  193. run_arr = np.array([run]*actions_arr.shape[0]).reshape(-1,1)
  194. rep_arr = np.array([rep]*actions_arr.shape[0]).reshape(-1,1)
  195. runnumber_arr = np.array([runnumber]*actions_arr.shape[0]).reshape(-1,1)
  198. hf["state"].resize((hf["state"].shape[0] + frames.shape[0]), axis = 0)
  199. hf["state"][-frames.shape[0]:] = frames
  201. hf["action"].resize((hf["action"].shape[0] + actions_arr.shape[0]), axis = 0)
  202. hf["action"][-actions_arr.shape[0]:] = actions_arr
  204. hf["session"].resize((hf["session"].shape[0] + sess_arr.shape[0]), axis = 0)
  205. hf["session"][-sess_arr.shape[0]:] = sess_arr
  207. hf["run"].resize((hf["run"].shape[0] + run_arr.shape[0]), axis = 0)
  208. hf["run"][-run_arr.shape[0]:] = run_arr
  210. hf["rep"].resize((hf["rep"].shape[0] + rep_arr.shape[0]), axis = 0)
  211. hf["rep"][-rep_arr.shape[0]:] = rep_arr
  213. hf["runnumber"].resize((hf["runnumber"].shape[0] + runnumber_arr.shape[0]), axis = 0)
  214. hf["runnumber"][-runnumber_arr.shape[0]:] = runnumber_arr
  217. print(frames.shape,sess,runnumber,run,rep,level,flush=True)
  218. all_shapes.append(frames.shape[0])
  219. del frames
  220. del actions
  221. del actions_arr
  223. env.close()
  224. hf.close()
  227. # In[ ]:
  229. with open(runshapes_store_name, 'wb') as f:
  230. pickle.dump(all_shapes, f)
  232. if __name__ == '__main__':
  233. main(sys.argv[:])

stage-2-genframes-bw.py at commit 0ef0d7c, no license · at the source

Overview

Authors: Anirudha Kemtur1,2,3, Francois Paugam1,2,3, Basile Pinsard3, Yann Harel3,4, Pravish Sainath1,2,3, Maximilien Le Clei3, Julie Boyle3, Karim Jerbi1,2,3,4, Lune Bellec1,2,3,4
  1. Computer Science Department, Université de Montréal, Montréal, Canada
  2. Mila – Quebec AI Institute, Montréal, Canada
  3. Centre de Recherche de l’Institut Universitaire de Gériatrie de Montréal, Montréal, Canada
  4. Psychology Department, Université de Montréal, Montréal, Canada
Institutions: Institut Universitaire de Gériatrie de Montréal (Canada); Mila - Quebec Artificial Intelligence Institute (Canada); Université de Montréal (Canada)
Journal: Imaging neuroscience (Cambridge, Mass.), volume 4, article IMAG.a.1286
Dates: received 24 July 2024; accepted 18 May 2026; published online 2 July 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1162/imag.a.1286 · PMID 42440925 · PMCID PMC13334379 · OpenAlex W4388127747
Open access: diamond, a free copy (OpenAlex)
Preprint: doi:10.1101/2023.10.28.564546
Status: code verified
Categories: fMRI (modality), human (organism)
Methods: Connectivity, Smoothing, state filtering, decompositions, Statistics, Machine learning, fMRI & imaging
Keywords: brain encoding, fMRI, imitation learning, naturalistic paradigms, personalized models
Topic: EEG and Brain-Computer Interfaces (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: Courtois Foundation; Mitacs
Citations: not cited yet (Europe PMC); 46 references in the paper

Abstract

Videogames provide a promising framework to understand brain activity in a rich, engaging, and active environment, in contrast to mostly passive tasks currently dominating the field, such as image viewing. Analyzing videogames neuroimaging data is, however, challenging, and relies on time-intensive manual annotations of game events, based on somewhat arbitrary rules. Here, we introduce an innovative approach using Artificial Neural networks (ANN) and brain encoding techniques to generate activation maps associated with videogame behavior using functional magnetic resonance imaging (fMRI). As individual behavior is highly variable across subjects in complex environments, we hypothesized that ANNs need to account for subject-specific behavior to capture brain dynamics properly. In this study, we used data collected while subjects played Shinobi III: Return of the Ninja Master (an action-platformer released by Sega in 1993), an action-platformer videogame. Using imitation learning, we trained an ANN to play the game while closely replicating the unique gameplay style of individual participants. We found that hidden layers of our imitation learning model successfully encoded task-relevant neural representations, and predicted individual brain dynamics with higher accuracy than models trained on other subjects’ gameplay. Individual-specific models also outperformed several baselines to predict brain activity, such as pixel inputs, or button presses. The highest correlations between layer activations and brain signals were observed in biologically plausible brain areas, that is, somatosensory, attention, and visual networks. This work thus demonstrates that subject-specific imitation models can be trained from scratch and improve brain encoding in an active naturalistic task. Our framework builds on a commercial videogame of unprecedented complexity in the fMRI brain encoding literature. We used a flexible game emulator that supports a broad range of commercial videogames, opening new naturalistic interactive environments for cognitive neuroscience.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 3 matches between paragraphs and lines of code.

anirudhk686/shinobi_imitation_learning

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 0ef0d7c7e90e6e77233d2714e0f1e96517e0341c, 14 October 2025
Languages: Jupyter (5), Python (4)
Size: 10 files, 9 scripts
Software Heritage: not archived
Found in: “Data and Code Availability”
Holds: README, 5 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (9 files), pandas (8 files), Matplotlib (7 files), h5py (6 files), Nilearn (5 files), PyTorch (5 files), scikit-learn (5 files), NiBabel (4 files), Pillow (1 file), SciPy (1 file), seaborn (1 file), statsmodels (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
10 files
At the source: github.com/anirudhk686/shinobi_imitation_learning

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 9 scripts, each with its path and the digest of its content;
  • 3 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data and Code Availability

Data used in the paper can be requested through Cneuromod website: https://www.cneuromod.ca/. The code for the study can be found in a dedicated GitHub repository: https://github.com/anirudhk686/shinobi_imitation_learning.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 9 authors, 5 keywords, 2 funders, 45 references.

Cite

This paper

Kemtur, A., Paugam, F., Pinsard, B., Harel, Y., Sainath, P., Clei, M. L., Boyle, J., Jerbi, K., & Bellec, L. (2026). Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play. Imaging neuroscience (Cambridge, Mass.), 4, IMAG.a.1286. https://doi.org/10.1162/imag.a.1286

BibTeX

@article{kemtur2026behavioral,
author = {Kemtur, Anirudha and Paugam, Francois and Pinsard, Basile and Harel, Yann and Sainath, Pravish and Clei, Maximilien Le and Boyle, Julie and Jerbi, Karim and Bellec, Lune},
title = {{Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play}},
journal = {Imaging neuroscience (Cambridge, Mass.)},
year = {2026},
month = jul,
volume = {4},
pages = {IMAG.a.1286},
publisher = {MIT Press},
issn = {2837-6056},
doi = {10.1162/imag.a.1286},
url = {https://doi.org/10.1162/imag.a.1286},
pmid = {42440925},
pmcid = {PMC13334379}
}

RIS

TY - JOUR
AU - Kemtur, Anirudha
AU - Paugam, Francois
AU - Pinsard, Basile
AU - Harel, Yann
AU - Sainath, Pravish
AU - Clei, Maximilien Le
AU - Boyle, Julie
AU - Jerbi, Karim
AU - Bellec, Lune
TI - Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play
T2 - Imaging neuroscience (Cambridge, Mass.)
J2 - Imaging Neurosci (Camb)
PY - 2026
DA - 2026/07/02
VL - 4
SP - IMAG.a.1286
SN - 2837-6056
PB - MIT Press
DO - 10.1162/imag.a.1286
UR - https://doi.org/10.1162/imag.a.1286
LA - en
ER -

CSL-JSON

{
"id": "10.1162/imag.a.1286",
"type": "article-journal",
"title": "Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play",
"container-title": "Imaging neuroscience (Cambridge, Mass.)",
"author": [
{
"family": "Kemtur",
"given": "Anirudha"
},
{
"family": "Paugam",
"given": "Francois"
},
{
"family": "Pinsard",
"given": "Basile"
},
{
"family": "Harel",
"given": "Yann"
},
{
"family": "Sainath",
"given": "Pravish"
},
{
"family": "Clei",
"given": "Maximilien Le"
},
{
"family": "Boyle",
"given": "Julie"
},
{
"family": "Jerbi",
"given": "Karim"
},
{
"family": "Bellec",
"given": "Lune"
}
],
"container-title-short": "Imaging Neurosci (Camb)",
"volume": "4",
"page": "IMAG.a.1286",
"DOI": "10.1162/imag.a.1286",
"PMID": "42440925",
"PMCID": "PMC13334379",
"ISSN": "2837-6056",
"publisher": "MIT Press",
"URL": "https://doi.org/10.1162/imag.a.1286",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
2
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1162/imag.a.1256 [code]
Gamer in the scanner: Event-related analysis of fMRI activity during retro videogame play guided by automated annotations of game content.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Nilearn, Pillow, statsmodels, 8 other tools, fMRI, 14 references
[2] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: h5py, Pillow, statsmodels, 8 other tools, 4 references
[3] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Nilearn, h5py, Pillow, 9 other tools, fMRI, 2 references
[4] doi:10.1016/j.isci.2026.117180 [code]
Developmental changes in similarity between neural representations of mental arithmetic and artificial neural networks.
Journal: iScience
In common: Nilearn, h5py, statsmodels, 8 other tools, fMRI, 3 references
[5] doi:10.1167/jov.26.5.7 [code]
Representations in vision and language converge in a shared, multidimensional space of perceived similarities.
Journal: Journal of vision
In common: Nilearn, h5py, Pillow, 9 other tools, 2 references
[6] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: h5py, Pillow, statsmodels, 8 other tools, 3 references
[7] doi:10.1162/nol.a.271 [code]
Compositional Complexity in Text and Images.
Journal: Neurobiology of language (Cambridge, Mass.)
In common: Nilearn, Pillow, NiBabel, 7 other tools, fMRI, 4 references
[8] doi:10.1162/imag.a.1309 [code]
Probing the content of semantic representations in body-selective regions.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: h5py, Pillow, statsmodels, 8 other tools, 2 references
[9] doi:10.1007/s12021-026-09803-3 [code]
NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.
Journal: Neuroinformatics
In common: Nilearn, h5py, Pillow, 7 other tools, fMRI, 2 references
[10] doi:10.1002/agm2.70073 [code]
Foreign Language Learning in Older Adults Modifies Resting-State Functional Connectivity Between the Subcortical Structures and the Cortex.
Journal: Aging medicine (Milton (N.S.W))
In common: Nilearn, h5py, Pillow, 9 other tools, fMRI

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.