OSCR

MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.

Code ↔ Paper

6 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 6 matches
  1. [1] § 2. Results ↔ src/final_evaluations_mi_multi.ipynb, lines 500–559 · score 0.77 · AlexNet, EffNet, InceptionV3, PixCorr, SwAV, brain correlation
  2. [2] § 2. Results ↔ src/plots_across_methods.sh, lines 1–47 · score 0.75 · AlexNet, EffNet, InceptionV3, PixCorr, SwAV, brain correlation
  3. [3] § 4. Methods › 4.2. MIRAGE › 4.2.6 Reconstruction. ↔ src/Train.ipynb, lines 214–238 · score 0.52 · Stable Cascade, text embeddings, image embeddings, latent, train, CLIP
  4. [4] § 4. Methods › 4.2. MIRAGE › 4.2.2. Methodology. ↔ src/Train.ipynb, lines 214–238 · score 0.51 · Stable Cascade, text embedding, image embedding, VDVAE, retrieval, GiT
  5. [5] § 4. Methods › 4.2. MIRAGE › 4.2.2. Methodology. ↔ src/recon_inference_mi.ipynb, lines 192–223 · score 0.51 · Stable Cascade, text embedding, image embedding, VDVAE, retrieval, GiT
  6. [6] § 4. Methods › 4.2. MIRAGE › 4.2.1. Datasets. ↔ src/utils.py, lines 766–850 · score 0.51 · visual cortex, NSD subjects, voxel, mask, stimuli, trained

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 439 lines · 17 KB · MIT · 2 matches

  1. # %% [markdown]
  2. # # Import packages & functions
  4. # %%
  5. import os
  6. import sys
  7. import json
  8. import argparse
  9. import numpy as np
  10. import math
  11. from einops import rearrange
  12. import time
  13. import random
  14. import string
  15. import h5py
  16. from tqdm import tqdm
  17. import webdataset as wds
  18. import gc
  19. from PIL import Image
  20. import matplotlib.pyplot as plt
  21. import torch
  22. import torch.nn as nn
  23. from torchvision import transforms
  24. from sklearn.linear_model import SGDRegressor
  25. from sklearn.preprocessing import StandardScaler
  26. from sklearn.utils import shuffle
  27. # tf32 data type is faster than standard float32
  28. torch.backends.cuda.matmul.allow_tf32 = True
  29. from sklearn.linear_model import Ridge
  30. import pickle
  31. # custom functions #
  32. import utils
  33. from sc_reconstructor import SC_Reconstructor
  34. from vdvae import VDVAE
  36. # %% [markdown]
  37. # # Configurations
  39. # %%
  40. # if running this interactively, can specify jupyter_args here for argparser to use
  41. if utils.is_interactive():
  42. model_name = "subj01_40sess_hypatia_ridge_scsubj01_40sess_hypatia_ridge_sc_medium_captions"
  43. print("model_name:", model_name)
  45. # global_batch_size and batch_size should already be defined in the 2nd cell block
  46. jupyter_args = f"--data_path=../dataset/ \
  47. --cache_dir=../cache/ \
  48. --model_name={model_name} \
  49. --batch_size=64 \
  50. --no-multi_subject --subj=1 --num_sessions=40 \
  51. --dual_guidance --prompt_recon --caption_type medium"
  53. print(jupyter_args)
  54. jupyter_args = jupyter_args.split()
  56. from IPython.display import clear_output # function to clear print outputs in cell
  57. %load_ext autoreload
  58. # this allows you to change functions in models.py or utils.py and have this notebook automatically update with your revisions
  59. %autoreload 2
  61. # %%
  62. parser = argparse.ArgumentParser(description="Model Training Configuration")
  63. parser.add_argument(
  64. "--model_name", type=str, default="testing",
  65. help="name of model, used for ckpt saving and wandb logging (if enabled)",
  66. )
  67. parser.add_argument(
  68. "--data_path", type=str, default=os.getcwd(),
  69. help="Path to where NSD data is stored / where to download it to",
  70. )
  71. parser.add_argument(
  72. "--cache_dir", type=str, default=os.getcwd(),
  73. help="Path to where misc. files downloaded from huggingface are stored. Defaults to current src directory.",
  74. )
  75. parser.add_argument(
  76. "--subj",type=int, default=1, choices=[1,2,3,4,5,6,7,8],
  77. help="Validate on which subject?",
  78. )
  79. parser.add_argument(
  80. "--num_sessions", type=float, default=40,
  81. help="Number of training sessions to include",
  82. )
  83. parser.add_argument(
  84. "--prompt_recon",action=argparse.BooleanOptionalAction, default=True,
  85. help="Use for prompt generating",
  86. )
  87. parser.add_argument(
  88. "--blurry_recon",action=argparse.BooleanOptionalAction,default=True,
  89. help="whether to output blurry reconstructions",
  90. )
  91. parser.add_argument(
  92. "--seed",type=int,default=42,
  93. )
  94. parser.add_argument(
  95. "--weight_decay",type=int,default=100000,
  96. )
  97. parser.add_argument(
  98. "--max_iter",type=int,default=50000,
  99. )
  100. parser.add_argument(
  101. "--dual_guidance",action=argparse.BooleanOptionalAction,default=True,
  102. help="Use the decoded captions for dual guidance",
  103. )
  104. parser.add_argument(
  105. "--caption_type",type=str,default='medium',choices=['coco','short', 'medium', 'schmedium'],
  106. )
  107. parser.add_argument(
  108. "--retrieval",action=argparse.BooleanOptionalAction,default=True,
  109. help="Use the decoded captions for dual guidance",
  110. )
  111. if utils.is_interactive():
  112. args = parser.parse_args(jupyter_args)
  113. else:
  114. args = parser.parse_args()
  115. print(f"args: {args}")
  116. # create global variables without the args prefix
  117. for attribute_name in vars(args).keys():
  118. globals()[attribute_name] = getattr(args, attribute_name)
  120. # seed all random functions
  121. utils.seed_everything(seed)
  123. outdir = os.path.abspath(f'../train_logs/{model_name}')
  124. os.makedirs(outdir,exist_ok=True)
  125. device = "cuda"
  127. # %% [markdown]
  128. # # Prep data, models, and dataloaders
  130. # %%
  131. x_train, valid_nsd_ids_train, x_test, test_nsd_ids = utils.load_nsd(subject=subj, num_sessions=num_sessions, data_path=data_path)
  132. print(x_train.shape, valid_nsd_ids_train.shape)
  134. print(f"Loaded subj {subj} betas!\n")
  136. # %% [markdown]
  137. # ## Prepare git feature
  139. # %%
  140. if not os.path.exists(f'{data_path}/git_image_features.hdf5'):
  141. print("Creating Git Feature...")
  142. from PIL import Image
  143. import requests
  144. from transformers import AutoProcessor, GitVisionModel, AutoModelForCausalLM, GitModel
  145. from modeling_git import GitForCausalLMClipEmb
  146. # Load 73k NSD images
  147. f = h5py.File(f'{data_path}/coco_images_224_float16.hdf5', 'r')
  148. beta_images = f['images']
  149. print("Loaded all 73k possible NSD images to cpu!", beta_images.shape)
  151. git_images = []
  152. processor = AutoProcessor.from_pretrained("microsoft/git-large-coco")
  154. git_text_model = GitForCausalLMClipEmb.from_pretrained("microsoft/git-large-coco")
  155. git_text_model.to(device)
  156. git_text_model.eval().requires_grad_(False)
  157. print("success load Git model")
  158. for i, image in enumerate(tqdm(beta_images)):
  159. pil_image = (image.transpose((1, 2, 0))*255).astype(np.uint8)
  160. inputs = processor(images=pil_image, return_tensors="pt").pixel_values.to(device)
  161. outputs = git_text_model.git.image_encoder(inputs).last_hidden_state
  162. # valid the captions
  163. if i <= 5:
  164. generated_ids = git_text_model.generate(pixel_values=outputs, max_length=50)
  165. generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)
  166. print(generated_caption)
  167. git_images.append(outputs.detach().cpu().numpy())
  170. with h5py.File(f'{data_path}/git_image_features.hdf5', 'w') as f:
  171. f.create_dataset('features', data=np.array(git_images))
  172. print("Finished!")
  173. del beta_images, git_images
  174. else:
  175. print("git_image_features.hdf5 already exist!")
  177. # %%
  178. # Load 73k NSD images
  179. f = h5py.File(f'{data_path}/coco_images_224_float16.hdf5', 'r')
  180. images = f['images'] # if you go OOM you can remove the [:] so it isnt preloaded to cpu! (will require a few edits elsewhere tho)
  181. # images = torch.Tensor(images).to("cpu").to(data_type)
  182. print("Loaded all 73k possible NSD images to cpu!", images.shape)
  184. # Load 73k NSD captions
  185. if caption_type == "schmedium":
  186. captions_small = np.load(f'{data_path}/preprocessed_data/short_length_captions.npy')
  187. captions_medium = np.load(f'{data_path}/preprocessed_data/mid_length_captions_73K.npy')
  188. # Create a mask to randomly select elements from both arrays
  189. mask = np.random.rand(len(captions_small)) > 0.5
  190. # Mix the arrays based on the mask
  191. captions = np.where(mask, captions_small, captions_medium)
  192. else:
  193. if caption_type == "coco":
  194. caption_file = "annots_73k.npy"
  195. elif caption_type == "short":
  196. caption_file = "short_length_captions.npy"
  197. elif caption_type == "medium":
  198. caption_file = "mid_length_captions_73K.npy"
  199. else:
  200. raise ValueError("Invalid caption type")
  201. captions = np.load(f'{data_path}/preprocessed_data/{caption_file}')
  202. print("Loaded all 73k NSD captions to cpu!", captions.shape)
  204. train_images = torch.zeros((len(valid_nsd_ids_train), 3, 224, 224))
  205. train_captions = np.zeros((len(valid_nsd_ids_train),), dtype=object)
  207. # Load specific training data
  208. for i, idx in enumerate(valid_nsd_ids_train):
  209. train_images[i] = torch.from_numpy(images[idx])
  210. train_captions[i] = captions[idx]
  212. print(f"Filtered down to only the {len(valid_nsd_ids_train)} training images for subject {subj}!")
  214. # %% [markdown]
  215. # ## Load models
  217. # %% [markdown]
  218. # ### Feature extractor model
  220. # %%
  221. clip_extractor = SC_Reconstructor(compile_models=False, embedder_only=True, device=device, cache_dir=cache_dir)
  222. vdvae = VDVAE(device=device, cache_dir=cache_dir)
  223. image_embedding_variant = "stable_cascade"
  224. clip_emb_dim = 768
  225. clip_seq_dim = 1
  226. retrieval_embedding_variant = "stable_cascade_hidden"
  227. retrieval_emb_dim = 1024
  228. retrieval_seq_dim = 257
  229. text_embedding_variant = "stable_cascade"
  230. clip_text_seq_dim=77
  231. clip_text_emb_dim=1280
  232. latent_embedding_variant = "vdvae"
  233. latent_emb_dim = 91168
  234. prompt_embedding_variant = "git"
  235. git_seq_dim = 257
  236. git_emb_dim = 1024
  237. if caption_type != "coco":
  238. text_embedding_variant += f"_{caption_type}"
  240. # %% [markdown]
  241. # # Creating block of CLIP embeddings
  243. # %%
  244. file_path = f"{data_path}/preprocessed_data/subject{subj}/{image_embedding_variant}_image_embeddings_train_{num_sessions}sess.pt"
  245. emb_batch_size = 50
  246. if not os.path.exists(file_path):
  247. # Generate CLIP Image embeddings
  248. print("Generating Image embeddings!")
  249. clip_image_train = torch.zeros((len(train_images), clip_seq_dim, clip_emb_dim)).to("cpu")
  250. for i in tqdm(range(len(train_images) // emb_batch_size), desc="Encoding clip images..."):
  251. batch_list = []
  252. for img in train_images[i * emb_batch_size:i * emb_batch_size + emb_batch_size]:
  253. batch_list.append(transforms.ToPILImage()(img))
  254. clip_image_train[i * emb_batch_size:i * emb_batch_size + emb_batch_size] = clip_extractor.embed_image(batch_list).to("cpu")
  256. torch.save(clip_image_train, file_path)
  257. else:
  258. clip_image_train = torch.load(file_path)
  261. if dual_guidance:
  262. emb_batch_size = 50
  263. file_path_txt = f"{data_path}/preprocessed_data/subject{subj}/{text_embedding_variant}_text_embeddings_train_{num_sessions}sess.pt"
  264. if not os.path.exists(file_path_txt):
  265. # Generate CLIP Text embeddings
  266. print("Generating Text embeddings!")
  267. clip_text_train = torch.zeros((len(train_captions), clip_text_seq_dim, clip_text_emb_dim)).to("cpu")
  268. for i in tqdm(range(len(train_captions) // emb_batch_size), desc="Encoding captions..."):
  269. batch_captions = train_captions[i * emb_batch_size:i * emb_batch_size + emb_batch_size].tolist()
  270. clip_text_train[i * emb_batch_size:i * emb_batch_size + emb_batch_size] = clip_extractor.embed_text(batch_captions).to("cpu")
  271. torch.save(clip_text_train, file_path_txt)
  272. else:
  273. clip_text_train = torch.load(file_path_txt)
  276. if blurry_recon:
  277. emb_batch_size = 1
  278. file_path = f"{data_path}/preprocessed_data/subject{subj}/{latent_embedding_variant}_latent_embeddings_train_{num_sessions}sess.pt"
  279. if not os.path.exists(file_path):
  280. print("Generating Latent Image embeddings!")
  281. vae_image_train = torch.zeros((len(train_images), latent_emb_dim)).to("cpu")
  282. for i in tqdm(range(len(train_images)), desc="Encoding blurry images..."):
  283. img = transforms.ToPILImage()(train_images[i])
  284. vae_image_train[i * emb_batch_size:i * emb_batch_size + emb_batch_size] = vdvae.embed_latent(img).reshape(-1, latent_emb_dim).to("cpu")
  285. torch.save(vae_image_train, file_path)
  286. else:
  287. vae_image_train = torch.load(file_path)
  289. if retrieval:
  290. file_path = f"{data_path}/preprocessed_data/subject{subj}/{retrieval_embedding_variant}_retrieval_embeddings_train_{num_sessions}sess.pt"
  291. emb_batch_size = 50
  292. if not os.path.exists(file_path):
  293. # Generate CLIP Retrieval embeddings
  294. print("Generating Retrieval embeddings!")
  295. retrieval_image_train = torch.zeros((len(train_images), retrieval_seq_dim, retrieval_emb_dim)).to("cpu")
  296. for i in tqdm(range(len(train_images) // emb_batch_size), desc="Encoding images..."):
  297. batch_list = []
  298. for img in train_images[i * emb_batch_size:i * emb_batch_size + emb_batch_size]:
  299. batch_list.append(transforms.ToPILImage()(img))
  300. retrieval_image_train[i * emb_batch_size:i * emb_batch_size + emb_batch_size] = clip_extractor.embed_image(batch_list, hidden=True).to("cpu")
  301. # Normalize for optimal cosine similarity
  302. retrieval_image_train = torch.nn.functional.normalize(retrieval_image_train, p=2, dim=2)
  303. torch.save(retrieval_image_train, file_path)
  304. else:
  305. retrieval_image_train = torch.load(file_path)
  307. # Load 73k GiT NSD features
  308. if prompt_recon:
  309. file_path_git = f"{data_path}/preprocessed_data/subject{subj}/{prompt_embedding_variant}_prompt_embeddings_train_{num_sessions}sess.pt"
  310. if not os.path.exists(file_path_git):
  311. with h5py.File(f'{data_path}/git_image_features.hdf5', 'r') as f:
  312. git_features = f['features'][:]
  313. train_git_images = torch.zeros((len(valid_nsd_ids_train), 257,1024))
  314. for i, idx in enumerate(valid_nsd_ids_train):
  315. train_git_images[i] = torch.from_numpy(git_features[idx])
  316. torch.save(train_git_images, file_path_git)
  317. del git_features
  318. else:
  319. train_git_images = torch.load(file_path_git)
  321. print(f"Loaded vectors for subj{subj}!")
  323. # %% [markdown]
  324. # # Train Ridge regression models
  326. # %%
  327. start = time.time()
  328. ridge_weights = np.zeros((clip_seq_dim * clip_emb_dim, x_train.shape[-1])).astype(np.float32)
  329. ridge_biases = np.zeros((clip_seq_dim * clip_emb_dim)).astype(np.float32)
  330. print(f"Training Ridge Image model with alpha={weight_decay}")
  331. model = Ridge(
  332. alpha=weight_decay,
  333. max_iter=max_iter,
  334. random_state=42,
  335. )
  337. model.fit(x_train, clip_image_train.reshape(len(clip_image_train), -1))
  338. ridge_weights = model.coef_
  339. ridge_biases = model.intercept_
  340. datadict = {"coef" : ridge_weights, "intercept" : ridge_biases}
  341. # Save the regression weights
  342. with open(f'{outdir}/ridge_image_weights.pkl', 'wb') as f:
  343. pickle.dump(datadict, f)
  344. del clip_image_train
  345. del ridge_weights
  346. del ridge_biases
  347. del datadict
  349. if dual_guidance:
  350. ridge_weights_txt = np.zeros((clip_text_seq_dim * clip_text_emb_dim, x_train.shape[-1])).astype(np.float32)
  351. ridge_biases_txt = np.zeros((clip_text_seq_dim * clip_text_emb_dim)).astype(np.float32)
  352. print(f"Training Ridge Text model with alpha={weight_decay}")
  353. model = Ridge(
  354. alpha=weight_decay,
  355. max_iter=max_iter,
  356. random_state=42,
  357. )
  359. model.fit(x_train, clip_text_train.reshape(len(clip_text_train), -1))
  360. ridge_weights_txt = model.coef_
  361. ridge_biases_txt = model.intercept_
  362. datadict = {"coef" : ridge_weights_txt, "intercept" : ridge_biases_txt}
  363. # Save the regression weights
  364. with open(f'{outdir}/ridge_text_weights.pkl', 'wb') as f:
  365. pickle.dump(datadict, f)
  367. del clip_text_train
  368. del ridge_weights_txt
  369. del ridge_biases_txt
  370. del datadict
  372. if blurry_recon:
  373. ridge_weights_blurry = np.zeros((latent_emb_dim, x_train.shape[-1])).astype(np.float32)
  374. ridge_biases_blurry = np.zeros((latent_emb_dim,)).astype(np.float32)
  375. print(f"Training Ridge Blurry recon model with alpha={weight_decay}")
  376. model = Ridge(
  377. alpha=weight_decay,
  378. max_iter=max_iter,
  379. random_state=42,
  380. )
  381. model.fit(x_train, vae_image_train)
  382. ridge_weights_blurry = model.coef_
  383. ridge_biases_blurry = model.intercept_
  384. datadict = {"coef" : ridge_weights_blurry, "intercept" : ridge_biases_blurry}
  385. # Save the regression weights
  386. with open(f'{outdir}/ridge_blurry_weights.pkl', 'wb') as f:
  387. pickle.dump(datadict, f)
  389. del vae_image_train
  390. del ridge_weights_blurry
  391. del ridge_biases_blurry
  392. del datadict
  394. if retrieval:
  395. ridge_weights = np.zeros((retrieval_seq_dim * retrieval_emb_dim, x_train.shape[-1])).astype(np.float32)
  396. ridge_biases = np.zeros((retrieval_seq_dim * retrieval_emb_dim)).astype(np.float32)
  397. print(f"Training Ridge Retrieval model with alpha={weight_decay}")
  398. model = Ridge(
  399. alpha=weight_decay,
  400. max_iter=max_iter,
  401. random_state=42,
  402. )
  403. x_train_norm = torch.nn.functional.normalize(x_train, p=2, dim=1)
  404. model.fit(x_train_norm, retrieval_image_train.reshape(len(retrieval_image_train), -1))
  405. ridge_weights = model.coef_
  406. ridge_biases = model.intercept_
  407. datadict = {"coef" : ridge_weights, "intercept" : ridge_biases}
  408. # Save the regression weights
  409. with open(f'{outdir}/ridge_retrieval_weights.pkl', 'wb') as f:
  410. pickle.dump(datadict, f)
  412. del retrieval_image_train
  413. del ridge_weights
  414. del ridge_biases
  415. del datadict
  417. if prompt_recon:
  418. ridge_weights_prompt = np.zeros((git_seq_dim*git_emb_dim, x_train.shape[-1])).astype(np.float32)
  419. ridge_biases_prompt = np.zeros((git_seq_dim*git_emb_dim)).astype(np.float32)
  420. print(f"Training Ridge prompt recon model with alpha={weight_decay}")
  421. model = Ridge(
  422. alpha=weight_decay,
  423. max_iter=max_iter,
  424. random_state=42,
  425. )
  426. model.fit(x_train, train_git_images.reshape(len(train_git_images), -1))
  427. ridge_weights_prompt = model.coef_
  428. ridge_biases_prompt = model.intercept_
  429. datadict = {"coef" : ridge_weights_prompt, "intercept" : ridge_biases_prompt}
  430. # Save the regression weights
  431. with open(f'{outdir}/ridge_prompt_weights.pkl', 'wb') as f:
  432. pickle.dump(datadict, f)
  434. del train_git_images
  435. del ridge_weights_prompt
  436. del ridge_biases_prompt
  437. del datadict
  439. print(f"Elapsed training time for {model_name}: {time.strftime('%H:%M:%S', time.gmtime(time.time() - start))}")

Train.ipynb at commit 7dbc13f, under MIT · at the source

Overview

Authors: Reese Kneeland1,2, Cesar Kadir Torrico Villanueva2, Tong Chen2,3, Jordyn Ojeda1, Shubh Khanna4, Jonathan Xu2,5, Paul S. Scotti2,6,7, Thomas Naselaris1
ORCID iDs: Reese Kneeland
  1. University of Minnesota, Minneapolis, Minnesota, United States of America
  2. Medical AI Research Center (MedARC), San Francisco, California, United States of America
  3. University of Sydney, Sydney, Australia
  4. Stanford University, Palo Alto, California, United States of America
  5. Alljoined, San Francisco, California, Unites States of America
  6. Sophont, San Francisco, California‌‌, Unites States of America
  7. Princeton Neuroscience Institute, Princeton, New Jersey,‌‌ United States of America
Institutions: University of Minnesota (United States); The University of Sydney (Australia); Stanford University (United States); Princeton University (United States)
Journal: PLoS computational biology, volume 22, issue 5, article e1014263
Dates: received 1 September 2025; accepted 23 April 2026; published online 22 May 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1371/journal.pcbi.1014263 · PMID 42172302 · PMCID PMC13218618 · OpenAlex W7162087174
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: fMRI (modality), human (organism)
Methods: Statistics, Machine learning, fMRI & imaging
MeSH: Brain*, Image Processing, Computer-Assisted*, Imagination*, Magnetic Resonance Imaging*, Algorithms, Brain Mapping, Computational Biology, Humans (* major topic)
Journal subjects: Biology and Life Sciences, Neuroscience, Brain Mapping, Functional Magnetic Resonance Imaging, Medicine and Health Sciences, Diagnostic Medicine, Diagnostic Radiology, Magnetic Resonance Imaging, Research and Analysis Methods, Imaging Techniques, Radiology and Imaging, Neuroimaging, Cognitive Science, Cognitive Psychology, Perception, Sensory Perception, Vision, Psychology, Social Sciences, Anatomy, Brain, Visual Cortex, Engineering and Technology, Electronics Engineering, Electronics, Signal Decoders, Signal Processing, Signal to Noise Ratio
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: National Eye Institute (R01EY023384 and R01EY038606)
Citations: not cited yet (Europe PMC); 72 references in the paper

Abstract

To be useful for downstream applications, vision decoding models that are trained to reconstruct seen images from human brain activity must be able to generalize to internally generated visual representations, i.e., mental images. In an analysis of the recently released NSD-Imagery dataset, we demonstrated that while some modern vision decoders can perform quite well on mental image reconstruction, some fail, and that state-of-the-art (SOTA) performance on seen image reconstruction is no guarantee of SOTA performance on mental image reconstruction. Motivated by these findings, we developed MIRAGE, a method explicitly designed to train on vision datasets and cross-decode mental images from brain activity. MIRAGE employs a linear backbone and multi-modal text and image features as input to a diffusion model. Feature metrics and human raters establish MIRAGE as SOTA for mental image reconstruction on the NSD-Imagery benchmark. With ablation analysis we show that mental image reconstruction works best when decoders use image features with relatively few dimensions and include guidance from text-based and both high- and low-level image-based features. Our work indicates that–given the right architecture–existing large-scale datasets using external stimuli are viable training data for decoding mental images, and warrant optimism about the future success and utility of mental image reconstruction.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 6 matches between paragraphs and lines of code.

naturalscenesdataset.org

License: none: the authors keep all their rights
State: the link answers, verified on 28 September 2026
Evidence: the link answers
Software Heritage: not checked
Found in: “Data Availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 28 September 2026: the link answers (HTTP 200)
  • 28 September 2026: the link answers (HTTP 200)
At the source: naturalscenesdataset.org/

MedARC-AI/MIRAGE

License: MIT
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Commit: 7dbc13fb888df0845ea40c4a8c7375cf0bce626f, 16 February 2025
Languages: Python (82), Shell (10), Jupyter (6)
Size: 154 files, 98 scripts
Software Heritage: not archived
Found in: “Data Availability”
Holds: README, license file, environment (src/requirements.txt, src/generative_models/pyproject.toml, src/generative_models/requirements/pt2.txt), tests, 6 notebooks
Not found: CITATION.cff, continuous integration, documentation
Tools: PyTorch (61 files), NumPy (35 files), Pillow (15 files), Matplotlib (10 files), PyTorch Lightning (9 files), h5py (7 files), Hugging Face Transformers (6 files), pandas (5 files), scikit-learn (5 files), OpenCV (4 files), SciPy (4 files), imageio (2 files), NiBabel (2 files), scikit-image (2 files), UMAP (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers
  • 28 September 2026: the link answers
100 files
At the source: github.com/MedARC-AI/MIRAGE

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 98 scripts, each with its path and the digest of its content;
  • 6 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data Availability

NSD-Imagery data has been released publically, and can be accessed at https://naturalscenesdataset.org/ The code for the MIRAGE method is open source and available at https://github.com/MedARC-AI/MIRAGE.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 28 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 8 authors, 8 MeSH terms, 1 funder, 44 references.

Cite

This paper

Kneeland, R., Villanueva, C. K. T., Chen, T., Ojeda, J., Khanna, S., Xu, J., Scotti, P. S., & Naselaris, T. (2026). MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery. PLoS computational biology, 22(5), e1014263. https://doi.org/10.1371/journal.pcbi.1014263

BibTeX

@article{kneeland2026mirage,
author = {Kneeland, Reese and Villanueva, Cesar Kadir Torrico and Chen, Tong and Ojeda, Jordyn and Khanna, Shubh and Xu, Jonathan and Scotti, Paul S. and Naselaris, Thomas},
title = {{MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery}},
journal = {PLoS computational biology},
year = {2026},
month = may,
volume = {22},
number = {5},
pages = {e1014263},
publisher = {PLOS},
issn = {1553-734X},
doi = {10.1371/journal.pcbi.1014263},
url = {https://doi.org/10.1371/journal.pcbi.1014263},
pmid = {42172302},
pmcid = {PMC13218618}
}

RIS

TY - JOUR
AU - Kneeland, Reese
AU - Villanueva, Cesar Kadir Torrico
AU - Chen, Tong
AU - Ojeda, Jordyn
AU - Khanna, Shubh
AU - Xu, Jonathan
AU - Scotti, Paul S.
AU - Naselaris, Thomas
TI - MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery
T2 - PLoS computational biology
J2 - PLoS Comput Biol
PY - 2026
DA - 2026/05/22
VL - 22
IS - 5
SP - e1014263
SN - 1553-734X
PB - PLOS
DO - 10.1371/journal.pcbi.1014263
UR - https://doi.org/10.1371/journal.pcbi.1014263
LA - en
ER -

CSL-JSON

{
"id": "10.1371/journal.pcbi.1014263",
"type": "article-journal",
"title": "MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery",
"container-title": "PLoS computational biology",
"author": [
{
"family": "Kneeland",
"given": "Reese"
},
{
"family": "Villanueva",
"given": "Cesar Kadir Torrico"
},
{
"family": "Chen",
"given": "Tong"
},
{
"family": "Ojeda",
"given": "Jordyn"
},
{
"family": "Khanna",
"given": "Shubh"
},
{
"family": "Xu",
"given": "Jonathan"
},
{
"family": "Scotti",
"given": "Paul S."
},
{
"family": "Naselaris",
"given": "Thomas"
}
],
"container-title-short": "PLoS Comput Biol",
"volume": "22",
"issue": "5",
"page": "e1014263",
"DOI": "10.1371/journal.pcbi.1014263",
"PMID": "42172302",
"PMCID": "PMC13218618",
"ISSN": "1553-734X",
"publisher": "PLOS",
"URL": "https://doi.org/10.1371/journal.pcbi.1014263",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
22
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Hugging Face Transformers, OpenCV, scikit-image, 9 other tools, fMRI, 11 references
[2] doi:10.1007/s12021-026-09803-3 [code]
NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.
Journal: Neuroinformatics
In common: PyTorch Lightning, scikit-image, h5py, 8 other tools, fMRI, 6 references
[3] doi:10.1162/imag.a.1299 [code]
A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: PyTorch Lightning, Hugging Face Transformers, scikit-image, 8 other tools, 6 references
[4] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: imageio, OpenCV, scikit-image, 9 other tools, fMRI, 3 references
[5] doi:10.1162/imag.a.1326 [code]
RAVEN: Robust, generalizable, multi-resolution structural MRI upsampling using autoencoders.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: PyTorch Lightning, imageio, Hugging Face Transformers, 10 other tools
[6] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: Hugging Face Transformers, h5py, Pillow, 7 other tools, 3 references
[7] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: PyTorch Lightning, Hugging Face Transformers, UMAP, 9 other tools
[8] doi:10.1038/s41467-026-71267-5 [code]
Human-like cognitive generalization for large models via mental representation-guided supervision.
Journal: Nature communications
In common: Hugging Face Transformers, UMAP, scikit-image, 8 other tools, 1 reference
[9] doi:10.1371/journal.pcbi.1013499 [code]
VesiclePy: A machine learning vesicle analysis toolbox for volume electron microscopy.
Journal: PLoS computational biology
In common: imageio, UMAP, OpenCV, 9 other tools
[10] doi:10.1371/journal.pcbi.1014656 [code]
Contrastive learning to fine-tune feature extraction models for the visual cortex.
Journal: PLoS computational biology
In common: Pillow, NiBabel, PyTorch, 5 other tools, fMRI, 4 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.