OSCR

NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.

Code ↔ Paper

8 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 8 matches
  1. [1] § Results › Quantitative Results ↔ matrices.ipynb, lines 338–404 · score 0.92 · EffNet, Inception v3, EfficientNet, SwAV distance, AlexNet, PixCorr
  2. [2] § Methods › Visual Reconstruction via Ridge-refined Embeddings and IP-Adapter-SDXL ↔ GOD_dataloader.ipynb, lines 128–182 · score 0.91 · classifier free guidance, Stable Diffusion XL, adapter_sdxl.bin, IP Adapter scale, inference, float16
  3. [3] § Methods › Visual Reconstruction via Ridge-refined Embeddings and IP-Adapter-SDXL ↔ cross_model_aligment.ipynb, lines 382–399 · score 0.81 · Stable Diffusion XL, adapter_sdxl.bin, IP Adapter scale, float16, h94, refined
  4. [4] § Results ↔ cross_model_aligment.ipynb, lines 181–301 · score 0.79 · AdamW, weight decay, Lightning, temperature, validation, GELU
  5. [5] § Methods › Overview ↔ GOD_dataloader.ipynb, lines 128–182 · score 0.75 · stable diffusion XL, sdxl bin, sdxl models, IP Adapter, h94, image embedding
  6. [6] § Methods › Overview ↔ cross_model_aligment.ipynb, lines 382–399 · score 0.74 · stable diffusion XL, sdxl bin, sdxl models, IP Adapter, h94, weights
  7. [7] § Methods › Cross-subject and Cross-dataset Neural Vision Alignment ↔ cross_model_aligment.ipynb, lines 181–301 · score 0.71 · cross entropy, contrastive loss, temperature, optimization, module, batch
  8. [8] § Results › Quantitative Results ↔ matrices.ipynb, lines 338–404 · score 0.58 · AlexNet, PixCorr, Inception, SSIM, efficiency, metrics

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 399 lines · 15 KB · no license · 4 matches

  1. # %% [markdown]
  2. # ## Loading Libraries and data
  4. # %%
  5. import torch
  6. import numpy as np
  7. from torch.utils.data import DataLoader, TensorDataset
  8. import warnings
  9. warnings.filterwarnings('ignore')
  10. device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
  12. # %%
  13. ## Loading BOLD5000 data
  15. BOLD_brain_signals_train = torch.load("/BOLD5000/data_augmentation/train_25percent_noise_75.pt", map_location=device)
  16. # BOLD_brain_signals_train = torch.load("/BOLD5000_V2/brain_signals_train.pt", map_location=device)
  17. BOLD_brain_signals_test = torch.load("/BOLD5000_V2/brain_signals_test.pt", map_location=device)
  19. # BOLD_brain_signals_train = torch.from_numpy(BOLD_brain_signals_train).to(device)
  20. BOLD_brain_signals_test = torch.from_numpy(BOLD_brain_signals_test).to(device)
  22. # Normalizing the BOLD signals
  24. mean = BOLD_brain_signals_train.mean(0)
  25. std = BOLD_brain_signals_train.std(0)
  26. BOLD_brain_signals_train = (BOLD_brain_signals_train - mean) / std
  27. BOLD_brain_signals_test = (BOLD_brain_signals_test - mean) / std
  29. BOLD_brain_signals_train = torch.nan_to_num(BOLD_brain_signals_train)
  30. BOLD_brain_signals_test = torch.nan_to_num(BOLD_brain_signals_test)
  33. BOLD_stimulus_embeddings_train = torch.load( "/BOLD5000/data_augmentation/train_embds_25percent.pt", map_location=device)
  34. # BOLD_stimulus_embeddings_train = torch.load( "/BOLD5000_V2/image_embeddings_train.pt", map_location=device)
  35. BOLD_stimulus_embeddings_test = torch.load("/BOLD5000_V2/image_embeddings_test.pt", map_location=device)
  37. BOLD_subject_ids_train = torch.load("/BOLD5000/data_augmentation/train_ids_25percent.pt", map_location=device)
  38. # BOLD_subject_ids_train = torch.load("/BOLD5000_V2/subject_ids_train.pt", map_location=device)
  39. BOLD_subject_ids_train = torch.tensor(BOLD_subject_ids_train).to(device)
  41. BOLD_subject_ids_test = torch.load("/BOLD5000_V2/subject_ids_test.pt", map_location=device)
  43. # Convert BOLD5000 brain signals to tensor
  44. BOLD_brain_signals_train = torch.tensor(BOLD_brain_signals_train).float()
  45. BOLD_brain_signals_test = torch.tensor(BOLD_brain_signals_test).float()
  47. # Extract CLIP embeddings from BOLD5000
  48. BOLD_stimulus_embeddings_train = BOLD_stimulus_embeddings_train[:, 1, 0].float()
  49. BOLD_stimulus_embeddings_test = BOLD_stimulus_embeddings_test[:, 1, 0].float()
  53. # %%
  54. BOLD_brain_signals_train.shape, BOLD_stimulus_embeddings_train.shape, BOLD_subject_ids_train.shape, BOLD_brain_signals_test.shape, BOLD_stimulus_embeddings_test.shape, BOLD_subject_ids_test.shape
  56. # %%
  57. #Loading NSD dataset
  59. NSD_train_data = torch.load("/decoding_NSD/data_augmentation/train_25percent_noise_50.pt").to(device).float()
  61. NSD_test_data = np.load("/decoding_NSD/data_fmri_nsd/test_data.npy")
  62. NSD_test_data = torch.from_numpy(NSD_test_data).to(device).float()
  65. mean = NSD_train_data.mean(0)
  66. std = NSD_train_data.std(0)
  67. NSD_train_data = (NSD_train_data - mean) / std
  68. NSD_test_data = (NSD_test_data - mean) / std
  70. NSD_train_data = torch.nan_to_num(NSD_train_data)
  71. NSD_test_data = torch.nan_to_num(NSD_test_data)
  73. NSD_train_img_embeds = torch.load("/decoding_NSD/data_augmentation/train_embds_25percent.pt", map_location=device)
  74. NSD_test_img_embeds = torch.load("/decoding_NSD/data_fmri_nsd/test_clip_img_embeds.pt", map_location=device)
  76. NSD_train_img_embeds = NSD_train_img_embeds[:, 1, 0].float()
  77. NSD_test_img_embeds = NSD_test_img_embeds[:, 1, 0].float()
  80. NSD_subject_train_ids = torch.load("/decoding_NSD/data_augmentation/train_ids_25percent.pt")
  81. NSD_subject_test_ids = np.load("/decoding_NSD/data_fmri_nsd/subject_test_ids.npy")
  83. # NSD_subject_train_ids=[int(i[-1]) for i in NSD_subject_train_ids]
  84. NSD_subject_test_ids=[int(i[-1]) for i in NSD_subject_test_ids]
  86. NSD_subject_train_ids = torch.from_numpy(np.array(NSD_subject_train_ids)).to(device)
  87. NSD_subject_test_ids = torch.from_numpy(np.array(NSD_subject_test_ids)).to(device)
  89. # %%
  90. #Loading GOD dataset
  91. GOD_train_brain_signals = torch.load('/GOD/train_brain_signals.pt').to(device)
  92. GOD_test_brain_signals = torch.load('/GOD/test_brain_signals.pt').to(device)
  94. mean = GOD_train_brain_signals.mean(0)
  95. std = GOD_train_brain_signals.std(0)
  96. GOD_train_brain_signals = (GOD_train_brain_signals - mean) / std
  97. GOD_test_brain_signals = (GOD_test_brain_signals - mean) / std
  99. GOD_train_brain_signals = torch.nan_to_num(GOD_train_brain_signals).float()
  100. GOD_test_brain_signals = torch.nan_to_num(GOD_test_brain_signals).float()
  102. GOD_train_image_embds = torch.load('/GOD/train_image_embeddings.pt', map_location=device)
  103. GOD_test_image_embds = torch.load('/GOD/test_image_embeddings.pt', map_location=device)
  105. GOD_train_image_embds = GOD_train_image_embds[:, 1, 0].float()
  106. GOD_test_image_embds = GOD_test_image_embds[:, 1, 0].float()
  109. GOD_train_subject_ids = torch.load('/GOD/train_subject_ids.pt', map_location=device)
  110. # GOD_train_subject_ids = torch.tensor(GOD_train_subject_ids).to(device)
  111. GOD_test_subject_ids = torch.load('/GOD/test_subject_ids.pt', map_location=device)
  114. GOD_train_brain_signals.shape, GOD_train_image_embds.shape, GOD_train_subject_ids.shape, GOD_test_brain_signals.shape, GOD_test_image_embds.shape, GOD_test_subject_ids.shape
  116. # %%
  117. # Concatenate datasets
  118. cross_train_data = torch.cat([NSD_train_data, BOLD_brain_signals_train, GOD_train_brain_signals], dim=0)
  119. cross_test_data = torch.cat([NSD_test_data, BOLD_brain_signals_test, GOD_test_brain_signals], dim=0)
  121. cross_train_embeds = torch.cat([NSD_train_img_embeds, BOLD_stimulus_embeddings_train, GOD_train_image_embds], dim=0)
  122. cross_test_embeds = torch.cat([NSD_test_img_embeds, BOLD_stimulus_embeddings_test, GOD_test_image_embds], dim=0)
  124. cross_train_subjects = torch.cat([torch.tensor(NSD_subject_train_ids, device=device), torch.tensor(BOLD_subject_ids_train, device=device), torch.tensor(GOD_train_subject_ids, device=device)], dim=0)
  125. cross_test_subjects = torch.cat([torch.tensor(NSD_subject_test_ids, device=device), torch.tensor(BOLD_subject_ids_test, device=device), torch.tensor(GOD_test_subject_ids, device=device)], dim=0)
  127. # %% [markdown]
  128. # ## Defining model
  130. # %%
  131. import torch
  132. import torch.nn as nn
  133. import pytorch_lightning as pl
  136. class Encoder(nn.Module):
  137. def __init__(self, input_dim, hidden_dims, output_dim, act_fn=nn.ReLU, alignment_layers_keys=[12, 13, 14, 15, 16], common_dim=1024): #, 1, 2, 5, 7, 8, 9, 10, 11
  138. super(Encoder, self).__init__()
  139. self.common_dim = common_dim
  140. self.alignment_layers = nn.ModuleDict({str(k): nn.Linear(input_dim, common_dim) for k in alignment_layers_keys})
  141. self.dropout = nn.Dropout(p=0.3)
  143. # Transformer Encoder
  144. self.transformer = nn.TransformerEncoder(
  145. nn.TransformerEncoderLayer(d_model=common_dim, nhead=8, dim_feedforward=2048, activation='gelu'),
  146. num_layers=4
  147. )
  149. layers = [nn.LayerNorm(common_dim)]
  150. prev_dim = common_dim
  151. for hidden_dim in hidden_dims:
  152. layers.append(nn.Linear(prev_dim, hidden_dim))
  153. layers.append(nn.LayerNorm(hidden_dim))
  154. layers.append(act_fn())
  155. layers.append(nn.Linear(hidden_dim, prev_dim)) #Residual Connections
  156. prev_dim = hidden_dim
  157. layers.append(nn.Linear(prev_dim, output_dim))
  158. layers.append(nn.LayerNorm(output_dim))
  159. self.net = nn.Sequential(*layers)
  160. # Alignment layer
  161. self.alignment_layer = nn.Linear(output_dim, output_dim)
  163. def forward(self, x, k=None):
  164. if k is None:
  165. k = torch.ones(len(x), dtype=torch.long)
  167. result = torch.zeros(len(x), self.common_dim, device=x.device)
  168. for key in k.unique():
  169. mask = (k == key)
  170. result[mask] = self.alignment_layers[str(key.item())](x[mask])
  171. # result[mask] = self.dropout(result[mask]) #add dropoutout
  172. # # Add Transformer encoding here
  173. result = result.unsqueeze(1) # Add sequence dimension for Transformer (batch, seq_len, features)
  174. result = self.transformer(result)
  175. result = result.squeeze(1) # Remove sequence dimension
  176. #Alignment layer
  177. aligned_result = self.net(result)
  178. aligned_result = self.alignment_layer(aligned_result)
  179. return aligned_result #self.net(result)
  181. class ContrastiveModel(pl.LightningModule):
  182. def __init__(self, num_input_channels, base_channel_size, latent_dim, temperature=0.1, act_fn=nn.GELU, loss_type="contrastive"):
  183. super().__init__()
  184. self.temperature = temperature
  185. self.model = Encoder(num_input_channels, base_channel_size, latent_dim, act_fn)
  186. self.loss_type = loss_type
  188. if loss_type == "contrastive":
  189. self.loss_fn = self.contrastive_loss
  190. elif loss_type == "mean_contrastive":
  191. self.loss_fn = self.mean_contrastive
  192. elif loss_type == "mse":
  193. self.loss_fn = torch.nn.functional.mse_loss
  194. elif loss_type == "cosine":
  195. self.loss_fn = self.cosine_loss
  197. self.train_losses = []
  198. self.train_mse = []
  199. self.train_cosine = []
  200. self.val_losses = []
  201. self.val_mse = []
  202. self.val_cosine = []
  204. self.train_history = {
  205. "train_loss": [],
  206. "train_mse": [],
  207. "train_cosine": []
  208. }
  210. self.val_history = {
  211. "val_loss": [],
  212. "val_mse": [],
  213. "val_cosine": []
  214. }
  216. def forward(self, x, **kwargs):
  217. return self.model(x, **kwargs)
  219. def contrastive_loss(self, z_i, z_j):
  220. z_i = nn.functional.normalize(z_i, dim=1)
  221. z_j = nn.functional.normalize(z_j, dim=1)
  223. logits = (z_i @ z_j.T) / self.temperature
  224. targets = torch.arange(logits.shape[0]).long().to(logits.device)
  226. # Original cross-entropy loss
  227. loss1 = torch.nn.functional.cross_entropy(logits, targets)
  229. # Transposed cross-entropy loss
  230. loss2 = torch.nn.functional.cross_entropy(logits.T, targets)
  232. # Combined loss
  233. loss = 0.5 * loss1 + 0.5 * loss2
  235. return loss
  238. def mean_contrastive(self, z_i, z_j, temperature=1.0):
  239. return nn.functional.mse_loss(z_i, z_j) + self.contrastive_loss(z_i, z_j, temperature=temperature) / 8
  241. def cosine_loss(self, z_i, z_j, temperature=1.0):
  242. cosine_similarity = torch.nn.functional.cosine_similarity(z_i, z_j).mean()
  243. return 1 - cosine_similarity
  245. def training_step(self, batch, batch_idx):
  246. x, y, idx = batch #add noise inX
  248. noise = 0.15 * torch.randn_like(x)
  249. x = x + noise
  251. y_hat = self(x, k=idx)
  252. loss = self.loss_fn(y_hat, y)
  253. self.log('train_loss', loss, on_epoch=True, prog_bar=True)
  254. self.train_losses.append(loss.item())
  256. mse_loss = torch.nn.functional.mse_loss(y_hat, y)
  257. cosine_similarity = torch.nn.functional.cosine_similarity(y_hat, y).mean()
  258. self.train_mse.append(mse_loss.item())
  259. self.train_cosine.append(cosine_similarity.item())
  261. return loss
  263. def validation_step(self, batch, batch_idx):
  264. x, y, idx = batch
  265. y_hat = self(x, k=idx)
  266. loss = self.loss_fn(y_hat, y)
  267. self.log('val_loss', loss, on_epoch=True, prog_bar=True)
  269. mse_loss = torch.nn.functional.mse_loss(y_hat, y)
  270. self.log('val_mse_loss', mse_loss, on_epoch=True, prog_bar=True)
  272. cosine_similarity = torch.nn.functional.cosine_similarity(y_hat, y).mean()
  273. self.log('val_cosine_similarity', cosine_similarity, on_epoch=True, prog_bar=True)
  275. self.val_losses.append(loss.item())
  276. self.val_mse.append(mse_loss.item())
  277. self.val_cosine.append(cosine_similarity.item())
  278. return mse_loss
  280. def on_train_epoch_end(self):
  281. self.train_history["train_loss"].append(np.mean(self.train_losses))
  282. self.train_history["train_mse"].append(np.mean(self.train_mse))
  283. self.train_history["train_cosine"].append(np.mean(self.train_cosine))
  284. self.train_losses = []
  285. self.train_mse = []
  286. self.train_cosine = []
  287. super().on_train_epoch_end()
  289. def on_validation_epoch_end(self):
  290. self.val_history["val_loss"].append(np.mean(self.val_losses))
  291. self.val_history["val_mse"].append(np.mean(self.val_mse))
  292. self.val_history["val_cosine"].append(np.mean(self.val_cosine))
  293. self.val_losses = []
  294. self.val_mse = []
  295. self.val_cosine = []
  296. super().on_validation_epoch_end()
  298. def configure_optimizers(self):
  299. optimizer = torch.optim.AdamW(self.parameters(), lr=1e-4, weight_decay=1e-3)
  300. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=50, verbose=True)
  301. return {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "val_loss"}
  303. # %%
  304. BS = 256
  306. # Create the final dataset
  307. train_dataset = TensorDataset(cross_train_data, cross_train_embeds, cross_train_subjects)
  308. test_dataset = TensorDataset(cross_train_data, cross_test_embeds, cross_test_subjects)
  310. # Create DataLoaders
  311. train_dataloader = DataLoader(train_dataset, batch_size=BS, shuffle=True)
  312. test_dataloader = DataLoader(test_dataset, batch_size=BS, shuffle=False)
  314. # %%
  315. brain_model = ContrastiveModel(num_input_channels=cross_train_data.shape[-1], base_channel_size=[1024], latent_dim=1280, act_fn=nn.GELU, loss_type="contrastive")
  317. trainer = pl.Trainer(max_epochs=10, devices=[2])
  319. # Train the model
  320. trainer.fit(brain_model, train_dataloader, test_dataloader)
  322. # %%
  324. #RIDGE REGRESION
  325. from sklearn.linear_model import Ridge
  327. train_pred_embeddings = []
  328. train_gt_embeddings = []
  330. brain_model.eval()
  331. with torch.no_grad():
  332. for x, y, k in train_dataloader:
  333. x = x.to("cpu")
  334. y = y.to("cpu")
  335. k = k.to("cpu")
  336. y_hat = brain_model(x, k=k)
  337. train_pred_embeddings.append(y_hat.cpu().numpy())
  338. train_gt_embeddings.append(y.cpu().numpy())
  340. train_pred_embeddings = np.vstack(train_pred_embeddings)
  341. train_gt_embeddings = np.vstack(train_gt_embeddings)
  343. # Train Ridge regression model
  344. ridge_reg = Ridge(alpha=50000.0)
  345. ridge_reg.fit(train_pred_embeddings, train_gt_embeddings)
  347. # Predict embeddings for the test set
  348. test_pred_embeddings = []
  350. with torch.no_grad():
  351. for x, y, k in test_dataloader:
  352. x = x.to("cpu")
  353. y = y.to("cpu")
  354. k = k.to("cpu")
  355. y_hat = brain_model(x, k=k)
  356. test_pred_embeddings.append(y_hat.cpu().numpy())
  358. test_pred_embeddings = np.vstack(test_pred_embeddings)
  360. # Apply Ridge regression on the test set embeddings
  361. refined_test_embeddings = ridge_reg.predict(test_pred_embeddings)
  363. # Convert refined embeddings to tensor
  364. refined_test_embeddings = torch.tensor(refined_test_embeddings).float()
  368. # %%
  369. BOLD_stimulus_embeddings_test = BOLD_stimulus_embeddings_test.to("cpu")
  371. BOLD_pred_IP = refined_test_embeddings.to("cpu")
  374. # %%
  375. NSD_stimulus_embeddings_test = NSD_test_img_embeds.to("cpu")
  376. NSD_pred_IP = refined_test_embeddings.to("cpu")
  378. # %%
  379. GOD_stimulus_embeddings_test = GOD_test_image_embds.to("cpu")
  380. GOD_pred_IP = refined_test_embeddings.to("cpu")
  382. # %%
  383. from diffusers import AutoPipelineForText2Image
  384. from diffusers.utils import load_image
  385. import torch
  387. pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda:2")
  388. pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
  389. pipeline.set_ip_adapter_scale(1.0)
  391. #RESHAPING THE EMBEDDINGS
  393. y_pred_IP = torch.zeros((2, refined_test_embeddings.shape[0], 1, refined_test_embeddings.shape[-1]), dtype=torch.float16)
  394. y_pred_IP[1] = refined_test_embeddings.unsqueeze(1)
  395. y_pred_IP[0] = torch.zeros_like(refined_test_embeddings.unsqueeze(1))
  396. y_pred_IP = y_pred_IP.transpose(0,1)
  398. y_pred_IP.shape
  399. torch.save(y_pred_IP, "New_results/BOLD_predicted_embeddings_08042025.pt")

cross_model_aligment.ipynb at commit a011911, no license · at the source

Overview

Authors: Muhammad Kashif1, Matteo Ferrante1, Nicola Toschi1,2
  1. Department of Biomedicine and Prevention, University of Rome Tor Vergata,Rome, Italy
  2. Martinos Center for Biomedical Imaging, MGH and Harvard Medical School,Boston, MA USA
Institutions: University of Rome Tor Vergata (Italy); Harvard University (United States)
Journal: Neuroinformatics, volume 24, issue 3, article 45
Dates: received 25 May 2026; accepted 10 July 2026; published online 18 July 2026; in print 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1007/s12021-026-09803-3 · PMID 42469516 · PMCID PMC13379418 · OpenAlex W7169512943
Open access: hybrid, a free copy (OpenAlex)
Status: code verified
Categories: fMRI (modality), human (organism), methods / tools (subfield)
Methods: Statistics, Preprocessing, fMRI & imaging, Machine learning
Keywords: fMRI decoding, Cross-subject generalization, Cross-dataset transfer, Contrastive learning, Diffusion models, IP-Adapter, Stable Diffusion XL, Visual reconstruction
MeSH: Brain*, Brain Mapping*, Magnetic Resonance Imaging*, Photic Stimulation*, Visual Perception*, Databases, Factual, Humans, Image Processing, Computer-Assisted, Representation Machine Learning (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: Università degli Studi di Roma Tor Vergata
Citations: not cited yet (Europe PMC); 55 references in the paper

Abstract

Recent advancements in neural decoding have shown promising results in reconstructing visual experiences from brain activity. However, existing approaches focus primarily on decoding within a single dataset or subject, which limits generalization across various sources of neuroimaging. In this work, we propose a novel framework for the decoding of visual stimuli between subjects and between data sets, integrating neural recordings from multiple publicly available fMRI datasets. To address inherent intersubject and interdataset variability, we introduce a contrastive learning-based alignment strategy using image embeddings from a pre-trained IP-Adapter model. Our approach learns a shared latent space by aligning subject-specific neural representations with image features, enabling generalized decoding across both subjects and datasets. In addition, we propose a simple yet effective data augmentation method using ridge regression. This method synthesizes realistic fMRI-like signals from novel images by predicting voxel activity and injecting learned noise distributions, thus enhancing training diversity and model robustness. To the best of our knowledge, while several recent studies have explored cross-subject decoding, we extend recent cross-subject decoding efforts by training a single unified framework jointly across multiple public fMRI datasets and subjects, enabling cross-dataset transfer in addition to cross-subject generalization. We distinguish this multi-dataset unified training setting, where each dataset contributes training data, from a stricter leave-one-dataset-out transfer setting in which the target dataset is excluded from source pretraining and used only for lightweight alignment-layer adaptation. Empirically, our unified model achieves strong semantic reconstruction across datasets (e.g., up to 94.8% CLIP similarity on NSD (AUG) and 0.403 SSIM on BOLD5000 after lightweight finetuning), demonstrating robust cross-subject and cross-dataset transfer.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 8 matches between paragraphs and lines of code.

r8832711/Cross-subject-and-cross-dataset-brain-decoding

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: a01191158a14df08c24d5309953614d1fc9ab025, 15 May 2025
Languages: Jupyter (8), Python (1)
Size: 10 files, 9 scripts
Software Heritage: not archived
Found in: “Information Sharing Statement”
Holds: README, 8 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (9 files), PyTorch (9 files), scikit-learn (7 files), Matplotlib (6 files), SciPy (6 files), Pillow (5 files), pandas (4 files), h5py (2 files), NiBabel (2 files), PyTorch Lightning (1 file), MNE-Python (1 file), MNE-BIDS (1 file), Nilearn (1 file), scikit-image (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
10 files
At the source: github.com/r8832711/Cross-subject-and-cross-dataset-brain-decoding

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 9 scripts, each with its path and the digest of its content;
  • 8 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability

The NSD (Natural Scenes Dataset) version 1.0 used in this study is publicly available at https://naturalscenesdataset.org/. The dataset is released under the Creative Commons Attribution-NonCommercial (CC BY-NC 4.0) license.

The BOLD5000 dataset used in this study is publicly available at https://bold5000-dataset.github.io/website/ and is released under the Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA 4.0) license.

The GOD (Generic Object Decoding) dataset version 1.2.1 used in this study is available from OpenNeuro at https://openneuro.org/datasets/ds001246/versions/1.2.1 and is released under the Public Domain Dedication and License (PDDL).

The code used in this study has been anonymously uploaded to GitHub and is available at: https://github.com/r8832711/Cross-subject-and-cross-dataset-brain-decoding

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 2, 28 September 2026

  • Publisher: n/a → Springer Science+Business Media

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 3 authors, 8 keywords, 9 MeSH terms, 1 funder, 33 references.

Cite

This paper

Kashif, M., Ferrante, M., & Toschi, N. (2026). NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects. Neuroinformatics, 24(3), 45. https://doi.org/10.1007/s12021-026-09803-3

BibTeX

@article{kashif2026neurofusion,
author = {Kashif, Muhammad and Ferrante, Matteo and Toschi, Nicola},
title = {{NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects}},
journal = {Neuroinformatics},
year = {2026},
month = jul,
volume = {24},
number = {3},
pages = {45},
publisher = {Springer Science+Business Media},
issn = {1539-2791},
doi = {10.1007/s12021-026-09803-3},
url = {https://doi.org/10.1007/s12021-026-09803-3},
pmid = {42469516},
pmcid = {PMC13379418}
}

RIS

TY - JOUR
AU - Kashif, Muhammad
AU - Ferrante, Matteo
AU - Toschi, Nicola
TI - NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects
T2 - Neuroinformatics
J2 - Neuroinformatics
PY - 2026
DA - 2026/07/18
VL - 24
IS - 3
SP - 45
SN - 1539-2791
PB - Springer Science+Business Media
DO - 10.1007/s12021-026-09803-3
UR - https://doi.org/10.1007/s12021-026-09803-3
LA - en
ER -

CSL-JSON

{
"id": "10.1007/s12021-026-09803-3",
"type": "article-journal",
"title": "NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects",
"container-title": "Neuroinformatics",
"author": [
{
"family": "Kashif",
"given": "Muhammad"
},
{
"family": "Ferrante",
"given": "Matteo"
},
{
"family": "Toschi",
"given": "Nicola"
}
],
"container-title-short": "Neuroinformatics",
"volume": "24",
"issue": "3",
"page": "45",
"DOI": "10.1007/s12021-026-09803-3",
"PMID": "42469516",
"PMCID": "PMC13379418",
"ISSN": "1539-2791",
"publisher": "Springer Science+Business Media",
"URL": "https://doi.org/10.1007/s12021-026-09803-3",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
18
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1162/imag.a.1299 [code]
A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: PyTorch Lightning, scikit-image, h5py, 7 other tools, methods / tools, 8 references
[2] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Nilearn, scikit-image, h5py, 8 other tools, fMRI, 7 references
[3] doi:10.1371/journal.pcbi.1014263 [code]
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.
Journal: PLoS computational biology
In common: PyTorch Lightning, scikit-image, h5py, 8 other tools, fMRI, 6 references
[4] doi:10.1038/s41597-026-06869-1 [code]
Individual Brain Charting: fifth release of high-resolution fMRI data for cognitive mapping.
Journal: Scientific data
In common: Nilearn, scikit-image, Pillow, 6 other tools, fMRI, methods / tools, 4 references
[5] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: Nilearn, scikit-image, h5py, 8 other tools, fMRI, methods / tools, 2 references
[6] doi:10.1038/s41597-025-05174-7 [code]
A large-scale MEG and EEG dataset for object recognition in naturalistic scenes
Journal: n/a
In common: MNE-BIDS, MNE-Python, h5py, 8 other tools, fMRI, methods / tools, 1 reference
[7] doi:10.1016/j.isci.2026.117180 [code]
Developmental changes in similarity between neural representations of mental arithmetic and artificial neural networks.
Journal: iScience
In common: Nilearn, h5py, NiBabel, 6 other tools, fMRI, 3 references
[8] doi:10.1523/jneurosci.0038-26.2026 [code]
Multidimensional Feature Tuning in Category Selective Areas of Human Visual Cortex.
Journal: The Journal of neuroscience : the official journal of the Society for Neuroscience
In common: h5py, Pillow, NiBabel, 6 other tools, fMRI, 3 references
[9] doi:10.1162/imag.a.1286 [code]
Behavioral imitation with artificial neural networks leads to personalized models of brain dynamics during videogame play.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Nilearn, h5py, Pillow, 7 other tools, fMRI, 2 references
[10] doi:10.1162/nol.a.271 [code]
Compositional Complexity in Text and Images.
Journal: Neurobiology of language (Cambridge, Mass.)
In common: Nilearn, Pillow, NiBabel, 6 other tools, fMRI, 3 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.