OSCR

A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration.

Code ↔ Paper

4 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 4 matches
  1. [1] § Material and Methods › Task 2: Image generation ↔ monkeys_gener.ipynb, lines 81–144 · score 0.85 · classifier free guidance, IP Adapter scale, Stable Diffusion, VAE, prompts, pipeline
  2. [2] § Material and Methods › Task 2: Image generation ↔ monkeys_gener.ipynb, lines 848–915 · score 0.64 · Gaussian blur, SSIM score, smoothed, resized, reconstruct
  3. [3] § Material and Methods › Task 1: Stimulus retrieval ↔ monkeys_gener.ipynb, lines 751–773 · score 0.53 · closest neighbor, nearest neighbor, predicted
  4. [4] § Material and Methods ↔ monkeys_gener.ipynb, lines 81–144 · score 0.52 · IP Adapter, Stable Diffusion, pipeline, weighting, embedding, model

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 922 lines · 30 KB · no license · 4 matches

  1. # %%
  2. import numpy as np
  3. from scipy.io import loadmat
  4. import os
  5. from os.path import join as opj
  6. from h5py import File
  7. import pandas as pd
  9. # load CLIP from huggingface, load the first N images and extract the features
  10. from transformers import CLIPProcessor, CLIPModel
  11. import torch
  12. import tqdm
  13. from PIL import Image
  14. from diffusers import StableDiffusionXLPipeline
  15. from diffusers import AutoPipelineForText2Image
  16. from copy import deepcopy
  17. from PIL import Image, ImageFilter
  18. from skimage.metrics import structural_similarity as ssim
  20. # %%
  21. storage_path = "..." # path to the monkey storage directory
  23. base_path = storage_path + "/THINGS_Monkey"
  24. thing_base_path = storage_path + "/THINGS_img/"
  26. data_path = "..." # path to the monkey THINGS data
  27. monkey = "F"
  28. os.listdir(base_path)
  30. # %%
  31. def resolve_reference(hdf5_file, ref):
  32. """
  33. Resolve an HDF5 dataset reference and convert it into a string.
  34. """
  35. data = hdf5_file[ref][:]
  36. return ''.join(chr(i) for i in data.flatten() if i > 0)
  39. things_imgs = File(opj(base_path,f'things_imgs_{monkey}.mat'))
  40. train_imgs = things_imgs['train_imgs'] # group object --> <HDF5 group "/train_imgs" (3 members)>
  41. test_imgs = things_imgs['test_imgs']
  43. train_classes = []
  44. train_local_paths = []
  45. train_things_paths = []
  47. test_classes = []
  48. test_local_paths = []
  49. test_things_paths = []
  51. with File(opj(base_path, f"things_imgs_{monkey}.mat")) as f:
  52. train_imgs = f['train_imgs']
  54. train_classes = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['train_imgs']['class']]
  55. train_local_paths = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['train_imgs']['local_path']]
  56. train_things_paths = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['train_imgs']['things_path']]
  58. test_classes = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['test_imgs']['class']]
  59. test_local_paths = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['test_imgs']['local_path']]
  60. test_things_paths = [resolve_reference(things_imgs, ref[0]) for ref in things_imgs['test_imgs']['things_path']]
  63. # %%
  64. trials = File(opj(base_path, f"THINGS_MUA_trials_{monkey}.mat"))
  65. df =pd.DataFrame(trials["ALLMAT"][:].T, columns=["#trial_idx", "#train_idx", "#test_idx", "#rep", "#count", "#correct"])
  67. df
  69. # %%
  70. data = np.load(data_path + '/monkeys/trials_allmua.npy')
  72. print(data.shape)
  74. # %%
  75. train_indices = df[df["#train_idx"]!=0]["#train_idx"].values.astype(int) - 1
  76. test_indices = df[df["#test_idx"]!=0]["#test_idx"].values.astype(int) - 1
  78. sorted_train_img_path = [train_things_paths[i] for i in train_indices]
  79. sorted_test_img_path = [test_things_paths[i] for i in test_indices]
  81. # %% [markdown]
  82. # ## Pipeline for IMG generation
  84. # %%
  85. device = "cuda:3" if torch.cuda.is_available() else "cpu"
  86. print("Device:", device)
  88. pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to(device)
  89. pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
  90. pipeline.set_ip_adapter_scale(0.6)
  91. vae = deepcopy(pipeline.vae).to(device).float()
  93. # Function to process image embeddings
  94. def process_image_embeds(batch_images):
  95. with torch.no_grad(): # Disable gradient computation
  96. imgs = [Image.open(img).convert("RGB") for img in batch_images]
  98. # Compute image embeddings on GPU
  99. image_embeds = pipeline.prepare_ip_adapter_image_embeds(
  100. ip_adapter_image=[imgs],
  101. ip_adapter_image_embeds=None,
  102. device=device,
  103. num_images_per_prompt=1,
  104. do_classifier_free_guidance=True,
  105. )
  107. return image_embeds[0].permute(0,1,2).cpu()
  110. # Function to process latents
  111. def process_latents(batch_images):
  112. with torch.no_grad(): # Disable gradient computation
  113. imgs = [Image.open(img).convert("RGB").resize((256,256)) for img in batch_images]
  115. # Convert images to tensors
  116. image_tensors = torch.stack([
  117. torch.from_numpy(
  118. np.array(img).astype(np.float32) / 255.0 * 2 - 1
  119. ).permute(2, 0, 1) for img in imgs
  120. ]).to(device)
  122. # Compute latents on GPU
  123. posterior = vae.encode(image_tensors).latent_dist
  124. latents = posterior.sample()
  126. # Move results to CPU and store
  127. return latents.cpu()
  130. def extract_latents(batch_images):
  131. with torch.no_grad(): # Disable gradient computation
  132. # Convert images to tensors
  133. image_tensors = torch.stack([
  134. torch.from_numpy(
  135. np.array(img).astype(np.float32) / 255.0 * 2 - 1
  136. ).permute(2, 0, 1) for img in batch_images
  137. ]).to(device)
  139. # Compute latents on GPU
  140. posterior = vae.encode(image_tensors).latent_dist
  141. latents = posterior.sample()
  143. # Move results to CPU and store
  144. return latents.cpu()
  146. # %%
  147. to_save_files = data_path + "/monkeys/generative/"
  148. train_images = [opj(thing_base_path,"THINGS","Images", img).replace("\\","/") for img in sorted_train_img_path]
  149. test_images = [opj(thing_base_path,"THINGS","Images", img).replace("\\","/") for img in sorted_test_img_path]
  150. batch_size = 128 # Set your desired batch size
  151. train = False
  153. if train:
  155. train_latents = []
  157. test_latents = []
  159. # Process images in batches
  160. for i in tqdm.trange(0, len(train_images), batch_size):
  161. batch_images = train_images[i : i + batch_size]
  162. latents = process_latents(batch_images) # fill test_features and test_latents
  163. torch.cuda.empty_cache()
  164. train_latents.append(latents)
  166. train_latents = torch.cat(train_latents,0)
  168. # Process images in batches
  169. for i in tqdm.trange(0, len(test_images), batch_size):
  170. batch_images = test_images[i : i + batch_size]
  171. latents = process_latents(batch_images) # fill test_features and test_latents
  172. torch.cuda.empty_cache()
  173. test_latents.append(latents)
  176. test_latents = torch.cat(test_latents,0)
  178. ### Save in things_base_path as pt files
  179. torch.save(train_latents, opj(to_save_files,f"train_latents_{monkey}.pt"))
  180. torch.save(test_latents, opj(to_save_files,f"test_latents_{monkey}.pt"))
  181. print("Latents saved in", to_save_files)
  183. else:
  184. train_latents = torch.load(opj(to_save_files,f"train_latents_{monkey}.pt")) # thing_base_path
  185. test_latents = torch.load(opj(to_save_files,f"test_latents_{monkey}.pt"))
  187. print("Latents loaded from", to_save_files)
  189. # %%
  190. ### Extract features
  191. if train:
  192. train_features = []
  193. test_features = []
  195. for i in tqdm.trange(0, len(train_images), batch_size):
  196. batch_images = train_images[i : i + batch_size]
  197. features = process_image_embeds(batch_images)
  198. train_features.append(features)
  199. torch.cuda.empty_cache()
  201. else:
  202. train_features = torch.load(opj(to_save_files,f"train_features_{monkey}.pt"))
  203. test_features = torch.load(opj(to_save_files,f"test_features_{monkey}.pt"))
  205. print("Features loaded from", to_save_files)
  207. # %%
  208. if train:
  209. for i in tqdm.trange(0, len(test_images), batch_size):
  210. batch_images = test_images[i : i + batch_size]
  211. features = process_image_embeds(batch_images) # fill test_features and test_latents
  212. test_features.append(features)
  213. torch.cuda.empty_cache()
  215. if train:
  216. train_features = torch.cat(train_features,1).permute(1,0,2)
  217. test_features = torch.cat(test_features,1).permute(1,0,2)
  219. ## save features and check shapes
  220. if train:
  221. torch.save(train_features, opj(to_save_files,f"train_features_{monkey}.pt"))
  222. torch.save(test_features, opj(to_save_files,f"test_features_{monkey}.pt"))
  223. print("Features saved in", to_save_files)
  225. print("Train features shape", train_features.shape)
  226. print("Test features shape", test_features.shape)
  228. print("Train latents shape", train_latents.shape)
  229. print("Test latents shape", test_latents.shape)
  232. # %% [markdown]
  233. # ## Neural Data
  235. # %%
  236. neural_train_trial_idx = df[df["#train_idx"]!=0]["#trial_idx"].values.astype(int) - 1
  237. neural_test_trial_idx = df[df["#test_idx"]!=0]["#trial_idx"].values.astype(int) - 1
  239. train_neural = data[:,neural_train_trial_idx] # prendo tutte le osservazioni --> no data[:,neural_train_trial_idx[:N]]
  240. test_neural = data[:,neural_test_trial_idx]
  241. print(train_neural.shape, test_neural.shape)
  243. # %%
  244. train_indices = df[df["#train_idx"]!=0]["#train_idx"].values.astype(int) - 1
  245. test_indices = df[df["#test_idx"]!=0]["#test_idx"].values.astype(int) - 1
  246. sorted_train_img_path = [train_things_paths[i] for i in train_indices]
  247. sorted_test_img_path = [test_things_paths[i] for i in test_indices]
  249. ## Avg activity over test repetitions
  250. test_neural_avg = []
  251. test_features_avg = []
  252. test_latents_avg = []
  253. selected_test_imgs = []
  254. for idx in tqdm.tqdm(np.unique(test_indices)):
  255. test_neural_avg.append(test_neural[:,test_indices==idx].mean(1))
  256. test_features_avg.append(np.array(test_features)[test_indices==idx].mean(0))
  257. test_latents_avg.append(np.array(test_latents)[test_indices==idx].mean(0))
  258. selected_test_imgs.append(np.array(sorted_test_img_path)[test_indices==idx][0])
  260. test_neural_avg = np.array(test_neural_avg).transpose(1,0,-1)
  261. test_features_avg = np.array(test_features_avg)
  262. test_latents_avg = np.array(test_latents_avg)
  263. selected_test_imgs = np.array(selected_test_imgs)
  265. print(test_neural_avg.shape, selected_test_imgs.shape, test_features_avg.shape, test_latents_avg.shape)
  267. # %%
  268. # test_neural_avg = np.load(data_path + '/monkeys/test_neural_avg.npy')
  269. # train_neural = np.load(data_path + '/monkeys/train_neural.npy')
  271. # %%
  272. train_neural = train_neural[100:] # (200, 15000, 1024)
  273. test_neural_avg = test_neural_avg[100:] # (200, 100, 1024)
  275. # %%
  276. print("Train features shape", train_features.shape)
  277. print("Test features shape", test_features_avg.shape)
  279. print("Train latents shape", train_latents.shape)
  280. print("Test latents shape", test_latents_avg.shape)
  282. print("Train neural shape", train_neural.shape)
  283. print("Test neural shape", test_neural_avg.shape)
  285. # %% [markdown]
  286. # ## Soft Mapping Model
  288. # %%
  289. from torch.utils.data import TensorDataset, DataLoader
  290. from torch.utils.data import random_split
  291. from sklearn.preprocessing import StandardScaler
  293. batch_size = 128
  294. seed = 42
  295. device = "cuda:3" if torch.cuda.is_available() else "cpu"
  297. X_train = train_neural.transpose(1, 0, 2)
  298. Y_train = train_features[:,-1]
  299. Y_train_lat = train_latents.reshape(len(train_latents),-1)
  300. X_test = test_neural_avg.transpose(1, 0, 2) # shape: (22248, 200, 1024)
  301. Y_test = torch.tensor(test_features_avg[:,-1])
  302. Y_test_lat = test_latents_avg.reshape(len(test_latents_avg),-1)
  304. scaler_X = StandardScaler()
  305. X_reshaped = X_train.reshape(-1, X_train.shape[-1])
  306. # X_reshaped = X_train.reshape(X_train.shape[0], -1)
  307. X_scaled = scaler_X.fit_transform(X_reshaped)
  308. X_train_tensor = torch.tensor(X_scaled.reshape(22248, 200, 1024), dtype=torch.float32, device=device)
  309. X_test_reshaped = X_test.reshape(-1, X_test.shape[-1])
  310. X_test_scaled = scaler_X.transform(X_test_reshaped)
  311. X_test_tensor = torch.tensor(X_test_scaled.reshape(100, 200, 1024), dtype=torch.float32, device=device)
  313. # %%
  314. dataset = TensorDataset(X_train_tensor, torch.tensor(Y_train, dtype=torch.float32, device=device))
  315. test_dataset = TensorDataset(X_test_tensor, torch.tensor(Y_test, dtype=torch.float32, device=device))
  316. val_size = int(0.2 * len(dataset))
  317. train_size = len(dataset) - val_size
  318. generator1 = torch.Generator().manual_seed(seed)
  319. train_dataset, val_dataset = random_split(dataset, [train_size, val_size], generator=generator1)
  321. train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
  322. val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
  323. test_loader_feat = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
  325. # %%
  326. from pytorch_lightning import Trainer
  327. from pytorch_lightning.loggers import CSVLogger
  328. from pytorch_lightning.callbacks import EarlyStopping
  329. import pytorch_lightning as pl
  330. import torch
  331. import torch.nn as nn
  332. import torch.nn.functional as F
  333. import geomloss
  336. class SoftMapping(pl.LightningModule):
  337. def __init__(self, input_dim=1024, output_dim=1280, lr=1e-4, tau=0.05):
  338. super().__init__()
  340. self.mlp = nn.Sequential(
  341. nn.Linear(input_dim, 768),
  342. nn.GELU(),
  343. nn.Dropout(0.5),
  344. nn.Linear(768, output_dim)
  345. )
  346. self.attn_linear = nn.Sequential(
  347. nn.Linear(input_dim, 1),
  348. nn.Dropout(0.5)
  349. )
  350. self.lin = nn.Sequential(
  351. nn.Linear(input_dim, output_dim)
  352. )
  353. self.loss_mse = nn.MSELoss()
  354. self.lr = lr
  355. self.tau = tau
  356. self.log_tau = nn.Parameter(torch.tensor(np.log(tau), dtype=torch.float32))
  358. def forward(self, x): # shape: (batch, 200, 1024)
  359. # attn_weights = torch.softmax(self.attn_linear(x), dim=1)
  360. attn_weights = torch.sigmoid(self.attn_linear(x))
  361. attn_out = torch.mean(attn_weights * x, dim=1)
  362. output = self.mlp(attn_out)
  363. return output, attn_weights
  365. def cosine_similarity_matrix(self, A, B):
  366. A_norm = F.normalize(A, dim=1)
  367. B_norm = F.normalize(B, dim=1)
  368. return torch.mm(A_norm, B_norm.T)
  370. def contrastive_loss_nt(self, S, tau):
  371. tau = torch.exp(self.log_tau)
  372. S_exp = torch.exp(S / tau)
  373. loss = -torch.log(torch.diag(S_exp) / S_exp.sum(dim=1))
  374. return loss.mean()
  376. def training_step(self, batch, batch_idx):
  377. x, y = batch
  378. # loss = self.loss_fn(self(x), y)
  379. output, attn_weights = self(x)
  380. cos_matrix = self.cosine_similarity_matrix(output, y)
  381. loss = self.contrastive_loss_nt(cos_matrix, self.tau)
  382. self.log("train_loss", loss, on_epoch=True, prog_bar=True)
  383. self.log("tau", torch.exp(self.log_tau).item(), prog_bar=True)
  384. return loss
  386. def validation_step(self, batch, batch_idx):
  387. x, y = batch
  388. # loss = self.loss_cl(self(x), y)
  389. output, attn_weights = self(x)
  390. cos_matrix = self.cosine_similarity_matrix(output, y)
  391. loss = self.contrastive_loss_nt(cos_matrix, self.tau)
  392. self.log("val_loss", loss, on_epoch=True, prog_bar=True)
  393. return loss
  395. def configure_optimizers(self):
  396. return torch.optim.Adam(self.parameters(), lr=self.lr, weight_decay=1e-4)
  398. # %%
  399. from models_nnlinear import SimpleTCN
  400. from models_linear import MlpAvgTime
  401. import pytorch_lightning as pl
  402. from pytorch_lightning.loggers import CSVLogger
  403. from pytorch_lightning.callbacks import EarlyStopping
  404. from pytorch_lightning import Trainer
  407. pl.seed_everything(seed, workers=True)
  408. model_feat = SoftMapping(input_dim=1024, output_dim=1280, lr=1e-4, tau=0.05)
  409. logger = CSVLogger("/home/repo/nlinear-monkeys/logs/", name="my_model")
  410. early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=0.09, patience=10, verbose=True, mode="min")
  411. trainer = Trainer(max_epochs=50, devices=[1], logger=logger, callbacks=[early_stop_callback])
  412. trainer.fit(model_feat, train_loader, val_loader)
  414. # %%
  415. import pandas as pd
  416. import matplotlib.pyplot as plt
  418. log_path = logger.log_dir + "/metrics.csv"
  419. df = pd.read_csv(log_path)
  421. val_loss_df = df[~df["val_loss"].isna()]
  422. train_loss_df = df[~df["train_loss_epoch"].isna()]
  423. plt.figure(figsize=(8, 5))
  424. plt.plot(np.array(train_loss_df["epoch"]), np.array(train_loss_df["train_loss_epoch"]), label="Train Loss")
  425. plt.plot(np.array(val_loss_df["epoch"]), np.array(val_loss_df["val_loss"]), label="Val Loss")
  426. plt.xlabel("Epoch")
  427. plt.ylabel("MSE Loss")
  428. plt.title("Training vs Validation Loss")
  429. plt.legend()
  430. plt.grid(True)
  431. plt.tight_layout()
  432. plt.show()
  434. # %%
  435. x,y = next(iter(test_loader_feat))
  437. y_pred_feat=[]
  438. y_true_feat=[]
  439. attn_weights_feat = []
  441. model_feat.eval()
  442. model_feat.to(device)
  443. with torch.no_grad():
  444. for x,y in tqdm.tqdm(test_loader_feat):
  445. y_hat, attn_weights = model_feat(x)
  446. # y_hat = model_feat(x)
  447. y_true_feat.append(y)
  448. y_pred_feat.append(y_hat)
  449. attn_weights_feat.append(attn_weights.cpu())
  451. y_pred_feat=torch.cat(y_pred_feat,0)
  452. y_true_feat=torch.cat(y_true_feat,0)
  453. attn_weights_feat = torch.cat(attn_weights_feat, dim=0).squeeze(-1)
  455. # %%
  456. import seaborn as sns
  458. sns.heatmap(attn_weights_feat)
  460. # %%
  461. from sklearn.neighbors import NearestNeighbors
  463. n_neighbors = 5
  464. y_true_np = y_true_feat.cpu().numpy()
  465. y_pred_np = y_pred_feat.cpu().numpy()
  466. nbrs = NearestNeighbors(n_neighbors=n_neighbors, metric='cosine').fit(y_true_np)
  468. distances, top_indices = nbrs.kneighbors(y_pred_np)
  469. true_indices = torch.arange(len(y_true_np)).cpu().numpy()
  471. top1_count = (top_indices[:, 0] == true_indices).sum()
  472. top3_count = sum(true_idx in top_indices[i] for i, true_idx in enumerate(true_indices))
  474. print(f"Top-1 accuracy: {top1_count}/{len(y_true_np)} ({top1_count / len(y_true_np) * 100:.2f}%)")
  475. print(f"Top-5 accuracy: {top3_count}/{len(y_true_np)} ({top3_count / len(y_true_np) * 100:.2f}%)")
  477. # %%
  478. idx = 0
  480. fig, axs = plt.subplots(5, n_neighbors+1, figsize=(10, 10))
  482. for j in range(5):
  483. axs[j, 0].imshow(Image.open(opj(thing_base_path,"THINGS","Images", test_things_paths[idx+j]).replace("\\","/")))
  484. axs[j, 0].axis("off")
  485. axs[j, 0].set_title("Original Image")
  487. for i, neighbor_idx in enumerate(top_indices[idx+j]):
  488. img = Image.open(opj(thing_base_path,"THINGS","Images", test_things_paths[neighbor_idx]).replace("\\","/"))
  489. axs[j, i+1].imshow(img)
  490. axs[j, i+1].axis("off")
  491. axs[j, i+1].set_title(f"Neighbor {i+1}")
  493. plt.tight_layout()
  495. # %%
  496. import random
  497. import matplotlib.pyplot as plt
  498. from PIL import Image
  499. from matplotlib.patches import Rectangle
  500. from os.path import join as opj
  502. random.seed(34) # 11
  503. num_rows = 5 # Numero di immagini da mostrare
  504. random_indices = random.sample(range(len(test_things_paths)), num_rows)
  506. fig, axs = plt.subplots(num_rows, n_neighbors + 1, figsize=(15, 10))
  508. for j, idx in enumerate(random_indices):
  509. original_path = test_things_paths[idx]
  510. original_full_path = opj(thing_base_path, "THINGS", "Images", original_path).replace("\\", "/")
  512. axs[j, 0].imshow(Image.open(original_full_path))
  513. axs[j, 0].axis("off")
  514. axs[j, 0].set_title("Original Image")
  516. highlight_original = False # Flag per sapere se devo evidenziare l'originale
  518. for i, neighbor_idx in enumerate(top_indices[idx]):
  519. if neighbor_idx >= len(test_things_paths):
  520. continue
  522. neighbor_path = test_things_paths[neighbor_idx]
  523. neighbor_full_path = opj(thing_base_path, "THINGS", "Images", neighbor_path).replace("\\", "/")
  525. img = Image.open(neighbor_full_path)
  526. axs[j, i + 1].imshow(img)
  527. axs[j, i + 1].axis("off")
  528. axs[j, i + 1].set_title(f"Neighbor {i + 1}")
  530. if neighbor_path == original_path:
  531. highlight_original = True
  532. rect = Rectangle(
  533. (0, 0), 1, 1,
  534. transform=axs[j, i + 1].transAxes,
  535. fill=False, color='lime', linewidth=8
  536. )
  537. axs[j, i + 1].add_patch(rect)
  539. if highlight_original:
  540. rect = Rectangle(
  541. (0, 0), 1, 1,
  542. transform=axs[j, 0].transAxes,
  543. fill=False, color='lime', linewidth=8
  544. )
  545. axs[j, 0].add_patch(rect)
  547. plt.tight_layout()
  548. plt.subplots_adjust(wspace=-0.6)
  550. plt.show()
  553. # %% [markdown]
  554. # ## Soft Mapping Latens
  556. # %%
  557. dataset = TensorDataset(X_train_tensor, torch.tensor(Y_train_lat, dtype=torch.float32, device=device))
  558. test_dataset = TensorDataset(X_test_tensor, torch.tensor(Y_test_lat, dtype=torch.float32, device=device))
  559. val_size = int(0.2 * len(dataset))
  560. train_size = len(dataset) - val_size
  561. generator1 = torch.Generator().manual_seed(seed)
  562. train_dataset, val_dataset = random_split(dataset, [train_size, val_size], generator=generator1)
  564. train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
  565. val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
  566. test_loader_lat = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
  568. # %%
  569. pl.seed_everything(seed, workers=True)
  570. model_latents = SoftMapping(input_dim=1024, output_dim=4*32*32, lr=1e-4, tau=0.05) # 1e-4 per generativo
  571. logger = CSVLogger("/home/repo/nlinear-monkeys/logs/", name="my_model")
  572. early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=0.09, patience=10, verbose=True, mode="min")
  573. trainer = Trainer(max_epochs=50, devices=[1], logger=logger, callbacks=[early_stop_callback])
  574. trainer.fit(model_latents, train_loader, val_loader)
  576. # %%
  577. import pandas as pd
  578. import matplotlib.pyplot as plt
  580. log_path = logger.log_dir + "/metrics.csv"
  581. df = pd.read_csv(log_path)
  583. val_loss_df = df[~df["val_loss"].isna()]
  584. train_loss_df = df[~df["train_loss_epoch"].isna()]
  585. plt.figure(figsize=(8, 5))
  586. plt.plot(np.array(train_loss_df["epoch"]), np.array(train_loss_df["train_loss_epoch"]), label="Train Loss")
  587. plt.plot(np.array(val_loss_df["epoch"]), np.array(val_loss_df["val_loss"]), label="Val Loss")
  588. plt.xlabel("Epoch")
  589. plt.ylabel("MSE Loss")
  590. plt.title("Training vs Validation Loss")
  591. plt.legend()
  592. plt.grid(True)
  593. plt.tight_layout()
  594. plt.show()
  596. # %%
  597. x,y = next(iter(test_loader_lat))
  599. y_pred_lat=[]
  600. y_true_lat=[]
  601. attn_weights_lat = []
  603. model_latents.eval()
  604. model_latents.to(device)
  605. with torch.no_grad():
  606. for x,y in tqdm.tqdm(test_loader_lat):
  607. y_hat, attn_weights = model_latents(x)
  608. # y_hat = model_latents(x)
  609. y_true_lat.append(y.reshape(len(y),4,32,32))
  610. y_pred_lat.append(y_hat.reshape(len(y_hat),4,32,32))
  611. attn_weights_lat.append(attn_weights.cpu())
  613. y_pred_lat=torch.cat(y_pred_lat,0)
  614. y_true_lat=torch.cat(y_true_lat,0)
  615. attn_weights_lat = torch.cat(attn_weights_lat, dim=0).squeeze(-1)
  617. # %%
  618. y_pred_lat.shape, y_true_lat.shape, y_pred_lat.device
  620. # %%
  621. import seaborn as sns
  623. sns.heatmap(attn_weights_lat)
  625. # %% [markdown]
  626. # ## Structural Decoding
  628. # %%
  629. y_pred_lat_train=[]
  631. model_latents.eval()
  632. model_latents.to(device)
  633. with torch.no_grad():
  634. for x,y in tqdm.tqdm(train_loader):
  635. y_hat, attn_weights = model_latents(x)
  636. # y_hat = model_latents(x)
  637. y_pred_lat_train.append(y_hat.reshape(len(y_hat),4,32,32))
  639. y_pred_lat_train=torch.cat(y_pred_lat_train,0)
  641. # %%
  642. y_pred_lat_train.shape
  644. # %%
  645. expected_latents_mean = train_latents.mean(0).to(device)
  646. expected_latents_std = train_latents.std(0).to(device)
  648. predicted_latents_mean = y_pred_lat_train.mean(0)
  649. predicted_latents_std = y_pred_lat_train.std(0)
  651. # %%
  652. def adjust_latents(latents, expected_latents_mean=expected_latents_mean, expected_latents_std=expected_latents_std, predicted_latents_mean=predicted_latents_mean, predicted_latents_std=predicted_latents_std):
  653. return ((latents - predicted_latents_mean) / predicted_latents_std) * expected_latents_std + expected_latents_mean
  655. test_latents_pred = adjust_latents(y_pred_lat).float()
  657. # %%
  658. test_latents_pred.shape, test_latents_pred.mean(), test_latents_pred.std(), expected_latents_mean.mean(), expected_latents_std.mean()
  660. # %%
  661. def reconstruct_images(latents):
  662. with torch.no_grad():
  663. # Scale the latents (SDXL typically scales latents by a factor)
  664. latents = latents * vae.scaling_factor
  666. # Decode latents back to image
  667. decoded_image = vae.decode(latents / vae.scaling_factor).sample
  669. # Convert decoded tensor back to PIL Image
  670. decoded_image = decoded_image.squeeze(0).permute(1, 2, 0)
  671. decoded_image = (decoded_image.clamp(-1, 1) + 1) / 2
  672. decoded_image = (decoded_image.cpu().numpy() * 255).astype(np.uint8)
  673. decoded_pil = Image.fromarray(decoded_image)
  674. return decoded_pil
  676. # %%
  677. idx = 5
  679. fig, axs = plt.subplots(5, 2, figsize=(10, 10))
  681. images = []
  682. latents = test_latents_pred[idx:idx+5].to(device)
  684. for lat in latents:
  685. images.append(reconstruct_images(lat.unsqueeze(0)))
  688. for i, img in enumerate(images):
  689. axs[i,0].imshow(Image.open(opj(thing_base_path, "THINGS", "Images", test_things_paths[idx+i]).replace("\\", "/")))
  690. axs[i,0].axis("off")
  691. axs[i,0].set_title(f"Original {i+1}")
  693. axs[i,1].imshow(img)
  694. axs[i,1].axis("off")
  695. axs[i,1].set_title(f"Reconstructed {i+1}")
  697. plt.tight_layout()
  699. # %%
  700. y_pred_feat_train=[]
  702. model_feat.eval()
  703. model_feat.to(device)
  704. with torch.no_grad():
  705. for x,y in tqdm.tqdm(train_loader):
  706. y_hat, attn_weights = model_feat(x)
  707. # y_hat = model_feat(x)
  708. y_pred_feat_train.append(y_hat)
  710. y_pred_feat_train=torch.cat(y_pred_feat_train,0)
  712. # %%
  713. y_pred_feat_train.shape
  715. # %%
  716. expected_features_mean = train_features[:,-1].mean(0).to(device)
  717. expected_features_std = train_features[:,-1].std(0).to(device)
  719. predicted_features_mean = y_pred_feat_train.mean(0)
  720. predicted_features_std = y_pred_feat_train.std(0)
  722. # %%
  723. def adjust_features(features, expected_features_mean=expected_features_mean, expected_features_std=expected_features_std, predicted_features_mean=predicted_features_mean, predicted_features_std=predicted_features_std):
  725. print(features.shape)
  726. scaled_pred = ((features - predicted_features_mean) / predicted_features_std) * expected_features_std + expected_features_mean
  727. zeros = torch.tensor(np.zeros_like(features.cpu()), device=device)
  729. return torch.stack([zeros,scaled_pred], axis=1)
  731. # %%
  732. test_features_pred = adjust_features(y_pred_feat)
  734. # %%
  735. test_features_pred.shape
  737. # %% [markdown]
  738. # ## Retrieve Structural similar images
  740. # %%
  741. # Number of nearest neighbors to find
  742. n_neighbors = 5
  744. # Create and fit the NearestNeighbors model
  745. nn_struct_model = NearestNeighbors(n_neighbors=n_neighbors, metric='cosine')
  746. nn_struct_model.fit(train_latents.reshape(len(train_latents),-1))
  748. # Find the nearest neighbors for the adjusted embeddings
  749. distances_struct, nearest_neighbors_struct_indices = nn_struct_model.kneighbors(test_latents_pred.cpu().reshape(len(test_latents_pred),-1))
  751. # %%
  752. ## Show original test image idx, reconstructed image idx from pred latents and 5 closest neighbors in train images
  754. idx = 30
  756. fig, axs = plt.subplots(5, n_neighbors+2, figsize=(10, 10))
  758. for j in range(5):
  759. axs[j, 0].imshow(Image.open(opj(thing_base_path, "THINGS", "Images", test_things_paths[idx+j]).replace("\\", "/")))
  760. axs[j, 0].axis("off")
  761. axs[j, 0].set_title("Original Image")
  763. axs[j, 1].imshow(reconstruct_images(test_latents_pred[idx+j].unsqueeze(0).to(device)))
  764. axs[j, 1].axis("off")
  765. # axs[j, 1].set_title("Reconstructed Image")
  767. for i, neighbor_idx in enumerate(nearest_neighbors_struct_indices[idx+j]):
  768. img = Image.open(train_images[neighbor_idx])
  769. axs[j, i+2].imshow(img)
  770. axs[j, i+2].axis("off")
  771. axs[j, i+2].set_title(f"Neighbor {i+1}")
  773. plt.tight_layout()
  775. # %% [markdown]
  776. # ## Reconstruct Images
  778. # %%
  779. idx_rec = 77 # 9, 12, 22, 28, 40, 46, 48, 55, 58, 77
  781. # %%
  782. test_features_pred.shape, test_features.shape, test_features_pred[idx_rec].half().unsqueeze(0).shape, idx_rec
  784. # %%
  785. seed = 55 # 42, 55, 999
  786. generator = torch.Generator(device=device).manual_seed(seed)
  788. prepared_latents = pipeline.prepare_latents(batch_size=1,latents=test_latents_pred[idx_rec].unsqueeze(0).half(), num_channels_latents=4, height=32, width=32, dtype =torch.float16,device=device, generator=generator)
  790. recon = pipeline(
  791. prompt="",
  792. ip_adapter_image_embeds=[test_features_pred[idx_rec].half().unsqueeze(-2)],
  793. negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
  794. num_inference_steps = 50,
  795. guide_strength = 0.9,
  796. guidance_scale = 10,
  797. num_images_per_prompt = 4,
  798. generator=generator,
  799. ).images
  801. # %%
  802. init_latents = test_latents_pred[idx_rec].unsqueeze(0).half().to(device)
  803. low_res = reconstruct_images(init_latents.float().to(device))
  805. low_res
  807. # %%
  808. from skimage.metrics import structural_similarity as ssim
  810. recon_small = [i.resize((256,256)) for i in recon]
  811. recon_smooth = [i.filter(ImageFilter.GaussianBlur(6)) for i in recon_small]
  813. ssim_scores = [ssim(np.array(recon_smooth[i]), np.array(low_res), win_size=3) for i in range(4)]
  814. recon_sorted = [recon[i] for i in np.argsort(ssim_scores)[::-1]]
  816. # %%
  817. fig, axs = plt.subplots(1, 3, figsize=(20, 20)) # 1 riga, 3 colonne
  819. # Mostra immagine originale
  820. axs[0].imshow(Image.open(opj(thing_base_path, "THINGS", "Images", test_things_paths[idx_rec]).replace("\\", "/")))
  821. axs[0].axis("off")
  822. axs[0].set_title("Original Image", fontsize=25)
  824. # Mostra immagine a bassa risoluzione
  825. axs[1].imshow(low_res)
  826. axs[1].axis("off")
  827. axs[1].set_title("Low Resolution", fontsize=25)
  829. # Seleziona solo la ricostruzione con SSIM più alto
  830. best_recon_idx = sorted(range(len(ssim_scores)), key=lambda i: ssim_scores[i], reverse=True)[0]
  831. best_recon_img = recon_sorted[best_recon_idx]
  832. best_ssim_score = ssim_scores[best_recon_idx]
  834. # Mostra la miglior ricostruzione
  835. axs[2].imshow(best_recon_img)
  836. axs[2].axis("off")
  837. axs[2].set_title(f"Reconstructed, SSIM: {best_ssim_score:.2f}", fontsize=25)
  839. # Migliora la spaziatura
  840. plt.tight_layout()
  841. plt.subplots_adjust(wspace=0.3)
  843. plt.show()
  848. # %% [markdown]
  849. # ## Save Reconstructed
  851. # %%
  852. import os
  853. import torch
  854. import numpy as np
  855. from skimage.metrics import structural_similarity as ssim
  856. from PIL import Image, ImageFilter
  857. import matplotlib.pyplot as plt
  858. from os.path import join as opj
  860. save_recon_dir = data_path + "/monkeys/generative/img_gen_attmlp"
  861. # os.makedirs(save_true_dir, exist_ok=True)
  862. os.makedirs(save_recon_dir, exist_ok=True)
  864. seed = 55
  865. generator = torch.Generator(device=device).manual_seed(seed)
  867. for idx_rec in tqdm.tqdm(range(100)):
  869. # Prepara i latents
  870. prepared_latents = pipeline.prepare_latents(
  871. batch_size=1,
  872. latents=test_latents_pred[idx_rec].unsqueeze(0).half(),
  873. num_channels_latents=4,
  874. height=32,
  875. width=32,
  876. dtype=torch.float16,
  877. device=device,
  878. generator=generator
  879. )
  881. # Genera 4 ricostruzioni
  882. recon = pipeline(
  883. prompt="",
  884. ip_adapter_image_embeds=[test_features_pred[idx_rec].half().unsqueeze(-2)],
  885. negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
  886. num_inference_steps=50,
  887. guide_strength=0.9,
  888. guidance_scale=10,
  889. num_images_per_prompt=4,
  890. generator=generator,
  891. ).images
  893. # Prepara immagine low-res
  894. init_latents = test_latents_pred[idx_rec].unsqueeze(0).half()
  895. low_res = reconstruct_images(init_latents.float().to(device))
  897. # Prepara recon per SSIM
  898. recon_small = [i.resize((256, 256)) for i in recon]
  899. recon_smooth = [i.filter(ImageFilter.GaussianBlur(6)) for i in recon_small]
  901. # Calcola SSIM tra low_res e ogni ricostruzione
  902. ssim_scores = [ssim(np.array(recon_smooth[i]), np.array(low_res), win_size=3) for i in range(4)]
  904. # Seleziona la ricostruzione migliore
  905. best_recon_idx = np.argmax(ssim_scores)
  906. best_recon_img = recon[best_recon_idx]
  908. # Immagine vera
  909. # true_img = Image.open(opj(thing_base_path, "THINGS", "Images", test_things_paths[idx_rec]).replace("\\", "/"))
  910. # true_img.save(os.path.join(save_true_dir, f"true_{idx_rec:03d}.png"))
  912. # Salva immagine ricostruita
  913. best_recon_img.save(os.path.join(save_recon_dir, f"recon_{idx_rec:03d}.png"))
  915. print("✅ Tutte le immagini salvate!")
  918. # %%
  921. # %%

monkeys_gener.ipynb at commit 8ad7b9a, no license · at the source

Overview

Authors: Matteo Ciferri1, Matteo Ferrante1,2, Nicola Toschi1,3
ORCID iDs: Matteo Ciferri
  1. Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy
  2. Tether Evo, Edificio Centro Corporativo Presidente Plaza, San Salvador, El Salvador
  3. A.A. Martinos Center for Biomedical Imaging, Harvard Medical School/MGH, Boston, MA, United States
Institutions: University of Rome Tor Vergata (Italy); Harvard University (United States); Athinoula A. Martinos Center for Biomedical Imaging (United States)
Journal: Imaging neuroscience (Cambridge, Mass.), volume 4, article IMAG.a.1299
Dates: received 26 November 2025; accepted 19 June 2026; published online 13 July 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1162/imag.a.1299 · PMID 42453642 · PMCID PMC13366611 · OpenAlex W7124817775
Open access: green, a free copy (OpenAlex)
Preprint: arxiv:2601.11108
Status: code verified
Categories: methods / tools (subfield)
Methods: Smoothing, state filtering, decompositions, Statistics, Preprocessing, Machine learning, fMRI & imaging
Keywords: visual decoding, intracortical recordings, primate visual cortex, machine learning, generative reconstruction, brain–computer interfaces
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Citations: not cited yet (Europe PMC); 35 references in the paper

Abstract

Characterizing the information content of intracortical signals during visual processing is a central challenge in systems neuroscience. We address the problem of decoding visual information from high-density intracortical recordings in primates, using the THINGS Ventral Stream Spiking Dataset. We systematically evaluate the effects of model architecture, training objectives, and data scaling on decoding performance. Results show that decoding accuracy is jointly driven by non-linearity and selective temporal aggregation, rather than heavier sequence modelling in this data regime. A simple model combining temporal attention with a shallow MLP achieves up to 70% top-1 image retrieval accuracy, outperforming linear baselines as well as recurrent and convolutional approaches. Scaling analyses reveal predictable diminishing returns with increasing input dimensionality and dataset size. Building on these findings, we design a modular generative decoding pipeline that combines low-resolution latent reconstruction with semantically conditioned diffusion, generating plausible images from 200 ms of brain activity. This framework provides principles for brain-computer interfaces and semantic neural decoding.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 4 matches between paragraphs and lines of code.

gin.g-node.org/paolo_papale/tvsd

License: CC-BY-4.0
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: f26bd45f86f5413432e42f30880418be6930ec19, 5 December 2025
Languages: MATLAB (126), Python (54), C (2), C/C++ (1), Jupyter (1)
Size: 1,345 files, 184 scripts
Software Heritage: not archived
Found in: “Data and Code Availability”
Holds: README, license file, environment (_code/lucent-things/setup.py), 1 notebook
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: PyTorch (34 files), NumPy (25 files), Statistics and Machine Learning Toolbox (6 files), Pillow (6 files), Signal Processing Toolbox (5 files), h5py (2 files), Matplotlib (2 files), scikit-learn (2 files), SciPy (2 files), Curve Fitting Toolbox (1 file), Image Processing Toolbox (1 file), scikit-image (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
186 files
At the source: gin.g-node.org/paolo_papale/TVSD

fidelioc55/primates-mua-decode

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 8ad7b9a2a00f5a7a61b84599fde1eaafcd532e4a, 24 September 2025
Languages: Jupyter (3), Python (2)
Size: 6 files, 5 scripts
Software Heritage: not archived
Found in: “Data and Code Availability”
Holds: README, 3 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (5 files), PyTorch (5 files), PyTorch Lightning (4 files), pandas (3 files), Pillow (3 files), SciPy (3 files), Hugging Face Transformers (3 files), h5py (2 files), Matplotlib (2 files), scikit-image (2 files), scikit-learn (2 files), seaborn (2 files)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
6 files
At the source: github.com/fidelioc55/primates-mua-decode

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 189 scripts, each with its path and the digest of its content;
  • 4 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data and Code Availability

All data are from a publicly available dataset (THINGS Ventral Stream Spiking Dataset) at: https://gin.g-node.org/paolo_papale/TVSD. Implementation code for reproducibility is available at the repository: https://github.com/fidelioc55/primates-mua-decode.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 3 authors, 6 keywords, 34 references.

Cite

This paper

Ciferri, M., Ferrante, M., & Toschi, N. (2026). A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration. Imaging neuroscience (Cambridge, Mass.), 4, IMAG.a.1299. https://doi.org/10.1162/imag.a.1299

BibTeX

@article{ciferri2026modular,
author = {Ciferri, Matteo and Ferrante, Matteo and Toschi, Nicola},
title = {{A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration}},
journal = {Imaging neuroscience (Cambridge, Mass.)},
year = {2026},
month = jul,
volume = {4},
pages = {IMAG.a.1299},
publisher = {MIT Press},
issn = {2837-6056},
doi = {10.1162/imag.a.1299},
url = {https://doi.org/10.1162/imag.a.1299},
pmid = {42453642},
pmcid = {PMC13366611}
}

RIS

TY - JOUR
AU - Ciferri, Matteo
AU - Ferrante, Matteo
AU - Toschi, Nicola
TI - A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration
T2 - Imaging neuroscience (Cambridge, Mass.)
J2 - Imaging Neurosci (Camb)
PY - 2026
DA - 2026/07/13
VL - 4
SP - IMAG.a.1299
SN - 2837-6056
PB - MIT Press
DO - 10.1162/imag.a.1299
UR - https://doi.org/10.1162/imag.a.1299
LA - en
ER -

CSL-JSON

{
"id": "10.1162/imag.a.1299",
"type": "article-journal",
"title": "A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration",
"container-title": "Imaging neuroscience (Cambridge, Mass.)",
"author": [
{
"family": "Ciferri",
"given": "Matteo"
},
{
"family": "Ferrante",
"given": "Matteo"
},
{
"family": "Toschi",
"given": "Nicola"
}
],
"container-title-short": "Imaging Neurosci (Camb)",
"volume": "4",
"page": "IMAG.a.1299",
"DOI": "10.1162/imag.a.1299",
"PMID": "42453642",
"PMCID": "PMC13366611",
"ISSN": "2837-6056",
"publisher": "MIT Press",
"URL": "https://doi.org/10.1162/imag.a.1299",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
13
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1007/s12021-026-09803-3 [code]
NeuroFusion: A Unified Framework for Generalized Visual Stimulus Decoding from fMRI Across Datasets and Subjects.
Journal: Neuroinformatics
In common: PyTorch Lightning, scikit-image, h5py, 7 other tools, methods / tools, 8 references
[2] doi:10.1371/journal.pcbi.1014263 [code]
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.
Journal: PLoS computational biology
In common: PyTorch Lightning, Hugging Face Transformers, scikit-image, 8 other tools, 6 references
[3] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: Hugging Face Transformers, scikit-image, h5py, 8 other tools, 5 references
[4] doi:10.1038/s42003-026-10169-0 [code]
Shared representations in brains and models reveal a two-route cortical organization during scene perception.
Journal: Communications biology
In common: Hugging Face Transformers, h5py, Pillow, 7 other tools, 4 references
[5] doi:10.1038/s41467-026-71267-5 [code]
Human-like cognitive generalization for large models via mental representation-guided supervision.
Journal: Nature communications
In common: Hugging Face Transformers, scikit-image, h5py, 7 other tools, 3 references
[6] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: PyTorch Lightning, Hugging Face Transformers, scikit-image, 10 other tools
[7] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: scikit-image, h5py, Pillow, 6 other tools, methods / tools, 4 references
[8] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: Hugging Face Transformers, scikit-image, h5py, 9 other tools, 1 reference
[9] doi:10.1038/s41593-026-02285-1 [code]
Fixation duration on natural scenes is explained by memory encoding not processing demand.
Journal: Nature neuroscience
In common: scikit-image, h5py, Pillow, 7 other tools, 3 references
[10] doi:10.1038/s41593-026-02232-0 [code]
Entorhinal cortex represents task-relevant remote locations independently of CA1.
Journal: Nature neuroscience
In common: scikit-image, h5py, Image Processing Toolbox, 10 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.