OSCR

Learnable Diffusion Framework for Mouse V1 Neural Decoding.

Code ↔ Paper

17 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 17 matches
  1. [1] § Methods › Devices and Settings for Training the Models ↔ train.py, lines 193–248 · score 0.89 · default Adam betas, AdamW, weight decay, training steps, DDP, optimized
  2. [2] § Methods › Model Architecture › The Diffusion Transformer (DiT) ↔ models.py, lines 147–168 · score 0.76 · adaptive layer norm, adaLN, MSA, MLP, head, blocks
  3. [3] § Methods › Model Architecture › The AutoEncoder ↔ gen_eval.py, lines 19–128 · score 0.76 · sd vae ft, AutoencoderKL, ema, decoded, model
  4. [4] § Methods › Model Architecture › The AutoEncoder ↔ gen_eval_aggTest.py, lines 19–116 · score 0.76 · sd vae ft, AutoencoderKL, ema, decoded, model
  5. [5] § Methods › The Classifier‐Free Guidance (CFG) ↔ models.py, lines 191–316 · score 0.65 · classifier free guidance, diffusion models, unconditioned, embedded, training
  6. [6] § Methods › Model Architecture › The Diffusion Transformer (DiT) ↔ models.py, lines 191–316 · score 0.63 · sin cos, unpatchifies, embedder, blocks, latent, timesteps
  7. [7] § Results › Sensorium‐Viz Reconstructs Stimuli via DiT With Spatially Embedded Neuron Responses ↔ models.py, lines 147–168 · score 0.59 · Adaptive Layer Norm, adaLN, blocks, Zero, model
  8. [8] § Methods › Devices and Settings for Training the Models ↔ train.py, lines 250–323 · score 0.58 · exponential moving, monitored, explode, EMA, loss, Devices
  9. [9] § Methods › The Training Losses ↔ diffusion/gaussian_diffusion.py, lines 682–713 · score 0.55 · log likelihood, log variance, discretizing, Gaussian, model
  10. [10] § Methods › Model Architecture › The Response Projector Network ↔ sensorium/neuralpredictors_cp/layers/cores/conv2d.py, lines 34–169 · score 0.54 · convolution layer, feature map, connection, nonlinearity, channel
  11. [11] § Methods › Model Architecture › The Response Projector Network ↔ sensorium/cores/conv2d.py, lines 28–165 · score 0.54 · convolution layer, feature map, connection, nonlinearity, channel
  12. [12] § Methods › Evaluation ↔ res_23656_idw32_all/metrics_full.py, lines 124–137 · score 0.54 · learned perceptual image, LPIPS, metrics, patch
  13. [13] § Methods › Process the Input Data for Model Training and Inference ↔ dataloaders/sensorium_synthetic.py, lines 19–39 · score 0.52 · Inverse Distance Weighting, PyInterp, neighbors, IDW
  14. [14] § Methods › Process the Input Data for Model Training and Inference ↔ dataloaders/sensorium_test_agg.py, lines 14–34 · score 0.52 · Inverse Distance Weighting, PyInterp, neighbors, IDW
  15. [15] § Methods › The Training Losses ↔ diffusion/diffusion_utils.py, lines 62–88 · score 0.52 · log likelihood, Gaussian distribution, discretizing
  16. [16] § Methods › The Classifier‐Free Guidance (CFG) ↔ train.py, lines 193–248 · score 0.52 · classifier free guidance, embedded, diffusion, training, models
  17. [17] § Methods › The Training Losses ↔ diffusion/gaussian_diffusion.py, lines 682–713 · score 0.51 · variational lower bound, posterior, KL, model

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 420 lines · 17 KB · MIT · 4 matches

  1. # Copyright (c) Meta Platforms, Inc. and affiliates.
  2. # All rights reserved.
  4. # This source code is licensed under the license found in the
  5. # LICENSE file in the root directory of this source tree.
  6. # --------------------------------------------------------
  7. # References:
  8. # GLIDE: https://github.com/openai/glide-text2im
  9. # MAE: https://github.com/facebookresearch/mae/blob/main/models_mae.py
  10. # --------------------------------------------------------
  12. import torch
  13. import torch.nn as nn
  14. import numpy as np
  15. import math
  16. from timm.models.vision_transformer import PatchEmbed, Attention, Mlp
  19. def modulate(x, shift, scale):
  20. return x * (1 + scale.unsqueeze(1)) + shift.unsqueeze(1)
  23. #################################################################################
  24. # Embedding Layers for Timesteps and Class Labels #
  25. #################################################################################
  27. class TimestepEmbedder(nn.Module):
  28. """
  29. Embeds scalar timesteps into vector representations.
  30. """
  31. def __init__(self, hidden_size, frequency_embedding_size=256):
  32. super().__init__()
  33. self.mlp = nn.Sequential(
  34. nn.Linear(frequency_embedding_size, hidden_size, bias=True),
  35. nn.SiLU(),
  36. nn.Linear(hidden_size, hidden_size, bias=True),
  37. )
  38. self.frequency_embedding_size = frequency_embedding_size
  40. @staticmethod
  41. def timestep_embedding(t, dim, max_period=10000):
  42. """
  43. Create sinusoidal timestep embeddings.
  44. :param t: a 1-D Tensor of N indices, one per batch element.
  45. These may be fractional.
  46. :param dim: the dimension of the output.
  47. :param max_period: controls the minimum frequency of the embeddings.
  48. :return: an (N, D) Tensor of positional embeddings.
  49. """
  50. # https://github.com/openai/glide-text2im/blob/main/glide_text2im/nn.py
  51. half = dim // 2
  52. freqs = torch.exp(
  53. -math.log(max_period) * torch.arange(start=0, end=half, dtype=torch.float32) / half
  54. ).to(device=t.device)
  55. args = t[:, None].float() * freqs[None]
  56. embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
  57. if dim % 2:
  58. embedding = torch.cat([embedding, torch.zeros_like(embedding[:, :1])], dim=-1)
  59. return embedding
  61. def forward(self, t):
  62. t_freq = self.timestep_embedding(t, self.frequency_embedding_size)
  63. t_emb = self.mlp(t_freq)
  64. return t_emb
  67. class GridResponseEncoder(nn.Module):
  68. """
  69. Encode the response grids into vector representations utilize some structures of
  70. DownBlock of DDPM Unet
  71. """
  72. def __init__(self, num_grids, in_channels, hidden_size, drop_prob=0):
  73. super().__init__()
  74. self.num_grids = num_grids
  75. self.in_channels = in_channels
  76. self.drop_prob = drop_prob
  78. self.conv_in = nn.Conv2d(in_channels, 32, kernel_size=3, padding=1)
  80. self.resnet_conv_first = nn.Sequential(
  81. nn.GroupNorm(8, 32),
  82. nn.SiLU(),
  83. nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
  84. )
  85. self.residual_input_conv = nn.Conv2d(32, 64, kernel_size=1)
  87. self.attn_norm = nn.GroupNorm(8, 64)
  88. self.attn = nn.MultiheadAttention(64, num_heads=16, batch_first=True)
  90. self.norm_out = nn.GroupNorm(8, 64)
  91. self.conv_out = nn.Conv2d(64, 1, kernel_size=1)
  93. self.linear = nn.Linear(num_grids ** 2, hidden_size, bias=False)
  95. def drop_response(self, grid_responses):
  96. """
  97. Simulate the classifier-free guidance with 0 matrix
  98. """
  99. # create null response
  100. # null_response = torch.zeros_like(grid_responses[:, :1, :, :]) # Zero out only the first channel
  101. # null_response = torch.cat([null_response, grid_responses[:, 1:3, :, :].detach()], dim=1)
  102. null_response = torch.clone(grid_responses)
  103. null_response[:, :1, :, :] = 0
  105. drop_ids = torch.rand(grid_responses.shape[0], device=grid_responses.device) < self.drop_prob
  106. drop_ids = drop_ids.view(-1, 1, 1, 1) # Reshape for broadcasting across all dimensions
  108. # Use broadcasting to apply drop_ids across all channels and spatial dimensions
  109. grid_responses = torch.where(drop_ids, null_response, grid_responses)
  110. return grid_responses
  112. def forward(self, x, train):
  113. use_dropout = self.drop_prob > 0
  114. if train and use_dropout:
  115. # print(f"Dropout is {self.drop_prob}. Use CFG")
  116. x = self.drop_response(x)
  117. # else:
  118. # print(f"Dropout is {self.drop_prob}. No CFG")
  120. out = self.conv_in(x)
  122. resnet_input = out
  123. out = self.resnet_conv_first(out)
  124. out = out + self.residual_input_conv(resnet_input)
  126. batch_size, channels, h, w = out.shape
  127. in_attn = out.reshape(batch_size, channels, h * w).contiguous()
  128. in_attn = self.attn_norm(in_attn)
  129. in_attn = in_attn.transpose(1, 2).contiguous()
  130. out_attn, _ = self.attn(in_attn, in_attn, in_attn)
  131. out_attn = out_attn.transpose(1, 2).reshape(batch_size, channels, h, w).contiguous()
  132. out = out + out_attn
  134. out = self.norm_out(out)
  135. out = nn.SiLU()(out)
  136. out = self.conv_out(out)
  138. out = out.reshape(batch_size, -1).contiguous()
  139. out = self.linear(out)
  140. return out
  143. #################################################################################
  144. # Core DiT Model #
  145. #################################################################################
  147. class DiTBlock(nn.Module):
  148. """
  149. A DiT block with adaptive layer norm zero (adaLN-Zero) conditioning.
  150. """
  151. def __init__(self, hidden_size, num_heads, mlp_ratio=4.0, **block_kwargs):
  152. super().__init__()
  153. self.norm1 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
  154. self.attn = Attention(hidden_size, num_heads=num_heads, qkv_bias=True, **block_kwargs)
  155. self.norm2 = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
  156. mlp_hidden_dim = int(hidden_size * mlp_ratio)
  157. approx_gelu = lambda: nn.GELU(approximate="tanh")
  158. self.mlp = Mlp(in_features=hidden_size, hidden_features=mlp_hidden_dim, act_layer=approx_gelu, drop=0)
  159. self.adaLN_modulation = nn.Sequential(
  160. nn.SiLU(),
  161. nn.Linear(hidden_size, 6 * hidden_size, bias=True)
  162. )
  164. def forward(self, x, c):
  165. shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.adaLN_modulation(c).chunk(6, dim=1)
  166. x = x + gate_msa.unsqueeze(1) * self.attn(modulate(self.norm1(x), shift_msa, scale_msa))
  167. x = x + gate_mlp.unsqueeze(1) * self.mlp(modulate(self.norm2(x), shift_mlp, scale_mlp))
  168. return x
  171. class FinalLayer(nn.Module):
  172. """
  173. The final layer of DiT.
  174. """
  175. def __init__(self, hidden_size, patch_size, out_channels):
  176. super().__init__()
  177. self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
  178. self.linear = nn.Linear(hidden_size, patch_size * patch_size * out_channels, bias=True)
  179. self.adaLN_modulation = nn.Sequential(
  180. nn.SiLU(),
  181. nn.Linear(hidden_size, 2 * hidden_size, bias=True)
  182. )
  184. def forward(self, x, c):
  185. shift, scale = self.adaLN_modulation(c).chunk(2, dim=1)
  186. x = modulate(self.norm_final(x), shift, scale)
  187. x = self.linear(x)
  188. return x
  191. class DiT(nn.Module):
  192. """
  193. Diffusion model with a Transformer backbone.
  194. """
  195. def __init__(
  196. self,
  197. input_size=32,
  198. patch_size=2,
  199. in_channels=4,
  200. hidden_size=1152,
  201. depth=28,
  202. num_heads=16,
  203. mlp_ratio=4.0,
  204. response_grid=32,
  205. response_grid_channels=3,
  206. response_dropout_prob=0.1,
  207. # num_classes=1000,
  208. learn_sigma=True,
  209. ):
  210. super().__init__()
  211. self.learn_sigma = learn_sigma
  212. self.in_channels = in_channels
  213. self.out_channels = in_channels * 2 if learn_sigma else in_channels
  214. self.patch_size = patch_size
  215. self.num_heads = num_heads
  217. self.x_embedder = PatchEmbed(input_size, patch_size, in_channels, hidden_size, bias=True)
  218. self.t_embedder = TimestepEmbedder(hidden_size)
  219. # self.y_embedder = LabelEmbedder(num_classes, hidden_size, class_dropout_prob)
  220. self.y_embedder = GridResponseEncoder(response_grid, response_grid_channels, hidden_size, response_dropout_prob)
  221. num_patches = self.x_embedder.num_patches
  222. # Will use fixed sin-cos embedding:
  223. self.pos_embed = nn.Parameter(torch.zeros(1, num_patches, hidden_size), requires_grad=False)
  225. self.blocks = nn.ModuleList([
  226. DiTBlock(hidden_size, num_heads, mlp_ratio=mlp_ratio) for _ in range(depth)
  227. ])
  228. self.final_layer = FinalLayer(hidden_size, patch_size, self.out_channels)
  229. self.initialize_weights()
  231. def initialize_weights(self):
  232. # Initialize transformer layers:
  233. def _basic_init(module):
  234. if isinstance(module, nn.Linear):
  235. torch.nn.init.xavier_uniform_(module.weight)
  236. if module.bias is not None:
  237. nn.init.constant_(module.bias, 0)
  238. self.apply(_basic_init)
  240. # Initialize (and freeze) pos_embed by sin-cos embedding:
  241. pos_embed = get_2d_sincos_pos_embed(self.pos_embed.shape[-1], int(self.x_embedder.num_patches ** 0.5))
  242. self.pos_embed.data.copy_(torch.from_numpy(pos_embed).float().unsqueeze(0))
  244. # Initialize patch_embed like nn.Linear (instead of nn.Conv2d):
  245. w = self.x_embedder.proj.weight.data
  246. nn.init.xavier_uniform_(w.view([w.shape[0], -1]))
  247. nn.init.constant_(self.x_embedder.proj.bias, 0)
  249. # Initialize label embedding table:
  250. # nn.init.normal_(self.y_embedder.embedding_table.weight, std=0.02)
  251. nn.init.normal_(self.y_embedder.linear.weight, std=0.02)
  253. # Initialize timestep embedding MLP:
  254. nn.init.normal_(self.t_embedder.mlp[0].weight, std=0.02)
  255. nn.init.normal_(self.t_embedder.mlp[2].weight, std=0.02)
  257. # Zero-out adaLN modulation layers in DiT blocks:
  258. for block in self.blocks:
  259. nn.init.constant_(block.adaLN_modulation[-1].weight, 0)
  260. nn.init.constant_(block.adaLN_modulation[-1].bias, 0)
  262. # Zero-out output layers:
  263. nn.init.constant_(self.final_layer.adaLN_modulation[-1].weight, 0)
  264. nn.init.constant_(self.final_layer.adaLN_modulation[-1].bias, 0)
  265. nn.init.constant_(self.final_layer.linear.weight, 0)
  266. nn.init.constant_(self.final_layer.linear.bias, 0)
  268. def unpatchify(self, x):
  269. """
  270. x: (N, T, patch_size**2 * C)
  271. imgs: (N, H, W, C)
  272. """
  273. c = self.out_channels
  274. p = self.x_embedder.patch_size[0]
  275. h = w = int(x.shape[1] ** 0.5)
  276. assert h * w == x.shape[1]
  278. x = x.reshape(shape=(x.shape[0], h, w, p, p, c))
  279. x = torch.einsum('nhwpqc->nchpwq', x)
  280. imgs = x.reshape(shape=(x.shape[0], c, h * p, h * p))
  281. return imgs
  283. def forward(self, x, t, y):
  284. """
  285. Forward pass of DiT.
  286. x: (N, C, H, W) tensor of spatial inputs (images or latent representations of images)
  287. t: (N,) tensor of diffusion timesteps
  288. y: (N, CofG, G, G) tensor of grid responses
  289. """
  290. x = self.x_embedder(x) + self.pos_embed # (N, T, D), where T = H * W / patch_size ** 2
  291. t = self.t_embedder(t) # (N, D)
  292. y = self.y_embedder(y, self.training) # (N, D)
  293. c = t + y # (N, D)
  294. for block in self.blocks:
  295. x = block(x, c) # (N, T, D)
  296. x = self.final_layer(x, c) # (N, T, patch_size ** 2 * out_channels)
  297. x = self.unpatchify(x) # (N, out_channels, H, W)
  298. return x
  300. def forward_with_cfg(self, x, t, y, cfg_scale):
  301. """
  302. Forward pass of DiT, but also batches the unconditional forward pass for classifier-free guidance.
  303. """
  304. # https://github.com/openai/glide-text2im/blob/main/notebooks/text2im.ipynb
  305. half = x[: len(x) // 2]
  306. combined = torch.cat([half, half], dim=0)
  307. model_out = self.forward(combined, t, y)
  308. # For exact reproducibility reasons, we apply classifier-free guidance on only
  309. # three channels by default. The standard approach to cfg applies it to all channels.
  310. # This can be done by uncommenting the following line and commenting-out the line following that.
  311. eps, rest = model_out[:, :self.in_channels], model_out[:, self.in_channels:]
  312. # eps, rest = model_out[:, :3], model_out[:, 3:]
  313. cond_eps, uncond_eps = torch.split(eps, len(eps) // 2, dim=0)
  314. half_eps = uncond_eps + cfg_scale * (cond_eps - uncond_eps)
  315. eps = torch.cat([half_eps, half_eps], dim=0)
  316. return torch.cat([eps, rest], dim=1)
  319. #################################################################################
  320. # Sine/Cosine Positional Embedding Functions #
  321. #################################################################################
  322. # https://github.com/facebookresearch/mae/blob/main/util/pos_embed.py
  324. def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False, extra_tokens=0):
  325. """
  326. grid_size: int of the grid height and width
  327. return:
  328. pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token)
  329. """
  330. grid_h = np.arange(grid_size, dtype=np.float32)
  331. grid_w = np.arange(grid_size, dtype=np.float32)
  332. grid = np.meshgrid(grid_w, grid_h) # here w goes first
  333. grid = np.stack(grid, axis=0)
  335. grid = grid.reshape([2, 1, grid_size, grid_size])
  336. pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid)
  337. if cls_token and extra_tokens > 0:
  338. pos_embed = np.concatenate([np.zeros([extra_tokens, embed_dim]), pos_embed], axis=0)
  339. return pos_embed
  342. def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
  343. assert embed_dim % 2 == 0
  345. # use half of dimensions to encode grid_h
  346. emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)
  347. emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)
  349. emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
  350. return emb
  353. def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
  354. """
  355. embed_dim: output dimension for each position
  356. pos: a list of positions to be encoded: size (M,)
  357. out: (M, D)
  358. """
  359. assert embed_dim % 2 == 0
  360. omega = np.arange(embed_dim // 2, dtype=np.float64)
  361. omega /= embed_dim / 2.
  362. omega = 1. / 10000**omega # (D/2,)
  364. pos = pos.reshape(-1) # (M,)
  365. out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product
  367. emb_sin = np.sin(out) # (M, D/2)
  368. emb_cos = np.cos(out) # (M, D/2)
  370. emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D)
  371. return emb
  374. #################################################################################
  375. # DiT Configs #
  376. #################################################################################
  378. def DiT_XL_2(**kwargs):
  379. return DiT(depth=28, hidden_size=1152, patch_size=2, num_heads=16, **kwargs)
  381. def DiT_XL_4(**kwargs):
  382. return DiT(depth=28, hidden_size=1152, patch_size=4, num_heads=16, **kwargs)
  384. def DiT_XL_8(**kwargs):
  385. return DiT(depth=28, hidden_size=1152, patch_size=8, num_heads=16, **kwargs)
  387. def DiT_L_2(**kwargs):
  388. return DiT(depth=24, hidden_size=1024, patch_size=2, num_heads=16, **kwargs)
  390. def DiT_L_4(**kwargs):
  391. return DiT(depth=24, hidden_size=1024, patch_size=4, num_heads=16, **kwargs)
  393. def DiT_L_8(**kwargs):
  394. return DiT(depth=24, hidden_size=1024, patch_size=8, num_heads=16, **kwargs)
  396. def DiT_B_2(**kwargs):
  397. return DiT(depth=12, hidden_size=768, patch_size=2, num_heads=12, **kwargs)
  399. def DiT_B_4(**kwargs):
  400. return DiT(depth=12, hidden_size=768, patch_size=4, num_heads=12, **kwargs)
  402. def DiT_B_8(**kwargs):
  403. return DiT(depth=12, hidden_size=768, patch_size=8, num_heads=12, **kwargs)
  405. def DiT_S_2(**kwargs):
  406. return DiT(depth=12, hidden_size=384, patch_size=2, num_heads=6, **kwargs)
  408. def DiT_S_4(**kwargs):
  409. return DiT(depth=12, hidden_size=384, patch_size=4, num_heads=6, **kwargs)
  411. def DiT_S_8(**kwargs):
  412. return DiT(depth=12, hidden_size=384, patch_size=8, num_heads=6, **kwargs)
  415. DiT_models = {
  416. 'DiT-XL/2': DiT_XL_2, 'DiT-XL/4': DiT_XL_4, 'DiT-XL/8': DiT_XL_8,
  417. 'DiT-L/2': DiT_L_2, 'DiT-L/4': DiT_L_4, 'DiT-L/8': DiT_L_8,
  418. 'DiT-B/2': DiT_B_2, 'DiT-B/4': DiT_B_4, 'DiT-B/8': DiT_B_8,
  419. 'DiT-S/2': DiT_S_2, 'DiT-S/4': DiT_S_4, 'DiT-S/8': DiT_S_8,
  420. }

models.py at commit bb27c89, under MIT · at the source

Overview

Authors: Kaiwen Deng1, Peter S. Schwendeman1,2, Yuanfang Guan1,3
ORCID iDs: Yuanfang Guan
  1. Department of Computational Medicine and Bioinformatics University of Michigan Ann Arbor Michigan USA
  2. Department of Electrical Engineering and Computer Science University of Michigan Ann Arbor Michigan USA
  3. Department of Internal Medicine University of Michigan Ann Arbor Michigan USA
Institutions: University of Michigan (United States); Michigan Medicine (United States)
Journal: Advanced science (Weinheim, Baden-Wurttemberg, Germany), volume 13, issue 30, article e20220
Dates: received 12 October 2025; accepted 20 February 2026; published online 5 March 2026; in print May 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1002/advs.202520220 · PMID 41787846 · PMCID PMC13248838 · OpenAlex W7134066894
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: mouse (organism), systems (subfield)
Methods: Statistics, Machine learning, Connectivity, Single-unit activity, calcium imaging, Smoothing, state filtering, decompositions
Keywords: diffusion model, mouse, neural decoding, primary visual cortex
MeSH: Brain Mapping*, Image Processing, Computer-Assisted*, Magnetic Resonance Imaging*, Neurons*, Primary Visual Cortex*, Visual Cortex*, Animals, Mice, Photic Stimulation (* major topic)
Topic: Face Recognition and Perception (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: NIGMS (R35‐GM133346)
Citations: not cited yet (Europe PMC); 30 references in the paper

Abstract

Decoding visual stimuli from neural signals is an essential step toward understanding how sensory information is represented in the brain. While most existing approaches reconstruct visual stimuli from human functional magnetic resonance imaging (fMRI), utilizing calcium imaging in mice opens the door to single‐neuron‐level insights into non‐primate visual systems with distinct spectral sensitivities. Here, we present Sensorium‐Viz, a diffusion‐based framework specifically designed for decoding activity in the mouse primary visual cortex. The model is among the first to reliably reconstruct complex, high‐resolution images from previously unseen single‐neuron responses. At its core, Sensorium‐Viz introduces two key advances for neuron‐to‐image decoding: a synthetic‐response augmentation strategy that improves reconstruction performance by more than 30% while enabling cross‐mouse generalization through fine‐tuning, and an architectural design that integrates a Diffusion Transformer (DiT) with a spatial neuron‐embedding module, thereby achieving up to a 10.65% performance gain over leading fMRI‐based reconstruction methods across pixel‐ and content‐level benchmarks. Analysis of the neural responses and corresponding reconstructions reveals that neurons sensitive to low‐level visual features form the primary basis of V1's representation of external stimuli. These findings establish Sensorium‐Viz as a biologically grounded and technically robust tool for vision decoding.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 17 matches between paragraphs and lines of code.

GuanLab/Sensorium2022_Challenge

License: none: the authors keep all their rights
State: the link answers, verified on 30 September 2026
Evidence: files inventoried
Commit: def478db9e7ae519430acc4688614c49c6b07a05, 3 January 2024
Languages: Python (94), Jupyter (6), Shell (3)
Size: 214 files, 103 scripts
Software Heritage: not archived
Found in: the text, “Collect Images and Generate Synthetic Neuronal R”
Holds: README, 6 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: PyTorch (59 files), NumPy (41 files), pandas (13 files), SciPy (7 files), Matplotlib (6 files), seaborn (5 files), Pillow (4 files), h5py (3 files), scikit-image (2 files), scikit-learn (2 files), OpenCV (1 file), UMAP (1 file)
Availability: 1 check, the latest on 30 September 2026: the link answers
  • 30 September 2026: the link answers
104 files
At the source: github.com/GuanLab/Sensorium2022_Challenge

GuanLab/sensorium-viz

License: MIT
State: the link answers, verified on 30 September 2026
Evidence: files inventoried
Commit: bb27c89057998be71f49c0aa6c06a7637c1b5617, 1 July 2025
Languages: Python (17)
Size: 23 files, 17 scripts
Software Heritage: not archived
Found in: “Code Availability”
Holds: README, license file, environment (environment.yml)
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (15 files), PyTorch (14 files), Pillow (7 files), OpenCV (3 files), pandas (2 files), scikit-image (1 file), scikit-learn (1 file)
Availability: 1 check, the latest on 30 September 2026: the link answers
  • 30 September 2026: the link answers
19 files
At the source: github.com/GuanLab/sensorium-viz

Code Availability

The codes of this work are available at: https://github.com/GuanLab/sensorium‐viz (https://github.com/GuanLab/sensorium-viz).

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 120 scripts, each with its path and the digest of its content;
  • 17 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability Statement

Datasets of the mice can be retrieved from https://gin.g‐node.org/cajal/Sensorium2022 (https://gin.g-node.org/cajal/Sensorium2022). Synthetic responses for the COCO images and the model weights needed to reproduce the paper results can be obtained from Google Drive: https://drive.google.com/drive/folders/1GbJ7V2AzVezKW3U0lwhrKYaeQni2ntef.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 30 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 3 authors, 4 keywords, 9 MeSH terms, 1 funder, 22 references.

Cite

This paper

Deng, K., Schwendeman, P. S., & Guan, Y. (2026). Learnable Diffusion Framework for Mouse V1 Neural Decoding. Advanced science (Weinheim, Baden-Wurttemberg, Germany), 13(30), e20220. https://doi.org/10.1002/advs.202520220

BibTeX

@article{deng2026learnable,
author = {Deng, Kaiwen and Schwendeman, Peter S. and Guan, Yuanfang},
title = {{Learnable Diffusion Framework for Mouse V1 Neural Decoding}},
journal = {Advanced science (Weinheim, Baden-Wurttemberg, Germany)},
year = {2026},
month = mar,
volume = {13},
number = {30},
pages = {e20220},
publisher = {Wiley},
issn = {2198-3844},
doi = {10.1002/advs.202520220},
url = {https://doi.org/10.1002/advs.202520220},
pmid = {41787846},
pmcid = {PMC13248838}
}

RIS

TY - JOUR
AU - Deng, Kaiwen
AU - Schwendeman, Peter S.
AU - Guan, Yuanfang
TI - Learnable Diffusion Framework for Mouse V1 Neural Decoding
T2 - Advanced science (Weinheim, Baden-Wurttemberg, Germany)
J2 - Adv Sci (Weinh)
PY - 2026
DA - 2026/03/05
VL - 13
IS - 30
SP - e20220
SN - 2198-3844
PB - Wiley
DO - 10.1002/advs.202520220
UR - https://doi.org/10.1002/advs.202520220
LA - en
ER -

CSL-JSON

{
"id": "10.1002/advs.202520220",
"type": "article-journal",
"title": "Learnable Diffusion Framework for Mouse V1 Neural Decoding",
"container-title": "Advanced science (Weinheim, Baden-Wurttemberg, Germany)",
"author": [
{
"family": "Deng",
"given": "Kaiwen"
},
{
"family": "Schwendeman",
"given": "Peter S."
},
{
"family": "Guan",
"given": "Yuanfang"
}
],
"container-title-short": "Adv Sci (Weinh)",
"volume": "13",
"issue": "30",
"page": "e20220",
"DOI": "10.1002/advs.202520220",
"PMID": "41787846",
"PMCID": "PMC13248838",
"ISSN": "2198-3844",
"publisher": "Wiley",
"URL": "https://doi.org/10.1002/advs.202520220",
"language": "en",
"issued": {
"date-parts": [
[
2026,
3,
5
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.7554/elife.105081 [code]
Movie reconstruction from mouse visual cortex activity.
Journal: eLife
In common: OpenCV, Pillow, PyTorch, 4 other tools, systems, mouse, 6 references
[2] doi:10.1371/journal.pcbi.1014263 [code]
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery.
Journal: PLoS computational biology
In common: UMAP, OpenCV, scikit-image, 8 other tools, 2 references
[3] doi:10.1038/s41467-026-72057-9 [code]
Sex-specific behavioral feedback modulates sensorimotor processing and drives flexible social behavior.
Journal: Nature communications
In common: UMAP, OpenCV, scikit-image, 9 other tools, systems, 1 reference
[4] doi:10.1016/j.isci.2026.116825 [code]
Social hierarchy shapes behavioral and transcriptional responses to chronic stress and ketamine in male mice.
Journal: iScience
In common: UMAP, OpenCV, scikit-image, 9 other tools, systems, mouse
[5] doi:10.1371/journal.pbio.3003915 [code]
Noise-invariant representations of sound emerge along the canonical cortical hierarchy.
Journal: PLoS biology
In common: UMAP, OpenCV, scikit-image, 9 other tools, mouse
[6] doi:10.21203/rs.3.rs-9676637/v1 [code]
A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies
Journal: Research Square (preprint)
In common: UMAP, OpenCV, scikit-image, 9 other tools
[7] doi:10.1162/imag.a.1299 [code]
A modular semantic-structural pipeline for visual decoding from primate spiking data via selective temporal integration.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: scikit-image, h5py, Pillow, 7 other tools, 2 references
[8] doi:10.7554/elife.107933 [code]
Modality-agnostic decoding of vision and language from fMRI.
Journal: eLife
In common: OpenCV, scikit-image, h5py, 8 other tools, 1 reference
[9] doi:10.7554/elife.109717 [code]
Retrosplenial cortex enables context-dependent goal-directed sensorimotor transformation.
Journal: eLife
In common: OpenCV, scikit-image, h5py, 7 other tools, systems, mouse, 1 reference
[10] doi:10.1038/s41467-026-74823-1 [code]
Cerebellar activity is triggered by reach endpoint during learning of a complex locomotor task.
Journal: Nature communications
In common: UMAP, OpenCV, scikit-image, 7 other tools, mouse, 1 reference

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.