OSCR

How the visual brain can learn to parse images using a multiscale, incremental grouping process.

Code ↔ Paper

12 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 12 matches
  1. [1] § Model and task › Recurrent network › Hidden layers ( ). ↔ src/models/recurrent_network.py, lines 83–212 · score 0.72 · progressively coarser spatial, spatial resolution, Hidden layers, hierarchical, VIP, RF
  2. [2] § Model and task › Recurrent network › Input layer ( ). ↔ src/models/layers.py, lines 293–349 · score 0.69 · lateral inhibition, layer receives, higher layer, sensory, signals, modulated
  3. [3] § Model and task › Recurrent network › Input layer ( ). ↔ src/models/layers.py, lines 293–349 · score 0.64 · lateral inhibition, feedback modulated, sensory, signal, gated, Pyramidal
  4. [4] § Model and task ↔ src/models/layers.py, lines 351–438 · score 0.62 · lower layers, horizontal connections, higher layers, gated, feedback, model
  5. [5] § Model and task › Recurrent network ↔ src/models/recurrent_network.py, lines 83–212 · score 0.61 · pyramidal neurons, horizontal connections, recurrent networks, VIP, feedback, tracing
  6. [6] § Model and task › Feedforward network ↔ src/tasks/tasks.py, lines 110–248 · score 0.59 · distractor curves, curve tracing task, target curve, selection, RF, stimulus
  7. [7] § Model and task › Training › Training of the recurrent network. ↔ src/models/recurrent_network.py, lines 324–342 · score 0.57 · reward prediction error, recurrent network, inspired, Weights, Training
  8. [8] § Model and task › Recurrent network › Hidden layers ( ). ↔ src/models/layers.py, lines 351–438 · score 0.57 · hidden layers integrated, VIP activity, horizontal, feedback
  9. [9] § Model and task ↔ src/tasks/tasks.py, lines 110–248 · score 0.54 · distractor curve, curve tracing task, overlapping, coarser, loop, selection
  10. [10] § Model and task › Recurrent network › Input layer ( ). ↔ src/models/layers.py, lines 144–170 · score 0.53 · lateral inhibitory, inhibitory weights, kernel, connections, layer
  11. [11] § Model and task › Recurrent network › Hidden layers ( ). ↔ src/models/layers.py, lines 440–553 · score 0.53 · receptive field, feedforward weights, stride, selection, Hidden, layer
  12. [12] § Model and task › Training › Training of feedforward networks. ↔ src/models/recurrent_network.py, lines 16–22 · score 0.52 · reinforcement learning, recurrent network, curve tracing, architecture

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 600 lines · 28 KB · no license · 6 matches

  1. # -*- coding: utf-8 -*-
  2. """
  3. Created on Fri Oct 29 12:29:28 2021
  5. @author: Sami
  6. """
  7. from typing import Optional, List, Tuple
  8. import numpy as np
  9. import torch
  10. import torch.nn as nn
  11. import torch.nn.functional as F
  13. class CustomLayer(nn.Module):
  14. """Base class for custom neural network layers with specialized initialization and update rules.
  16. This class provides common functionality for input, hidden, and output layers including
  17. weight initialization, activation functions, and biologically-inspired learning rules.
  18. """
  20. def __init__(self) -> None:
  21. """Initialize the custom layer with default parameters."""
  22. self.initialisation_range = 0.1
  23. super().__init__()
  25. def activation_function(self, x: torch.Tensor) -> torch.Tensor:
  26. """Apply ReLU activation function.
  28. Args:
  29. x (torch.Tensor): Input tensor.
  31. Returns:
  32. torch.Tensor: Activated output.
  33. """
  34. return torch.relu(x)
  36. def step_function(self, x: torch.Tensor) -> torch.Tensor:
  37. """Apply a piecewise linear step function with saturation.
  39. Args:
  40. x (torch.Tensor): Input tensor.
  42. Returns:
  43. torch.Tensor: Transformed tensor with linear region and saturation.
  44. """
  45. eps = 1
  46. phi = 1
  47. x[x <= eps/phi] = phi * x[x <= eps/phi]
  48. x[x <= 0] = 0
  49. x[x >= eps/phi] = eps
  50. return x
  53. def gating_function(self, x: torch.Tensor) -> torch.Tensor:
  54. """Apply a smooth gating function for modulation.
  56. Args:
  57. x (torch.Tensor): Input tensor.
  59. Returns:
  60. torch.Tensor: Gated output with smooth saturation.
  61. """
  62. param = 100
  63. return torch.relu(param*x/(torch.sqrt(1+(param**2)*(x**2))))
  65. def initialize_feedforward_weights(self, layer: nn.Module, one_to_one: bool = False, change_scale: bool = False, receptive_field_size: int = 1) -> None:
  66. """Initialize feedforward connection weights with specific patterns.
  68. Args:
  69. layer (nn.Module): The layer whose weights to initialize.
  70. one_to_one (bool): If True, use one-to-one connectivity pattern.
  71. change_scale (bool): If True, initialize for scale-changing connections.
  72. receptive_field_size (int): Size of the receptive field for spatial connections.
  73. """
  74. # Initialize weight tensor with zeros
  75. weights = torch.zeros(layer.weight.shape)
  76. if layer.bias is not None:
  77. bias = torch.zeros(layer.bias.shape)
  79. if change_scale:
  80. # For scale-changing connections, initialize all weights uniformly
  81. for f in range(weights.shape[0]):
  82. for f2 in range(weights.shape[1]):
  83. weights[f, f2, :, :] = self.initialisation_range * np.random.rand() + 1
  84. else:
  85. # For same-scale connections
  86. for f in range(weights.shape[0]):
  87. for f2 in range(weights.shape[1]):
  88. if not one_to_one:
  89. # Localized receptive field: only connect nearby spatial locations
  90. lower_bound = self.grid_size - receptive_field_size//2 - 1
  91. upper_bound = self.grid_size + receptive_field_size//2
  92. weights[f, f2, lower_bound:upper_bound,lower_bound:upper_bound] = 0.001 * np.random.rand()
  93. else:
  94. # One-to-one connections: single weight per feature pair
  95. if self.layer_type != "output":
  96. weights[f,f2,0] = self.initialisation_range * np.random.rand() + 1
  97. else:
  98. # Output layer uses smaller initial weights
  99. weights[f, f2, 0] = 0.001 + 0.001 * np.random.rand()
  101. # Initialize biases to 1 (positive baseline)
  102. if layer.bias is not None:
  103. for f in range(bias.shape[0]):
  104. bias[f] = 1
  106. # Set as learnable parameters
  107. layer.weight = torch.nn.Parameter(weights)
  108. if layer.bias is not None:
  109. layer.bias = torch.nn.Parameter(bias)
  111. def initialize_feedback_weights(self, layer: nn.Module, change_scale: bool = False) -> None:
  112. """Initialize feedback connection weights.
  114. Args:
  115. layer (nn.Module): The layer whose weights to initialize.
  116. change_scale (bool): If True, initialize for scale-changing connections.
  117. """
  118. # Initialize weight tensor with zeros
  119. weights = torch.zeros(layer.weight.shape)
  120. if layer.bias is not None:
  121. bias = torch.zeros(layer.bias.shape)
  123. # Set up feedback connections (top-down modulation)
  124. for f in range(weights.shape[0]):
  125. for f2 in range(weights.shape[1]):
  126. if self.layer_type != "output":
  127. if change_scale:
  128. # Scale-changing feedback: full spatial connectivity
  129. weights[f, f2, :, :] = self.initialisation_range * np.random.rand() + 0.5
  130. else:
  131. # Local feedback: connect to 4 nearest neighbors (cross pattern)
  132. # This creates a plus-shaped connectivity pattern
  133. weights[f, f2, 0,1] = self.initialisation_range * np.random.rand() + 0.1 # Top
  134. weights[f, f2, 1,0] = self.initialisation_range * np.random.rand() + 0.1 # Left
  135. weights[f, f2, 1,2] = self.initialisation_range * np.random.rand() + 0.1 # Right
  136. weights[f, f2, 2,1] = self.initialisation_range * np.random.rand() + 0.1 # Bottom
  138. # Set as learnable parameters
  139. layer.weight = torch.nn.Parameter(weights)
  140. if layer.bias is not None:
  141. layer.bias = torch.nn.Parameter(bias)
  144. def initialize_inhibitory_weights(self, layer: nn.Module) -> None:
  145. """Initialize inhibitory connection weights with negative biases.
  147. Args:
  148. layer (nn.Module): The layer whose weights to initialize.
  149. """
  150. # Initialize weight tensor with zeros
  151. weights = torch.zeros(layer.weight.shape)
  152. if layer.bias is not None:
  153. bias = torch.zeros(layer.bias.shape)
  155. # Set up lateral inhibition: only within same feature channel (f == f2)
  156. for f in range(weights.shape[0]):
  157. for f2 in range(weights.shape[1]):
  158. if f == f2:
  159. # Self-connection at center of 3x3 kernel
  160. weights[f, f2, 1, 1] = 1
  162. # Initialize biases to negative values for inhibitory effect
  163. if layer.bias is not None:
  164. for f in range(bias.shape[0]):
  165. bias[f] = -1 - self.initialisation_range * np.random.rand()
  167. # Set as learnable parameters
  168. layer.weight = torch.nn.Parameter(weights)
  169. if layer.bias is not None:
  170. layer.bias = torch.nn.Parameter(bias)
  173. def average_traces(self, traces: torch.Tensor, mask: torch.Tensor, receptive_field_size: int = 1) -> torch.Tensor:
  174. """Average gradient traces over spatial dimensions for weight updates.
  176. Args:
  177. traces (torch.Tensor): Gradient traces to average.
  178. mask (torch.Tensor): Mask indicating valid connections.
  179. receptive_field_size (int): Size of receptive field for output layers.
  181. Returns:
  182. torch.Tensor: Averaged traces.
  183. """
  184. traces *= mask
  185. if self.layer_type != "output":
  186. m = torch.mean(traces[:,:,:,:], axis=(2,3))
  187. m = m[:, :, None,None]
  188. traces[:,:] = m
  189. else:
  190. intermmask = torch.ones((2 * self.grid_size - 1, 2 * self.grid_size - 1))
  191. lower_bound = self.grid_size - receptive_field_size//2 - 1
  192. upper_bound = self.grid_size + receptive_field_size//2
  194. intermmask = torch.zeros((2 * self.grid_size - 1, 2 * self.grid_size - 1))
  195. intermmask[lower_bound:upper_bound,lower_bound:upper_bound] = 1
  196. m = torch.mean(traces[:, :, intermmask == 1], axis=2)
  197. traces[:, :, intermmask == 1] = m[:,None]
  198. return(traces)
  201. def update_weight(self, layer: nn.Module, upper: torch.Tensor, beta: float, delta: float, mask: Optional[torch.Tensor] = None, z: Optional[torch.Tensor] = None, average: bool = True, receptive_field_size: int = 1, inhib: bool = False, change_scale: bool = False) -> nn.Module:
  202. """Update layer weights using reward prediction error.
  204. Args:
  205. layer (nn.Module): The layer to update.
  206. upper (torch.Tensor): Upper layer activity.
  207. beta (float): Learning rate.
  208. delta (float): Reward prediction error.
  209. mask (torch.Tensor, optional): Connection mask.
  210. z (torch.Tensor, optional): Gradient output.
  211. average (bool): Whether to average traces.
  212. receptive_field_size (int): Receptive field size.
  213. inhib (bool): If True, update inhibitory connections only.
  214. change_scale (bool): If True, handle scale-changing connections.
  216. Returns:
  217. nn.Module: Updated layer.
  218. """
  219. with torch.no_grad():
  220. # Compute gradients of output with respect to weights and biases
  221. # This implements eligibility traces for credit assignment
  222. if layer.bias is not None:
  223. delta_weight = torch.autograd.grad(upper, [layer.weight, layer.bias], grad_outputs=z, retain_graph=True, allow_unused=True)
  224. delta_bias = delta_weight[1]
  225. # Clamp gradient magnitudes to prevent instability
  226. delta_bias = torch.clamp(delta_bias, None, 1)
  227. else:
  228. delta_weight = torch.autograd.grad(upper, layer.weight, grad_outputs=z, retain_graph=True, allow_unused=True)
  230. delta_weight = delta_weight[0]
  231. # Clamp weight gradients to prevent large updates
  232. delta_weight = torch.clamp(delta_weight, None, 1)
  234. if inhib == False:
  235. # Update excitatory weights using the reward prediction error
  236. # delta_weight acts as eligibility trace, delta is reward prediction error
  237. if (self.layer_type == 'output' and average) or change_scale:
  238. # For output and scale-changing layers, average gradients spatially
  239. delta_weight = self.average_traces(delta_weight, mask, receptive_field_size=receptive_field_size)
  240. elif mask is not None:
  241. # Apply connection mask to enforce sparse connectivity
  242. delta_weight = delta_weight * mask
  244. # Weight update: w = w + learning_rate * RPE * eligibility_trace
  245. weight_update = layer.weight + beta * delta * delta_weight
  246. layer.weight.copy_(weight_update)
  247. else:
  248. # Update inhibitory biases only (weights remain fixed)
  249. bias_update = layer.bias + beta * delta * delta_bias
  250. layer.bias.copy_(bias_update)
  251. return(layer)
  253. def make_mask(self, weight: torch.Tensor) -> torch.Tensor:
  254. """Create a binary mask from weight tensor.
  256. Args:
  257. weight (torch.Tensor): Weight tensor.
  259. Returns:
  260. torch.Tensor: Binary mask (1 where weights are non-zero, 0 elsewhere).
  261. """
  262. mask = torch.clone(weight.detach())
  263. mask[mask != 0] = 1
  264. return mask
  266. def to(self, device: torch.device) -> None:
  267. """Move layer parameters to specified device.
  269. Args:
  270. device (torch.device): Target device (CPU or CUDA).
  271. """
  272. if self.layer_type == 'input':
  273. self.FB.to(device)
  274. self.lateral_inhibition.to(device)
  275. self.feedback_mask = self.feedback_mask.to(device)
  276. self.inhibition_mask = self.inhibition_mask.to(device)
  277. elif self.layer_type == 'hidden':
  278. self.FF.to(device)
  279. self.feedforward_mask = self.feedforward_mask.to(device)
  280. self.horizontal_mask = self.horizontal_mask.to(device)
  281. self.H.to(device)
  282. if self.has_feedback:
  283. self.FB.to(device)
  284. self.feedback_mask = self.feedback_mask.to(device)
  285. elif self.layer_type == 'output':
  286. for layer in range(len(self.skip_weights)):
  287. self.skip_weights[layer].to(device)
  288. for layer in range(1,len(self.skip_masks)):
  289. self.skip_masks[layer] = self.skip_masks[layer].to(device)
  293. class InputLayer(CustomLayer):
  294. """Input layer with feedback modulation and lateral inhibition.
  296. This layer receives sensory input and is modulated by feedback from higher layers
  297. through VIP and SOM interneuron populations.
  298. """
  300. def __init__(self, feature_in: int, feature_out: int) -> None:
  301. """Initialize input layer.
  303. Args:
  304. feature_in (int): Number of input features.
  305. feature_out (int): Number of output features from higher layer.
  306. """
  307. super().__init__()
  308. self.layer_type = 'input'
  309. K_size = 3
  311. self.FB = nn.Conv2d(feature_out, feature_in, K_size, stride=1, padding='same',bias=False)
  312. self.lateral_inhibition = nn.Conv2d(feature_in, feature_in, K_size, stride=1, padding='same',bias=True)
  314. # Initializing weights
  315. self.initialize_feedback_weights(self.FB)
  316. self.initialize_inhibitory_weights(self.lateral_inhibition)
  318. # Setting the mask to have connections only between neighboours
  319. self.feedback_mask = self.make_mask(self.FB.weight)
  320. self.inhibition_mask = self.make_mask(self.lateral_inhibition.weight)
  322. def forward(self, upper_y: torch.Tensor, input_stimulus: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
  323. """Forward pass through input layer.
  325. Args:
  326. upper_y (torch.Tensor): Modulation signal from upper layer.
  327. input_stimulus (torch.Tensor): Sensory input stimulus.
  329. Returns:
  330. tuple: (modulated_output, vip_activity, som_activity)
  331. """
  332. Y = input_stimulus
  333. vip_activity = self.step_function(self.FB(upper_y))
  334. som_activity = self.activation_function(1 - vip_activity)
  335. modulated_output = self.activation_function((-self.lateral_inhibition(som_activity)) * self.gating_function(Y))
  336. return(modulated_output, vip_activity, som_activity)
  339. def update_layer(self, upper: List[torch.Tensor], z: List[torch.Tensor], beta: float, delta: float) -> None:
  340. """Update layer weights based on reward prediction error.
  342. Args:
  343. upper (list): Upper layer activities [pyramidal, som, vip].
  344. z (list): Gradients [pyramidal, som, vip].
  345. beta (float): Learning rate.
  346. delta (float): Reward prediction error.
  347. """
  348. self.FB = self.update_weight(self.FB, upper[2], beta, delta, self.feedback_mask, z[2],change_scale=False)
  349. self.lateral_inhibition = self.update_weight(self.lateral_inhibition, upper[0], beta, delta, self.inhibition_mask, z[0],inhib=True)
  351. class HiddenLayer(CustomLayer):
  352. """Hidden layer with feedforward, feedback, and horizontal connections.
  354. This layer integrates information from lower layers (feedforward), higher layers
  355. (feedback), and within the same layer (horizontal connections).
  356. """
  358. def __init__(self, feature_in_lower: int, feature_out: int, feature_in_higher: int, big_pixels_size: int, grid_size: int, has_feedback: bool = True, change_scale_fb: bool = False, change_scale_ff: bool = False, higher_scale: bool = False) -> None:
  359. """Initialize hidden layer.
  361. Args:
  362. feature_in_lower (int): Number of features from lower layer.
  363. feature_out (int): Number of output features.
  364. feature_in_higher (int): Number of features from higher layer.
  365. big_pixels_size (int): Stride for scale-changing connections.
  366. grid_size (int): Size of spatial grid.
  367. has_feedback (bool): Whether to include upper layer modulation.
  368. change_scale_fb (bool): Whether feedback changes scale.
  369. change_scale_ff (bool): Whether feedforward changes scale.
  370. higher_scale (bool): Whether this is a higher-scale layer.
  371. """
  372. super().__init__()
  373. self.layer_type = 'hidden'
  374. self.has_feedback = has_feedback # If the higher layer has a modulated group
  375. self.change_scale_fb = change_scale_fb
  376. self.change_scale_ff = change_scale_ff
  377. self.higher_scale = higher_scale
  378. self.big_pixels_size = big_pixels_size
  379. self.grid_size = grid_size
  381. # Making the weights
  382. if self.change_scale_ff:
  383. self.FF = nn.Conv2d(feature_in_lower, feature_out, self.big_pixels_size, stride=self.big_pixels_size, bias=True)
  384. else:
  385. self.FF = nn.Conv2d(feature_in_lower, feature_out, 1, stride=1, padding='same', bias=True)
  387. if has_feedback:
  388. self.FB = nn.ConvTranspose2d(feature_in_higher, feature_out, self.big_pixels_size, stride=self.big_pixels_size, padding=0,bias=False)
  389. self.H = nn.Conv2d(feature_out,feature_out,3,stride = 1,padding='same',bias=False)
  391. # Initializing the weights
  392. self.initialize_feedforward_weights(self.FF,change_scale = change_scale_ff,one_to_one=True)
  393. if has_feedback:
  394. self.initialize_feedback_weights(self.FB,change_scale = change_scale_fb)
  395. self.initialize_feedback_weights(self.H,change_scale=False)
  397. # Making the masks
  398. if has_feedback:
  399. self.feedback_mask = self.make_mask(self.FB.weight)
  400. self.feedforward_mask = self.make_mask(self.FF.weight)
  401. self.horizontal_mask = self.make_mask(self.H.weight)
  404. def forward(self, current_y: torch.Tensor, lower_ymod: Optional[torch.Tensor] = None, upper_y: Optional[torch.Tensor] = None, horiz: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
  405. """Forward pass through hidden layer.
  407. Args:
  408. current_y (torch.Tensor): Current layer activity.
  409. lower_ymod (torch.Tensor, optional): Input from lower layer.
  410. upper_y (torch.Tensor, optional): Modulation from upper layer.
  411. horiz (torch.Tensor, optional): Horizontal connections.
  413. Returns:
  414. tuple: (modulated_output, vip_activity, som_activity)
  415. """
  416. feedforward_input = self.FF(lower_ymod)
  417. if upper_y is not None:
  418. vip_activity = self.step_function(self.FB(upper_y) + self.H(horiz))
  419. else:
  420. vip_activity = self.step_function(self.H(horiz))
  421. som_activity = self.activation_function(1 - vip_activity)
  422. modulated_output = self.activation_function((feedforward_input - som_activity) * self.gating_function(current_y))
  423. return(modulated_output, vip_activity, som_activity)
  425. def update_layer(self, upper: List[torch.Tensor], z: List[torch.Tensor], beta: float, delta: float, train_v: bool = True) -> None:
  426. """Update layer weights based on reward prediction error.
  428. Args:
  429. upper (list): Upper layer activities [pyramidal, som, vip].
  430. z (list): Gradients [pyramidal, som, vip].
  431. beta (float): Learning rate.
  432. delta (float): Reward prediction error.
  433. train_v (bool): Whether to train VIP connections.
  434. """
  435. self.FF = self.update_weight(self.FF, upper[0], beta, delta, self.feedforward_mask, z[0],change_scale=self.change_scale_ff)
  436. if self.has_feedback:
  437. self.FB = self.update_weight(self.FB, upper[2], beta, delta, self.feedback_mask, z[2],change_scale = self.change_scale_fb)
  438. self.H = self.update_weight(self.H, upper[2], beta, delta, self.horizontal_mask, z[2],change_scale = False)
  440. class OutputLayer(CustomLayer):
  441. """Output layer that aggregates information from all hierarchical levels.
  443. This layer combines skip connections from input, hidden, and high-level layers
  444. to produce action values (Q-values).
  445. """
  447. def __init__(self, high_features: int, hidden_features: int, input_features: int, feature_out: int, grid_size: int, RF_size_list: List[int]) -> None:
  448. """Initialize output layer.
  450. Args:
  451. high_features (int): Number of high-level features.
  452. hidden_features (int): Number of hidden features.
  453. input_features (int): Number of input features.
  454. feature_out (int): Number of output features.
  455. grid_size (int): Size of spatial grid.
  456. RF_size_list (list): List of receptive field sizes for each scale.
  457. """
  458. super().__init__()
  459. self.grid_size = grid_size
  460. K_size = 2 * self.grid_size - 1
  461. self.layer_type = 'output'
  462. self.grid_size = grid_size
  463. self.RF_size_list = RF_size_list
  464. self.hidden_features = hidden_features
  465. self.high_features = high_features
  466. self.feature_out = feature_out
  468. self.skip_weights = []
  469. for layer in range(len(self.RF_size_list)):
  470. if layer == 0:
  471. self.skip_weights.append(nn.Conv2d(input_features, feature_out, 1, stride=self.RF_size_list[layer], padding='same', bias=False))
  472. elif layer == 1:
  473. self.skip_weights.append(nn.Conv2d(hidden_features, feature_out, K_size, stride=self.RF_size_list[layer], padding='same',bias=False))
  474. else:
  475. self.skip_weights.append(nn.ConvTranspose2d(high_features, feature_out,K_size, stride=self.RF_size_list[layer], padding=int(0.5*(2 * self.grid_size - 1 - self.RF_size_list[layer])),bias=False))
  477. self.skip_masks = [None]
  478. for layer in range(1,len(self.RF_size_list)):
  479. self.skip_masks.append(torch.zeros_like(self.skip_weights[layer].weight))
  480. lower_bound = self.grid_size-(self.RF_size_list[layer]//2)-1
  481. upper_bound = self.grid_size+(self.RF_size_list[layer]//2)
  482. self.skip_masks[layer][:,:,lower_bound:upper_bound,lower_bound:upper_bound] = 1/50
  484. # Initializing weights
  485. self.initialize_feedforward_weights(self.skip_weights[0], one_to_one=True)
  486. for layer in range(1,len(self.RF_size_list)):
  487. self.initialize_feedforward_weights(self.skip_weights[layer], receptive_field_size=self.RF_size_list[layer])
  489. self.skip_weights = nn.ModuleList(self.skip_weights)
  491. def forward(self, pyramidal_recurrent: List[torch.Tensor]) -> torch.Tensor:
  492. """Forward pass through output layer.
  494. Args:
  495. pyramidal_recurrent (list): List of pyramidal activities from all layers.
  497. Returns:
  498. torch.Tensor: Output Q-values for action selection.
  499. """
  500. Y = self.skip_weights[0](pyramidal_recurrent[0])
  501. for layer in range(1,len(pyramidal_recurrent)):
  502. Y = Y + self.skip_weights[layer](pyramidal_recurrent[layer])
  503. return(Y)
  505. def rescale(self, new_grid_size: int, device: torch.device) -> None:
  506. """Rescale output layer for different grid size.
  508. Args:
  509. new_grid_size (int): New spatial grid size.
  510. device (torch.device): Device to move parameters to.
  511. """
  512. K_size = 2 * new_grid_size - 1
  513. skip_weights = [self.skip_weights[0]]
  514. for layer in range(1,len(self.RF_size_list)):
  515. if layer == 1:
  516. skip_weights.append(nn.Conv2d(self.hidden_features, self.feature_out, K_size, stride=self.RF_size_list[layer], padding='same',bias=False))
  517. else:
  518. skip_weights.append(nn.ConvTranspose2d(self.high_features, self.feature_out,K_size, stride=self.RF_size_list[layer], padding=int(0.5*(2 * new_grid_size - 1 - self.RF_size_list[layer])),bias=False))
  520. for layer in range(1,len(self.RF_size_list)):
  521. weight = torch.zeros_like(skip_weights[layer].weight)
  522. lower_bound = new_grid_size - self.RF_size_list[layer]//2 - 1
  523. upper_bound = new_grid_size + self.RF_size_list[layer]//2
  524. non_zero_weights = torch.unique(self.skip_weights[layer].weight[self.skip_weights[layer].weight!=0])
  525. non_zero_weights = non_zero_weights[:,None,None,None]
  526. weight[:,:,lower_bound:upper_bound,lower_bound:upper_bound] = non_zero_weights
  527. skip_weights[layer].weight = torch.nn.Parameter(weight)
  528. self.skip_weights[layer] = skip_weights[layer]
  529. self.skip_weights[layer].to(device)
  531. self.skip_masks = [None]
  532. for layer in range(1,len(self.RF_size_list)):
  533. self.skip_masks.append(torch.zeros_like(self.skip_weights[layer].weight))
  534. lower_bound = new_grid_size-(self.RF_size_list[layer]//2)-1
  535. upper_bound = new_grid_size+(self.RF_size_list[layer]//2)
  536. self.skip_masks[layer][:,:,lower_bound:upper_bound,lower_bound:upper_bound] = 1/50
  538. self.grid_size = new_grid_size
  541. def update_layer(self, upper: torch.Tensor, beta: float, delta: float) -> None:
  542. """Update all skip connection weights.
  544. Args:
  545. upper (torch.Tensor): Upper layer activity.
  546. beta (float): Learning rate.
  547. delta (float): Reward prediction error.
  548. """
  549. for layer in range(len(self.skip_weights)):
  550. if layer == 0:
  551. self.skip_weights[layer] = self.update_weight(self.skip_weights[layer],upper,beta,delta,average=False)
  552. else:
  553. self.skip_weights[layer] = self.update_weight(self.skip_weights[layer],upper,beta,delta,self.skip_masks[layer],receptive_field_size = self.RF_size_list[layer])
  555. class FeedforwardLayer(CustomLayer):
  556. """Wrapper for pretrained feedforward networks.
  558. This layer encapsulates pretrained feedforward networks for object and curve detection
  559. at multiple scales.
  560. """
  562. def __init__(self, feedforward: nn.ModuleList, feedforward_interm: nn.ModuleList, num_scales: int) -> None:
  563. """Initialize feedforward layer.
  565. Args:
  566. feedforward (nn.ModuleList): List of feedforward layers.
  567. feedforward_interm (nn.ModuleList): List of intermediate layers.
  568. num_scales (int): Number of spatial scales.
  569. """
  570. super().__init__()
  571. self.num_scales = num_scales
  572. self.feedforward = feedforward
  573. self.feedforward_interm = feedforward_interm
  575. self.sig = nn.Sigmoid()
  577. def forward(self, x: torch.Tensor) -> List[torch.Tensor]:
  578. """Forward pass through feedforward network.
  580. Args:
  581. x (torch.Tensor): Input image.
  583. Returns:
  584. list: Multi-scale feature representations.
  585. """
  586. intern_representation = [None] * self.num_scales
  587. for layer in range(self.num_scales):
  588. if layer == 0:
  589. x = F.relu(self.feedforward[0](x))
  590. intern_representation[layer] = x
  591. else:
  592. interm = F.relu(self.feedforward_interm[layer](x))
  593. intern_representation[layer] = self.sig(self.feedforward[layer](interm))
  595. intern_representation[0] = intern_representation[0].detach()
  596. for layer in range(1,len(intern_representation)):
  597. intern_representation[layer] = torch.relu(intern_representation[layer] - 0.7)
  598. intern_representation[layer] = intern_representation[layer].detach()
  600. return intern_representation

layers.py at commit cb55334, no license · at the source

Overview

Authors: Sami Mollard1, Sander M. Bohte2,3, Pieter R. Roelfsema1,4,5,6
ORCID iDs: Sami Mollard, Pieter R. Roelfsema
  1. Department of Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands
  2. Machine Learning Group, Centrum Wiskunde & Informatica, Amsterdam, The Netherlands
  3. Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, Netherlands
  4. Laboratory of Visual Brain Therapy, Sorbonne Université, Institut National de la Santé et de la Recherche Médicale, Centre National de la Recherche Scientifique, Institut de la Vision, Paris, France
  5. Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, VU University, Amsterdam, The Netherlands
  6. Department of Neurosurgery, Amsterdam University Medical Center, Amsterdam, The Netherlands
Institutions: Netherlands Institute for Neuroscience (Netherlands); Centrum Wiskunde & Informatica (Netherlands); University of Amsterdam (Netherlands); Centre National de la Recherche Scientifique (France); Institut de la Vision (France); Inserm (France); Sorbonne Université (France); Amsterdam University Medical Centers (Netherlands); Vrije Universiteit Amsterdam (Netherlands)
Journal: PLoS computational biology, volume 22, issue 4, article e1014193
Dates: received 17 June 2025; accepted 1 April 2026; published online 15 April 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1371/journal.pcbi.1014193 · PMID 41984938 · PMCID PMC13095124 · OpenAlex W7154471241
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: human (organism), non-human primate (organism)
Methods: Preprocessing
MeSH: Learning*, Models, Neurological*, Visual Cortex*, Animals, Computational Biology, Humans, Neurons, Visual Perception (* major topic)
Journal subjects: Biology and Life Sciences, Cell Biology, Cellular Types, Animal Cells, Neurons, Neuroscience, Cellular Neuroscience, Cognitive Science, Cognitive Psychology, Learning, Learning Curves, Psychology, Social Sciences, Learning and Memory, Anatomy, Brain, Visual Cortex, Medicine and Health Sciences, Perception, Sensory Perception, Vision, Computer and Information Sciences, Neural Networks, Organisms, Eukaryota, Animals, Vertebrates, Amniotes, Mammals, Primates, Monkeys, Zoology, Recurrent Neural Networks
Topic: Visual perception and processing mechanisms (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: EC | Horizon 2020 Framework Programme (EU Framework Programme for Research and Innovation H2020) (945539, 800858); EC | HORIZON EUROPE Framework Programme (Horizon Europe) (101052963); Nederlandse Organisatie voor Wetenschappelijk Onderzoek (17619, OCENW.KLEIN.178); Agence Nationale de la Recherche (French National Research Agency) (ANR-18- 590 IAHU-0001); Ministerie van Onderwijs, Cultuur en Wetenschap
Citations: not cited yet (Europe PMC); 60 references in the paper

Abstract

Natural scenes usually contain many objects that need to be segregated from each other and the background. Object-based attention is the process that groups image fragments belonging to the same objects. Curve-tracing tasks provide a special case, testing our ability to group image elements of an elongated curve. In the brain, curve-tracing is associated with the gradual spread of enhanced neuronal activity over the representation of the traced curve. Previous studies demonstrated that the tracing speed is higher if curves are far apart than if they are nearby. One hypothesis is that a larger distance between curves permits activity propagation in higher visual cortical areas. In these higher areas receptive fields are larger and connections exist between neurons representing image regions that are farther apart (Pooresmaeili et al., 2014). We propose a recurrent architecture for the scale-invariant tracing of curves and objects. The architecture is composed of a feedforward pathway that dynamically selects the appropriate scale for tracing, and a recurrent pathway for propagating enhanced neuronal activity through horizontal and feedback connections, enabled by a disinhibitory loop involving VIP and SOM interneurons. We trained the network using a biologically plausible reinforcement learning scheme and observed that training on short curves allowed the networks to generalize to longer curves and 2D-objects. The network chose the scale based on the distance between curves and the width of objects, just as in human psychophysics and the visual cortex of monkeys. The results provide a mechanistic account of the learning and execution of multiscale perceptual grouping in the brain.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 12 matches between paragraphs and lines of code.

samimol/multiscale_tracing

License: none: the authors keep all their rights
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: cb55334b4075ad35b3e1c891f64cab10a477c93f, 2 March 2026
Languages: Python (12)
Size: 28 files, 12 scripts
Software Heritage: not archived
Found in: “Data Availability”
Holds: README, environment (requirements.txt)
Not found: license file, CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (8 files), PyTorch (8 files), scikit-image (2 files), SciPy (1 file)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
13 files
At the source: github.com/samimol/multiscale_tracing

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 12 scripts, each with its path and the digest of its content;
  • 12 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data Availability

All the code used to train the networks and to analyze the data is available on the following GitHub address: https://github.com/samimol/multiscale_tracing.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 3 authors, 8 MeSH terms, 5 funders, 52 references.

Cite

This paper

Mollard, S., Bohte, S. M., & Roelfsema, P. R. (2026). How the visual brain can learn to parse images using a multiscale, incremental grouping process. PLoS computational biology, 22(4), e1014193. https://doi.org/10.1371/journal.pcbi.1014193

BibTeX

@article{mollard2026how,
author = {Mollard, Sami and Bohte, Sander M. and Roelfsema, Pieter R.},
title = {{How the visual brain can learn to parse images using a multiscale, incremental grouping process}},
journal = {PLoS computational biology},
year = {2026},
month = apr,
volume = {22},
number = {4},
pages = {e1014193},
publisher = {PLOS},
issn = {1553-734X},
doi = {10.1371/journal.pcbi.1014193},
url = {https://doi.org/10.1371/journal.pcbi.1014193},
pmid = {41984938},
pmcid = {PMC13095124}
}

RIS

TY - JOUR
AU - Mollard, Sami
AU - Bohte, Sander M.
AU - Roelfsema, Pieter R.
TI - How the visual brain can learn to parse images using a multiscale, incremental grouping process
T2 - PLoS computational biology
J2 - PLoS Comput Biol
PY - 2026
DA - 2026/04/15
VL - 22
IS - 4
SP - e1014193
SN - 1553-734X
PB - PLOS
DO - 10.1371/journal.pcbi.1014193
UR - https://doi.org/10.1371/journal.pcbi.1014193
LA - en
ER -

CSL-JSON

{
"id": "10.1371/journal.pcbi.1014193",
"type": "article-journal",
"title": "How the visual brain can learn to parse images using a multiscale, incremental grouping process",
"container-title": "PLoS computational biology",
"author": [
{
"family": "Mollard",
"given": "Sami"
},
{
"family": "Bohte",
"given": "Sander M."
},
{
"family": "Roelfsema",
"given": "Pieter R."
}
],
"container-title-short": "PLoS Comput Biol",
"volume": "22",
"issue": "4",
"page": "e1014193",
"DOI": "10.1371/journal.pcbi.1014193",
"PMID": "41984938",
"PMCID": "PMC13095124",
"ISSN": "1553-734X",
"publisher": "PLOS",
"URL": "https://doi.org/10.1371/journal.pcbi.1014193",
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
15
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1371/journal.pbio.3003831 [code]
Disinhibitory signaling enables flexible coding of top-down information in cortical networks.
Journal: PLoS biology
In common: PyTorch, SciPy, NumPy, 6 references
[2] doi:10.1371/journal.pbio.3003915 [code]
Noise-invariant representations of sound emerge along the canonical cortical hierarchy.
Journal: PLoS biology
In common: scikit-image, PyTorch, SciPy, 1 other tool, 3 references
[3] doi:10.1038/s41598-026-45730-8 [code]
Gestalt laws enhance the representation of figures over backgrounds in the visual cortex and influence contrast perception.
Journal: Scientific reports
In common: non-human primate, 4 references
[4] doi:10.1038/s41467-026-72146-9 [code]
Modeling attention and binding in the brain through bidirectional recurrent gating.
Journal: Nature communications
In common: PyTorch, SciPy, NumPy, 3 references
[5] doi:10.1038/s42003-026-10418-2 [code]
Cortical PV and VIP interneurons similarly influence SST neuron output despite distinct unitary properties.
Journal: Communications biology
In common: 5 references
[6] doi:10.1038/s41467-026-72619-x [code]
An inhibitory brainstem pathway reduces visual detection during background motion.
Journal: Nature communications
In common: 4 references
[7] doi:10.1038/s41467-026-76098-y [code]
A single computational objective can produce specialization of streams in visual cortex.
Journal: Nature communications
In common: scikit-image, PyTorch, SciPy, 1 other tool, 1 reference
[8] doi:10.1126/sciadv.aed6417 [code]
Intrinsic timing, not temporal prediction, underlies ramping dynamics in visual and parietal cortex during passive behavior.
Journal: Science advances
In common: scikit-image, SciPy, NumPy, 2 references
[9] doi:10.1038/s41467-026-71331-0 [code]
A multimodal approach for visualizing and identifying electrophysiological cell types in vivo.
Journal: Nature communications
In common: PyTorch, SciPy, NumPy, 2 references
[10] doi:10.1038/s41467-026-71918-7 [code]
Developmental disinhibition gates language lateralization in childhood.
Journal: Nature communications
In common: PyTorch, SciPy, NumPy, 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.