OSCR

Temporal coding enables hyperacuity in event-based vision.

Code ↔ Paper

8 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 8 matches
  1. [1] § Methods › Model implementation and training details ↔ eb_models.py, lines 523–557 · score 0.87 · residual connections, layer normalization, multi head, dropout rate, Transformer layers, query
  2. [2] § Methods › Model implementation and training details ↔ losses.py, lines 11–49 · score 0.67 · Cross Entropy, cosine, Temperature, Loss, embedding, logits
  3. [3] § Methods › Model implementation and training details ↔ eb_models.py, lines 111–251 · score 0.63 · projection head, classification head, logits, batches, dimensional, network
  4. [4] § Methods › Setup ↔ data_collection/events/syclop_acquire_dataset_20230623.ipynb, lines 552–577 · score 0.62 · TGY D003HV, Maestro, motors, rotations, event
  5. [5] § Methods › Model implementation and training details ↔ eb_train.py, lines 241–256 · score 0.59 · decay, exponential, Adam, warmup, Optimization, schedule
  6. [6] § Methods › Model implementation and training details ↔ eb_models.py, lines 289–383 · score 0.54 · GRU layer, classification head, model
  7. [7] § Methods › Model implementation and training details ↔ eb_contrastive.py, lines 81–127 · score 0.53 · decay, exponential, Adam, schedule, batch, linear
  8. [8] § Results › Accurate event timing contains task-relevant information ↔ notebooks/fig_acc_vs_jitter_allDS_supp.ipynb, lines 407–454 · score 0.50 · tailed Wilcoxon rank, jitter training, sum, accuracy, EB, event

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 588 lines · 25 KB · other · 3 matches

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  4. import torchvision.transforms as transforms
  5. from torchvision.models import resnet18, ResNet18_Weights
  7. class TimeSeriesTransformer(nn.Module):
  8. def __init__(self,
  9. n_timesteps = 64,
  10. d_timeseries = 4,
  11. d_k = 256,
  12. d_mlp = 512,
  13. d_mlp2 = 256,
  14. n_layers = 8,
  15. num_heads = 4,
  16. num_classes = 10,
  17. dropout_rate = 0.1,
  18. offsets = None,
  19. scalings = None,
  20. dropout_ffn_from_layer = 4,
  21. subseq_len = None,
  22. model_head = 'cls_mlp',
  23. model_head_init_method = None,
  24. input_dense_init_method = None):
  26. '''
  27. Transformer for time series data.
  28. :param n_timesteps: number of timesteps in the input
  29. :param d_timeseries: dimensionality of the timeseries
  30. :param d_k: dimensionality of the key and query vectors
  31. :param d_mlp: dimensionality of the MLP in the feedforward layer
  32. :param d_mlp2: dimensionality of the MLP in the classification head
  33. :param n_layers: number of layers in the Transformer
  34. :param num_heads: number of heads in the Multi-Head Attention
  35. :param num_classes: number of classes in the classification head
  36. :param dropout_rate: dropout rate
  37. :param offsets: offsets to be added to the input
  38. :param scalings: scalings to be multiplied to the input
  39. :param dropout_ffn_from_layer: layer from which to apply dropout to the feedforward network
  40. :param subseq_len: list or tuple. when not None, the input is split into subsequences according to this list, each subsequence is processed by the transformer, and the outputs are concatenated
  41. :param model_head: head architecture. 'cls_mlp' or 'cls_avgpool'
  42. Receives input of shape (batch_size, n_timesteps, d_timeseries), returns logits of shape (batch_size, num_classes)
  43. '''
  44. super(TimeSeriesTransformer, self).__init__()
  45. if offsets is None:
  46. offsets = torch.zeros(1, 1, d_timeseries)
  47. if scalings is None:
  48. scalings = torch.ones(1, 1, d_timeseries)
  49. self.offsets = offsets
  50. self.scalings = scalings
  51. self.n_layers = n_layers
  52. self.dropout_ffn_from_layer = dropout_ffn_from_layer
  53. self.subseq_len = subseq_len
  54. self.model_head_init_method = model_head_init_method
  56. # Initial dense layer to expand dimensions
  57. #todo: add option to use dedicated input dense layer per subsequence in a case of multiple subsequences
  58. self.input_dense = nn.Linear(d_timeseries, d_k)
  59. if input_dense_init_method is not None:
  60. apply_init_method([self.input_dense], input_dense_init_method)
  62. # Defining n_layers of Transformer Layers
  63. self.transformer_layers = nn.ModuleList([
  64. TransformerLayer(d_k, num_heads, d_mlp, dropout_rate) for _ in range(n_layers)
  65. ])
  67. if model_head == 'cls_mlp':
  68. self.cls_head = ClassifierHead_MLP(d_k * n_timesteps,
  69. d_mlp2, num_classes, dropout_rate,do_flatten=True)
  70. elif model_head == 'cls_avgpool':
  71. self.cls_head = ClassifierHead_AvgPool(d_k, d_mlp2, num_classes, dropout_rate, init_method=model_head_init_method)
  72. elif model_head == 'proj_avgpool':
  73. self.cls_head = ProjectionHead_AvgPool(d_k, d_mlp2)
  74. elif model_head == 'none':
  75. self.cls_head = nn.Identity()
  76. else:
  77. raise ValueError('Unknown cls_head_arch')
  79. def forward(self, x, attn_mask=None, padding_mask=None):
  81. # Add offsets and scaling
  82. #convert to torch tensor locally to avoid problems with multiple GPUs
  83. torch_offsets = torch.tensor(self.offsets, device=x.device, dtype=x.dtype)
  84. torch_scalings = torch.tensor(self.scalings, device=x.device, dtype=x.dtype)
  86. if self.subseq_len is not None and (attn_mask is not None or padding_mask is not None):
  87. raise ValueError('subseq_len cannot be used with attn_mask or key_padding_mask')
  89. if self.subseq_len is not None:
  90. #in a case of multiple subsequences, we create an attention mask to prevent attention between subsequences
  91. #this mask is a matrix of shape (n_timesteps, n_timesteps) it is false between all pairs of timesteps that belong to different subsequences
  92. #otherwise, the mask is None
  93. attn_mask = torch.ones(x.shape[1], x.shape[1], dtype=torch.bool, device=x.device)
  94. this_position = 0
  95. for subseq_len in self.subseq_len:
  96. attn_mask[this_position:this_position+subseq_len, this_position:this_position+subseq_len] = False
  97. this_position += subseq_len
  99. x = (x + torch_offsets) * torch_scalings
  101. x = self.input_dense(x)
  103. for i, layer in enumerate(self.transformer_layers):
  104. x = layer(x, apply_dropout=(i >= self.dropout_ffn_from_layer), attn_mask=attn_mask, key_padding_mask=padding_mask)
  106. x = self.cls_head(x, mask=padding_mask)
  108. return x
  111. class TimeSeriesTransformerWithSubseq(nn.Module):
  112. def __init__(self,
  113. n_timesteps=64,
  114. d_timeseries=4,
  115. d_k=256,
  116. d_mlp=512,
  117. d_mlp2=256,
  118. n_layers=8,
  119. num_heads=4,
  120. num_classes=10,
  121. dropout_rate=0.1,
  122. offsets=None,
  123. scalings=None,
  124. dropout_ffn_from_layer=4,
  125. subseq_len=None,
  126. model_head='cls_mlp',
  127. shared_input_dense=False,
  128. drop_subseq_prob=0.5,
  129. extra_dim_for_subsamples=False):
  130. '''
  131. Transformer for time series data. With option to split the input into multiple subsequences.
  132. :param n_timesteps: number of timesteps in the input
  133. :param d_timeseries: dimensionality of the timeseries
  134. :param d_k: dimensionality of the key and query vectors
  135. :param d_mlp: dimensionality of the MLP in the feedforward layer
  136. :param d_mlp2: dimensionality of the MLP in the classification head
  137. :param n_layers: number of layers in the Transformer
  138. :param num_heads: number of heads in the Multi-Head Attention
  139. :param num_classes: number of classes in the classification head
  140. :param dropout_rate: dropout rate
  141. :param offsets: offsets to be added to the input
  142. :param scalings: scalings to be multiplied to the input
  143. :param dropout_ffn_from_layer: layer from which to apply dropout to the feedforward network
  144. :param subseq_len: list or tuple. when not None, the input is split into subsequences according to this list, each subsequence is processed by the transformer, and the outputs are concatenated
  145. :param model_head: head architecture. 'cls_mlp' or 'cls_avgpool'
  146. :param shared_input_dense: if True, the input is processed by a single dense layer before being split into subsequences. If False, each subsequence is processed by a dedicated dense layer.
  147. :param drop_subseq_prob: probability of dropping a subsequence during training
  148. Receives input of shape (batch_size, n_timesteps, d_timeseries), returns logits of shape (batch_size, num_classes)
  149. '''
  150. super(TimeSeriesTransformerWithSubseq, self).__init__()
  151. if offsets is None:
  152. offsets = torch.zeros(1, 1, d_timeseries)
  153. if scalings is None:
  154. scalings = torch.ones(1, 1, d_timeseries)
  155. self.offsets = offsets
  156. self.scalings = scalings
  157. self.n_layers = n_layers
  158. self.dropout_ffn_from_layer = dropout_ffn_from_layer
  159. self.subseq_len = subseq_len
  160. self.shared_input_dense = shared_input_dense
  161. self.drop_subseq_prob = drop_subseq_prob
  162. self.extra_dim_for_subsamples = extra_dim_for_subsamples
  164. # Initial dense layer to expand dimensions
  165. if shared_input_dense:
  166. self.input_dense = nn.Linear(d_timeseries, d_k)
  167. else:
  168. self.input_dense = nn.ModuleList([nn.Linear(d_timeseries, d_k) for _ in range(len(subseq_len))])
  170. if subseq_len is None:
  171. raise NotImplementedError
  172. # Defining n_layers of Transformer Layers
  173. # self.transformer_layers = nn.ModuleList([
  174. # TransformerLayer(d_k, num_heads, d_mlp, dropout_rate) for _ in range(n_layers)
  175. # ])
  176. else:
  177. # in case of multiple subsequences, we need to split the input into subsequences
  178. # each subsequence is processed by a dedicated transformer, and the outputs are concatenated
  179. for i in range(len(subseq_len)):
  180. setattr(self, 'transformer_stack_{}'.format(i), TransformerCore(d_k=d_k,
  181. num_heads=num_heads,
  182. d_mlp=d_mlp,
  183. n_layers=n_layers,
  184. dropout_rate=dropout_rate,
  185. dropout_ffn_from_layer=dropout_ffn_from_layer))
  187. if model_head == 'cls_mlp':
  188. self.cls_head = ClassifierHead_MLP(d_k * (len(subseq_len) if subseq_len is not None else 1),
  189. d_mlp2, num_classes, dropout_rate,do_flatten=extra_dim_for_subsamples)
  190. elif model_head == 'cls_avgpool':
  191. self.cls_head = ClassifierHead_AvgPool(d_k, d_mlp2, num_classes, dropout_rate)
  192. elif model_head == 'proj_avgpool':
  193. self.cls_head = ProjectionHead_AvgPool(d_k, d_mlp2, skip_mean=True)
  194. elif model_head == 'none':
  195. self.cls_head = nn.Identity()
  196. else:
  197. raise ValueError('Unknown cls_head_arch')
  199. def forward(self, x):
  200. # Add offsets and scaling
  201. #convert to torch tensor locally to avoid problems with multiple GPUs
  202. torch_offsets = torch.tensor(self.offsets, device=x.device, dtype=x.dtype)
  203. torch_scalings = torch.tensor(self.scalings, device=x.device, dtype=x.dtype)
  205. x = (x + torch_offsets) * torch_scalings
  207. if self.shared_input_dense:
  208. x = self.input_dense(x)
  211. if self.subseq_len is None:
  212. raise NotImplementedError
  213. # Pass through each Transformer Layer
  214. # for i, layer in enumerate(self.transformer_layers):
  215. # x = layer(x, apply_dropout=(i >= self.dropout_ffn_from_layer))
  216. else:
  217. # in case of multiple subsequences, we need to split the input into subsequences
  218. # each subsequence is processed by a dedicated transformer, and the outputs are passed through average
  219. # pooling and concatenated
  220. x_list = []
  221. this_position = 0
  222. for i, subseq_len in enumerate(self.subseq_len):
  223. if self.extra_dim_for_subsamples:
  224. x_subseq = x[:, i, ...]
  225. else:
  226. x_subseq = x[:, this_position:this_position+subseq_len, :]
  227. this_position += subseq_len
  228. if not self.shared_input_dense:
  229. x_subseq = self.input_dense[i](x_subseq)
  230. x_subseq = getattr(self, 'transformer_stack_{}'.format(i))(x_subseq)
  231. x_subseq = torch.mean(x_subseq, dim=1)
  232. x_list.append(x_subseq)
  234. # if in training mode with probability self.drop_subseq_prob pick at most one subsequence and set it to zero
  235. # this is done by creating a subsequence of zeros and replacing one random subsequence
  236. # out of the original subsequences with zeros.
  237. if self.training:
  238. if torch.rand(1) < self.drop_subseq_prob:
  239. x_list[torch.randint(len(x_list),(1,))] = torch.zeros_like(x_list[0])
  240. #scale all other subsequences to compensate for the zero subsequence
  241. x_list = [x * (len(x_list) / (len(x_list) - 1)) for x in x_list]
  243. if self.extra_dim_for_subsamples:
  244. x = torch.stack(x_list, dim=1)
  245. else:
  246. x = torch.cat(x_list, dim=1)
  248. # Classification head
  249. x = self.cls_head(x)
  251. return x
  253. class TransformerCore(nn.Module):
  254. def __init__(self,
  255. d_k=256,
  256. d_mlp=512,
  257. n_layers=8,
  258. num_heads=4,
  259. dropout_rate=0.1,
  260. dropout_ffn_from_layer=4):
  261. '''
  262. Core of the Transformer. No classification head or input preprocessing.
  263. :param d_k: dimensionality of the key and query vectors
  264. :param d_mlp: dimensionality of the MLP in the feedforward layer
  265. :param n_layers: number of layers in the Transformer
  266. :param num_heads: number of heads in the Multi-Head Attention
  267. :param dropout_rate: dropout rate
  268. :param dropout_ffn_from_layer: layer from which to apply dropout to the feedforward network
  269. Receives input of shape (batch_size, n_timesteps, d_timeseries), returns logits of shape (batch_size, num_classes)
  270. '''
  271. super(TransformerCore, self).__init__()
  273. self.n_layers = n_layers
  274. self.dropout_ffn_from_layer = dropout_ffn_from_layer
  276. # Defining n_layers of Transformer Layers
  277. self.transformer_layers = nn.ModuleList([
  278. TransformerLayer(d_k, num_heads, d_mlp, dropout_rate) for _ in range(n_layers)
  279. ])
  282. def forward(self, x):
  283. # Pass through each Transformer Layer
  284. for i, layer in enumerate(self.transformer_layers):
  285. x = layer(x, apply_dropout=(i >= self.dropout_ffn_from_layer))
  286. return x
  289. class TimeSeriesGRU(nn.Module):
  290. def __init__(self,
  291. n_timesteps=64,
  292. d_timeseries=4,
  293. d_hidden=256,
  294. d_mlp=512,
  295. d_mlp2=256,
  296. n_layers=8,
  297. num_classes=10,
  298. dropout_rate=0.1,
  299. offsets=None,
  300. scalings=None,
  301. dropout_ffn_from_layer=4,
  302. subseq_len=None,
  303. model_head='cls_mlp',
  304. model_head_init_method=None,
  305. input_dense_init_method=None):
  306. '''
  307. GRU-based model for time series data.
  308. :param n_timesteps: number of timesteps in the input
  309. :param d_timeseries: dimensionality of the timeseries
  310. :param d_hidden: dimensionality of the hidden state in the GRU
  311. :param d_mlp: dimensionality of the MLP in the feedforward layer
  312. :param d_mlp2: dimensionality of the MLP in the classification head
  313. :param n_layers: number of GRU layers
  314. :param num_classes: number of classes in the classification head
  315. :param dropout_rate: dropout rate
  316. :param offsets: offsets to be added to the input
  317. :param scalings: scalings to be multiplied to the input
  318. :param dropout_ffn_from_layer: layer from which to apply dropout to the feedforward network
  319. :param subseq_len: list or tuple. when not None, the input is split into subsequences according to this list, each subsequence is processed by the GRU, and the outputs are concatenated
  320. :param model_head: head architecture. 'cls_mlp' or 'cls_avgpool'
  321. Receives input of shape (batch_size, n_timesteps, d_timeseries), returns logits of shape (batch_size, num_classes)
  322. '''
  323. super(TimeSeriesGRU, self).__init__()
  324. if offsets is None:
  325. offsets = torch.zeros(1, 1, d_timeseries)
  326. if scalings is None:
  327. scalings = torch.ones(1, 1, d_timeseries)
  328. self.offsets = offsets
  329. self.scalings = scalings
  330. self.n_layers = n_layers
  331. self.dropout_ffn_from_layer = dropout_ffn_from_layer
  332. self.subseq_len = subseq_len
  333. self.model_head_init_method = model_head_init_method
  335. # Initial dense layer to expand dimensions
  336. self.input_dense = nn.Linear(d_timeseries, d_hidden)
  337. if input_dense_init_method is not None:
  338. apply_init_method([self.input_dense], input_dense_init_method)
  340. # GRU layer
  341. self.gru_layers = nn.ModuleList([
  342. nn.GRU(input_size=d_hidden, hidden_size=d_hidden, num_layers=1, batch_first=True) for _ in range(n_layers)
  343. ])
  345. # Dropout layer
  346. self.dropout = nn.Dropout(dropout_rate)
  348. # Classification head
  349. if model_head == 'cls_mlp':
  350. self.cls_head = ClassifierHead_MLP(d_hidden * n_timesteps, d_mlp2, num_classes, dropout_rate,
  351. do_flatten=True)
  352. elif model_head == 'cls_avgpool':
  353. self.cls_head = ClassifierHead_AvgPool(d_hidden, d_mlp2, num_classes, dropout_rate,
  354. init_method=model_head_init_method)
  355. elif model_head == 'proj_avgpool':
  356. self.cls_head = ProjectionHead_AvgPool(d_hidden, d_mlp2)
  357. elif model_head == 'none':
  358. self.cls_head = nn.Identity()
  359. else:
  360. raise ValueError('Unknown cls_head_arch')
  362. def forward(self, x, attn_mask=None, padding_mask=None):
  363. # Add offsets and scaling
  364. torch_offsets = torch.tensor(self.offsets, device=x.device, dtype=x.dtype)
  365. torch_scalings = torch.tensor(self.scalings, device=x.device, dtype=x.dtype)
  367. if self.subseq_len is not None and (attn_mask is not None or padding_mask is not None):
  368. raise ValueError('subseq_len cannot be used with attn_mask or key_padding_mask')
  370. x = (x + torch_offsets) * torch_scalings
  371. x = self.input_dense(x)
  373. # GRU forward pass with conditional dropout
  374. for i, gru_layer in enumerate(self.gru_layers):
  375. x, _ = gru_layer(x)
  376. if i >= self.dropout_ffn_from_layer:
  377. x = self.dropout(x)
  379. # Apply the classification head
  381. x = self.cls_head(x.contiguous(), mask=padding_mask)
  383. return x
  386. class VanillaCNN(nn.Module):
  387. def __init__(self, input_shape = (10,10),
  388. num_classes = 10):
  389. super(VanillaCNN, self).__init__()
  391. self.resize = nn.Upsample(size=(56, 56), mode='bilinear', align_corners=False)
  393. self.conv1 = nn.Conv2d(1, 32, kernel_size=3)
  394. self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
  395. self.conv3 = nn.Conv2d(64, 64, kernel_size=2)
  397. self.pool = nn.MaxPool2d(2)
  399. # Calculate the size of the flattened features
  400. self.flat_features = self._get_flat_features()
  402. self.dropout1 = nn.Dropout(0.3)
  403. self.fc1 = nn.Linear(self.flat_features, 128)
  404. self.dropout2 = nn.Dropout(0.3)
  405. self.fc2 = nn.Linear(128, num_classes)
  407. def forward(self, x):
  408. if len(x.shape) == 3:
  409. x = x.unsqueeze(1)
  410. x = self.resize(x)
  412. x = F.relu(self.conv1(x))
  413. x = self.pool(x)
  415. x = F.relu(self.conv2(x))
  416. x = self.pool(x)
  418. x = F.relu(self.conv3(x))
  420. x = x.view(-1, self.flat_features)
  421. x = self.dropout1(x)
  422. x = F.relu(self.fc1(x))
  423. x = self.dropout2(x)
  424. x = self.fc2(x)
  426. return x
  428. def _get_flat_features(self):
  429. # Helper method to calculate the size of the flattened features
  430. with torch.no_grad():
  431. x = torch.zeros(1, 1, 56, 56) # Dummy input
  432. x = F.relu(self.conv1(x))
  433. x = self.pool(x)
  434. x = F.relu(self.conv2(x))
  435. x = self.pool(x)
  436. x = F.relu(self.conv3(x))
  437. return x.numel()
  440. class GrayscaleResNet18(nn.Module):
  441. def __init__(self, input_shape = (10,10),
  442. num_classes = 10):
  443. super(GrayscaleResNet18, self).__init__()
  445. # Load pretrained ResNet18
  446. self.resnet = resnet18(weights=None)
  448. # Modify the first convolutional layer to accept grayscale input
  449. self.resnet.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
  451. # Modify the final fully connected layer if needed
  452. if num_classes != 1000:
  453. self.resnet.fc = nn.Linear(self.resnet.fc.in_features, num_classes)
  455. # Define preprocessing transforms
  456. self.preprocess = transforms.Compose([
  457. transforms.Resize(224, antialias=True),
  458. # transforms.CenterCrop(224),
  459. transforms.Normalize(mean=[0.485], std=[0.229])
  460. ])
  462. def forward(self, x):
  463. # x is expected to be a grayscale image tensor of shape (B, 1, H, W)
  464. if len(x.shape) == 3:
  465. x = x.unsqueeze(1)
  466. x = self.preprocess(x)
  467. return self.resnet(x)
  469. class ClassifierHead_MLP(nn.Module):
  470. def __init__(self, d_flatten, d_mlp, num_classes, dropout_rate, do_flatten=True):
  471. super(ClassifierHead_MLP, self).__init__()
  472. self.do_flatten = do_flatten
  473. self.additional_dense = nn.Linear(d_flatten, d_mlp)
  474. self.output_dense = nn.Linear(d_mlp, num_classes)
  475. self.dropout = nn.Dropout(dropout_rate)
  477. def forward(self, x, mask=None):
  478. if mask is not None:
  479. raise ValueError('mask is not supported for MLP head')
  480. if self.do_flatten:
  481. x = x.view(x.size(0), -1) # Flatten
  483. x = self.dropout(x)
  484. x = F.relu(self.additional_dense(x))
  485. x = self.dropout(x)
  486. x = self.output_dense(x)
  487. return x
  489. class ClassifierHead_AvgPool(nn.Module):
  490. def __init__(self, d_k, d_mlp, num_classes, dropout_rate, init_method=None):
  491. super(ClassifierHead_AvgPool, self).__init__()
  492. self.additional_dense = nn.Linear(d_k, d_mlp)
  493. self.output_dense = nn.Linear(d_mlp, num_classes)
  494. self.dropout = nn.Dropout(dropout_rate)
  495. if init_method is not None:
  496. apply_init_method([self.additional_dense, self.output_dense], init_method)
  498. def forward(self, x, mask=None):
  499. x = masked_mean(x, mask, dim=1)
  500. x = self.dropout(x)
  501. x = F.relu(self.additional_dense(x))
  502. x = self.dropout(x)
  503. x = self.output_dense(x)
  504. return x
  506. class ProjectionHead_AvgPool(nn.Module):
  507. def __init__(self, d_k, d_mlp, skip_mean=False):
  508. super(ProjectionHead_AvgPool, self).__init__()
  509. self.additional_dense = nn.Linear(d_k, d_mlp)
  510. self.output_dense = nn.Linear(d_mlp, d_mlp)
  511. self.skip_mean = skip_mean
  513. def forward(self, x, mask=None):
  514. if not self.skip_mean:
  515. x = masked_mean(x, mask=mask, dim=1)
  516. x = F.relu(self.additional_dense(x))
  517. x = self.output_dense(x)
  518. return x
  523. class TransformerLayer(nn.Module):
  524. def __init__(self, d_k, num_heads, d_mlp, dropout_rate):
  525. super(TransformerLayer, self).__init__()
  526. self.query_projection = nn.Linear(d_k, d_k)
  527. self.key_projection = nn.Linear(d_k, d_k)
  528. self.value_projection = nn.Linear(d_k, d_k)
  530. self.attention = nn.MultiheadAttention(d_k, num_heads, batch_first=True)
  531. self.norm1 = nn.LayerNorm(d_k)
  532. self.norm2 = nn.LayerNorm(d_k)
  534. self.ffn = nn.Sequential(
  535. nn.Linear(d_k, d_mlp),
  536. nn.ReLU(),
  537. nn.Linear(d_mlp, d_k)
  538. )
  539. self.dropout = nn.Dropout(dropout_rate)
  541. def forward(self, x, apply_dropout=False,attn_mask=None,key_padding_mask=None):
  542. # Projecting to query, key, and value
  543. query = self.query_projection(x)
  544. key = self.key_projection(x)
  545. value = self.value_projection(x)
  547. # Multi-Head Attention and Residual Connection
  548. att_out, _ = self.attention(query, key, value,attn_mask=attn_mask,key_padding_mask=key_padding_mask)
  549. x = self.norm1(x + att_out)
  551. # Feed-Forward Network and Residual Connection
  552. ffn_out = self.ffn(x)
  553. if apply_dropout:
  554. ffn_out = self.dropout(ffn_out)
  555. x = self.norm2(x + ffn_out)
  557. return x
  560. def masked_mean(x,mask=None,dim=1):
  561. #mask indicates which elements should be excluded from the mean
  562. if mask is None:
  563. x = torch.mean(x, dim=dim)
  564. else:
  565. #expand mask to the same dimentionality as x
  566. mask = mask.unsqueeze(-1)
  567. x = torch.sum(x * ~mask, dim=dim) / torch.sum(~mask, dim=dim)
  568. return x
  570. def apply_init_method(layers, init_method):
  571. if init_method == 'xavier':
  572. for layer in layers:
  573. nn.init.xavier_uniform_(layer.weight)
  574. nn.init.zeros_(layer.bias)
  575. elif init_method == 'kaiming':
  576. for layer in layers:
  577. nn.init.kaiming_uniform_(layer.weight)
  578. nn.init.zeros_(layer.bias)
  579. elif init_method == 'small_weights':
  580. for layer in layers:
  581. nn.init.uniform_(layer.weight, -0.01, 0.01)
  582. nn.init.zeros_(layer.bias)
  583. elif init_method == 'diag':
  584. for layer in layers:
  585. nn.init.eye_(layer.weight)
  586. nn.init.zeros_(layer.bias)
  587. else:
  588. raise ValueError('Unknown init method')

eb_models.py at commit c2b7506, under other · at the source

Overview

Authors: Eldad Assa1, Alexander Rivkind1, Michael Kreiserman1, Fahad Shahbaz Khan2, Salman Khan2, Ehud Ahissar1
ORCID iDs: Eldad Assa, Alexander Rivkind, Ehud Ahissar
  1. Weizmann Institute of Science, Department of Brain Sciences, Rehovot, Israel
  2. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Institutions: Weizmann Institute of Science (Israel); Mohamed bin Zayed University of Artificial Intelligence (United Arab Emirates)
Journal: Nature communications, volume 17, issue 1, article 9951
Dates: received 14 May 2025; accepted 7 August 2026; published online 20 August 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s41467-026-76878-6 · PMID 42754584 · PMCID PMC13586331 · OpenAlex W4408866247
Open access: gold, a free copy (OpenAlex)
Preprint: doi:10.1101/2025.03.25.645190
Status: code verified
Categories: human (organism), cognitive (subfield)
Methods: Connectivity, Statistics, Machine learning
Keywords: Retina, Neural encoding, Oculomotor system
MeSH: Vision, Ocular*, Visual Acuity*, Visual Perception*, Humans, Neural Networks, Computer, Photic Stimulation, Retina (* major topic)
Topic: Computability, Logic, AI Algorithms (Computational Theory and Mathematics, Computer Science), according to OpenAlex
Funding: European Research Council (786949)
Citations: not cited yet (Europe PMC); 78 references in the paper

Abstract

Although the eye may appear static during fixation, it is in constant motion–transforming visual input into rich spatio-temporal streams of neuronal activity. These movements challenge the classical view that visual acuity derives from spatial sampling, suggesting that hyperacuity can emerge from precise temporal encoding. Here we show that artificial systems can exploit this principle using retina-like event-based (EB) sensing, a neuromorphic approach in which the sensor emits asynchronous spikes in response to luminance changes. Using an EB camera undergoing controlled “fixational” motion over tiny, pixelated images, we generated datasets in which recognition depends on sub-pixel information. We found that artificial neural networks trained on these spatio-temporal event streams relied crucially on the precise temporal information and outperformed conventional frame-based models. The learned representations also supported Vernier-style sub-pixel discrimination, demonstrating hyperacuity-like behavior in the artificial bio-mimetic system. These results show that active event-based sensing can use precise timing to recover spatial details that are unrecoverable from a static single frame. Together, these findings offer a new perspective on visual perception, open pathways for advancing neuromorphic engineering and energy-efficient AI vision systems, and provide a framework for testing hypotheses about biological vision.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 8 matches between paragraphs and lines of code.

zalandoresearch/fashion-mnist

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: b2617bb6d3ffa2e429640350f613e3291e10b141, 21 March 2022
Languages: Python (11), JavaScript (1)
Size: 51 files, 12 scripts
Software Heritage: archived
Found in: “Data availability”
Holds: README, license file, environment (Dockerfile, requirements.txt), documentation
Not found: CITATION.cff, tests, continuous integration
Tools: NumPy (5 files), scikit-learn (2 files), TensorFlow (2 files), Matplotlib (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
14 files
At the source: github.com/zalandoresearch/fashion-mnist

ahissar-lab/event-based-hyperacuity

License: other
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: c2b75060e1bb0f1334ca37da23b672483aae9e37, 9 July 2026
Languages: Jupyter (20), Python (8), Shell (2)
Size: 51 files, 30 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file, environment (environment.yml, environment_old.yml), 20 notebooks
Not found: CITATION.cff, tests, continuous integration, documentation
Tools: NumPy (23 files), Matplotlib (17 files), pandas (14 files), PyTorch (11 files), SciPy (9 files), scikit-learn (8 files), OpenCV (2 files), TensorFlow (2 files)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
32 files
At the source: github.com/ahissar-lab/event-based-hyperacuity

Zenodo 20777652

License: CC-BY-4.0
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: “Code availability”
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (23 files), Matplotlib (17 files), pandas (14 files), PyTorch (11 files), SciPy (9 files), scikit-learn (8 files), OpenCV (2 files), TensorFlow (2 files)
Availability: 1 check, the latest on 27 September 2026: the link answers (HTTP 200)
  • 27 September 2026: the link answers (HTTP 200)
32 files
At the source:

Code availability

The code generated in this study is publicly available at Github (https://github.com/ahissar-lab/event-based-hyperacuity) and Zenodo (https://doi.org/10.5281/zenodo.20777652)78. The repository include also the in-house python code used for datasets collections and the python code used to generate the figures. The code can be used freely for academic research.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 3 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 72 scripts, each with its path and the digest of its content;
  • 8 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability Statement

The datasets generated and used in this study (including tiny event-based and tiny frame-based versions) have been deposited in Zenodo and are publicly available at https://doi.org/10.5281/zenodo.2097375639. The three original FB datasets that were converted are also publicly available (MNIST (https://www.kaggle.com/datasets/hojjatk/mnist-dataset)36, Fashion-MNIST (https://github.com/zalandoresearch/fashion-mnist)37, and Kuzushiji-MNIST (https://github.com/rois-codh/kmnist)38), as well as N-MNIST (https://www.kaggle.com/datasets/khoahongg/n-mnist)30. Source data are provided with this paper.

The code generated in this study is publicly available at Github (https://github.com/ahissar-lab/event-based-hyperacuity) and Zenodo (https://doi.org/10.5281/zenodo.20777652)78. The repository include also the in-house python code used for datasets collections and the python code used to generate the figures. The code can be used freely for academic research.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 6 authors, 3 keywords, 7 MeSH terms, 1 funder, 49 references.

Cite

This paper

Assa, E., Rivkind, A., Kreiserman, M., Khan, F. S., Khan, S., & Ahissar, E. (2026). Temporal coding enables hyperacuity in event-based vision. Nature communications, 17(1), 9951. https://doi.org/10.1038/s41467-026-76878-6

BibTeX

@article{assa2026temporal,
author = {Assa, Eldad and Rivkind, Alexander and Kreiserman, Michael and Khan, Fahad Shahbaz and Khan, Salman and Ahissar, Ehud},
title = {{Temporal coding enables hyperacuity in event-based vision}},
journal = {Nature communications},
year = {2026},
month = aug,
volume = {17},
number = {1},
pages = {9951},
publisher = {Nature Publishing Group},
issn = {2041-1723},
doi = {10.1038/s41467-026-76878-6},
url = {https://doi.org/10.1038/s41467-026-76878-6},
pmid = {42754584},
pmcid = {PMC13586331}
}

RIS

TY - JOUR
AU - Assa, Eldad
AU - Rivkind, Alexander
AU - Kreiserman, Michael
AU - Khan, Fahad Shahbaz
AU - Khan, Salman
AU - Ahissar, Ehud
TI - Temporal coding enables hyperacuity in event-based vision
T2 - Nature communications
J2 - Nat Commun
PY - 2026
DA - 2026/08/20
VL - 17
IS - 1
SP - 9951
SN - 2041-1723
PB - Nature Publishing Group
DO - 10.1038/s41467-026-76878-6
UR - https://doi.org/10.1038/s41467-026-76878-6
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41467-026-76878-6",
"type": "article-journal",
"title": "Temporal coding enables hyperacuity in event-based vision",
"container-title": "Nature communications",
"author": [
{
"family": "Assa",
"given": "Eldad"
},
{
"family": "Rivkind",
"given": "Alexander"
},
{
"family": "Kreiserman",
"given": "Michael"
},
{
"family": "Khan",
"given": "Fahad Shahbaz"
},
{
"family": "Khan",
"given": "Salman"
},
{
"family": "Ahissar",
"given": "Ehud"
}
],
"container-title-short": "Nat Commun",
"volume": "17",
"issue": "1",
"page": "9951",
"DOI": "10.1038/s41467-026-76878-6",
"PMID": "42754584",
"PMCID": "PMC13586331",
"ISSN": "2041-1723",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41467-026-76878-6",
"language": "en",
"issued": {
"date-parts": [
[
2026,
8,
20
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41467-026-72057-9 [code]
Sex-specific behavioral feedback modulates sensorimotor processing and drives flexible social behavior.
Journal: Nature communications
In common: TensorFlow, OpenCV, PyTorch, 5 other tools, 1 reference
[2] doi:10.1038/s41467-026-74358-5 [code]
Brain-inspired spatial intelligence for embodied agents.
Journal: Nature communications
In common: TensorFlow, OpenCV, PyTorch, 5 other tools, cognitive
[3] doi:10.1038/s41598-026-43529-1 [code]
A spiking neural network inspired by neuroscience and psychology for Western mode- and key-conditioned music learning and composition.
Journal: Scientific reports
In common: TensorFlow, OpenCV, PyTorch, 5 other tools, cognitive
[4] doi:10.1162/imag.a.1207 [code]
Investigating the temporal dynamics and modeling of mid-level feature representations in humans.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: OpenCV, PyTorch, scikit-learn, 4 other tools, cognitive, 1 reference
[5] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: TensorFlow, OpenCV, PyTorch, 5 other tools
[6] doi:10.1371/journal.pcbi.1014571 [code]
SynAPSeg: A novel dataset and image analysis framework for deep learning-based synapse detection and quantification.
Journal: PLoS computational biology
In common: TensorFlow, OpenCV, PyTorch, 5 other tools
[7] doi:10.1093/jnen/nlaf152 [code]
Clinical and pathologic correlations of machine learning quantification of Aβ deposits across 3 brain regions of decedents with Alzheimer disease.
Journal: Journal of neuropathology and experimental neurology
In common: TensorFlow, OpenCV, PyTorch, 5 other tools
[8] doi:10.1016/j.patter.2026.101538 [code]
A multi-modal foundation model for brain disease diagnosis and medical imaging.
Journal: Patterns (New York, N.Y.)
In common: TensorFlow, OpenCV, PyTorch, 5 other tools
[9] doi:10.1038/s41597-025-05174-7 [code]
A large-scale MEG and EEG dataset for object recognition in naturalistic scenes
Journal: n/a
In common: TensorFlow, OpenCV, PyTorch, 5 other tools
[10] doi:10.1126/sciadv.aed3650 [code]
Truthful visualizations for mass spectrometry imaging enable high-spatial-resolution interactive &lt;i&gt;m/z&lt;/i&gt; mapping and exploration.
Journal: Science advances
In common: TensorFlow, OpenCV, PyTorch, 5 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.