OSCR

AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § Methods › Development of AI diagnostic model ↔ gradientCAM/swin_agcam/models/swin_transformer.py, lines 586–706 · score 0.52 · Swin Transformer, patch embedding, dimensions, classifier, model
  2. [2] § Methods › Development of AI diagnostic model ↔ gradientCAM/swin_agcam/models/swin_transformer_v2.py, lines 593–707 · score 0.52 · Swin Transformer, patch embedding, dimensions, classifier, model

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,085 lines · 43 KB · no license · 1 match

  1. """ Swin Transformer
  2. A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows`
  3. - https://arxiv.org/pdf/2103.14030
  5. Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below
  7. S3 (AutoFormerV2, https://arxiv.org/abs/2111.14725) Swin weights from
  8. - https://github.com/microsoft/Cream/tree/main/AutoFormerV2
  10. Modifications and additions for timm hacked together by / Copyright 2021, Ross Wightman
  11. """
  12. # --------------------------------------------------------
  13. # Swin Transformer
  14. # Copyright (c) 2021 Microsoft
  15. # Licensed under The MIT License [see LICENSE for details]
  16. # Written by Ze Liu
  17. # --------------------------------------------------------
  18. import logging
  19. import math
  20. from typing import Callable, List, Optional, Tuple, Union
  22. import torch
  23. import torch.nn as nn
  25. from ..data import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD
  26. from ..layers import PatchEmbed, Mlp, DropPath, ClassifierHead, to_2tuple, to_ntuple, trunc_normal_, \
  27. _assert, use_fused_attn, resize_rel_pos_bias_table, resample_patch_embed, ndgrid
  28. from ._builder import build_model_with_cfg
  29. from ._features import feature_take_indices
  30. from ._features_fx import register_notrace_function
  31. from ._manipulate import checkpoint_seq, named_apply
  32. from ._registry import generate_default_cfgs, register_model, register_model_deprecations
  33. from .vision_transformer import get_init_weights_vit
  35. __all__ = ['SwinTransformer'] # model_registry will add each entrypoint fn to this
  37. _logger = logging.getLogger(__name__)
  39. _int_or_tuple_2_t = Union[int, Tuple[int, int]]
  42. def window_partition(
  43. x: torch.Tensor,
  44. window_size: Tuple[int, int],
  45. ) -> torch.Tensor:
  46. """
  47. Partition into non-overlapping windows with padding if needed.
  48. Args:
  49. x (tensor): input tokens with [B, H, W, C].
  50. window_size (int): window size.
  52. Returns:
  53. windows: windows after partition with [B * num_windows, window_size, window_size, C].
  54. (Hp, Wp): padded height and width before partition
  55. """
  56. B, H, W, C = x.shape
  57. x = x.view(B, H // window_size[0], window_size[0], W // window_size[1], window_size[1], C)
  58. windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size[0], window_size[1], C)
  59. return windows
  62. @register_notrace_function # reason: int argument is a Proxy
  63. def window_reverse(windows, window_size: Tuple[int, int], H: int, W: int):
  64. """
  65. Args:
  66. windows: (num_windows*B, window_size, window_size, C)
  67. window_size (int): Window size
  68. H (int): Height of image
  69. W (int): Width of image
  71. Returns:
  72. x: (B, H, W, C)
  73. """
  74. C = windows.shape[-1]
  75. x = windows.view(-1, H // window_size[0], W // window_size[1], window_size[0], window_size[1], C)
  76. x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, H, W, C)
  77. return x
  80. def get_relative_position_index(win_h: int, win_w: int):
  81. # get pair-wise relative position index for each token inside the window
  82. coords = torch.stack(ndgrid(torch.arange(win_h), torch.arange(win_w))) # 2, Wh, Ww
  83. coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
  84. relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
  85. relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
  86. relative_coords[:, :, 0] += win_h - 1 # shift to start from 0
  87. relative_coords[:, :, 1] += win_w - 1
  88. relative_coords[:, :, 0] *= 2 * win_w - 1
  89. return relative_coords.sum(-1) # Wh*Ww, Wh*Ww
  92. class WindowAttention(nn.Module):
  93. """ Window based multi-head self attention (W-MSA) module with relative position bias.
  94. It supports shifted and non-shifted windows.
  95. """
  96. fused_attn: torch.jit.Final[bool]
  98. def __init__(
  99. self,
  100. dim: int,
  101. num_heads: int,
  102. head_dim: Optional[int] = None,
  103. window_size: _int_or_tuple_2_t = 7,
  104. qkv_bias: bool = True,
  105. attn_drop: float = 0.,
  106. proj_drop: float = 0.,
  107. ):
  108. """
  109. Args:
  110. dim: Number of input channels.
  111. num_heads: Number of attention heads.
  112. head_dim: Number of channels per head (dim // num_heads if not set)
  113. window_size: The height and width of the window.
  114. qkv_bias: If True, add a learnable bias to query, key, value.
  115. attn_drop: Dropout ratio of attention weight.
  116. proj_drop: Dropout ratio of output.
  117. """
  118. super().__init__()
  119. self.dim = dim
  120. self.window_size = to_2tuple(window_size) # Wh, Ww
  121. win_h, win_w = self.window_size
  122. self.window_area = win_h * win_w
  123. self.num_heads = num_heads
  124. head_dim = head_dim or dim // num_heads
  125. attn_dim = head_dim * num_heads
  126. self.scale = head_dim ** -0.5
  127. self.fused_attn = use_fused_attn(experimental=True) # NOTE not tested for prime-time yet
  129. # define a parameter table of relative position bias, shape: 2*Wh-1 * 2*Ww-1, nH
  130. self.relative_position_bias_table = nn.Parameter(torch.zeros((2 * win_h - 1) * (2 * win_w - 1), num_heads))
  132. # get pair-wise relative position index for each token inside the window
  133. self.register_buffer("relative_position_index", get_relative_position_index(win_h, win_w), persistent=False)
  135. self.qkv = nn.Linear(dim, attn_dim * 3, bias=qkv_bias)
  136. self.attn_drop = nn.Dropout(attn_drop)
  137. self.proj = nn.Linear(attn_dim, dim)
  138. self.proj_drop = nn.Dropout(proj_drop)
  140. trunc_normal_(self.relative_position_bias_table, std=.02)
  141. self.softmax = nn.Softmax(dim=-1)
  143. # Identity layers for pytorch hook
  144. self.forward_hook_before_softmax = nn.Identity()
  145. self.backward_hook_after_softmax = nn.Identity()
  148. def set_window_size(self, window_size: Tuple[int, int]) -> None:
  149. """Update window size & interpolate position embeddings
  150. Args:
  151. window_size (int): New window size
  152. """
  153. window_size = to_2tuple(window_size)
  154. if window_size == self.window_size:
  155. return
  156. self.window_size = window_size
  157. win_h, win_w = self.window_size
  158. self.window_area = win_h * win_w
  159. with torch.no_grad():
  160. new_bias_shape = (2 * win_h - 1) * (2 * win_w - 1), self.num_heads
  161. self.relative_position_bias_table = nn.Parameter(
  162. resize_rel_pos_bias_table(
  163. self.relative_position_bias_table,
  164. new_window_size=self.window_size,
  165. new_bias_shape=new_bias_shape,
  166. ))
  167. self.register_buffer("relative_position_index", get_relative_position_index(win_h, win_w), persistent=False)
  169. def _get_rel_pos_bias(self) -> torch.Tensor:
  170. relative_position_bias = self.relative_position_bias_table[
  171. self.relative_position_index.view(-1)].view(self.window_area, self.window_area, -1) # Wh*Ww,Wh*Ww,nH
  172. relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
  173. return relative_position_bias.unsqueeze(0)
  175. def forward(self, x, mask: Optional[torch.Tensor] = None):
  176. """
  177. Args:
  178. x: input features with shape of (num_windows*B, N, C)
  179. mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None
  180. """
  181. B_, N, C = x.shape
  182. qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
  183. q, k, v = qkv.unbind(0)
  185. if self.fused_attn:
  186. attn_mask = self._get_rel_pos_bias()
  187. if mask is not None:
  188. num_win = mask.shape[0]
  189. mask = mask.view(1, num_win, 1, N, N).expand(B_ // num_win, -1, self.num_heads, -1, -1)
  190. attn_mask = attn_mask + mask.reshape(-1, self.num_heads, N, N)
  191. x = torch.nn.functional.scaled_dot_product_attention(
  192. q, k, v,
  193. attn_mask=attn_mask,
  194. dropout_p=self.attn_drop.p if self.training else 0.,
  195. )
  196. else:
  197. q = q * self.scale
  198. attn = q @ k.transpose(-2, -1)
  199. attn = self.forward_hook_before_softmax(attn)
  200. attn = self.backward_hook_after_softmax(attn) # 왜 옮겼더니 된거지?
  201. attn = attn + self._get_rel_pos_bias()
  202. if mask is not None:
  203. num_win = mask.shape[0]
  204. attn = attn.view(-1, num_win, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
  205. attn = attn.view(-1, self.num_heads, N, N)
  206. #forward hook
  207. attn = self.softmax(attn)
  208. attn = self.attn_drop(attn)
  209. #backward hook
  211. x = attn @ v
  213. x = x.transpose(1, 2).reshape(B_, N, -1)
  214. x = self.proj(x)
  215. x = self.proj_drop(x)
  216. return x
  219. class SwinTransformerBlock(nn.Module):
  220. """ Swin Transformer Block.
  221. """
  223. def __init__(
  224. self,
  225. dim: int,
  226. input_resolution: _int_or_tuple_2_t,
  227. num_heads: int = 4,
  228. head_dim: Optional[int] = None,
  229. window_size: _int_or_tuple_2_t = 7,
  230. shift_size: int = 0,
  231. always_partition: bool = False,
  232. dynamic_mask: bool = False,
  233. mlp_ratio: float = 4.,
  234. qkv_bias: bool = True,
  235. proj_drop: float = 0.,
  236. attn_drop: float = 0.,
  237. drop_path: float = 0.,
  238. act_layer: Callable = nn.GELU,
  239. norm_layer: Callable = nn.LayerNorm,
  240. ):
  241. """
  242. Args:
  243. dim: Number of input channels.
  244. input_resolution: Input resolution.
  245. window_size: Window size.
  246. num_heads: Number of attention heads.
  247. head_dim: Enforce the number of channels per head
  248. shift_size: Shift size for SW-MSA.
  249. always_partition: Always partition into full windows and shift
  250. mlp_ratio: Ratio of mlp hidden dim to embedding dim.
  251. qkv_bias: If True, add a learnable bias to query, key, value.
  252. proj_drop: Dropout rate.
  253. attn_drop: Attention dropout rate.
  254. drop_path: Stochastic depth rate.
  255. act_layer: Activation layer.
  256. norm_layer: Normalization layer.
  257. """
  258. super().__init__()
  259. self.dim = dim
  260. self.input_resolution = input_resolution
  261. self.target_shift_size = to_2tuple(shift_size) # store for later resize
  262. self.always_partition = always_partition
  263. self.dynamic_mask = dynamic_mask
  264. self.window_size, self.shift_size = self._calc_window_shift(window_size, shift_size)
  265. self.window_area = self.window_size[0] * self.window_size[1]
  266. self.mlp_ratio = mlp_ratio
  268. self.norm1 = norm_layer(dim)
  269. self.attn = WindowAttention(
  270. dim,
  271. num_heads=num_heads,
  272. head_dim=head_dim,
  273. window_size=self.window_size,
  274. qkv_bias=qkv_bias,
  275. attn_drop=attn_drop,
  276. proj_drop=proj_drop,
  277. )
  278. self.drop_path1 = DropPath(drop_path) if drop_path > 0. else nn.Identity()
  280. self.norm2 = norm_layer(dim)
  281. self.mlp = Mlp(
  282. in_features=dim,
  283. hidden_features=int(dim * mlp_ratio),
  284. act_layer=act_layer,
  285. drop=proj_drop,
  286. )
  287. self.drop_path2 = DropPath(drop_path) if drop_path > 0. else nn.Identity()
  289. self.register_buffer(
  290. "attn_mask",
  291. None if self.dynamic_mask else self.get_attn_mask(),
  292. persistent=False,
  293. )
  295. self.backward_hook_before_mlp = nn.Identity()
  297. def get_attn_mask(self, x: Optional[torch.Tensor] = None) -> Optional[torch.Tensor]:
  298. if any(self.shift_size):
  299. # calculate attention mask for SW-MSA
  300. if x is not None:
  301. H, W = x.shape[1], x.shape[2]
  302. device = x.device
  303. dtype = x.dtype
  304. else:
  305. H, W = self.input_resolution
  306. device = None
  307. dtype = None
  308. H = math.ceil(H / self.window_size[0]) * self.window_size[0]
  309. W = math.ceil(W / self.window_size[1]) * self.window_size[1]
  310. img_mask = torch.zeros((1, H, W, 1), dtype=dtype, device=device) # 1 H W 1
  311. cnt = 0
  312. for h in (
  313. (0, -self.window_size[0]),
  314. (-self.window_size[0], -self.shift_size[0]),
  315. (-self.shift_size[0], None),
  316. ):
  317. for w in (
  318. (0, -self.window_size[1]),
  319. (-self.window_size[1], -self.shift_size[1]),
  320. (-self.shift_size[1], None),
  321. ):
  322. img_mask[:, h[0]:h[1], w[0]:w[1], :] = cnt
  323. cnt += 1
  324. mask_windows = window_partition(img_mask, self.window_size) # nW, window_size, window_size, 1
  325. mask_windows = mask_windows.view(-1, self.window_area)
  326. attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
  327. attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
  328. else:
  329. attn_mask = None
  331. return attn_mask
  333. def _calc_window_shift(
  334. self,
  335. target_window_size: Union[int, Tuple[int, int]],
  336. target_shift_size: Optional[Union[int, Tuple[int, int]]] = None,
  337. ) -> Tuple[Tuple[int, int], Tuple[int, int]]:
  338. target_window_size = to_2tuple(target_window_size)
  339. if target_shift_size is None:
  340. # if passed value is None, recalculate from default window_size // 2 if it was previously non-zero
  341. target_shift_size = self.target_shift_size
  342. if any(target_shift_size):
  343. target_shift_size = (target_window_size[0] // 2, target_window_size[1] // 2)
  344. else:
  345. target_shift_size = to_2tuple(target_shift_size)
  347. if self.always_partition:
  348. return target_window_size, target_shift_size
  350. window_size = [r if r <= w else w for r, w in zip(self.input_resolution, target_window_size)]
  351. shift_size = [0 if r <= w else s for r, w, s in zip(self.input_resolution, window_size, target_shift_size)]
  352. return tuple(window_size), tuple(shift_size)
  354. def set_input_size(
  355. self,
  356. feat_size: Tuple[int, int],
  357. window_size: Tuple[int, int],
  358. always_partition: Optional[bool] = None,
  359. ):
  360. """
  361. Args:
  362. feat_size: New input resolution
  363. window_size: New window size
  364. always_partition: Change always_partition attribute if not None
  365. """
  366. self.input_resolution = feat_size
  367. if always_partition is not None:
  368. self.always_partition = always_partition
  369. self.window_size, self.shift_size = self._calc_window_shift(window_size)
  370. self.window_area = self.window_size[0] * self.window_size[1]
  371. self.attn.set_window_size(self.window_size)
  372. self.register_buffer(
  373. "attn_mask",
  374. None if self.dynamic_mask else self.get_attn_mask(),
  375. persistent=False,
  376. )
  378. def _attn(self, x):
  379. B, H, W, C = x.shape
  381. # cyclic shift
  382. has_shift = any(self.shift_size)
  383. if has_shift:
  384. shifted_x = torch.roll(x, shifts=(-self.shift_size[0], -self.shift_size[1]), dims=(1, 2))
  385. else:
  386. shifted_x = x
  388. # pad for resolution not divisible by window size
  389. pad_h = (self.window_size[0] - H % self.window_size[0]) % self.window_size[0]
  390. pad_w = (self.window_size[1] - W % self.window_size[1]) % self.window_size[1]
  391. shifted_x = torch.nn.functional.pad(shifted_x, (0, 0, 0, pad_w, 0, pad_h))
  392. _, Hp, Wp, _ = shifted_x.shape
  394. # partition windows
  395. x_windows = window_partition(shifted_x, self.window_size) # nW*B, window_size, window_size, C
  396. x_windows = x_windows.view(-1, self.window_area, C) # nW*B, window_size*window_size, C
  398. # W-MSA/SW-MSA
  399. if getattr(self, 'dynamic_mask', False):
  400. attn_mask = self.get_attn_mask(shifted_x)
  401. else:
  402. attn_mask = self.attn_mask
  403. attn_windows = self.attn(x_windows, mask=attn_mask) # nW*B, window_size*window_size, C
  405. # merge windows
  406. attn_windows = attn_windows.view(-1, self.window_size[0], self.window_size[1], C)
  407. shifted_x = window_reverse(attn_windows, self.window_size, Hp, Wp) # B H' W' C
  408. shifted_x = shifted_x[:, :H, :W, :].contiguous()
  410. # reverse cyclic shift
  411. if has_shift:
  413. x = torch.roll(shifted_x, shifts=self.shift_size, dims=(1, 2))
  414. else:
  415. x = shifted_x
  417. return x
  419. def forward(self, x):
  420. B, H, W, C = x.shape
  421. x = x + self.drop_path1(self._attn(self.norm1(x)))
  422. x = x.reshape(B, -1, C)
  423. x = x + self.drop_path2(self.mlp(self.norm2(x)))
  424. x = self.backward_hook_before_mlp(x)
  425. x = x.reshape(B, H, W, C)
  426. return x
  429. class PatchMerging(nn.Module):
  430. """ Patch Merging Layer.
  431. """
  433. def __init__(
  434. self,
  435. dim: int,
  436. out_dim: Optional[int] = None,
  437. norm_layer: Callable = nn.LayerNorm,
  438. ):
  439. """
  440. Args:
  441. dim: Number of input channels.
  442. out_dim: Number of output channels (or 2 * dim if None)
  443. norm_layer: Normalization layer.
  444. """
  445. super().__init__()
  446. self.dim = dim
  447. self.out_dim = out_dim or 2 * dim
  448. self.norm = norm_layer(4 * dim)
  449. self.reduction = nn.Linear(4 * dim, self.out_dim, bias=False)
  451. def forward(self, x):
  452. B, H, W, C = x.shape
  454. pad_values = (0, 0, 0, W % 2, 0, H % 2)
  455. x = nn.functional.pad(x, pad_values)
  456. _, H, W, _ = x.shape
  458. x = x.reshape(B, H // 2, 2, W // 2, 2, C).permute(0, 1, 3, 4, 2, 5).flatten(3)
  459. x = self.norm(x)
  460. x = self.reduction(x)
  461. return x
  464. class SwinTransformerStage(nn.Module):
  465. """ A basic Swin Transformer layer for one stage.
  466. """
  468. def __init__(
  469. self,
  470. dim: int,
  471. out_dim: int,
  472. input_resolution: Tuple[int, int],
  473. depth: int,
  474. downsample: bool = True,
  475. num_heads: int = 4,
  476. head_dim: Optional[int] = None,
  477. window_size: _int_or_tuple_2_t = 7,
  478. always_partition: bool = False,
  479. dynamic_mask: bool = False,
  480. mlp_ratio: float = 4.,
  481. qkv_bias: bool = True,
  482. proj_drop: float = 0.,
  483. attn_drop: float = 0.,
  484. drop_path: Union[List[float], float] = 0.,
  485. norm_layer: Callable = nn.LayerNorm,
  486. ):
  487. """
  488. Args:
  489. dim: Number of input channels.
  490. out_dim: Number of output channels.
  491. input_resolution: Input resolution.
  492. depth: Number of blocks.
  493. downsample: Downsample layer at the end of the layer.
  494. num_heads: Number of attention heads.
  495. head_dim: Channels per head (dim // num_heads if not set)
  496. window_size: Local window size.
  497. mlp_ratio: Ratio of mlp hidden dim to embedding dim.
  498. qkv_bias: If True, add a learnable bias to query, key, value.
  499. proj_drop: Projection dropout rate.
  500. attn_drop: Attention dropout rate.
  501. drop_path: Stochastic depth rate.
  502. norm_layer: Normalization layer.
  503. """
  504. super().__init__()
  505. self.dim = dim
  506. self.input_resolution = input_resolution
  507. self.output_resolution = tuple(i // 2 for i in input_resolution) if downsample else input_resolution
  508. self.depth = depth
  509. self.grad_checkpointing = False
  510. window_size = to_2tuple(window_size)
  511. shift_size = tuple([w // 2 for w in window_size])
  513. # patch merging layer
  514. if downsample:
  515. self.downsample = PatchMerging(
  516. dim=dim,
  517. out_dim=out_dim,
  518. norm_layer=norm_layer,
  519. )
  520. else:
  521. assert dim == out_dim
  522. self.downsample = nn.Identity()
  524. # build blocks
  525. self.blocks = nn.Sequential(*[
  526. SwinTransformerBlock(
  527. dim=out_dim,
  528. input_resolution=self.output_resolution,
  529. num_heads=num_heads,
  530. head_dim=head_dim,
  531. window_size=window_size,
  532. shift_size=0 if (i % 2 == 0) else shift_size,
  533. always_partition=always_partition,
  534. dynamic_mask=dynamic_mask,
  535. mlp_ratio=mlp_ratio,
  536. qkv_bias=qkv_bias,
  537. proj_drop=proj_drop,
  538. attn_drop=attn_drop,
  539. drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path,
  540. norm_layer=norm_layer,
  541. )
  542. for i in range(depth)])
  544. def set_input_size(
  545. self,
  546. feat_size: Tuple[int, int],
  547. window_size: int,
  548. always_partition: Optional[bool] = None,
  549. ):
  550. """ Updates the resolution, window size and so the pair-wise relative positions.
  552. Args:
  553. feat_size: New input (feature) resolution
  554. window_size: New window size
  555. always_partition: Always partition / shift the window
  556. """
  557. self.input_resolution = feat_size
  558. if isinstance(self.downsample, nn.Identity):
  559. self.output_resolution = feat_size
  560. else:
  561. self.output_resolution = tuple(i // 2 for i in feat_size)
  562. for block in self.blocks:
  563. block.set_input_size(
  564. feat_size=self.output_resolution,
  565. window_size=window_size,
  566. always_partition=always_partition,
  567. )
  569. def forward(self, x):
  570. x = self.downsample(x)
  572. if self.grad_checkpointing and not torch.jit.is_scripting():
  573. x = checkpoint_seq(self.blocks, x)
  574. else:
  575. x = self.blocks(x)
  576. return x
  579. class SwinTransformer(nn.Module):
  580. """ Swin Transformer
  582. A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` -
  583. https://arxiv.org/pdf/2103.14030
  584. """
  586. def __init__(
  587. self,
  588. img_size: _int_or_tuple_2_t = 224,
  589. patch_size: int = 4,
  590. in_chans: int = 3,
  591. num_classes: int = 1000,
  592. global_pool: str = 'avg',
  593. embed_dim: int = 96,
  594. depths: Tuple[int, ...] = (2, 2, 6, 2),
  595. num_heads: Tuple[int, ...] = (3, 6, 12, 24),
  596. head_dim: Optional[int] = None,
  597. window_size: _int_or_tuple_2_t = 7,
  598. always_partition: bool = False,
  599. strict_img_size: bool = True,
  600. mlp_ratio: float = 4.,
  601. qkv_bias: bool = True,
  602. drop_rate: float = 0.,
  603. proj_drop_rate: float = 0.,
  604. attn_drop_rate: float = 0.,
  605. drop_path_rate: float = 0.1,
  606. embed_layer: Callable = PatchEmbed,
  607. norm_layer: Union[str, Callable] = nn.LayerNorm,
  608. weight_init: str = '',
  609. **kwargs,
  610. ):
  611. """
  612. Args:
  613. img_size: Input image size.
  614. patch_size: Patch size.
  615. in_chans: Number of input image channels.
  616. num_classes: Number of classes for classification head.
  617. embed_dim: Patch embedding dimension.
  618. depths: Depth of each Swin Transformer layer.
  619. num_heads: Number of attention heads in different layers.
  620. head_dim: Dimension of self-attention heads.
  621. window_size: Window size.
  622. mlp_ratio: Ratio of mlp hidden dim to embedding dim.
  623. qkv_bias: If True, add a learnable bias to query, key, value.
  624. drop_rate: Dropout rate.
  625. attn_drop_rate (float): Attention dropout rate.
  626. drop_path_rate (float): Stochastic depth rate.
  627. embed_layer: Patch embedding layer.
  628. norm_layer (nn.Module): Normalization layer.
  629. """
  630. super().__init__()
  631. assert global_pool in ('', 'avg')
  632. self.num_classes = num_classes
  633. self.global_pool = global_pool
  634. self.output_fmt = 'NHWC'
  636. self.num_layers = len(depths)
  637. self.embed_dim = embed_dim
  638. self.num_features = self.head_hidden_size = int(embed_dim * 2 ** (self.num_layers - 1))
  639. self.feature_info = []
  641. if not isinstance(embed_dim, (tuple, list)):
  642. embed_dim = [int(embed_dim * 2 ** i) for i in range(self.num_layers)]
  644. # split image into non-overlapping patches
  645. self.patch_embed = embed_layer(
  646. img_size=img_size,
  647. patch_size=patch_size,
  648. in_chans=in_chans,
  649. embed_dim=embed_dim[0],
  650. norm_layer=norm_layer,
  651. strict_img_size=strict_img_size,
  652. output_fmt='NHWC',
  653. )
  654. patch_grid = self.patch_embed.grid_size
  656. # build layers
  657. head_dim = to_ntuple(self.num_layers)(head_dim)
  658. if not isinstance(window_size, (list, tuple)):
  659. window_size = to_ntuple(self.num_layers)(window_size)
  660. elif len(window_size) == 2:
  661. window_size = (window_size,) * self.num_layers
  662. assert len(window_size) == self.num_layers
  663. mlp_ratio = to_ntuple(self.num_layers)(mlp_ratio)
  664. dpr = [x.tolist() for x in torch.linspace(0, drop_path_rate, sum(depths)).split(depths)]
  665. layers = []
  666. in_dim = embed_dim[0]
  667. scale = 1
  668. for i in range(self.num_layers):
  669. out_dim = embed_dim[i]
  670. layers += [SwinTransformerStage(
  671. dim=in_dim,
  672. out_dim=out_dim,
  673. input_resolution=(
  674. patch_grid[0] // scale,
  675. patch_grid[1] // scale
  676. ),
  677. depth=depths[i],
  678. downsample=i > 0,
  679. num_heads=num_heads[i],
  680. head_dim=head_dim[i],
  681. window_size=window_size[i],
  682. always_partition=always_partition,
  683. dynamic_mask=not strict_img_size,
  684. mlp_ratio=mlp_ratio[i],
  685. qkv_bias=qkv_bias,
  686. proj_drop=proj_drop_rate,
  687. attn_drop=attn_drop_rate,
  688. drop_path=dpr[i],
  689. norm_layer=norm_layer,
  690. )]
  691. in_dim = out_dim
  692. if i > 0:
  693. scale *= 2
  694. self.feature_info += [dict(num_chs=out_dim, reduction=patch_size * scale, module=f'layers.{i}')]
  695. self.layers = nn.Sequential(*layers)
  697. self.norm = norm_layer(self.num_features)
  698. self.head = ClassifierHead(
  699. self.num_features,
  700. num_classes,
  701. pool_type=global_pool,
  702. drop_rate=drop_rate,
  703. input_fmt=self.output_fmt,
  704. )
  705. if weight_init != 'skip':
  706. self.init_weights(weight_init)
  708. @torch.jit.ignore
  709. def init_weights(self, mode=''):
  710. assert mode in ('jax', 'jax_nlhb', 'moco', '')
  711. head_bias = -math.log(self.num_classes) if 'nlhb' in mode else 0.
  712. named_apply(get_init_weights_vit(mode, head_bias=head_bias), self)
  714. @torch.jit.ignore
  715. def no_weight_decay(self):
  716. nwd = set()
  717. for n, _ in self.named_parameters():
  718. if 'relative_position_bias_table' in n:
  719. nwd.add(n)
  720. return nwd
  722. def set_input_size(
  723. self,
  724. img_size: Optional[Tuple[int, int]] = None,
  725. patch_size: Optional[Tuple[int, int]] = None,
  726. window_size: Optional[Tuple[int, int]] = None,
  727. window_ratio: int = 8,
  728. always_partition: Optional[bool] = None,
  729. ) -> None:
  730. """ Updates the image resolution and window size.
  732. Args:
  733. img_size: New input resolution, if None current resolution is used
  734. patch_size (Optional[Tuple[int, int]): New patch size, if None use current patch size
  735. window_size: New window size, if None based on new_img_size // window_div
  736. window_ratio: divisor for calculating window size from grid size
  737. always_partition: always partition into windows and shift (even if window size < feat size)
  738. """
  739. if img_size is not None or patch_size is not None:
  740. self.patch_embed.set_input_size(img_size=img_size, patch_size=patch_size)
  741. patch_grid = self.patch_embed.grid_size
  743. if window_size is None:
  744. window_size = tuple([pg // window_ratio for pg in patch_grid])
  746. for index, stage in enumerate(self.layers):
  747. stage_scale = 2 ** max(index - 1, 0)
  748. stage.set_input_size(
  749. feat_size=(patch_grid[0] // stage_scale, patch_grid[1] // stage_scale),
  750. window_size=window_size,
  751. always_partition=always_partition,
  752. )
  754. @torch.jit.ignore
  755. def group_matcher(self, coarse=False):
  756. return dict(
  757. stem=r'^patch_embed', # stem and embed
  758. blocks=r'^layers\.(\d+)' if coarse else [
  759. (r'^layers\.(\d+).downsample', (0,)),
  760. (r'^layers\.(\d+)\.\w+\.(\d+)', None),
  761. (r'^norm', (99999,)),
  762. ]
  763. )
  765. @torch.jit.ignore
  766. def set_grad_checkpointing(self, enable=True):
  767. for l in self.layers:
  768. l.grad_checkpointing = enable
  770. @torch.jit.ignore
  771. def get_classifier(self) -> nn.Module:
  772. return self.head.fc
  774. def reset_classifier(self, num_classes: int, global_pool: Optional[str] = None):
  775. self.num_classes = num_classes
  776. self.head.reset(num_classes, pool_type=global_pool)
  778. def forward_intermediates(
  779. self,
  780. x: torch.Tensor,
  781. indices: Optional[Union[int, List[int]]] = None,
  782. norm: bool = False,
  783. stop_early: bool = False,
  784. output_fmt: str = 'NCHW',
  785. intermediates_only: bool = False,
  786. ) -> Union[List[torch.Tensor], Tuple[torch.Tensor, List[torch.Tensor]]]:
  787. """ Forward features that returns intermediates.
  789. Args:
  790. x: Input image tensor
  791. indices: Take last n blocks if int, all if None, select matching indices if sequence
  792. norm: Apply norm layer to compatible intermediates
  793. stop_early: Stop iterating over blocks when last desired intermediate hit
  794. output_fmt: Shape of intermediate feature outputs
  795. intermediates_only: Only return intermediate features
  796. Returns:
  798. """
  799. assert output_fmt in ('NCHW',), 'Output shape must be NCHW.'
  800. intermediates = []
  801. take_indices, max_index = feature_take_indices(len(self.layers), indices)
  803. # forward pass
  804. x = self.patch_embed(x)
  806. num_stages = len(self.layers)
  807. if torch.jit.is_scripting() or not stop_early: # can't slice blocks in torchscript
  808. stages = self.layers
  809. else:
  810. stages = self.layers[:max_index + 1]
  811. for i, stage in enumerate(stages):
  812. x = stage(x)
  813. if i in take_indices:
  814. if norm and i == num_stages - 1:
  815. x_inter = self.norm(x) # applying final norm last intermediate
  816. else:
  817. x_inter = x
  818. x_inter = x_inter.permute(0, 3, 1, 2).contiguous()
  819. intermediates.append(x_inter)
  821. if intermediates_only:
  822. return intermediates
  824. x = self.norm(x)
  826. return x, intermediates
  828. def prune_intermediate_layers(
  829. self,
  830. indices: Union[int, List[int]] = 1,
  831. prune_norm: bool = False,
  832. prune_head: bool = True,
  833. ):
  834. """ Prune layers not required for specified intermediates.
  835. """
  836. take_indices, max_index = feature_take_indices(len(self.layers), indices)
  837. self.layers = self.layers[:max_index + 1] # truncate blocks
  838. if prune_norm:
  839. self.norm = nn.Identity()
  840. if prune_head:
  841. self.reset_classifier(0, '')
  842. return take_indices
  844. def forward_features(self, x):
  845. x = self.patch_embed(x)
  846. x = self.layers(x)
  847. x = self.norm(x)
  848. return x
  850. def forward_head(self, x, pre_logits: bool = False):
  851. return self.head(x, pre_logits=True) if pre_logits else self.head(x)
  853. def forward(self, x):
  854. x = self.forward_features(x)
  855. x = self.forward_head(x)
  856. return x
  859. def checkpoint_filter_fn(state_dict, model):
  860. """ convert patch embedding weight from manual patchify + linear proj to conv"""
  861. old_weights = True
  862. if 'head.fc.weight' in state_dict:
  863. old_weights = False
  864. import re
  865. out_dict = {}
  866. state_dict = state_dict.get('model', state_dict)
  867. state_dict = state_dict.get('state_dict', state_dict)
  868. for k, v in state_dict.items():
  869. if any([n in k for n in ('relative_position_index', 'attn_mask')]):
  870. continue # skip buffers that should not be persistent
  872. if 'patch_embed.proj.weight' in k:
  873. _, _, H, W = model.patch_embed.proj.weight.shape
  874. if v.shape[-2] != H or v.shape[-1] != W:
  875. v = resample_patch_embed(
  876. v,
  877. (H, W),
  878. interpolation='bicubic',
  879. antialias=True,
  880. verbose=True,
  881. )
  883. if k.endswith('relative_position_bias_table'):
  884. m = model.get_submodule(k[:-29])
  885. if v.shape != m.relative_position_bias_table.shape or m.window_size[0] != m.window_size[1]:
  886. v = resize_rel_pos_bias_table(
  887. v,
  888. new_window_size=m.window_size,
  889. new_bias_shape=m.relative_position_bias_table.shape,
  890. )
  892. if old_weights:
  893. k = re.sub(r'layers.(\d+).downsample', lambda x: f'layers.{int(x.group(1)) + 1}.downsample', k)
  894. k = k.replace('head.', 'head.fc.')
  896. out_dict[k] = v
  897. return out_dict
  900. def _create_swin_transformer(variant, pretrained=False, **kwargs):
  901. default_out_indices = tuple(i for i, _ in enumerate(kwargs.get('depths', (1, 1, 3, 1))))
  902. out_indices = kwargs.pop('out_indices', default_out_indices)
  904. model = build_model_with_cfg(
  905. SwinTransformer, variant, pretrained,
  906. pretrained_filter_fn=checkpoint_filter_fn,
  907. feature_cfg=dict(flatten_sequential=True, out_indices=out_indices),
  908. **kwargs)
  910. return model
  913. def _cfg(url='', **kwargs):
  914. return {
  915. 'url': url,
  916. 'num_classes': 1000, 'input_size': (3, 224, 224), 'pool_size': (7, 7),
  917. 'crop_pct': .9, 'interpolation': 'bicubic', 'fixed_input_size': True,
  918. 'mean': IMAGENET_DEFAULT_MEAN, 'std': IMAGENET_DEFAULT_STD,
  919. 'first_conv': 'patch_embed.proj', 'classifier': 'head.fc',
  920. 'license': 'mit', **kwargs
  921. }
  924. default_cfgs = generate_default_cfgs({
  925. 'swin_small_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
  926. hf_hub_id='timm/',
  927. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth', ),
  928. 'swin_base_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
  929. hf_hub_id='timm/',
  930. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth',),
  931. 'swin_base_patch4_window12_384.ms_in22k_ft_in1k': _cfg(
  932. hf_hub_id='timm/',
  933. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth',
  934. input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
  935. 'swin_large_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
  936. hf_hub_id='timm/',
  937. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pth',),
  938. 'swin_large_patch4_window12_384.ms_in22k_ft_in1k': _cfg(
  939. hf_hub_id='timm/',
  940. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pth',
  941. input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
  943. 'swin_tiny_patch4_window7_224.ms_in1k': _cfg(
  944. hf_hub_id='timm/',
  945. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth',),
  946. 'swin_small_patch4_window7_224.ms_in1k': _cfg(
  947. hf_hub_id='timm/',
  948. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth',),
  949. 'swin_base_patch4_window7_224.ms_in1k': _cfg(
  950. hf_hub_id='timm/',
  951. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth',),
  952. 'swin_base_patch4_window12_384.ms_in1k': _cfg(
  953. hf_hub_id='timm/',
  954. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pth',
  955. input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
  957. # tiny 22k pretrain is worse than 1k, so moved after (untagged priority is based on order)
  958. 'swin_tiny_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
  959. hf_hub_id='timm/',
  960. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pth',),
  962. 'swin_tiny_patch4_window7_224.ms_in22k': _cfg(
  963. hf_hub_id='timm/',
  964. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth',
  965. num_classes=21841),
  966. 'swin_small_patch4_window7_224.ms_in22k': _cfg(
  967. hf_hub_id='timm/',
  968. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pth',
  969. num_classes=21841),
  970. 'swin_base_patch4_window7_224.ms_in22k': _cfg(
  971. hf_hub_id='timm/',
  972. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth',
  973. num_classes=21841),
  974. 'swin_base_patch4_window12_384.ms_in22k': _cfg(
  975. hf_hub_id='timm/',
  976. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth',
  977. input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0, num_classes=21841),
  978. 'swin_large_patch4_window7_224.ms_in22k': _cfg(
  979. hf_hub_id='timm/',
  980. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth',
  981. num_classes=21841),
  982. 'swin_large_patch4_window12_384.ms_in22k': _cfg(
  983. hf_hub_id='timm/',
  984. url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pth',
  985. input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0, num_classes=21841),
  987. 'swin_s3_tiny_224.ms_in1k': _cfg(
  988. hf_hub_id='timm/',
  989. url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_t-1d53f6a8.pth'),
  990. 'swin_s3_small_224.ms_in1k': _cfg(
  991. hf_hub_id='timm/',
  992. url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_s-3bb4c69d.pth'),
  993. 'swin_s3_base_224.ms_in1k': _cfg(
  994. hf_hub_id='timm/',
  995. url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_b-a1e95db4.pth'),
  996. })
  999. @register_model
  1000. def swin_tiny_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
  1001. """ Swin-T @ 224x224, trained ImageNet-1k
  1002. """
  1003. model_args = dict(patch_size=4, window_size=7, embed_dim=96, depths=(2, 2, 6, 2), num_heads=(3, 6, 12, 24))
  1004. return _create_swin_transformer(
  1005. 'swin_tiny_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1008. @register_model
  1009. def swin_small_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
  1010. """ Swin-S @ 224x224
  1011. """
  1012. model_args = dict(patch_size=4, window_size=7, embed_dim=96, depths=(2, 2, 18, 2), num_heads=(3, 6, 12, 24))
  1013. return _create_swin_transformer(
  1014. 'swin_small_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1017. @register_model
  1018. def swin_base_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
  1019. """ Swin-B @ 224x224
  1020. """
  1021. model_args = dict(patch_size=4, window_size=7, embed_dim=128, depths=(2, 2, 18, 2), num_heads=(4, 8, 16, 32))
  1022. return _create_swin_transformer(
  1023. 'swin_base_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1026. @register_model
  1027. def swin_base_patch4_window12_384(pretrained=False, **kwargs) -> SwinTransformer:
  1028. """ Swin-B @ 384x384
  1029. """
  1030. model_args = dict(patch_size=4, window_size=12, embed_dim=128, depths=(2, 2, 18, 2), num_heads=(4, 8, 16, 32))
  1031. return _create_swin_transformer(
  1032. 'swin_base_patch4_window12_384', pretrained=pretrained, **dict(model_args, **kwargs))
  1035. @register_model
  1036. def swin_large_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
  1037. """ Swin-L @ 224x224
  1038. """
  1039. model_args = dict(patch_size=4, window_size=7, embed_dim=192, depths=(2, 2, 18, 2), num_heads=(6, 12, 24, 48))
  1040. return _create_swin_transformer(
  1041. 'swin_large_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1044. @register_model
  1045. def swin_large_patch4_window12_384(pretrained=False, **kwargs) -> SwinTransformer:
  1046. """ Swin-L @ 384x384
  1047. """
  1048. model_args = dict(patch_size=4, window_size=12, embed_dim=192, depths=(2, 2, 18, 2), num_heads=(6, 12, 24, 48))
  1049. return _create_swin_transformer(
  1050. 'swin_large_patch4_window12_384', pretrained=pretrained, **dict(model_args, **kwargs))
  1053. @register_model
  1054. def swin_s3_tiny_224(pretrained=False, **kwargs) -> SwinTransformer:
  1055. """ Swin-S3-T @ 224x224, https://arxiv.org/abs/2111.14725
  1056. """
  1057. model_args = dict(
  1058. patch_size=4, window_size=(7, 7, 14, 7), embed_dim=96, depths=(2, 2, 6, 2), num_heads=(3, 6, 12, 24))
  1059. return _create_swin_transformer('swin_s3_tiny_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1062. @register_model
  1063. def swin_s3_small_224(pretrained=False, **kwargs) -> SwinTransformer:
  1064. """ Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725
  1065. """
  1066. model_args = dict(
  1067. patch_size=4, window_size=(14, 14, 14, 7), embed_dim=96, depths=(2, 2, 18, 2), num_heads=(3, 6, 12, 24))
  1068. return _create_swin_transformer('swin_s3_small_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1071. @register_model
  1072. def swin_s3_base_224(pretrained=False, **kwargs) -> SwinTransformer:
  1073. """ Swin-S3-B @ 224x224, https://arxiv.org/abs/2111.14725
  1074. """
  1075. model_args = dict(
  1076. patch_size=4, window_size=(7, 7, 14, 7), embed_dim=96, depths=(2, 2, 30, 2), num_heads=(3, 6, 12, 24))
  1077. return _create_swin_transformer('swin_s3_base_224', pretrained=pretrained, **dict(model_args, **kwargs))
  1080. register_model_deprecations(__name__, {
  1081. 'swin_base_patch4_window7_224_in22k': 'swin_base_patch4_window7_224.ms_in22k',
  1082. 'swin_base_patch4_window12_384_in22k': 'swin_base_patch4_window12_384.ms_in22k',
  1083. 'swin_large_patch4_window7_224_in22k': 'swin_large_patch4_window7_224.ms_in22k',
  1084. 'swin_large_patch4_window12_384_in22k': 'swin_large_patch4_window12_384.ms_in22k',
  1085. })

swin_transformer.py at commit 9cfc3f0, no license · at the source

Overview

Authors: Yoon Hwan Byun1,2, Hyunseok Seo3, Jae-Kyung Won4, Boram Lee5, Duk Hyun Hong6, Sun Mo Nam7, Jong Ha Hwang7, Min-Sung Kim7, Yong-Hwy Kim7, Jang Hun Kim6, Mi Ok Yu6, Kyung-Jae Park6, HoJoon Kim3, Sunit Das8, Doo-Sik Kong9, Chul-Kee Park7, Shin-Hyuk Kang6
  1. Department of Neurosurgery, SMG- SNU Boramae Medical Center,Seoul, Republic of Korea
  2. Department of Neurosurgery, Seoul National University College of Medicine,Seoul, Republic of Korea
  3. Department of Artificial Intelligence, Korea University College of Informatics,Seoul, Republic of Korea
  4. Department of Pathology, Seoul National University Hospital, Seoul National University College of Medicine,Seoul, Republic of Korea
  5. Department of Pathology, Samsung Medical Center, Sungkyunkwan University School of Medicine,Seoul, Republic of Korea
  6. Department of Neurosurgery, Korea University Anam Hospital, Korea University College of Medicine,Seoul, Republic of Korea
  7. Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine,Seoul, Republic of Korea
  8. Division of Neurosurgery, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, University of Toronto,Toronto, Canada
  9. Department of Neurosurgery, Samsung Medical Center, Sungkyunkwan University School of Medicine,Seoul, Republic of Korea
Institutions: SNUH SMG-SNU Boramae Medical Center (South Korea); Seoul National University (South Korea); Korea University (South Korea); Seoul National University Hospital (South Korea); Sungkyunkwan University (South Korea); Korea University Anam Hospital (South Korea); University of Toronto (Canada)
Journal: NPJ digital medicine, volume 9, issue 1, article 471
Dates: received 14 December 2025; accepted 10 April 2026; published online 18 April 2026
Type: Research article · Language: English
License: CC BY-NC-ND
Identifiers: DOI 10.1038/s41746-026-02651-0 · PMID 42000880 · PMCID PMC13276397 · OpenAlex W7154848825
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: histology / microscopy (modality), other condition (population), clinical / translational (subfield)
Methods: Statistics, Machine learning
Keywords: Cancer, Computational biology and bioinformatics, Medical research, Oncology
Topic: Glioma Diagnosis and Treatment (Genetics, Medicine), according to OpenAlex
Funding: Ministry of Science and ICT (RS202400338025); Ministry of Health and Welfare (RS-2022-KH129293); Korea University Anam Hospital (O241376); Technology Development Program of the Ministry of SMEs and Startups (RS-2024-00487593)
Citations: not cited yet (Europe PMC); 50 references in the paper
Research resources: RRID:SCR_008567

Abstract

The abstract is not reproduced here: the paper's license (CC BY-NC-ND) does not allow it. Read it in the paper, at the publisher or on Europe PMC.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

ImpelKorea/AI-agumented-CLE-Imaging-npj-Digital-Medicine

License: none: the authors keep all their rights
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 9cfc3f0914b0ad7fae4a9a7ac432c671198b0cc3, 15 March 2026
Languages: Python (251)
Size: 280 files, 251 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, environment (environment.yml)
Not found: license file, CITATION.cff, tests, continuous integration, documentation
Tools: PyTorch (206 files), NumPy (20 files), Pillow (9 files), TensorFlow (2 files), Matplotlib (1 file), pandas (1 file), SciPy (1 file)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
252 files
At the source: github.com/ImpelKorea/AI-agumented-CLE-Imaging-npj-Digital-Medicine

Code availability statement

The paper has a code availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

Read it in the paper: doi.org/10.1038/s41746-026-02651-0.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 251 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability statement

The paper has a data availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

  • it says that the data are available on request

Read it in the paper: doi.org/10.1038/s41746-026-02651-0.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 17 authors, 4 keywords, 4 funders, 42 references, 1 RRID.

Cite

This paper

Byun, Y. H., Seo, H., Won, J.-K., Lee, B., Hong, D. H., Nam, S. M., Hwang, J. H., Kim, M.-S., Kim, Y.-H., Kim, J. H., Yu, M. O., Park, K.-J., Kim, H., Das, S., Kong, D.-S., Park, C.-K., & Kang, S.-H. (2026). AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors. NPJ digital medicine, 9(1), 471. https://doi.org/10.1038/s41746-026-02651-0

BibTeX

@article{byun2026ai,
author = {Byun, Yoon Hwan and Seo, Hyunseok and Won, Jae-Kyung and Lee, Boram and Hong, Duk Hyun and Nam, Sun Mo and Hwang, Jong Ha and Kim, Min-Sung and Kim, Yong-Hwy and Kim, Jang Hun and Yu, Mi Ok and Park, Kyung-Jae and Kim, HoJoon and Das, Sunit and Kong, Doo-Sik and Park, Chul-Kee and Kang, Shin-Hyuk},
title = {{AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors}},
journal = {NPJ digital medicine},
year = {2026},
month = apr,
volume = {9},
number = {1},
pages = {471},
publisher = {Nature Publishing Group},
issn = {2398-6352},
doi = {10.1038/s41746-026-02651-0},
url = {https://doi.org/10.1038/s41746-026-02651-0},
pmid = {42000880},
pmcid = {PMC13276397}
}

RIS

TY - JOUR
AU - Byun, Yoon Hwan
AU - Seo, Hyunseok
AU - Won, Jae-Kyung
AU - Lee, Boram
AU - Hong, Duk Hyun
AU - Nam, Sun Mo
AU - Hwang, Jong Ha
AU - Kim, Min-Sung
AU - Kim, Yong-Hwy
AU - Kim, Jang Hun
AU - Yu, Mi Ok
AU - Park, Kyung-Jae
AU - Kim, HoJoon
AU - Das, Sunit
AU - Kong, Doo-Sik
AU - Park, Chul-Kee
AU - Kang, Shin-Hyuk
TI - AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors
T2 - NPJ digital medicine
J2 - NPJ Digit Med
PY - 2026
DA - 2026/04/18
VL - 9
IS - 1
SP - 471
SN - 2398-6352
PB - Nature Publishing Group
DO - 10.1038/s41746-026-02651-0
UR - https://doi.org/10.1038/s41746-026-02651-0
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41746-026-02651-0",
"type": "article-journal",
"title": "AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors",
"container-title": "NPJ digital medicine",
"author": [
{
"family": "Byun",
"given": "Yoon Hwan"
},
{
"family": "Seo",
"given": "Hyunseok"
},
{
"family": "Won",
"given": "Jae-Kyung"
},
{
"family": "Lee",
"given": "Boram"
},
{
"family": "Hong",
"given": "Duk Hyun"
},
{
"family": "Nam",
"given": "Sun Mo"
},
{
"family": "Hwang",
"given": "Jong Ha"
},
{
"family": "Kim",
"given": "Min-Sung"
},
{
"family": "Kim",
"given": "Yong-Hwy"
},
{
"family": "Kim",
"given": "Jang Hun"
},
{
"family": "Yu",
"given": "Mi Ok"
},
{
"family": "Park",
"given": "Kyung-Jae"
},
{
"family": "Kim",
"given": "HoJoon"
},
{
"family": "Das",
"given": "Sunit"
},
{
"family": "Kong",
"given": "Doo-Sik"
},
{
"family": "Park",
"given": "Chul-Kee"
},
{
"family": "Kang",
"given": "Shin-Hyuk"
}
],
"container-title-short": "NPJ Digit Med",
"volume": "9",
"issue": "1",
"page": "471",
"DOI": "10.1038/s41746-026-02651-0",
"PMID": "42000880",
"PMCID": "PMC13276397",
"ISSN": "2398-6352",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41746-026-02651-0",
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
18
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1016/j.patter.2026.101538 [code]
A multi-modal foundation model for brain disease diagnosis and medical imaging.
Journal: Patterns (New York, N.Y.)
In common: TensorFlow, Pillow, PyTorch, 4 other tools, clinical / translational, 1 reference
[2] doi:10.1093/jnen/nlaf152 [code]
Clinical and pathologic correlations of machine learning quantification of Aβ deposits across 3 brain regions of decedents with Alzheimer disease.
Journal: Journal of neuropathology and experimental neurology
In common: TensorFlow, Pillow, PyTorch, 4 other tools, clinical / translational
[3] doi:10.7554/elife.110588 [code]
Opening the black box toward a modular approach to spike sorting.
Journal: eLife
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[4] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[5] doi:10.3390/s26175501 [code]
Evaluation of a Hybrid Neural-Polynomial Deep Q-Network for Switching-Aware Spectrum Selection in a Controlled Radio-Frequency Measurement-Replay Testbed.
Journal: Sensors (Basel, Switzerland)
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[6] doi:10.1126/sciadv.aed3650 [code]
Truthful visualizations for mass spectrometry imaging enable high-spatial-resolution interactive &lt;i&gt;m/z&lt;/i&gt; mapping and exploration.
Journal: Science advances
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[7] doi:10.1038/s41592-026-03194-8 [code]
Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.
Journal: Nature methods
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[8] doi:10.1371/journal.pcbi.1014571 [code]
SynAPSeg: A novel dataset and image analysis framework for deep learning-based synapse detection and quantification.
Journal: PLoS computational biology
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[9] doi:10.1364/boe.605322 [code]
Generalized plaque digitization framework for multi-dimensional mesoscopic images.
Journal: Biomedical optics express
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[10] doi:10.1523/eneuro.0023-26.2026 [code]
Real-Time Segmentation and Classification of Birdsong Syllables for Learning Experiments.
Journal: eNeuro
In common: TensorFlow, Pillow, PyTorch, 4 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.