AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches

[1] § Methods › Development of AI diagnostic model ↔ gradientCAM/swin_agcam/models/swin_transformer.py, lines 586–706 · score 0.52 · Swin Transformer, patch embedding, dimensions, classifier, model
[2] § Methods › Development of AI diagnostic model ↔ gradientCAM/swin_agcam/models/swin_transformer_v2.py, lines 593–707 · score 0.52 · Swin Transformer, patch embedding, dimensions, classifier, model

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The paper is loaded by your browser, with JavaScript: read it at doi.org or on Europe PMC.

The authors' code

Python · 1,085 lines · 43 KB · no license · 1 match

""" Swin Transformer
A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows`
- https://arxiv.org/pdf/2103.14030
Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below
S3 (AutoFormerV2, https://arxiv.org/abs/2111.14725) Swin weights from
- https://github.com/microsoft/Cream/tree/main/AutoFormerV2
Modifications and additions for timm hacked together by / Copyright 2021, Ross Wightman
"""
# --------------------------------------------------------
# Swin Transformer
# Copyright (c) 2021 Microsoft
# Licensed under The MIT License [see LICENSE for details]
# Written by Ze Liu
# --------------------------------------------------------
import logging
import math
from typing import Callable, List, Optional, Tuple, Union
import torch
import torch.nn as nn
from ..data import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD
from ..layers import PatchEmbed, Mlp, DropPath, ClassifierHead, to_2tuple, to_ntuple, trunc_normal_, \
_assert, use_fused_attn, resize_rel_pos_bias_table, resample_patch_embed, ndgrid
from ._builder import build_model_with_cfg
from ._features import feature_take_indices
from ._features_fx import register_notrace_function
from ._manipulate import checkpoint_seq, named_apply
from ._registry import generate_default_cfgs, register_model, register_model_deprecations
from .vision_transformer import get_init_weights_vit
__all__ = ['SwinTransformer'] # model_registry will add each entrypoint fn to this
_logger = logging.getLogger(__name__)
_int_or_tuple_2_t = Union[int, Tuple[int, int]]
def window_partition(
x: torch.Tensor,
window_size: Tuple[int, int],
) -> torch.Tensor:
"""
Partition into non-overlapping windows with padding if needed.
Args:
x (tensor): input tokens with [B, H, W, C].
window_size (int): window size.
Returns:
windows: windows after partition with [B * num_windows, window_size, window_size, C].
(Hp, Wp): padded height and width before partition
"""
B, H, W, C = x.shape
x = x.view(B, H // window_size[0], window_size[0], W // window_size[1], window_size[1], C)
windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size[0], window_size[1], C)
return windows
@register_notrace_function # reason: int argument is a Proxy
def window_reverse(windows, window_size: Tuple[int, int], H: int, W: int):
"""
Args:
windows: (num_windows*B, window_size, window_size, C)
window_size (int): Window size
H (int): Height of image
W (int): Width of image
Returns:
x: (B, H, W, C)
"""
C = windows.shape[-1]
x = windows.view(-1, H // window_size[0], W // window_size[1], window_size[0], window_size[1], C)
x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, H, W, C)
return x
def get_relative_position_index(win_h: int, win_w: int):
# get pair-wise relative position index for each token inside the window
coords = torch.stack(ndgrid(torch.arange(win_h), torch.arange(win_w))) # 2, Wh, Ww
coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
relative_coords[:, :, 0] += win_h - 1 # shift to start from 0
relative_coords[:, :, 1] += win_w - 1
relative_coords[:, :, 0] *= 2 * win_w - 1
return relative_coords.sum(-1) # Wh*Ww, Wh*Ww
class WindowAttention(nn.Module):
""" Window based multi-head self attention (W-MSA) module with relative position bias.
It supports shifted and non-shifted windows.
"""
fused_attn: torch.jit.Final[bool]
def __init__(
self,
dim: int,
num_heads: int,
head_dim: Optional[int] = None,
window_size: _int_or_tuple_2_t = 7,
qkv_bias: bool = True,
attn_drop: float = 0.,
proj_drop: float = 0.,
):
"""
Args:
dim: Number of input channels.
num_heads: Number of attention heads.
head_dim: Number of channels per head (dim // num_heads if not set)
window_size: The height and width of the window.
qkv_bias: If True, add a learnable bias to query, key, value.
attn_drop: Dropout ratio of attention weight.
proj_drop: Dropout ratio of output.
"""
super().__init__()
self.dim = dim
self.window_size = to_2tuple(window_size) # Wh, Ww
win_h, win_w = self.window_size
self.window_area = win_h * win_w
self.num_heads = num_heads
head_dim = head_dim or dim // num_heads
attn_dim = head_dim * num_heads
self.scale = head_dim ** -0.5
self.fused_attn = use_fused_attn(experimental=True) # NOTE not tested for prime-time yet
# define a parameter table of relative position bias, shape: 2*Wh-1 * 2*Ww-1, nH
self.relative_position_bias_table = nn.Parameter(torch.zeros((2 * win_h - 1) * (2 * win_w - 1), num_heads))
# get pair-wise relative position index for each token inside the window
self.register_buffer("relative_position_index", get_relative_position_index(win_h, win_w), persistent=False)
self.qkv = nn.Linear(dim, attn_dim * 3, bias=qkv_bias)
self.attn_drop = nn.Dropout(attn_drop)
self.proj = nn.Linear(attn_dim, dim)
self.proj_drop = nn.Dropout(proj_drop)
trunc_normal_(self.relative_position_bias_table, std=.02)
self.softmax = nn.Softmax(dim=-1)
# Identity layers for pytorch hook
self.forward_hook_before_softmax = nn.Identity()
self.backward_hook_after_softmax = nn.Identity()
def set_window_size(self, window_size: Tuple[int, int]) -> None:
"""Update window size & interpolate position embeddings
Args:
window_size (int): New window size
"""
window_size = to_2tuple(window_size)
if window_size == self.window_size:
return
self.window_size = window_size
win_h, win_w = self.window_size
self.window_area = win_h * win_w
with torch.no_grad():
new_bias_shape = (2 * win_h - 1) * (2 * win_w - 1), self.num_heads
self.relative_position_bias_table = nn.Parameter(
resize_rel_pos_bias_table(
self.relative_position_bias_table,
new_window_size=self.window_size,
new_bias_shape=new_bias_shape,
))
self.register_buffer("relative_position_index", get_relative_position_index(win_h, win_w), persistent=False)
def _get_rel_pos_bias(self) -> torch.Tensor:
relative_position_bias = self.relative_position_bias_table[
self.relative_position_index.view(-1)].view(self.window_area, self.window_area, -1) # Wh*Ww,Wh*Ww,nH
relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
return relative_position_bias.unsqueeze(0)
def forward(self, x, mask: Optional[torch.Tensor] = None):
"""
Args:
x: input features with shape of (num_windows*B, N, C)
mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None
"""
B_, N, C = x.shape
qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
q, k, v = qkv.unbind(0)
if self.fused_attn:
attn_mask = self._get_rel_pos_bias()
if mask is not None:
num_win = mask.shape[0]
mask = mask.view(1, num_win, 1, N, N).expand(B_ // num_win, -1, self.num_heads, -1, -1)
attn_mask = attn_mask + mask.reshape(-1, self.num_heads, N, N)
x = torch.nn.functional.scaled_dot_product_attention(
q, k, v,
attn_mask=attn_mask,
dropout_p=self.attn_drop.p if self.training else 0.,
)
else:
q = q * self.scale
attn = q @ k.transpose(-2, -1)
attn = self.forward_hook_before_softmax(attn)
attn = self.backward_hook_after_softmax(attn) # 왜 옮겼더니 된거지?
attn = attn + self._get_rel_pos_bias()
if mask is not None:
num_win = mask.shape[0]
attn = attn.view(-1, num_win, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
attn = attn.view(-1, self.num_heads, N, N)
#forward hook
attn = self.softmax(attn)
attn = self.attn_drop(attn)
#backward hook
x = attn @ v
x = x.transpose(1, 2).reshape(B_, N, -1)
x = self.proj(x)
x = self.proj_drop(x)
return x
class SwinTransformerBlock(nn.Module):
""" Swin Transformer Block.
"""
def __init__(
self,
dim: int,
input_resolution: _int_or_tuple_2_t,
num_heads: int = 4,
head_dim: Optional[int] = None,
window_size: _int_or_tuple_2_t = 7,
shift_size: int = 0,
always_partition: bool = False,
dynamic_mask: bool = False,
mlp_ratio: float = 4.,
qkv_bias: bool = True,
proj_drop: float = 0.,
attn_drop: float = 0.,
drop_path: float = 0.,
act_layer: Callable = nn.GELU,
norm_layer: Callable = nn.LayerNorm,
):
"""
Args:
dim: Number of input channels.
input_resolution: Input resolution.
window_size: Window size.
num_heads: Number of attention heads.
head_dim: Enforce the number of channels per head
shift_size: Shift size for SW-MSA.
always_partition: Always partition into full windows and shift
mlp_ratio: Ratio of mlp hidden dim to embedding dim.
qkv_bias: If True, add a learnable bias to query, key, value.
proj_drop: Dropout rate.
attn_drop: Attention dropout rate.
drop_path: Stochastic depth rate.
act_layer: Activation layer.
norm_layer: Normalization layer.
"""
super().__init__()
self.dim = dim
self.input_resolution = input_resolution
self.target_shift_size = to_2tuple(shift_size) # store for later resize
self.always_partition = always_partition
self.dynamic_mask = dynamic_mask
self.window_size, self.shift_size = self._calc_window_shift(window_size, shift_size)
self.window_area = self.window_size[0] * self.window_size[1]
self.mlp_ratio = mlp_ratio
self.norm1 = norm_layer(dim)
self.attn = WindowAttention(
dim,
num_heads=num_heads,
head_dim=head_dim,
window_size=self.window_size,
qkv_bias=qkv_bias,
attn_drop=attn_drop,
proj_drop=proj_drop,
)
self.drop_path1 = DropPath(drop_path) if drop_path > 0. else nn.Identity()
self.norm2 = norm_layer(dim)
self.mlp = Mlp(
in_features=dim,
hidden_features=int(dim * mlp_ratio),
act_layer=act_layer,
drop=proj_drop,
)
self.drop_path2 = DropPath(drop_path) if drop_path > 0. else nn.Identity()
self.register_buffer(
"attn_mask",
None if self.dynamic_mask else self.get_attn_mask(),
persistent=False,
)
self.backward_hook_before_mlp = nn.Identity()
def get_attn_mask(self, x: Optional[torch.Tensor] = None) -> Optional[torch.Tensor]:
if any(self.shift_size):
# calculate attention mask for SW-MSA
if x is not None:
H, W = x.shape[1], x.shape[2]
device = x.device
dtype = x.dtype
else:
H, W = self.input_resolution
device = None
dtype = None
H = math.ceil(H / self.window_size[0]) * self.window_size[0]
W = math.ceil(W / self.window_size[1]) * self.window_size[1]
img_mask = torch.zeros((1, H, W, 1), dtype=dtype, device=device) # 1 H W 1
cnt = 0
for h in (
(0, -self.window_size[0]),
(-self.window_size[0], -self.shift_size[0]),
(-self.shift_size[0], None),
):
for w in (
(0, -self.window_size[1]),
(-self.window_size[1], -self.shift_size[1]),
(-self.shift_size[1], None),
):
img_mask[:, h[0]:h[1], w[0]:w[1], :] = cnt
cnt += 1
mask_windows = window_partition(img_mask, self.window_size) # nW, window_size, window_size, 1
mask_windows = mask_windows.view(-1, self.window_area)
attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
else:
attn_mask = None
return attn_mask
def _calc_window_shift(
self,
target_window_size: Union[int, Tuple[int, int]],
target_shift_size: Optional[Union[int, Tuple[int, int]]] = None,
) -> Tuple[Tuple[int, int], Tuple[int, int]]:
target_window_size = to_2tuple(target_window_size)
if target_shift_size is None:
# if passed value is None, recalculate from default window_size // 2 if it was previously non-zero
target_shift_size = self.target_shift_size
if any(target_shift_size):
target_shift_size = (target_window_size[0] // 2, target_window_size[1] // 2)
else:
target_shift_size = to_2tuple(target_shift_size)
if self.always_partition:
return target_window_size, target_shift_size
window_size = [r if r <= w else w for r, w in zip(self.input_resolution, target_window_size)]
shift_size = [0 if r <= w else s for r, w, s in zip(self.input_resolution, window_size, target_shift_size)]
return tuple(window_size), tuple(shift_size)
def set_input_size(
self,
feat_size: Tuple[int, int],
window_size: Tuple[int, int],
always_partition: Optional[bool] = None,
):
"""
Args:
feat_size: New input resolution
window_size: New window size
always_partition: Change always_partition attribute if not None
"""
self.input_resolution = feat_size
if always_partition is not None:
self.always_partition = always_partition
self.window_size, self.shift_size = self._calc_window_shift(window_size)
self.window_area = self.window_size[0] * self.window_size[1]
self.attn.set_window_size(self.window_size)
self.register_buffer(
"attn_mask",
None if self.dynamic_mask else self.get_attn_mask(),
persistent=False,
)
def _attn(self, x):
B, H, W, C = x.shape
# cyclic shift
has_shift = any(self.shift_size)
if has_shift:
shifted_x = torch.roll(x, shifts=(-self.shift_size[0], -self.shift_size[1]), dims=(1, 2))
else:
shifted_x = x
# pad for resolution not divisible by window size
pad_h = (self.window_size[0] - H % self.window_size[0]) % self.window_size[0]
pad_w = (self.window_size[1] - W % self.window_size[1]) % self.window_size[1]
shifted_x = torch.nn.functional.pad(shifted_x, (0, 0, 0, pad_w, 0, pad_h))
_, Hp, Wp, _ = shifted_x.shape
# partition windows
x_windows = window_partition(shifted_x, self.window_size) # nW*B, window_size, window_size, C
x_windows = x_windows.view(-1, self.window_area, C) # nW*B, window_size*window_size, C
# W-MSA/SW-MSA
if getattr(self, 'dynamic_mask', False):
attn_mask = self.get_attn_mask(shifted_x)
else:
attn_mask = self.attn_mask
attn_windows = self.attn(x_windows, mask=attn_mask) # nW*B, window_size*window_size, C
# merge windows
attn_windows = attn_windows.view(-1, self.window_size[0], self.window_size[1], C)
shifted_x = window_reverse(attn_windows, self.window_size, Hp, Wp) # B H' W' C
shifted_x = shifted_x[:, :H, :W, :].contiguous()
# reverse cyclic shift
if has_shift:
x = torch.roll(shifted_x, shifts=self.shift_size, dims=(1, 2))
else:
x = shifted_x
return x
def forward(self, x):
B, H, W, C = x.shape
x = x + self.drop_path1(self._attn(self.norm1(x)))
x = x.reshape(B, -1, C)
x = x + self.drop_path2(self.mlp(self.norm2(x)))
x = self.backward_hook_before_mlp(x)
x = x.reshape(B, H, W, C)
return x
class PatchMerging(nn.Module):
""" Patch Merging Layer.
"""
def __init__(
self,
dim: int,
out_dim: Optional[int] = None,
norm_layer: Callable = nn.LayerNorm,
):
"""
Args:
dim: Number of input channels.
out_dim: Number of output channels (or 2 * dim if None)
norm_layer: Normalization layer.
"""
super().__init__()
self.dim = dim
self.out_dim = out_dim or 2 * dim
self.norm = norm_layer(4 * dim)
self.reduction = nn.Linear(4 * dim, self.out_dim, bias=False)
def forward(self, x):
B, H, W, C = x.shape
pad_values = (0, 0, 0, W % 2, 0, H % 2)
x = nn.functional.pad(x, pad_values)
_, H, W, _ = x.shape
x = x.reshape(B, H // 2, 2, W // 2, 2, C).permute(0, 1, 3, 4, 2, 5).flatten(3)
x = self.norm(x)
x = self.reduction(x)
return x
class SwinTransformerStage(nn.Module):
""" A basic Swin Transformer layer for one stage.
"""
def __init__(
self,
dim: int,
out_dim: int,
input_resolution: Tuple[int, int],
depth: int,
downsample: bool = True,
num_heads: int = 4,
head_dim: Optional[int] = None,
window_size: _int_or_tuple_2_t = 7,
always_partition: bool = False,
dynamic_mask: bool = False,
mlp_ratio: float = 4.,
qkv_bias: bool = True,
proj_drop: float = 0.,
attn_drop: float = 0.,
drop_path: Union[List[float], float] = 0.,
norm_layer: Callable = nn.LayerNorm,
):
"""
Args:
dim: Number of input channels.
out_dim: Number of output channels.
input_resolution: Input resolution.
depth: Number of blocks.
downsample: Downsample layer at the end of the layer.
num_heads: Number of attention heads.
head_dim: Channels per head (dim // num_heads if not set)
window_size: Local window size.
mlp_ratio: Ratio of mlp hidden dim to embedding dim.
qkv_bias: If True, add a learnable bias to query, key, value.
proj_drop: Projection dropout rate.
attn_drop: Attention dropout rate.
drop_path: Stochastic depth rate.
norm_layer: Normalization layer.
"""
super().__init__()
self.dim = dim
self.input_resolution = input_resolution
self.output_resolution = tuple(i // 2 for i in input_resolution) if downsample else input_resolution
self.depth = depth
self.grad_checkpointing = False
window_size = to_2tuple(window_size)
shift_size = tuple([w // 2 for w in window_size])
# patch merging layer
if downsample:
self.downsample = PatchMerging(
dim=dim,
out_dim=out_dim,
norm_layer=norm_layer,
)
else:
assert dim == out_dim
self.downsample = nn.Identity()
# build blocks
self.blocks = nn.Sequential(*[
SwinTransformerBlock(
dim=out_dim,
input_resolution=self.output_resolution,
num_heads=num_heads,
head_dim=head_dim,
window_size=window_size,
shift_size=0 if (i % 2 == 0) else shift_size,
always_partition=always_partition,
dynamic_mask=dynamic_mask,
mlp_ratio=mlp_ratio,
qkv_bias=qkv_bias,
proj_drop=proj_drop,
attn_drop=attn_drop,
drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path,
norm_layer=norm_layer,
)
for i in range(depth)])
def set_input_size(
self,
feat_size: Tuple[int, int],
window_size: int,
always_partition: Optional[bool] = None,
):
""" Updates the resolution, window size and so the pair-wise relative positions.
Args:
feat_size: New input (feature) resolution
window_size: New window size
always_partition: Always partition / shift the window
"""
self.input_resolution = feat_size
if isinstance(self.downsample, nn.Identity):
self.output_resolution = feat_size
else:
self.output_resolution = tuple(i // 2 for i in feat_size)
for block in self.blocks:
block.set_input_size(
feat_size=self.output_resolution,
window_size=window_size,
always_partition=always_partition,
)
def forward(self, x):
x = self.downsample(x)
if self.grad_checkpointing and not torch.jit.is_scripting():
x = checkpoint_seq(self.blocks, x)
else:
x = self.blocks(x)
return x
class SwinTransformer(nn.Module):
""" Swin Transformer
A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` -
https://arxiv.org/pdf/2103.14030
"""
def __init__(
self,
img_size: _int_or_tuple_2_t = 224,
patch_size: int = 4,
in_chans: int = 3,
num_classes: int = 1000,
global_pool: str = 'avg',
embed_dim: int = 96,
depths: Tuple[int, ...] = (2, 2, 6, 2),
num_heads: Tuple[int, ...] = (3, 6, 12, 24),
head_dim: Optional[int] = None,
window_size: _int_or_tuple_2_t = 7,
always_partition: bool = False,
strict_img_size: bool = True,
mlp_ratio: float = 4.,
qkv_bias: bool = True,
drop_rate: float = 0.,
proj_drop_rate: float = 0.,
attn_drop_rate: float = 0.,
drop_path_rate: float = 0.1,
embed_layer: Callable = PatchEmbed,
norm_layer: Union[str, Callable] = nn.LayerNorm,
weight_init: str = '',
**kwargs,
):
"""
Args:
img_size: Input image size.
patch_size: Patch size.
in_chans: Number of input image channels.
num_classes: Number of classes for classification head.
embed_dim: Patch embedding dimension.
depths: Depth of each Swin Transformer layer.
num_heads: Number of attention heads in different layers.
head_dim: Dimension of self-attention heads.
window_size: Window size.
mlp_ratio: Ratio of mlp hidden dim to embedding dim.
qkv_bias: If True, add a learnable bias to query, key, value.
drop_rate: Dropout rate.
attn_drop_rate (float): Attention dropout rate.
drop_path_rate (float): Stochastic depth rate.
embed_layer: Patch embedding layer.
norm_layer (nn.Module): Normalization layer.
"""
super().__init__()
assert global_pool in ('', 'avg')
self.num_classes = num_classes
self.global_pool = global_pool
self.output_fmt = 'NHWC'
self.num_layers = len(depths)
self.embed_dim = embed_dim
self.num_features = self.head_hidden_size = int(embed_dim * 2 ** (self.num_layers - 1))
self.feature_info = []
if not isinstance(embed_dim, (tuple, list)):
embed_dim = [int(embed_dim * 2 ** i) for i in range(self.num_layers)]
# split image into non-overlapping patches
self.patch_embed = embed_layer(
img_size=img_size,
patch_size=patch_size,
in_chans=in_chans,
embed_dim=embed_dim[0],
norm_layer=norm_layer,
strict_img_size=strict_img_size,
output_fmt='NHWC',
)
patch_grid = self.patch_embed.grid_size
# build layers
head_dim = to_ntuple(self.num_layers)(head_dim)
if not isinstance(window_size, (list, tuple)):
window_size = to_ntuple(self.num_layers)(window_size)
elif len(window_size) == 2:
window_size = (window_size,) * self.num_layers
assert len(window_size) == self.num_layers
mlp_ratio = to_ntuple(self.num_layers)(mlp_ratio)
dpr = [x.tolist() for x in torch.linspace(0, drop_path_rate, sum(depths)).split(depths)]
layers = []
in_dim = embed_dim[0]
scale = 1
for i in range(self.num_layers):
out_dim = embed_dim[i]
layers += [SwinTransformerStage(
dim=in_dim,
out_dim=out_dim,
input_resolution=(
patch_grid[0] // scale,
patch_grid[1] // scale
),
depth=depths[i],
downsample=i > 0,
num_heads=num_heads[i],
head_dim=head_dim[i],
window_size=window_size[i],
always_partition=always_partition,
dynamic_mask=not strict_img_size,
mlp_ratio=mlp_ratio[i],
qkv_bias=qkv_bias,
proj_drop=proj_drop_rate,
attn_drop=attn_drop_rate,
drop_path=dpr[i],
norm_layer=norm_layer,
)]
in_dim = out_dim
if i > 0:
scale *= 2
self.feature_info += [dict(num_chs=out_dim, reduction=patch_size * scale, module=f'layers.{i}')]
self.layers = nn.Sequential(*layers)
self.norm = norm_layer(self.num_features)
self.head = ClassifierHead(
self.num_features,
num_classes,
pool_type=global_pool,
drop_rate=drop_rate,
input_fmt=self.output_fmt,
)
if weight_init != 'skip':
self.init_weights(weight_init)
@torch.jit.ignore
def init_weights(self, mode=''):
assert mode in ('jax', 'jax_nlhb', 'moco', '')
head_bias = -math.log(self.num_classes) if 'nlhb' in mode else 0.
named_apply(get_init_weights_vit(mode, head_bias=head_bias), self)
@torch.jit.ignore
def no_weight_decay(self):
nwd = set()
for n, _ in self.named_parameters():
if 'relative_position_bias_table' in n:
nwd.add(n)
return nwd
def set_input_size(
self,
img_size: Optional[Tuple[int, int]] = None,
patch_size: Optional[Tuple[int, int]] = None,
window_size: Optional[Tuple[int, int]] = None,
window_ratio: int = 8,
always_partition: Optional[bool] = None,
) -> None:
""" Updates the image resolution and window size.
Args:
img_size: New input resolution, if None current resolution is used
patch_size (Optional[Tuple[int, int]): New patch size, if None use current patch size
window_size: New window size, if None based on new_img_size // window_div
window_ratio: divisor for calculating window size from grid size
always_partition: always partition into windows and shift (even if window size < feat size)
"""
if img_size is not None or patch_size is not None:
self.patch_embed.set_input_size(img_size=img_size, patch_size=patch_size)
patch_grid = self.patch_embed.grid_size
if window_size is None:
window_size = tuple([pg // window_ratio for pg in patch_grid])
for index, stage in enumerate(self.layers):
stage_scale = 2 ** max(index - 1, 0)
stage.set_input_size(
feat_size=(patch_grid[0] // stage_scale, patch_grid[1] // stage_scale),
window_size=window_size,
always_partition=always_partition,
)
@torch.jit.ignore
def group_matcher(self, coarse=False):
return dict(
stem=r'^patch_embed', # stem and embed
blocks=r'^layers\.(\d+)' if coarse else [
(r'^layers\.(\d+).downsample', (0,)),
(r'^layers\.(\d+)\.\w+\.(\d+)', None),
(r'^norm', (99999,)),
]
)
@torch.jit.ignore
def set_grad_checkpointing(self, enable=True):
for l in self.layers:
l.grad_checkpointing = enable
@torch.jit.ignore
def get_classifier(self) -> nn.Module:
return self.head.fc
def reset_classifier(self, num_classes: int, global_pool: Optional[str] = None):
self.num_classes = num_classes
self.head.reset(num_classes, pool_type=global_pool)
def forward_intermediates(
self,
x: torch.Tensor,
indices: Optional[Union[int, List[int]]] = None,
norm: bool = False,
stop_early: bool = False,
output_fmt: str = 'NCHW',
intermediates_only: bool = False,
) -> Union[List[torch.Tensor], Tuple[torch.Tensor, List[torch.Tensor]]]:
""" Forward features that returns intermediates.
Args:
x: Input image tensor
indices: Take last n blocks if int, all if None, select matching indices if sequence
norm: Apply norm layer to compatible intermediates
stop_early: Stop iterating over blocks when last desired intermediate hit
output_fmt: Shape of intermediate feature outputs
intermediates_only: Only return intermediate features
Returns:
"""
assert output_fmt in ('NCHW',), 'Output shape must be NCHW.'
intermediates = []
take_indices, max_index = feature_take_indices(len(self.layers), indices)
# forward pass
x = self.patch_embed(x)
num_stages = len(self.layers)
if torch.jit.is_scripting() or not stop_early: # can't slice blocks in torchscript
stages = self.layers
else:
stages = self.layers[:max_index + 1]
for i, stage in enumerate(stages):
x = stage(x)
if i in take_indices:
if norm and i == num_stages - 1:
x_inter = self.norm(x) # applying final norm last intermediate
else:
x_inter = x
x_inter = x_inter.permute(0, 3, 1, 2).contiguous()
intermediates.append(x_inter)
if intermediates_only:
return intermediates
x = self.norm(x)
return x, intermediates
def prune_intermediate_layers(
self,
indices: Union[int, List[int]] = 1,
prune_norm: bool = False,
prune_head: bool = True,
):
""" Prune layers not required for specified intermediates.
"""
take_indices, max_index = feature_take_indices(len(self.layers), indices)
self.layers = self.layers[:max_index + 1] # truncate blocks
if prune_norm:
self.norm = nn.Identity()
if prune_head:
self.reset_classifier(0, '')
return take_indices
def forward_features(self, x):
x = self.patch_embed(x)
x = self.layers(x)
x = self.norm(x)
return x
def forward_head(self, x, pre_logits: bool = False):
return self.head(x, pre_logits=True) if pre_logits else self.head(x)
def forward(self, x):
x = self.forward_features(x)
x = self.forward_head(x)
return x
def checkpoint_filter_fn(state_dict, model):
""" convert patch embedding weight from manual patchify + linear proj to conv"""
old_weights = True
if 'head.fc.weight' in state_dict:
old_weights = False
import re
out_dict = {}
state_dict = state_dict.get('model', state_dict)
state_dict = state_dict.get('state_dict', state_dict)
for k, v in state_dict.items():
if any([n in k for n in ('relative_position_index', 'attn_mask')]):
continue # skip buffers that should not be persistent
if 'patch_embed.proj.weight' in k:
_, _, H, W = model.patch_embed.proj.weight.shape
if v.shape[-2] != H or v.shape[-1] != W:
v = resample_patch_embed(
v,
(H, W),
interpolation='bicubic',
antialias=True,
verbose=True,
)
if k.endswith('relative_position_bias_table'):
m = model.get_submodule(k[:-29])
if v.shape != m.relative_position_bias_table.shape or m.window_size[0] != m.window_size[1]:
v = resize_rel_pos_bias_table(
v,
new_window_size=m.window_size,
new_bias_shape=m.relative_position_bias_table.shape,
)
if old_weights:
k = re.sub(r'layers.(\d+).downsample', lambda x: f'layers.{int(x.group(1)) + 1}.downsample', k)
k = k.replace('head.', 'head.fc.')
out_dict[k] = v
return out_dict
def _create_swin_transformer(variant, pretrained=False, **kwargs):
default_out_indices = tuple(i for i, _ in enumerate(kwargs.get('depths', (1, 1, 3, 1))))
out_indices = kwargs.pop('out_indices', default_out_indices)
model = build_model_with_cfg(
SwinTransformer, variant, pretrained,
pretrained_filter_fn=checkpoint_filter_fn,
feature_cfg=dict(flatten_sequential=True, out_indices=out_indices),
**kwargs)
return model
def _cfg(url='', **kwargs):
return {
'url': url,
'num_classes': 1000, 'input_size': (3, 224, 224), 'pool_size': (7, 7),
'crop_pct': .9, 'interpolation': 'bicubic', 'fixed_input_size': True,
'mean': IMAGENET_DEFAULT_MEAN, 'std': IMAGENET_DEFAULT_STD,
'first_conv': 'patch_embed.proj', 'classifier': 'head.fc',
'license': 'mit', **kwargs
}
default_cfgs = generate_default_cfgs({
'swin_small_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth', ),
'swin_base_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth',),
'swin_base_patch4_window12_384.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth',
input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
'swin_large_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pth',),
'swin_large_patch4_window12_384.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pth',
input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
'swin_tiny_patch4_window7_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth',),
'swin_small_patch4_window7_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth',),
'swin_base_patch4_window7_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth',),
'swin_base_patch4_window12_384.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pth',
input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0),
# tiny 22k pretrain is worse than 1k, so moved after (untagged priority is based on order)
'swin_tiny_patch4_window7_224.ms_in22k_ft_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pth',),
'swin_tiny_patch4_window7_224.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth',
num_classes=21841),
'swin_small_patch4_window7_224.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pth',
num_classes=21841),
'swin_base_patch4_window7_224.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth',
num_classes=21841),
'swin_base_patch4_window12_384.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth',
input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0, num_classes=21841),
'swin_large_patch4_window7_224.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth',
num_classes=21841),
'swin_large_patch4_window12_384.ms_in22k': _cfg(
hf_hub_id='timm/',
url='https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pth',
input_size=(3, 384, 384), pool_size=(12, 12), crop_pct=1.0, num_classes=21841),
'swin_s3_tiny_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_t-1d53f6a8.pth'),
'swin_s3_small_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_s-3bb4c69d.pth'),
'swin_s3_base_224.ms_in1k': _cfg(
hf_hub_id='timm/',
url='https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_b-a1e95db4.pth'),
})
@register_model
def swin_tiny_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-T @ 224x224, trained ImageNet-1k
"""
model_args = dict(patch_size=4, window_size=7, embed_dim=96, depths=(2, 2, 6, 2), num_heads=(3, 6, 12, 24))
return _create_swin_transformer(
'swin_tiny_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_small_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-S @ 224x224
"""
model_args = dict(patch_size=4, window_size=7, embed_dim=96, depths=(2, 2, 18, 2), num_heads=(3, 6, 12, 24))
return _create_swin_transformer(
'swin_small_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_base_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-B @ 224x224
"""
model_args = dict(patch_size=4, window_size=7, embed_dim=128, depths=(2, 2, 18, 2), num_heads=(4, 8, 16, 32))
return _create_swin_transformer(
'swin_base_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_base_patch4_window12_384(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-B @ 384x384
"""
model_args = dict(patch_size=4, window_size=12, embed_dim=128, depths=(2, 2, 18, 2), num_heads=(4, 8, 16, 32))
return _create_swin_transformer(
'swin_base_patch4_window12_384', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_large_patch4_window7_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-L @ 224x224
"""
model_args = dict(patch_size=4, window_size=7, embed_dim=192, depths=(2, 2, 18, 2), num_heads=(6, 12, 24, 48))
return _create_swin_transformer(
'swin_large_patch4_window7_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_large_patch4_window12_384(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-L @ 384x384
"""
model_args = dict(patch_size=4, window_size=12, embed_dim=192, depths=(2, 2, 18, 2), num_heads=(6, 12, 24, 48))
return _create_swin_transformer(
'swin_large_patch4_window12_384', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_s3_tiny_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-S3-T @ 224x224, https://arxiv.org/abs/2111.14725
"""
model_args = dict(
patch_size=4, window_size=(7, 7, 14, 7), embed_dim=96, depths=(2, 2, 6, 2), num_heads=(3, 6, 12, 24))
return _create_swin_transformer('swin_s3_tiny_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_s3_small_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725
"""
model_args = dict(
patch_size=4, window_size=(14, 14, 14, 7), embed_dim=96, depths=(2, 2, 18, 2), num_heads=(3, 6, 12, 24))
return _create_swin_transformer('swin_s3_small_224', pretrained=pretrained, **dict(model_args, **kwargs))
@register_model
def swin_s3_base_224(pretrained=False, **kwargs) -> SwinTransformer:
""" Swin-S3-B @ 224x224, https://arxiv.org/abs/2111.14725
"""
model_args = dict(
patch_size=4, window_size=(7, 7, 14, 7), embed_dim=96, depths=(2, 2, 30, 2), num_heads=(3, 6, 12, 24))
return _create_swin_transformer('swin_s3_base_224', pretrained=pretrained, **dict(model_args, **kwargs))
register_model_deprecations(__name__, {
'swin_base_patch4_window7_224_in22k': 'swin_base_patch4_window7_224.ms_in22k',
'swin_base_patch4_window12_384_in22k': 'swin_base_patch4_window12_384.ms_in22k',
'swin_large_patch4_window7_224_in22k': 'swin_large_patch4_window7_224.ms_in22k',
'swin_large_patch4_window12_384_in22k': 'swin_large_patch4_window12_384.ms_in22k',
})

swin_transformer.py at commit 9cfc3f0, no license · at the source

Overview

Authors: Yoon Hwan Byun^1,2, Hyunseok Seo³, Jae-Kyung Won⁴, Boram Lee⁵, Duk Hyun Hong⁶, Sun Mo Nam⁷, Jong Ha Hwang⁷, Min-Sung Kim⁷, Yong-Hwy Kim⁷, Jang Hun Kim⁶, Mi Ok Yu⁶, Kyung-Jae Park⁶, HoJoon Kim³, Sunit Das⁸, Doo-Sik Kong⁹, Chul-Kee Park⁷, Shin-Hyuk Kang⁶

Institutions: SNUH SMG-SNU Boramae Medical Center (South Korea); Seoul National University (South Korea); Korea University (South Korea); Seoul National University Hospital (South Korea); Sungkyunkwan University (South Korea); Korea University Anam Hospital (South Korea); University of Toronto (Canada)

Journal: NPJ digital medicine, volume 9, issue 1, article 471

Dates: received 14 December 2025; accepted 10 April 2026; published online 18 April 2026

Type: Research article · Language: English

License: CC BY-NC-ND

Identifiers: DOI 10.1038/s41746-026-02651-0 · PMID 42000880 · PMCID PMC13276397 · OpenAlex W7154848825

Open access: gold, a free copy (OpenAlex)

Status: code verified

Categories: histology / microscopy (modality), other condition (population), clinical / translational (subfield)

Methods: Statistics, Machine learning

Keywords: Cancer, Computational biology and bioinformatics, Medical research, Oncology

Topic: Glioma Diagnosis and Treatment (Genetics, Medicine), according to OpenAlex

Funding: Ministry of Science and ICT (RS202400338025); Ministry of Health and Welfare (RS-2022-KH129293); Korea University Anam Hospital (O241376); Technology Development Program of the Ministry of SMEs and Startups (RS-2024-00487593)

Citations: not cited yet (Europe PMC); 50 references in the paper

Research resources: RRID:SCR_008567

Abstract

The abstract is not reproduced here: the paper's license (CC BY-NC-ND) does not allow it. Read it in the paper, at the publisher or on Europe PMC.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

ImpelKorea/AI-agumented-CLE-Imaging-npj-Digital-Medicine

License: none: the authors keep all their rights

State: the link answers, verified on 29 September 2026

Evidence: files inventoried

Commit: 9cfc3f0914b0ad7fae4a9a7ac432c671198b0cc3, 15 March 2026

Languages: Python (251)

Size: 280 files, 251 scripts

Software Heritage: not archived

Found in: “Code availability”

Holds: README, environment (environment.yml)

Not found: license file, CITATION.cff, tests, continuous integration, documentation

Tools: PyTorch (206 files), NumPy (20 files), Pillow (9 files), TensorFlow (2 files), Matplotlib (1 file), pandas (1 file), SciPy (1 file)

Availability: 1 check, the latest on 29 September 2026: the link answers

29 September 2026: the link answers

252 files

data_loader.py, Python, 98 lines
gradientCAM/AGCAM.py, Python, 224 lines
gradientCAM/swin_agcam/__init__.py, Python, 4 lines
gradientCAM/swin_agcam/data/__init__.py, Python, 15 lines
gradientCAM/swin_agcam/data/auto_augment.py, Python, 997 lines
gradientCAM/swin_agcam/data/config.py, Python, 129 lines
gradientCAM/swin_agcam/data/constants.py, Python, 10 lines
gradientCAM/swin_agcam/data/dataset.py, Python, 202 lines
gradientCAM/swin_agcam/data/dataset_factory.py, Python, 224 lines
gradientCAM/swin_agcam/data/dataset_info.py, Python, 73 lines
gradientCAM/swin_agcam/data/distributed_sampler.py, Python, 135 lines
gradientCAM/swin_agcam/data/imagenet_info.py, Python, 95 lines
gradientCAM/swin_agcam/data/loader.py, Python, 409 lines
gradientCAM/swin_agcam/data/mixup.py, Python, 316 lines
gradientCAM/swin_agcam/data/random_erasing.py, Python, 117 lines
gradientCAM/swin_agcam/data/readers/__init__.py, Python, 2 lines
gradientCAM/swin_agcam/data/readers/class_map.py, Python, 23 lines
gradientCAM/swin_agcam/data/readers/img_extensions.py, Python, 50 lines
gradientCAM/swin_agcam/data/readers/reader.py, Python, 17 lines
gradientCAM/swin_agcam/data/readers/reader_factory.py, Python, 45 lines
gradientCAM/swin_agcam/data/readers/reader_hfds.py, Python, 86 lines
gradientCAM/swin_agcam/data/readers/reader_hfids.py, Python, 215 lines
gradientCAM/swin_agcam/data/readers/reader_image_folder.py, Python, 99 lines
gradientCAM/swin_agcam/data/readers/reader_image_in_tar.py, Python, 229 lines
gradientCAM/swin_agcam/data/readers/reader_image_tar.py, Python, 74 lines
gradientCAM/swin_agcam/data/readers/reader_tfds.py, Python, 355 lines
gradientCAM/swin_agcam/data/readers/reader_wds.py, Python, 466 lines
gradientCAM/swin_agcam/data/readers/shared_count.py, Python, 14 lines
gradientCAM/swin_agcam/data/real_labels.py, Python, 47 lines
gradientCAM/swin_agcam/data/tf_preprocessing.py, Python, 233 lines
gradientCAM/swin_agcam/data/transforms.py, Python, 583 lines
gradientCAM/swin_agcam/data/transforms_factory.py, Python, 460 lines
gradientCAM/swin_agcam/layers/__init__.py, Python, 61 lines
gradientCAM/swin_agcam/layers/activations.py, Python, 173 lines
gradientCAM/swin_agcam/layers/activations_me.py, Python, 208 lines
gradientCAM/swin_agcam/layers/adaptive_avgmax_pool.py, Python, 183 lines
gradientCAM/swin_agcam/layers/attention2d.py, Python, 351 lines
gradientCAM/swin_agcam/layers/attention_pool.py, Python, 105 lines
gradientCAM/swin_agcam/layers/attention_pool2d.py, Python, 278 lines
gradientCAM/swin_agcam/layers/blur_pool.py, Python, 91 lines
gradientCAM/swin_agcam/layers/bottleneck_attn.py, Python, 157 lines
gradientCAM/swin_agcam/layers/cbam.py, Python, 112 lines
gradientCAM/swin_agcam/layers/classifier.py, Python, 283 lines
gradientCAM/swin_agcam/layers/cond_conv2d.py, Python, 123 lines
gradientCAM/swin_agcam/layers/config.py, Python, 149 lines
gradientCAM/swin_agcam/layers/conv2d_same.py, Python, 110 lines
gradientCAM/swin_agcam/layers/conv_bn_act.py, Python, 91 lines
gradientCAM/swin_agcam/layers/create_act.py, Python, 138 lines
gradientCAM/swin_agcam/layers/create_attn.py, Python, 89 lines
gradientCAM/swin_agcam/layers/create_conv2d.py, Python, 36 lines
gradientCAM/swin_agcam/layers/create_norm.py, Python, 58 lines
gradientCAM/swin_agcam/layers/create_norm_act.py, Python, 95 lines
gradientCAM/swin_agcam/layers/drop.py, Python, 182 lines
gradientCAM/swin_agcam/layers/eca.py, Python, 145 lines
gradientCAM/swin_agcam/layers/evo_norm.py, Python, 352 lines
gradientCAM/swin_agcam/layers/fast_norm.py, Python, 150 lines
gradientCAM/swin_agcam/layers/filter_response_norm.py, Python, 68 lines
gradientCAM/swin_agcam/layers/format.py, Python, 58 lines
gradientCAM/swin_agcam/layers/gather_excite.py, Python, 90 lines
gradientCAM/swin_agcam/layers/global_context.py, Python, 67 lines
gradientCAM/swin_agcam/layers/grid.py, Python, 49 lines
gradientCAM/swin_agcam/layers/grn.py, Python, 39 lines
gradientCAM/swin_agcam/layers/halo_attn.py, Python, 233 lines
gradientCAM/swin_agcam/layers/helpers.py, Python, 43 lines
gradientCAM/swin_agcam/layers/hybrid_embed.py, Python, 253 lines
gradientCAM/swin_agcam/layers/inplace_abn.py, Python, 87 lines
gradientCAM/swin_agcam/layers/interpolate.py, Python, 68 lines
gradientCAM/swin_agcam/layers/lambda_layer.py, Python, 134 lines
gradientCAM/swin_agcam/layers/layer_scale.py, Python, 38 lines
gradientCAM/swin_agcam/layers/linear.py, Python, 19 lines
gradientCAM/swin_agcam/layers/median_pool.py, Python, 49 lines
gradientCAM/swin_agcam/layers/mixed_conv2d.py, Python, 51 lines
gradientCAM/swin_agcam/layers/ml_decoder.py, Python, 146 lines
gradientCAM/swin_agcam/layers/mlp.py, Python, 260 lines
gradientCAM/swin_agcam/layers/non_local_attn.py, Python, 145 lines
gradientCAM/swin_agcam/layers/norm.py, Python, 192 lines
gradientCAM/swin_agcam/layers/norm_act.py, Python, 461 lines
gradientCAM/swin_agcam/layers/padding.py, Python, 87 lines
gradientCAM/swin_agcam/layers/patch_dropout.py, Python, 53 lines
gradientCAM/swin_agcam/layers/patch_embed.py, Python, 307 lines
gradientCAM/swin_agcam/layers/pool2d_same.py, Python, 73 lines
gradientCAM/swin_agcam/layers/pos_embed.py, Python, 79 lines
gradientCAM/swin_agcam/layers/pos_embed_rel.py, Python, 491 lines
gradientCAM/swin_agcam/layers/pos_embed_sincos.py, Python, 443 lines
gradientCAM/swin_agcam/layers/selective_kernel.py, Python, 119 lines
gradientCAM/swin_agcam/layers/separable_conv.py, Python, 76 lines
gradientCAM/swin_agcam/layers/space_to_depth.py, Python, 32 lines
gradientCAM/swin_agcam/layers/split_attn.py, Python, 84 lines
gradientCAM/swin_agcam/layers/split_batchnorm.py, Python, 75 lines
gradientCAM/swin_agcam/layers/squeeze_excite.py, Python, 102 lines
gradientCAM/swin_agcam/layers/std_conv.py, Python, 133 lines
gradientCAM/swin_agcam/layers/test_time_pool.py, Python, 52 lines
gradientCAM/swin_agcam/layers/trace_utils.py, Python, 13 lines
gradientCAM/swin_agcam/layers/typing.py, Python, 7 lines
gradientCAM/swin_agcam/layers/weight_init.py, Python, 167 lines
gradientCAM/swin_agcam/loss/__init__.py, Python, 4 lines
gradientCAM/swin_agcam/loss/asymmetric_loss.py, Python, 97 lines
gradientCAM/swin_agcam/loss/binary_cross_entropy.py, Python, 65 lines
gradientCAM/swin_agcam/loss/cross_entropy.py, Python, 36 lines
gradientCAM/swin_agcam/loss/jsd.py, Python, 39 lines
gradientCAM/swin_agcam/models/__init__.py, Python, 100 lines
gradientCAM/swin_agcam/models/_builder.py, Python, 472 lines
gradientCAM/swin_agcam/models/_efficientnet_blocks.py, Python, 702 lines
gradientCAM/swin_agcam/models/_efficientnet_builder.py, Python, 576 lines
gradientCAM/swin_agcam/models/_factory.py, Python, 127 lines
gradientCAM/swin_agcam/models/_features.py, Python, 484 lines
gradientCAM/swin_agcam/models/_features_fx.py, Python, 179 lines
gradientCAM/swin_agcam/models/_helpers.py, Python, 167 lines
gradientCAM/swin_agcam/models/_hub.py, Python, 417 lines
gradientCAM/swin_agcam/models/_manipulate.py, Python, 278 lines
gradientCAM/swin_agcam/models/_pretrained.py, Python, 94 lines
gradientCAM/swin_agcam/models/_prune.py, Python, 116 lines
gradientCAM/swin_agcam/models/_registry.py, Python, 352 lines
gradientCAM/swin_agcam/models/beit.py, Python, 716 lines
gradientCAM/swin_agcam/models/byoanet.py, Python, 455 lines
gradientCAM/swin_agcam/models/byobnet.py, Python, 2,784 lines
gradientCAM/swin_agcam/models/cait.py, Python, 591 lines
gradientCAM/swin_agcam/models/coat.py, Python, 803 lines
gradientCAM/swin_agcam/models/convit.py, Python, 428 lines
gradientCAM/swin_agcam/models/convmixer.py, Python, 146 lines
gradientCAM/swin_agcam/models/convnext.py, Python, 1,241 lines
gradientCAM/swin_agcam/models/crossvit.py, Python, 627 lines
gradientCAM/swin_agcam/models/cspnet.py, Python, 1,111 lines
gradientCAM/swin_agcam/models/davit.py, Python, 816 lines
gradientCAM/swin_agcam/models/deit.py, Python, 417 lines
gradientCAM/swin_agcam/models/densenet.py, Python, 424 lines
gradientCAM/swin_agcam/models/dla.py, Python, 515 lines
gradientCAM/swin_agcam/models/dpn.py, Python, 371 lines
gradientCAM/swin_agcam/models/edgenext.py, Python, 576 lines
gradientCAM/swin_agcam/models/efficientformer.py, Python, 649 lines
gradientCAM/swin_agcam/models/efficientformer_v2.py, Python, 736 lines
gradientCAM/swin_agcam/models/efficientnet.py, Python, 2,860 lines
gradientCAM/swin_agcam/models/efficientvit_mit.py, Python, 1,072 lines
gradientCAM/swin_agcam/models/efficientvit_msra.py, Python, 659 lines
gradientCAM/swin_agcam/models/eva.py, Python, 1,325 lines
gradientCAM/swin_agcam/models/factory.py, Python, 4 lines
gradientCAM/swin_agcam/models/fastvit.py, Python, 1,634 lines
gradientCAM/swin_agcam/models/features.py, Python, 4 lines
gradientCAM/swin_agcam/models/focalnet.py, Python, 652 lines
gradientCAM/swin_agcam/models/fx_features.py, Python, 4 lines
gradientCAM/swin_agcam/models/gcvit.py, Python, 592 lines
gradientCAM/swin_agcam/models/ghostnet.py, Python, 433 lines
gradientCAM/swin_agcam/models/hardcorenas.py, Python, 156 lines
gradientCAM/swin_agcam/models/helpers.py, Python, 7 lines
gradientCAM/swin_agcam/models/hgnet.py, Python, 738 lines
gradientCAM/swin_agcam/models/hiera.py, Python, 996 lines
gradientCAM/swin_agcam/models/hieradet_sam2.py, Python, 635 lines
gradientCAM/swin_agcam/models/hrnet.py, Python, 979 lines
gradientCAM/swin_agcam/models/hub.py, Python, 4 lines
gradientCAM/swin_agcam/models/inception_next.py, Python, 432 lines
gradientCAM/swin_agcam/models/inception_resnet_v2.py, Python, 341 lines
gradientCAM/swin_agcam/models/inception_v3.py, Python, 458 lines
gradientCAM/swin_agcam/models/inception_v4.py, Python, 325 lines
gradientCAM/swin_agcam/models/layers/__init__.py, Python, 48 lines
gradientCAM/swin_agcam/models/levit.py, Python, 997 lines
gradientCAM/swin_agcam/models/mambaout.py, Python, 642 lines
gradientCAM/swin_agcam/models/maxxvit.py, Python, 2,402 lines
gradientCAM/swin_agcam/models/metaformer.py, Python, 1,059 lines
gradientCAM/swin_agcam/models/mlp_mixer.py, Python, 710 lines
gradientCAM/swin_agcam/models/mobilenetv3.py, Python, 1,350 lines
gradientCAM/swin_agcam/models/mobilevit.py, Python, 681 lines
gradientCAM/swin_agcam/models/mvitv2.py, Python, 1,120 lines
gradientCAM/swin_agcam/models/nasnet.py, Python, 600 lines
gradientCAM/swin_agcam/models/nest.py, Python, 583 lines
gradientCAM/swin_agcam/models/nextvit.py, Python, 688 lines
gradientCAM/swin_agcam/models/nfnet.py, Python, 1,045 lines
gradientCAM/swin_agcam/models/pit.py, Python, 460 lines
gradientCAM/swin_agcam/models/pnasnet.py, Python, 378 lines
gradientCAM/swin_agcam/models/pvt_v2.py, Python, 503 lines
gradientCAM/swin_agcam/models/rdnet.py, Python, 505 lines
gradientCAM/swin_agcam/models/registry.py, Python, 4 lines
gradientCAM/swin_agcam/models/regnet.py, Python, 1,191 lines
gradientCAM/swin_agcam/models/repghost.py, Python, 481 lines
gradientCAM/swin_agcam/models/repvit.py, Python, 513 lines
gradientCAM/swin_agcam/models/res2net.py, Python, 227 lines
gradientCAM/swin_agcam/models/resnest.py, Python, 251 lines
gradientCAM/swin_agcam/models/resnet.py, Python, 2,109 lines
gradientCAM/swin_agcam/models/resnetv2.py, Python, 907 lines
gradientCAM/swin_agcam/models/rexnet.py, Python, 358 lines
gradientCAM/swin_agcam/models/selecsls.py, Python, 378 lines
gradientCAM/swin_agcam/models/senet.py, Python, 465 lines
gradientCAM/swin_agcam/models/sequencer.py, Python, 540 lines
gradientCAM/swin_agcam/models/sknet.py, Python, 240 lines
gradientCAM/swin_agcam/models/swin_transformer.py, Python, 1,085 lines, 1 match
gradientCAM/swin_agcam/models/swin_transformer_v2.py, Python, 1,094 lines, 1 match
gradientCAM/swin_agcam/models/swin_transformer_v2_cr.py, Python, 1,153 lines
gradientCAM/swin_agcam/models/tiny_vit.py, Python, 715 lines
gradientCAM/swin_agcam/models/tnt.py, Python, 374 lines
gradientCAM/swin_agcam/models/tresnet.py, Python, 346 lines
gradientCAM/swin_agcam/models/twins.py, Python, 581 lines
gradientCAM/swin_agcam/models/vgg.py, Python, 298 lines
gradientCAM/swin_agcam/models/visformer.py, Python, 549 lines
gradientCAM/swin_agcam/models/vision_transformer.py, Python, 3,424 lines
gradientCAM/swin_agcam/models/vision_transformer_hybrid.py, Python, 408 lines
gradientCAM/swin_agcam/models/vision_transformer_relpos.py, Python, 703 lines
gradientCAM/swin_agcam/models/vision_transformer_sam.py, Python, 759 lines
gradientCAM/swin_agcam/models/vitamin.py, Python, 602 lines
gradientCAM/swin_agcam/models/volo.py, Python, 994 lines
gradientCAM/swin_agcam/models/vovnet.py, Python, 480 lines
gradientCAM/swin_agcam/models/xception.py, Python, 255 lines
gradientCAM/swin_agcam/models/xception_aligned.py, Python, 435 lines
gradientCAM/swin_agcam/models/xcit.py, Python, 1,003 lines
gradientCAM/swin_agcam/optim/__init__.py, Python, 17 lines
gradientCAM/swin_agcam/optim/adabelief.py, Python, 201 lines
gradientCAM/swin_agcam/optim/adafactor.py, Python, 167 lines
gradientCAM/swin_agcam/optim/adahessian.py, Python, 156 lines
gradientCAM/swin_agcam/optim/adamp.py, Python, 105 lines
gradientCAM/swin_agcam/optim/adamw.py, Python, 122 lines
gradientCAM/swin_agcam/optim/adan.py, Python, 124 lines
gradientCAM/swin_agcam/optim/lamb.py, Python, 192 lines
gradientCAM/swin_agcam/optim/lars.py, Python, 135 lines
gradientCAM/swin_agcam/optim/lion.py, Python, 226 lines
gradientCAM/swin_agcam/optim/lookahead.py, Python, 66 lines
gradientCAM/swin_agcam/optim/madgrad.py, Python, 184 lines
gradientCAM/swin_agcam/optim/nadam.py, Python, 97 lines
gradientCAM/swin_agcam/optim/nadamw.py, Python, 349 lines
gradientCAM/swin_agcam/optim/nvnovograd.py, Python, 120 lines
gradientCAM/swin_agcam/optim/optim_factory.py, Python, 423 lines
gradientCAM/swin_agcam/optim/radam.py, Python, 89 lines
gradientCAM/swin_agcam/optim/rmsprop_tf.py, Python, 139 lines
gradientCAM/swin_agcam/optim/sgdp.py, Python, 70 lines
gradientCAM/swin_agcam/optim/sgdw.py, Python, 266 lines
gradientCAM/swin_agcam/scheduler/__init__.py, Python, 8 lines
gradientCAM/swin_agcam/scheduler/cosine_lr.py, Python, 117 lines
gradientCAM/swin_agcam/scheduler/multistep_lr.py, Python, 63 lines
gradientCAM/swin_agcam/scheduler/plateau_lr.py, Python, 111 lines
gradientCAM/swin_agcam/scheduler/poly_lr.py, Python, 113 lines
gradientCAM/swin_agcam/scheduler/scheduler.py, Python, 127 lines
gradientCAM/swin_agcam/scheduler/scheduler_factory.py, Python, 210 lines
gradientCAM/swin_agcam/scheduler/step_lr.py, Python, 63 lines
gradientCAM/swin_agcam/scheduler/tanh_lr.py, Python, 114 lines
gradientCAM/swin_agcam/utils/__init__.py, Python, 16 lines
gradientCAM/swin_agcam/utils/agc.py, Python, 42 lines
gradientCAM/swin_agcam/utils/attention_extract.py, Python, 85 lines
gradientCAM/swin_agcam/utils/checkpoint_saver.py, Python, 150 lines
gradientCAM/swin_agcam/utils/clip_grad.py, Python, 23 lines
gradientCAM/swin_agcam/utils/cuda.py, Python, 78 lines
gradientCAM/swin_agcam/utils/decay_batch.py, Python, 43 lines
gradientCAM/swin_agcam/utils/distributed.py, Python, 182 lines
gradientCAM/swin_agcam/utils/jit.py, Python, 58 lines
gradientCAM/swin_agcam/utils/log.py, Python, 28 lines
gradientCAM/swin_agcam/utils/metrics.py, Python, 32 lines
gradientCAM/swin_agcam/utils/misc.py, Python, 32 lines
gradientCAM/swin_agcam/utils/model.py, Python, 245 lines
gradientCAM/swin_agcam/utils/model_ema.py, Python, 260 lines
gradientCAM/swin_agcam/utils/onnx.py, Python, 109 lines
gradientCAM/swin_agcam/utils/random.py, Python, 9 lines
gradientCAM/swin_agcam/utils/summary.py, Python, 51 lines
gradientCAM/swin_agcam/version.py, Python, 1 line
model.py, Python, 105 lines
visualization_instance.py, Python, 267 lines
README.md, Text, 37 lines

At the source: github.com/ImpelKorea/AI-agumented-CLE-Imaging-npj-Digital-Medicine

Code availability statement

The paper has a code availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

it points to the authors' code: ImpelKorea/AI-agumented-CLE-Imaging-npj-Digital-Medicine

Read it in the paper: doi.org/10.1038/s41746-026-02651-0.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
251 scripts, each with its path and the digest of its content;
2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability statement

The paper has a data availability statement. Its license (CC BY-NC-ND) does not allow reproducing it here; in short, from what the harvester recognized in it:

it says that the data are available on request

Read it in the paper: doi.org/10.1038/s41746-026-02651-0.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 17 authors, 4 keywords, 4 funders, 42 references, 1 RRID.

Cite

This paper

Byun, Y. H., Seo, H., Won, J.-K., Lee, B., Hong, D. H., Nam, S. M., Hwang, J. H., Kim, M.-S., Kim, Y.-H., Kim, J. H., Yu, M. O., Park, K.-J., Kim, H., Das, S., Kong, D.-S., Park, C.-K., & Kang, S.-H. (2026). AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors. NPJ digital medicine, 9(1), 471. https://doi.org/10.1038/s41746-026-02651-0

BibTeX

@article{byun2026ai, author = {Byun, Yoon Hwan and Seo, Hyunseok and Won, Jae-Kyung and Lee, Boram and Hong, Duk Hyun and Nam, Sun Mo and Hwang, Jong Ha and Kim, Min-Sung and Kim, Yong-Hwy and Kim, Jang Hun and Yu, Mi Ok and Park, Kyung-Jae and Kim, HoJoon and Das, Sunit and Kong, Doo-Sik and Park, Chul-Kee and Kang, Shin-Hyuk}, title = {{AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors}}, journal = {NPJ digital medicine}, year = {2026}, month = apr, volume = {9}, number = {1}, pages = {471}, publisher = {Nature Publishing Group}, issn = {2398-6352}, doi = {10.1038/s41746-026-02651-0}, url = {https://doi.org/10.1038/s41746-026-02651-0}, pmid = {42000880}, pmcid = {PMC13276397} }

RIS

TY - JOUR AU - Byun, Yoon Hwan AU - Seo, Hyunseok AU - Won, Jae-Kyung AU - Lee, Boram AU - Hong, Duk Hyun AU - Nam, Sun Mo AU - Hwang, Jong Ha AU - Kim, Min-Sung AU - Kim, Yong-Hwy AU - Kim, Jang Hun AU - Yu, Mi Ok AU - Park, Kyung-Jae AU - Kim, HoJoon AU - Das, Sunit AU - Kong, Doo-Sik AU - Park, Chul-Kee AU - Kang, Shin-Hyuk TI - AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors T2 - NPJ digital medicine J2 - NPJ Digit Med PY - 2026 DA - 2026/04/18 VL - 9 IS - 1 SP - 471 SN - 2398-6352 PB - Nature Publishing Group DO - 10.1038/s41746-026-02651-0 UR - https://doi.org/10.1038/s41746-026-02651-0 LA - en ER -

CSL-JSON

{ "id": "10.1038/s41746-026-02651-0", "type": "article-journal", "title": "AI Augmented Confocal Laser Endomicroscopy for Rapid Intraoperative Diagnosis of Brain Tumors", "container-title": "NPJ digital medicine", "author": [ { "family": "Byun", "given": "Yoon Hwan" }, { "family": "Seo", "given": "Hyunseok" }, { "family": "Won", "given": "Jae-Kyung" }, { "family": "Lee", "given": "Boram" }, { "family": "Hong", "given": "Duk Hyun" }, { "family": "Nam", "given": "Sun Mo" }, { "family": "Hwang", "given": "Jong Ha" }, { "family": "Kim", "given": "Min-Sung" }, { "family": "Kim", "given": "Yong-Hwy" }, { "family": "Kim", "given": "Jang Hun" }, { "family": "Yu", "given": "Mi Ok" }, { "family": "Park", "given": "Kyung-Jae" }, { "family": "Kim", "given": "HoJoon" }, { "family": "Das", "given": "Sunit" }, { "family": "Kong", "given": "Doo-Sik" }, { "family": "Park", "given": "Chul-Kee" }, { "family": "Kang", "given": "Shin-Hyuk" } ], "container-title-short": "NPJ Digit Med", "volume": "9", "issue": "1", "page": "471", "DOI": "10.1038/s41746-026-02651-0", "PMID": "42000880", "PMCID": "PMC13276397", "ISSN": "2398-6352", "publisher": "Nature Publishing Group", "URL": "https://doi.org/10.1038/s41746-026-02651-0", "language": "en", "issued": { "date-parts": [ [ 2026, 4, 18 ] ] } }

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1016/j.patter.2026.101538 [code]: A multi-modal foundation model for brain disease diagnosis and medical imaging.
Journal: Patterns (New York, N.Y.)
In common: TensorFlow, Pillow, PyTorch, 4 other tools, clinical / translational, 1 reference
[2] doi:10.1093/jnen/nlaf152 [code]: Clinical and pathologic correlations of machine learning quantification of Aβ deposits across 3 brain regions of decedents with Alzheimer disease.
Journal: Journal of neuropathology and experimental neurology
In common: TensorFlow, Pillow, PyTorch, 4 other tools, clinical / translational
[3] doi:10.7554/elife.110588 [code]: Opening the black box toward a modular approach to spike sorting.
Journal: eLife
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[4] doi:10.1038/s42003-026-10957-8 [code]: Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[5] doi:10.3390/s26175501 [code]: Evaluation of a Hybrid Neural-Polynomial Deep Q-Network for Switching-Aware Spectrum Selection in a Controlled Radio-Frequency Measurement-Replay Testbed.
Journal: Sensors (Basel, Switzerland)
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[6] doi:10.1126/sciadv.aed3650 [code]: Truthful visualizations for mass spectrometry imaging enable high-spatial-resolution interactive <i>m/z</i> mapping and exploration.
Journal: Science advances
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[7] doi:10.1038/s41592-026-03194-8 [code]: Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.
Journal: Nature methods
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[8] doi:10.1371/journal.pcbi.1014571 [code]: SynAPSeg: A novel dataset and image analysis framework for deep learning-based synapse detection and quantification.
Journal: PLoS computational biology
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[9] doi:10.1364/boe.605322 [code]: Generalized plaque digitization framework for multi-dimensional mesoscopic images.
Journal: Biomedical optics express
In common: TensorFlow, Pillow, PyTorch, 4 other tools
[10] doi:10.1523/eneuro.0023-26.2026 [code]: Real-Time Segmentation and Classification of Birdsong Syllables for Learning Experiments.
Journal: eNeuro
In common: TensorFlow, Pillow, PyTorch, 4 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.