OSCR

Deep learning for early detection of cerebral small vessel disease using self-supervised graph embeddings and retinal image analysis.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § Detection methodology › Phase 2: global transformer-based encoding ↔ rnv_t_empirical_validation.py, lines 228–249 · score 0.55 · Multi Head Correlative, MHCA
  2. [2] § Materials utilized › Experimental components ↔ rnv_t_empirical_validation.py, lines 62–71 · score 0.52 · top hat, Gaussian, CLAHE, preprocessing, vessel, retinal

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 683 lines · 27 KB · GPL-3.0 · 2 matches

  1. """
  2. RNV-T empirical validation reference implementation.
  4. This script implements the five phases described in the manuscript:
  5. 1) Local Vascular Extraction (LVE)
  6. 2) Global Transformer-based Encoding (GTE)
  7. 3) Graph-based Convolutional Attention Network (G-CAN)
  8. 4) Local-Global Attention Fusion (LGAF)
  9. 5) Optimized training with SSL + federated-style aggregation
  11. It is designed as a reproducible empirical validation scaffold rather than the
  12. original laboratory code. Dataset paths, labels, and site partitions can be
  13. plugged in through the config section / CLI.
  14. """
  15. from __future__ import annotations
  17. import argparse
  18. import math
  19. import os
  20. import random
  21. from dataclasses import dataclass
  22. from pathlib import Path
  23. from typing import Dict, Iterable, List, Optional, Sequence, Tuple
  25. import cv2
  26. import numpy as np
  27. import torch
  28. import torch.nn as nn
  29. import torch.nn.functional as F
  30. from torch.utils.data import DataLoader, Dataset
  33. # -----------------------------------------------------------------------------
  34. # Reproducibility helpers
  35. # -----------------------------------------------------------------------------
  37. def seed_everything(seed: int = 42) -> None:
  38. random.seed(seed)
  39. np.random.seed(seed)
  40. torch.manual_seed(seed)
  41. torch.cuda.manual_seed_all(seed)
  44. # -----------------------------------------------------------------------------
  45. # Preprocessing utilities
  46. # -----------------------------------------------------------------------------
  48. def apply_clahe_rgb(image: np.ndarray) -> np.ndarray:
  49. lab = cv2.cvtColor(image, cv2.COLOR_RGB2LAB)
  50. l, a, b = cv2.split(lab)
  51. clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
  52. l = clahe.apply(l)
  53. out = cv2.merge([l, a, b])
  54. return cv2.cvtColor(out, cv2.COLOR_LAB2RGB)
  57. def top_hat_enhance(gray: np.ndarray, kernel_size: int = 15) -> np.ndarray:
  58. kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (kernel_size, kernel_size))
  59. return cv2.morphologyEx(gray, cv2.MORPH_TOPHAT, kernel)
  62. def preprocess_retinal_image(image: np.ndarray, image_size: int = 224) -> np.ndarray:
  63. image = cv2.resize(image, (image_size, image_size), interpolation=cv2.INTER_AREA)
  64. image = apply_clahe_rgb(image)
  65. image = cv2.GaussianBlur(image, (3, 3), 0)
  66. gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
  67. vessel_boost = top_hat_enhance(gray, kernel_size=11)
  68. vessel_boost = np.stack([vessel_boost] * 3, axis=-1)
  69. image = cv2.addWeighted(image, 0.85, vessel_boost, 0.15, 0.0)
  70. image = image.astype(np.float32) / 255.0
  71. return image
  74. def load_rgb(path: str, image_size: int = 224) -> np.ndarray:
  75. img = cv2.imread(path, cv2.IMREAD_COLOR)
  76. if img is None:
  77. raise FileNotFoundError(path)
  78. img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  79. return preprocess_retinal_image(img, image_size=image_size)
  82. # -----------------------------------------------------------------------------
  83. # Dataset
  84. # -----------------------------------------------------------------------------
  86. @dataclass
  87. class SampleRecord:
  88. fundus: str
  89. octa: str
  90. fa: str
  91. label: int
  92. site: str = "site_0"
  95. class MultiModalRetinalDataset(Dataset):
  96. def __init__(self, records: Sequence[SampleRecord], image_size: int = 224, augment: bool = False):
  97. self.records = list(records)
  98. self.image_size = image_size
  99. self.augment = augment
  101. def __len__(self) -> int:
  102. return len(self.records)
  104. def _augment(self, x: np.ndarray) -> np.ndarray:
  105. if not self.augment:
  106. return x
  107. if random.random() < 0.5:
  108. x = np.flip(x, axis=1).copy()
  109. if random.random() < 0.5:
  110. x = np.flip(x, axis=0).copy()
  111. if random.random() < 0.2:
  112. noise = np.random.normal(0, 0.02, x.shape).astype(np.float32)
  113. x = np.clip(x + noise, 0.0, 1.0)
  114. return x
  116. def __getitem__(self, idx: int) -> Dict[str, torch.Tensor]:
  117. rec = self.records[idx]
  118. fundus = self._augment(load_rgb(rec.fundus, self.image_size))
  119. octa = self._augment(load_rgb(rec.octa, self.image_size))
  120. fa = self._augment(load_rgb(rec.fa, self.image_size))
  122. return {
  123. "fundus": torch.from_numpy(fundus).permute(2, 0, 1),
  124. "octa": torch.from_numpy(octa).permute(2, 0, 1),
  125. "fa": torch.from_numpy(fa).permute(2, 0, 1),
  126. "label": torch.tensor(rec.label, dtype=torch.long),
  127. "site": rec.site,
  128. }
  131. # -----------------------------------------------------------------------------
  132. # Phase 1: Local Vascular Extraction (LVE)
  133. # -----------------------------------------------------------------------------
  135. class ConvBlock(nn.Module):
  136. def __init__(self, in_ch: int, out_ch: int):
  137. super().__init__()
  138. self.block = nn.Sequential(
  139. nn.Conv2d(in_ch, out_ch, kernel_size=3, stride=1, padding=1),
  140. nn.BatchNorm2d(out_ch),
  141. nn.ReLU(inplace=True),
  142. nn.Conv2d(out_ch, out_ch, kernel_size=3, stride=1, padding=1),
  143. nn.BatchNorm2d(out_ch),
  144. nn.ReLU(inplace=True),
  145. )
  147. def forward(self, x: torch.Tensor) -> torch.Tensor:
  148. return self.block(x)
  151. class LVEBranch(nn.Module):
  152. def __init__(self, in_ch: int = 3, widths: Sequence[int] = (32, 64, 128, 256), emb_dim: int = 256):
  153. super().__init__()
  154. self.blocks = nn.ModuleList()
  155. prev = in_ch
  156. for w in widths:
  157. self.blocks.append(ConvBlock(prev, w))
  158. prev = w
  159. self.pool = nn.MaxPool2d(2)
  160. self.project = nn.Sequential(
  161. nn.Linear(sum(widths), emb_dim),
  162. nn.ReLU(inplace=True),
  163. nn.Linear(emb_dim, emb_dim),
  164. )
  166. def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, List[torch.Tensor]]:
  167. pyramids: List[torch.Tensor] = []
  168. h = x
  169. pooled_feats = []
  170. for i, block in enumerate(self.blocks):
  171. h = block(h)
  172. pyramids.append(h)
  173. pooled_feats.append(F.adaptive_avg_pool2d(h, 1).flatten(1))
  174. if i != len(self.blocks) - 1:
  175. h = self.pool(h)
  176. local_vec = self.project(torch.cat(pooled_feats, dim=1))
  177. return local_vec, pyramids
  180. class LVEEncoder(nn.Module):
  181. def __init__(self, emb_dim: int = 256):
  182. super().__init__()
  183. self.fundus_branch = LVEBranch(3, emb_dim=emb_dim)
  184. self.octa_branch = LVEBranch(3, emb_dim=emb_dim)
  185. self.fa_branch = LVEBranch(3, emb_dim=emb_dim)
  186. self.cross_modal = nn.Sequential(
  187. nn.Linear(emb_dim * 3, emb_dim * 2),
  188. nn.ReLU(inplace=True),
  189. nn.Linear(emb_dim * 2, emb_dim),
  190. )
  192. def forward(self, fundus: torch.Tensor, octa: torch.Tensor, fa: torch.Tensor) -> Dict[str, torch.Tensor | List[torch.Tensor]]:
  193. f_vec, f_maps = self.fundus_branch(fundus)
  194. o_vec, o_maps = self.octa_branch(octa)
  195. a_vec, a_maps = self.fa_branch(fa)
  197. hadamard_fo = f_vec * o_vec
  198. hadamard_fa = f_vec * a_vec
  199. hadamard_oa = o_vec * a_vec
  200. fused = self.cross_modal(torch.cat([hadamard_fo, hadamard_fa, hadamard_oa], dim=1))
  201. return {
  202. "local_vec": fused,
  203. "fundus_vec": f_vec,
  204. "octa_vec": o_vec,
  205. "fa_vec": a_vec,
  206. "fundus_maps": f_maps,
  207. "octa_maps": o_maps,
  208. "fa_maps": a_maps,
  209. }
  212. # -----------------------------------------------------------------------------
  213. # Phase 2: Global Transformer-based Encoding (GTE)
  214. # -----------------------------------------------------------------------------
  216. class PatchEmbed(nn.Module):
  217. def __init__(self, img_size: int = 224, patch_size: int = 16, in_chans: int = 3, embed_dim: int = 384):
  218. super().__init__()
  219. self.grid_size = img_size // patch_size
  220. self.num_patches = self.grid_size * self.grid_size
  221. self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
  223. def forward(self, x: torch.Tensor) -> torch.Tensor:
  224. x = self.proj(x) # [B, D, H', W']
  225. return x.flatten(2).transpose(1, 2) # [B, N, D]
  228. class MHCA(nn.Module):
  229. """Multi-head correlative attention."""
  230. def __init__(self, dim: int, heads: int = 8, dropout: float = 0.1):
  231. super().__init__()
  232. self.heads = heads
  233. self.scale = (dim // heads) ** -0.5
  234. self.qkv = nn.Linear(dim, dim * 3)
  235. self.corr = nn.Parameter(torch.zeros(heads, 1, 1))
  236. self.proj = nn.Linear(dim, dim)
  237. self.drop = nn.Dropout(dropout)
  239. def forward(self, x: torch.Tensor) -> torch.Tensor:
  240. b, n, c = x.shape
  241. qkv = self.qkv(x).reshape(b, n, 3, self.heads, c // self.heads).permute(2, 0, 3, 1, 4)
  242. q, k, v = qkv[0], qkv[1], qkv[2]
  243. sim = (q @ k.transpose(-2, -1)) * self.scale
  244. div = -torch.abs(q.unsqueeze(-2) - k.unsqueeze(-3)).mean(dim=-1)
  245. attn = sim + self.corr + 0.1 * div
  246. attn = attn.softmax(dim=-1)
  247. attn = self.drop(attn)
  248. out = (attn @ v).transpose(1, 2).reshape(b, n, c)
  249. return self.proj(out)
  252. class TransformerBlock(nn.Module):
  253. def __init__(self, dim: int, heads: int = 8, mlp_ratio: float = 4.0, dropout: float = 0.1):
  254. super().__init__()
  255. self.norm1 = nn.LayerNorm(dim)
  256. self.attn = MHCA(dim, heads=heads, dropout=dropout)
  257. self.norm2 = nn.LayerNorm(dim)
  258. self.mlp = nn.Sequential(
  259. nn.Linear(dim, int(dim * mlp_ratio)),
  260. nn.GELU(),
  261. nn.Dropout(dropout),
  262. nn.Linear(int(dim * mlp_ratio), dim),
  263. nn.Dropout(dropout),
  264. )
  266. def forward(self, x: torch.Tensor) -> torch.Tensor:
  267. x = x + self.attn(self.norm1(x))
  268. x = x + self.mlp(self.norm2(x))
  269. return x
  272. class GTEEncoder(nn.Module):
  273. def __init__(self, img_size: int = 224, patch_size: int = 16, dim: int = 384, depth: int = 4, heads: int = 8):
  274. super().__init__()
  275. self.patch_embed = PatchEmbed(img_size, patch_size, 9, dim)
  276. num_patches = self.patch_embed.num_patches
  277. self.cls_token = nn.Parameter(torch.zeros(1, 1, dim))
  278. self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, dim))
  279. self.blocks = nn.ModuleList([TransformerBlock(dim, heads=heads) for _ in range(depth)])
  280. self.norm = nn.LayerNorm(dim)
  282. def forward(self, fused_modal_image: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
  283. x = self.patch_embed(fused_modal_image)
  284. b = x.shape[0]
  285. cls = self.cls_token.expand(b, -1, -1)
  286. x = torch.cat([cls, x], dim=1) + self.pos_embed
  287. for block in self.blocks:
  288. x = block(x)
  289. x = self.norm(x)
  290. return x[:, 0], x[:, 1:]
  293. # -----------------------------------------------------------------------------
  294. # Phase 3: Graph-based Convolutional Attention Network (G-CAN)
  295. # -----------------------------------------------------------------------------
  297. class GraphAttentionLayer(nn.Module):
  298. def __init__(self, in_dim: int, out_dim: int, dropout: float = 0.2, alpha: float = 0.2):
  299. super().__init__()
  300. self.lin = nn.Linear(in_dim, out_dim, bias=False)
  301. self.attn = nn.Linear(2 * out_dim, 1, bias=False)
  302. self.dropout = nn.Dropout(dropout)
  303. self.leaky_relu = nn.LeakyReLU(alpha)
  305. def forward(self, x: torch.Tensor, adj: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
  306. h = self.lin(x)
  307. b, n, d = h.shape
  308. h_i = h.unsqueeze(2).expand(b, n, n, d)
  309. h_j = h.unsqueeze(1).expand(b, n, n, d)
  310. e = self.leaky_relu(self.attn(torch.cat([h_i, h_j], dim=-1)).squeeze(-1))
  311. e = e.masked_fill(adj <= 0, float("-inf"))
  312. alpha = F.softmax(e, dim=-1)
  313. alpha = self.dropout(alpha)
  314. out = alpha @ h
  315. return F.elu(out), alpha
  318. class GCANEncoder(nn.Module):
  319. def __init__(self, node_dim: int, hidden_dim: int = 128, num_layers: int = 3, dropout: float = 0.2):
  320. super().__init__()
  321. dims = [node_dim] + [hidden_dim] * num_layers
  322. self.layers = nn.ModuleList([
  323. GraphAttentionLayer(dims[i], dims[i + 1], dropout=dropout) for i in range(num_layers)
  324. ])
  325. self.out_proj = nn.Linear(hidden_dim + 1, hidden_dim)
  327. @staticmethod
  328. def build_graph(node_feats: torch.Tensor, coords: torch.Tensor, sigma: float = 0.35) -> torch.Tensor:
  329. # coords: [B, N, 2], normalized to [0,1]
  330. dists = torch.cdist(coords, coords, p=2)
  331. adj = torch.exp(-(dists ** 2) / (2 * sigma * sigma))
  332. eye = torch.eye(adj.size(-1), device=adj.device).unsqueeze(0)
  333. adj = torch.maximum(adj, eye)
  334. return adj
  336. def forward(self, node_feats: torch.Tensor, coords: torch.Tensor) -> Tuple[torch.Tensor, Dict[str, torch.Tensor]]:
  337. adj = self.build_graph(node_feats, coords)
  338. h = node_feats
  339. attn_maps = []
  340. for layer in self.layers:
  341. h, alpha = layer(h, adj)
  342. attn_maps.append(alpha)
  343. predicted_adj = torch.sigmoid(torch.matmul(F.normalize(h, dim=-1), F.normalize(h, dim=-1).transpose(-1, -2)))
  344. anomaly = torch.mean(torch.abs(adj - predicted_adj), dim=(-1, -2), keepdim=True)
  345. graph_repr = h.mean(dim=1)
  346. graph_repr = self.out_proj(torch.cat([graph_repr, anomaly], dim=-1))
  347. return graph_repr, {
  348. "adj": adj,
  349. "predicted_adj": predicted_adj,
  350. "anomaly": anomaly,
  351. "attn_maps": attn_maps[-1],
  352. }
  355. # -----------------------------------------------------------------------------
  356. # Phase 4: Local-Global Attention Fusion (LGAF)
  357. # -----------------------------------------------------------------------------
  359. class LGAF(nn.Module):
  360. def __init__(self, local_dim: int, global_dim: int, graph_dim: int, hidden_dim: int = 512, num_classes: int = 2):
  361. super().__init__()
  362. self.local_proj = nn.Linear(local_dim, hidden_dim)
  363. self.global_proj = nn.Linear(global_dim, hidden_dim)
  364. self.graph_proj = nn.Linear(graph_dim, hidden_dim)
  365. self.gate = nn.Sequential(
  366. nn.Linear(hidden_dim * 2, hidden_dim),
  367. nn.ReLU(inplace=True),
  368. nn.Linear(hidden_dim, hidden_dim),
  369. nn.Sigmoid(),
  370. )
  371. self.attn = nn.MultiheadAttention(hidden_dim, num_heads=4, batch_first=True)
  372. self.norm = nn.LayerNorm(hidden_dim)
  373. self.classifier = nn.Sequential(
  374. nn.Linear(hidden_dim, hidden_dim // 2),
  375. nn.ReLU(inplace=True),
  376. nn.Dropout(0.2),
  377. nn.Linear(hidden_dim // 2, num_classes),
  378. )
  380. def forward(self, local_vec: torch.Tensor, global_vec: torch.Tensor, graph_vec: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
  381. l = self.local_proj(local_vec)
  382. g = self.global_proj(global_vec)
  383. r = self.graph_proj(graph_vec)
  384. gate = self.gate(torch.cat([l, g], dim=-1))
  385. fused = gate * l + (1.0 - gate) * g
  386. tokens = torch.stack([fused, r], dim=1)
  387. attended, _ = self.attn(tokens, tokens, tokens)
  388. fused = self.norm(attended.mean(dim=1) + fused + 0.5 * r)
  389. logits = self.classifier(fused)
  390. return logits, fused
  393. # -----------------------------------------------------------------------------
  394. # Full RNV-T model
  395. # -----------------------------------------------------------------------------
  397. class RNVT(nn.Module):
  398. def __init__(self, image_size: int = 224, local_dim: int = 256, global_dim: int = 384, graph_dim: int = 128, num_classes: int = 2):
  399. super().__init__()
  400. self.lve = LVEEncoder(emb_dim=local_dim)
  401. self.gte = GTEEncoder(img_size=image_size, patch_size=16, dim=global_dim, depth=4, heads=8)
  402. self.gcan = GCANEncoder(node_dim=global_dim, hidden_dim=graph_dim, num_layers=3, dropout=0.2)
  403. self.lgaf = LGAF(local_dim=local_dim, global_dim=global_dim, graph_dim=graph_dim, hidden_dim=512, num_classes=num_classes)
  404. self.proj_ssl = nn.Sequential(
  405. nn.Linear(512, 256), nn.ReLU(inplace=True), nn.Linear(256, 128)
  406. )
  408. @staticmethod
  409. def _token_coords(num_tokens: int, device: torch.device) -> torch.Tensor:
  410. side = int(math.sqrt(num_tokens))
  411. ys, xs = torch.meshgrid(
  412. torch.linspace(0, 1, side, device=device),
  413. torch.linspace(0, 1, side, device=device),
  414. indexing="ij",
  415. )
  416. coords = torch.stack([xs.reshape(-1), ys.reshape(-1)], dim=-1)
  417. return coords
  419. def forward(self, fundus: torch.Tensor, octa: torch.Tensor, fa: torch.Tensor) -> Dict[str, torch.Tensor]:
  420. lve_out = self.lve(fundus, octa, fa)
  421. fused_modal_image = torch.cat([fundus, octa, fa], dim=1)
  422. global_vec, patch_tokens = self.gte(fused_modal_image)
  424. b, n, d = patch_tokens.shape
  425. coords = self._token_coords(n, patch_tokens.device).unsqueeze(0).repeat(b, 1, 1)
  426. graph_vec, graph_aux = self.gcan(patch_tokens, coords)
  427. logits, fused_repr = self.lgaf(lve_out["local_vec"], global_vec, graph_vec)
  429. return {
  430. "logits": logits,
  431. "local_vec": lve_out["local_vec"],
  432. "global_vec": global_vec,
  433. "graph_vec": graph_vec,
  434. "fused_repr": fused_repr,
  435. "ssl_proj": F.normalize(self.proj_ssl(fused_repr), dim=-1),
  436. "graph_anomaly": graph_aux["anomaly"].squeeze(-1),
  437. "adj": graph_aux["adj"],
  438. "predicted_adj": graph_aux["predicted_adj"],
  439. }
  442. # -----------------------------------------------------------------------------
  443. # Phase 5: losses and optimization
  444. # -----------------------------------------------------------------------------
  446. def info_nce(z1: torch.Tensor, z2: torch.Tensor, temperature: float = 0.2) -> torch.Tensor:
  447. b = z1.size(0)
  448. z = torch.cat([z1, z2], dim=0)
  449. sim = torch.mm(z, z.t()) / temperature
  450. mask = torch.eye(2 * b, device=z.device, dtype=torch.bool)
  451. sim = sim.masked_fill(mask, -1e9)
  452. targets = torch.arange(b, device=z.device)
  453. targets = torch.cat([targets + b, targets], dim=0)
  454. return F.cross_entropy(sim, targets)
  457. def graph_regularization(adj: torch.Tensor, pred_adj: torch.Tensor) -> torch.Tensor:
  458. return F.mse_loss(pred_adj, adj)
  461. def total_loss_fn(outputs_1: Dict[str, torch.Tensor], outputs_2: Dict[str, torch.Tensor], labels: torch.Tensor,
  462. lambda_ssl: float = 0.2, lambda_reg: float = 0.1) -> Tuple[torch.Tensor, Dict[str, float]]:
  463. ce = F.cross_entropy(outputs_1["logits"], labels)
  464. ssl = info_nce(outputs_1["ssl_proj"], outputs_2["ssl_proj"])
  465. reg = graph_regularization(outputs_1["adj"], outputs_1["predicted_adj"])
  466. total = ce + lambda_ssl * ssl + lambda_reg * reg
  467. return total, {
  468. "ce": float(ce.detach().cpu()),
  469. "ssl": float(ssl.detach().cpu()),
  470. "reg": float(reg.detach().cpu()),
  471. "total": float(total.detach().cpu()),
  472. }
  475. # -----------------------------------------------------------------------------
  476. # Utilities for federated-style training
  477. # -----------------------------------------------------------------------------
  479. def clone_state_dict(model: nn.Module) -> Dict[str, torch.Tensor]:
  480. return {k: v.detach().cpu().clone() for k, v in model.state_dict().items()}
  483. def load_state_dict_strict(model: nn.Module, state: Dict[str, torch.Tensor]) -> None:
  484. model.load_state_dict(state, strict=True)
  487. def fedavg(states: List[Dict[str, torch.Tensor]]) -> Dict[str, torch.Tensor]:
  488. avg = {}
  489. for key in states[0].keys():
  490. avg[key] = sum(s[key] for s in states) / len(states)
  491. return avg
  494. def accuracy_from_logits(logits: torch.Tensor, labels: torch.Tensor) -> float:
  495. preds = logits.argmax(dim=1)
  496. return float((preds == labels).float().mean().item())
  499. def sensitivity_specificity(logits: torch.Tensor, labels: torch.Tensor) -> Tuple[float, float]:
  500. preds = logits.argmax(dim=1)
  501. tp = ((preds == 1) & (labels == 1)).sum().item()
  502. tn = ((preds == 0) & (labels == 0)).sum().item()
  503. fp = ((preds == 1) & (labels == 0)).sum().item()
  504. fn = ((preds == 0) & (labels == 1)).sum().item()
  505. sensitivity = tp / max(tp + fn, 1)
  506. specificity = tn / max(tn + fp, 1)
  507. return sensitivity, specificity
  510. # -----------------------------------------------------------------------------
  511. # Demo generation for validation when real data is unavailable
  512. # -----------------------------------------------------------------------------
  514. def make_multimodal_dataset(root: str, n: int = 48, image_size: int = 224) -> List[SampleRecord]:
  515. os.makedirs(root, exist_ok=True)
  516. records: List[SampleRecord] = []
  517. for i in range(n):
  518. label = i % 2
  519. site = f"site_{i % 3}"
  520. paths = {}
  521. for modality in ["fundus", "octa", "fa"]:
  522. canvas = np.zeros((image_size, image_size, 3), dtype=np.uint8)
  523. # background tone
  524. canvas[:] = (20 + 15 * label, 30 + 5 * (i % 5), 20 + 10 * (i % 3))
  525. # vessel-like line patterns
  526. center = (image_size // 2, image_size // 2)
  527. num_lines = 18 + 6 * label
  528. thickness = 1 + label
  529. for a in np.linspace(0, 2 * np.pi, num_lines, endpoint=False):
  530. r = image_size // 2 - 8
  531. end = (int(center[0] + r * np.cos(a)), int(center[1] + r * np.sin(a)))
  532. cv2.line(canvas, center, end, (180, 180, 180), thickness)
  533. if modality == "octa" and label == 1:
  534. cv2.circle(canvas, center, 26, (10, 10, 10), -1) # FAZ enlargement cue
  535. if modality == "fa" and label == 1:
  536. cv2.circle(canvas, (image_size // 3, image_size // 3), 14, (230, 230, 230), -1)
  537. if modality == "fundus" and label == 1:
  538. cv2.circle(canvas, (2 * image_size // 3, image_size // 2), 10, (220, 50, 50), -1)
  539. p = os.path.join(root, f"{modality}_{i}.png")
  540. cv2.imwrite(p, cv2.cvtColor(canvas, cv2.COLOR_RGB2BGR))
  541. paths[modality] = p
  542. records.append(SampleRecord(paths["fundus"], paths["octa"], paths["fa"], label, site))
  543. return records
  546. # -----------------------------------------------------------------------------
  547. # Training / evaluation
  548. # -----------------------------------------------------------------------------
  550. def build_site_loaders(records: Sequence[SampleRecord], image_size: int, batch_size: int) -> Dict[str, DataLoader]:
  551. sites: Dict[str, List[SampleRecord]] = {}
  552. for rec in records:
  553. sites.setdefault(rec.site, []).append(rec)
  554. loaders = {}
  555. for site, site_records in sites.items():
  556. ds = MultiModalRetinalDataset(site_records, image_size=image_size, augment=True)
  557. loaders[site] = DataLoader(ds, batch_size=batch_size, shuffle=True, num_workers=0)
  558. return loaders
  561. def make_augmented_views(batch: Dict[str, torch.Tensor]) -> Tuple[Dict[str, torch.Tensor], Dict[str, torch.Tensor]]:
  562. def jitter(x: torch.Tensor) -> torch.Tensor:
  563. noise = 0.03 * torch.randn_like(x)
  564. return torch.clamp(x + noise, 0.0, 1.0)
  565. v1 = {k: jitter(v) if k in ["fundus", "octa", "fa"] else v for k, v in batch.items()}
  566. v2 = {k: jitter(v) if k in ["fundus", "octa", "fa"] else v for k, v in batch.items()}
  567. return v1, v2
  570. def run_local_epoch(model: RNVT, loader: DataLoader, device: torch.device, lr: float = 1e-4) -> Tuple[Dict[str, torch.Tensor], Dict[str, float]]:
  571. model.train()
  572. optimizer = torch.optim.Adam(model.parameters(), lr=lr)
  573. meters = {"loss": 0.0, "acc": 0.0, "count": 0}
  574. for batch in loader:
  575. batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}
  576. v1, v2 = make_augmented_views(batch)
  578. out1 = model(v1["fundus"], v1["octa"], v1["fa"])
  579. out2 = model(v2["fundus"], v2["octa"], v2["fa"])
  580. loss, _ = total_loss_fn(out1, out2, batch["label"])
  582. optimizer.zero_grad()
  583. loss.backward()
  584. optimizer.step()
  586. meters["loss"] += loss.item() * batch["label"].size(0)
  587. meters["acc"] += accuracy_from_logits(out1["logits"], batch["label"]) * batch["label"].size(0)
  588. meters["count"] += batch["label"].size(0)
  589. stats = {
  590. "loss": meters["loss"] / max(meters["count"], 1),
  591. "acc": meters["acc"] / max(meters["count"], 1),
  592. }
  593. return clone_state_dict(model), stats
  596. @torch.no_grad()
  597. def evaluate(model: RNVT, loader: DataLoader, device: torch.device) -> Dict[str, float]:
  598. model.eval()
  599. all_logits, all_labels = [], []
  600. for batch in loader:
  601. fundus = batch["fundus"].to(device)
  602. octa = batch["octa"].to(device)
  603. fa = batch["fa"].to(device)
  604. labels = batch["label"].to(device)
  605. out = model(fundus, octa, fa)
  606. all_logits.append(out["logits"])
  607. all_labels.append(labels)
  608. logits = torch.cat(all_logits, dim=0)
  609. labels = torch.cat(all_labels, dim=0)
  610. acc = accuracy_from_logits(logits, labels)
  611. sens, spec = sensitivity_specificity(logits, labels)
  612. return {"accuracy": acc, "sensitivity": sens, "specificity": spec}
  615. def federated_train(records: Sequence[SampleRecord], image_size: int = 224, batch_size: int = 8, rounds: int = 3,
  616. lr: float = 1e-4, device: str = "cpu") -> Dict[str, float]:
  617. device_t = torch.device(device)
  618. loaders = build_site_loaders(records, image_size=image_size, batch_size=batch_size)
  619. global_model = RNVT(image_size=image_size).to(device_t)
  620. global_state = clone_state_dict(global_model)
  622. for rnd in range(1, rounds + 1):
  623. local_states = []
  624. print(f"[Federated Round {rnd}/{rounds}]")
  625. for site, loader in loaders.items():
  626. local_model = RNVT(image_size=image_size).to(device_t)
  627. load_state_dict_strict(local_model, global_state)
  628. state, stats = run_local_epoch(local_model, loader, device_t, lr=lr)
  629. local_states.append(state)
  630. print(f" Site={site:<8} loss={stats['loss']:.4f} acc={stats['acc']:.4f}")
  631. global_state = fedavg(local_states)
  632. load_state_dict_strict(global_model, global_state)
  634. # Evaluate on pooled dataset
  635. eval_loader = DataLoader(MultiModalRetinalDataset(records, image_size=image_size, augment=False),
  636. batch_size=batch_size, shuffle=False, num_workers=0)
  637. metrics = evaluate(global_model, eval_loader, device_t)
  638. print("[Final Metrics]", metrics)
  639. return metrics
  642. # -----------------------------------------------------------------------------
  643. # CLI entry
  644. # -----------------------------------------------------------------------------
  646. def parse_args() -> argparse.Namespace:
  647. parser = argparse.ArgumentParser(description="RNV-T empirical validation code")
  648. parser.add_argument("--device", default="cpu", help="cpu or cuda")
  649. parser.add_argument("--image-size", type=int, default=224)
  650. parser.add_argument("--batch-size", type=int, default=8)
  651. parser.add_argument("--rounds", type=int, default=3)
  652. parser.add_argument("--lr", type=float, default=1e-4)
  653. parser.add_argument("--seed", type=int, default=42)
  654. parser.add_argument("--demo", action="store_true", help="Run on generated multimodal data")
  655. parser.add_argument("--root", default="./_rnvt_data")
  656. return parser.parse_args()
  659. def main() -> None:
  660. args = parse_args()
  661. seed_everything(args.seed)
  663. if args._demo:
  664. records = make_multimodal_dataset(args.root, n=48, image_size=args.image_size)
  665. metrics = federated_train(
  666. records,
  667. image_size=args.image_size,
  668. batch_size=args.batch_size,
  669. rounds=args.rounds,
  670. lr=args.lr,
  671. device=args.device,
  672. )
  673. print("empirical validation completed.")
  674. print(metrics)
  675. else:
  676. print(
  677. "This reference implementation is ready, but real dataset CSV / file paths were not provided.\n"
  678. "Use --demo for a runnable demonstration, or adapt the SampleRecord loader to your datasets."
  679. )
  682. if __name__ == "__main__":
  683. main()

rnv_t_empirical_validation.py at commit 284c4a2, under GPL-3.0 · at the source

Overview

Authors: S. Nandhini1, K. Vanitha1
  1. Department of Computer Science and Engineering, Faculty of Engineering, Karpagam Academy of Higher Education,Coimbatore, Tamil Nadu India
Institutions: Karpagam Academy of Higher Education (India)
Journal: Scientific reports, volume 16, issue 1, article 17579
Dates: received 26 June 2025; accepted 8 April 2026; published online 15 April 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s41598-026-48421-6 · PMID 41986421 · PMCID PMC13243560 · OpenAlex W7154448337
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: human (organism), stroke (population)
Methods: Machine learning
Keywords: Neural, Transformers, CSVD, Neuro-ophthalmic, Blood vessels, Eye diseases, Cognitive neuroscience, Diseases of the nervous system
MeSH: Cerebral Small Vessel Diseases*, Deep Learning*, Image Processing, Computer-Assisted*, Retina*, Convolutional Neural Networks, Early Diagnosis, Humans, Retinal Vessels (* major topic)
Topic: Retinal Imaging and Analysis (Radiology, Nuclear Medicine and Imaging, Medicine), according to OpenAlex
Citations: not cited yet (Europe PMC); 38 references in the paper

Abstract

The primary driver or cause of cognitive decline and stroke is Cerebral Small Vessel Disease (CSVD), which currently requires neuroimaging tests, which are expensive to obtain and inaccessible in standard clinical settings. Low-cost retinal imaging techniques offer non-invasive assessments that mirror the condition of the brain’s small blood vessels (cerebral microvasculature). State-of-the-art diagnostic methods currently have no accessible, non-invasive, or cost-effective solution to identify CSVD at its earliest stages through visual assessment of retinal biomarkers. This study presents the Retino-Neuro Vision Transformer (RNV-T) framework as a proposed method to detect CSVD utilizing multimodality retinal imaging. The model system comprises five fundamental phases, beginning with Local Vascular Extraction (LVE), followed by Global Transformer-based Encoding (GTE), then proceeding to Graph-Based Relational Learning through Graph-based Convolutional Attention Network (G-CAN) before implementing Local-Global Attention Fusion (LGAF) as well as Optimized training procedures to obtain precise micro-vascular abnormality detection. The diagnostic performance of this model reaches 98.8% accuracy and shows 97.4% sensitivity along with 98.1% specificity, surpassing previous detection approaches. This diagnostic system represents a major leap forward in neuro-ophthalmic care because it enables early prediction of CSVD while expanding medical accessibility through retinal scans that are easy to conduct.

Supplementary Information: The online version contains supplementary material available at 10.1038/s41598-026-48421-6.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

Nandhiniphd07/RNVT

License: GPL-3.0
State: the link answers, verified on 29 September 2026
Evidence: files inventoried
Commit: 284c4a276ec4237aefc0175c44e011d1410cd611, 21 March 2026
Languages: Python (1)
Size: 3 files, 1 script
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file
Not found: CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (1 file), OpenCV (1 file), PyTorch (1 file)
Availability: 1 check, the latest on 29 September 2026: the link answers
  • 29 September 2026: the link answers
3 files
At the source: github.com/Nandhiniphd07/RNVT.git

Code availability

The custom code used for the implementation and empirical validation of the proposed method is publicly available at https://github.com/Nandhiniphd07/RNVT.git. The version corresponding to this study is archived and can be accessed for reproducibility purposes. The code is provided for research and academic purposes and is compatible with standard Python-based deep learning environments.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 1 script, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 29 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 2 authors, 8 keywords, 8 MeSH terms, 30 references.

Cite

This paper

Nandhini, S., & Vanitha, K. (2026). Deep learning for early detection of cerebral small vessel disease using self-supervised graph embeddings and retinal image analysis. Scientific reports, 16(1), 17579. https://doi.org/10.1038/s41598-026-48421-6

BibTeX

@article{nandhini2026deep,
author = {Nandhini, S. and Vanitha, K.},
title = {{Deep learning for early detection of cerebral small vessel disease using self-supervised graph embeddings and retinal image analysis}},
journal = {Scientific reports},
year = {2026},
month = apr,
volume = {16},
number = {1},
pages = {17579},
publisher = {Nature Publishing Group},
issn = {2045-2322},
doi = {10.1038/s41598-026-48421-6},
url = {https://doi.org/10.1038/s41598-026-48421-6},
pmid = {41986421},
pmcid = {PMC13243560}
}

RIS

TY - JOUR
AU - Nandhini, S.
AU - Vanitha, K.
TI - Deep learning for early detection of cerebral small vessel disease using self-supervised graph embeddings and retinal image analysis
T2 - Scientific reports
J2 - Sci Rep
PY - 2026
DA - 2026/04/15
VL - 16
IS - 1
SP - 17579
SN - 2045-2322
PB - Nature Publishing Group
DO - 10.1038/s41598-026-48421-6
UR - https://doi.org/10.1038/s41598-026-48421-6
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41598-026-48421-6",
"type": "article-journal",
"title": "Deep learning for early detection of cerebral small vessel disease using self-supervised graph embeddings and retinal image analysis",
"container-title": "Scientific reports",
"author": [
{
"family": "Nandhini",
"given": "S."
},
{
"family": "Vanitha",
"given": "K."
}
],
"container-title-short": "Sci Rep",
"volume": "16",
"issue": "1",
"page": "17579",
"DOI": "10.1038/s41598-026-48421-6",
"PMID": "41986421",
"PMCID": "PMC13243560",
"ISSN": "2045-2322",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41598-026-48421-6",
"language": "en",
"issued": {
"date-parts": [
[
2026,
4,
15
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1371/journal.pone.0347867 [code]
LiteFeatNet: A parameter-efficient and performance-centric deep learning model for multi-ocular disease identification using intermediate feature reduction from fundus images.
Journal: PloS one
In common: OpenCV, PyTorch, NumPy, 1 reference
[2] doi:10.3390/diagnostics16172861 [code]
Microstructural Changes in the Corpus Callosum in Different Forms of Sporadic Age-Related Cerebral Small Vessel Disease.
Journal: Diagnostics (Basel, Switzerland)
In common: OpenCV, NumPy, stroke, 1 reference
[3] doi:10.1038/s41598-026-41137-7 [code]
Deep optimization-guided hybrid neural network for accurate detection and segmentation of white matter hyperintensities in clinical MRI images.
Journal: Scientific reports
In common: OpenCV, PyTorch, NumPy, stroke
[4] doi:10.1038/s41598-026-43798-w [code]
A Machine learning pipeline to investigate tissue ingrowth in cerebral aneurysms using preclinical animal models.
Journal: Scientific reports
In common: OpenCV, PyTorch, NumPy, stroke
[5] doi:10.1038/s41598-026-50020-4
nnLoGoNet: a hybrid local-global network for retinal vessel segmentation with Skeleton Recall Loss.
Journal: Scientific reports
In common: 2 references
[6] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: OpenCV, PyTorch, NumPy
[7] doi:10.1007/s12021-026-09815-z [code]
BasNet: Attention U-Net-Based Automated Axon Segmentation in Bielschowsky Silver-Stained Histology.
Journal: Neuroinformatics
In common: OpenCV, PyTorch, NumPy
[8] doi:10.1038/s41467-026-76956-9 [code]
Innervated human cardiac muscle model reveals sympathetic drivers of KCNH2-associated arrhythmias.
Journal: Nature communications
In common: OpenCV, PyTorch, NumPy
[9] doi:10.1038/s41598-026-61605-4 [code]
Learning precise segmentation of neurofibrillary tangles from rapid manual point annotations.
Journal: Scientific reports
In common: OpenCV, PyTorch, NumPy
[10] doi:10.1038/s41467-026-76837-1 [code]
Drug screen and machine learning predict neuroprotective agents in a preclinical human model of childhood dementia.
Journal: Nature communications
In common: OpenCV, PyTorch, NumPy

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.