OSCR

Concept2Brain: an AI model for predicting neurophysiological responses to text and pictures.

Code ↔ Paper

8 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 8 matches
  1. [1] § Methods › The conditioned variational autoencoder model for compressing and predicting electrophysiological data ↔ Concept2Brain_image_datarecovery.py, lines 24–139 · score 0.86 · reparameterization trick, bottleneck layer, standard deviation, latent vector, variational, decoded
  2. [2] § Methods › Training of the cVAE model to abstract electrophysiological features ↔ Concept2Brain_training_script.py, lines 342–374 · score 0.80 · Kullback Leibler, Squared Error, reconstruction loss, divergence, MSE, VAE
  3. [3] § Methods › Training of the cVAE model to abstract electrophysiological features ↔ Concept2Brain_training_script.py, lines 213–340 · score 0.79 · convolutional layers, RELU, kernel, padding, stride, flattened
  4. [4] § Methods › The conditioned variational autoencoder model for compressing and predicting electrophysiological data ↔ Concept2Brain_training_script.py, lines 213–340 · score 0.79 · reparameterization trick, bottleneck layer, latent vector, variational, decoded, encodes
  5. [5] § Methods › Training of the cVAE model to abstract electrophysiological features ↔ Concept2Brain_image_datarecovery.py, lines 24–139 · score 0.69 · RELU, kernel, padding, stride, flattened, concatenated
  6. [6] § Methods › Training of the cVAE model to abstract electrophysiological features ↔ Concept2Brain_training_script.py, lines 342–374 · score 0.67 · KL loss, reconstruction loss, summed, annealing, sigmoid, MSE
  7. [7] § Methods › Inference of the conceptual space and training of the cross-domain network ↔ Concept2Brain_training_script.py, lines 877–915 · score 0.53 · neural network, ELU, latent space, activation, matching, linear
  8. [8] § Methods › The cross-domain bridge between the concept-level and electrophysiological latent spaces ↔ Concept2Brain_training_script.py, lines 877–915 · score 0.51 · latent vector, neural network, linear, layer, mapping, space

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,260 lines · 56 KB · GPL-3.0 · 6 matches

  1. # -*- coding: utf-8 -*-
  2. """
  3. Concept2Brain script
  5. Code for processing EEG data, extracting image features with CLIP,
  6. training a Conditional Variational Autoencoder (CVAE) on EEG data,
  7. and training a neural network to map CLIP features to the CVAE's latent space.
  8. """
  10. # =============================================================================
  11. # Section 1: Imports and Helper Functions
  12. # =============================================================================
  13. # Description: Imports necessary libraries and defines utility functions.
  15. import matplotlib.pyplot as plt
  16. import torch
  17. import torch.nn as nn
  18. import torch.optim as optim
  19. from torch.utils.data import DataLoader, TensorDataset
  20. import clip
  21. from PIL import Image
  22. import numpy as np
  23. import pandas as pd
  24. import random
  25. import scipy.io as sio
  26. from scipy.stats import shapiro
  27. import gc # Garbage Collector to explicitly free memory if needed
  28. import os # To check if files exist (e.g., for loading best model)
  30. # Function to set a seed for reproducibility
  31. def set_seed(seed):
  32. """Sets the seed for PyTorch (CPU and CUDA), NumPy, and random."""
  33. torch.manual_seed(seed)
  34. if torch.cuda.is_available():
  35. torch.cuda.manual_seed(seed)
  36. torch.cuda.manual_seed_all(seed) # if using multi-GPU.
  37. np.random.seed(seed)
  38. random.seed(seed)
  39. # These settings help reproducibility on CUDA
  40. torch.backends.cudnn.benchmark = False
  41. torch.backends.cudnn.deterministic = True
  43. # =============================================================================
  44. # Section 2: Initial Configuration and Environment Setup
  45. # =============================================================================
  46. # Description: Sets the reproducibility seed and configures the device (CPU/GPU).
  48. # Set seed for reproducibility
  49. set_seed(42)
  51. # Configure device to use GPU if available, otherwise CPU
  52. device = "cuda" if torch.cuda.is_available() else "cpu"
  53. print(f"Using device: {device}")
  55. # =============================================================================
  56. # Section 3: Image Feature Extraction with CLIP
  57. # =============================================================================
  58. # Description: Loads the CLIP model, processes images from a specific directory,
  59. # extracts their features (embeddings), and saves them to a .npy file.
  61. print("\n--- Starting CLIP feature extraction ---")
  62. # Load Excel file with image information
  63. # Ensure this path is correct for your system
  64. file_path = '../PictureLabels.xlsx'
  65. excel = pd.read_excel(file_path)
  67. # Load the CLIP model and image preprocessing function
  68. # Using ViT-B/32 model and moving it to the configured device (GPU/CPU)
  69. model_clip, preprocess_clip = clip.load("ViT-B/32", device=device)
  71. # Path to the folder containing the images
  72. # Ensure this path is correct for your system
  73. folder_path = '../Pictures_Version12'
  74. # Initialize array to store features (360 images, 512 features per image)
  75. image_features = np.zeros((360, 512))
  77. # Iterate over the images specified in the Excel file
  78. for i in range(360):
  79. pic_filename = f"{excel.iloc[i, 1]}.jpg"
  80. pic2analyze = f"{folder_path}/{pic_filename}"
  81. # print(f"Processing: {pic2analyze}") # Uncomment if debugging needed
  82. try:
  83. # Open, preprocess the image, and move it to the device
  84. image = preprocess_clip(Image.open(pic2analyze)).unsqueeze(0).to(device)
  85. # Extract image features using CLIP model without gradient calculation
  86. with torch.no_grad():
  87. features = model_clip.encode_image(image)
  88. image_features[i, :] = features.cpu().numpy() # Move features to CPU and store
  89. except FileNotFoundError:
  90. print(f"Warning: File not found {pic2analyze}. Skipping.")
  91. except Exception as e:
  92. print(f"Error processing {pic2analyze}: {e}")
  94. # Save the extracted features to a NumPy file
  95. np.save('image_features.npy', image_features)
  96. print("Image features saved to image_features.npy")
  97. # Free CLIP model memory if possible (optional)
  98. del model_clip
  99. del preprocess_clip
  100. if torch.cuda.is_available():
  101. torch.cuda.empty_cache()
  103. # =============================================================================
  104. # Section 4: EEG Data Loading and Preprocessing
  105. # =============================================================================
  106. # Description: Loads EEG data, labels, and subject information from .mat files.
  107. # Performs normalization on the EEG data.
  109. print("\n--- Loading and preprocessing EEG data ---")
  110. # Base path where scripts and data are located
  111. # Ensure this path is correct for your system
  112. path = './' # Example path, adjust as needed
  114. # Load Clean voltage EEG data (raw) from .mat files and concatenate them
  115. # All trials (> 10k) were divided for python loading reasons into 3 parts
  116. # Ensure these file paths are correct
  117. mat_contents = sio.loadmat(path + 'DATA_EEG_PARTS.ma.mat')
  118. DATA_EEG = np.concatenate((
  119. np.asarray(mat_contents['DATA_RAW_ALL_PART1']),
  120. np.asarray(mat_contents['DATA_RAW_ALL_PART2']),
  121. np.asarray(mat_contents['DATA_RAW_ALL_PART3'])
  122. ), axis=2)
  124. print(f"Shape of loaded EEG data: {DATA_EEG.shape}") # E.g., (Electrodes, Timepoints, Trials)
  126. # Load image labels (numbers corresponding to rows in Excel/CLIP features)
  127. # Ensure this file path is correct
  128. tmp = sio.loadmat(path + 'LABEL_ALL_MyAPS12.mat')
  129. LABELS = tmp['PICTURE_XCEL_NUM'].T
  130. print(f"Shape of loaded labels: {LABELS.shape}") # E.g., (Trials, 1)
  132. # Load subject identifiers
  133. # Ensure this file path is correct
  134. tmp = sio.loadmat(path + 'SUBJECTS_ALL_MyAPS12.mat')
  135. SUBJECTS = tmp['SUBJECT_NUM'].T
  136. SUBJECTS_IDS = np.unique(SUBJECTS)
  137. print(f"Shape of loaded subject IDs: {SUBJECTS.shape}") # E.g., (Trials, 1)
  138. print(f"Number of unique subjects: {len(SUBJECTS_IDS)}")
  140. # Create condition vector (one-hot encoding for subjects)
  141. ConditionedVector = np.zeros((len(SUBJECTS), len(SUBJECTS_IDS)))
  142. for i in range(len(SUBJECTS)):
  143. ConditionedVector[i, SUBJECTS_IDS == SUBJECTS[i]] = 1
  144. print(f"Shape of condition vector (subjects one-hot): {ConditionedVector.shape}") # E.g., (Trials, NumSubjects)
  146. # Z-score normalization of EEG data based on the baseline (first 50 timepoints)
  147. print("Normalizing EEG data...")
  148. # Calculate mean and standard deviation of the baseline period.
  149. # It corresponds with first 50 time points, i.e 100 ms, for each channel and trial
  150. meanData = DATA_EEG[:, :50, :].mean(axis=1, keepdims=True)
  151. stdData = DATA_EEG[:, :50, :].std(axis=1, keepdims=True)
  152. # Avoid division by zero if standard deviation is very small
  153. stdData[stdData < 1e-6] = 1e-6
  154. # Apply Z-score normalization
  155. DATA_EEG_NORM = (DATA_EEG - meanData) / stdData
  156. print("Normalization completed.")
  158. # Free up memory
  159. del mat_contents, tmp, meanData, stdData, DATA_EEG
  160. gc.collect()
  162. # =============================================================================
  163. # Section 5: Data Splitting into Training and Testing Sets
  164. # =============================================================================
  165. # Description: Randomly permutes the data and splits it into training (80%)
  166. # and testing (20%) sets.
  168. print("\n--- Splitting data into training and testing sets ---")
  169. # Generate randomly permuted indices
  170. num_trials = DATA_EEG_NORM.shape[2]
  171. newInds = np.random.permutation(num_trials)
  173. # Calculate the split index for 80/20 division
  174. th_traintest = int(num_trials * 0.8)
  176. # Split the normalized EEG data
  177. # Shape: (Electrodes, Timepoints, Trials) -> (E, T, N_train/N_test)
  178. X_train = DATA_EEG_NORM[:, :, newInds[:th_traintest]]
  179. X_test = DATA_EEG_NORM[:, :, newInds[th_traintest:]]
  181. # Split labels, condition vectors, and subject IDs
  182. # Shape: (Trials, Features) -> (N_train/N_test, Features)
  183. y_train = LABELS[newInds[:th_traintest]]
  184. y_test = LABELS[newInds[th_traintest:]]
  185. c_train = ConditionedVector[newInds[:th_traintest]]
  186. c_test = ConditionedVector[newInds[th_traintest:]]
  187. s_train = SUBJECTS[newInds[:th_traintest]]
  188. s_test = SUBJECTS[newInds[th_traintest:]]
  190. print(f"Training set size: {X_train.shape[2]} trials")
  191. print(f"Testing set size: {X_test.shape[2]} trials")
  193. # Free up memory
  194. del DATA_EEG_NORM, LABELS, ConditionedVector, SUBJECTS, newInds
  195. gc.collect()
  197. # =============================================================================
  198. # Section 6: Conditional Variational Autoencoder (CVAE) Definition
  199. # =============================================================================
  200. # Description: Defines the CVAE architecture using PyTorch, including the
  201. # encoder, decoder, and loss function. The model is designed for multi-GPU use.
  203. print("\n--- Defining the CVAE model ---")
  205. # Function to free GPU memory (if needed)
  206. def free_gpu_memory():
  207. """Clears the GPU memory cache."""
  208. if torch.cuda.is_available():
  209. # print("Clearing GPU cache...") # Can be verbose, uncomment if needed
  210. torch.cuda.empty_cache()
  211. gc.collect()
  213. # Definition of the conditional CVAE structure
  214. class ConditionalVAE(nn.Module):
  215. """
  216. Conditional Variational Autoencoder (CVAE) with convolutional layers.
  217. Designed to operate across multiple GPUs:
  218. - Encoder on GPU 0
  219. - Bottleneck layers (mu, logvar) on GPU 1
  220. - Decoder on GPU 2
  221. Adjusts to the main device if fewer than 3 GPUs are available.
  222. """
  223. def __init__(self, input_channels, condition_dim, latent_dim):
  224. super(ConditionalVAE, self).__init__()
  225. self.condition_dim = condition_dim
  226. self.latent_dim = latent_dim
  228. # Check availability of sufficient GPUs
  229. num_gpus = torch.cuda.device_count()
  230. if num_gpus < 3:
  231. print(f"Warning: Found {num_gpus} GPUs, but the model is designed for 3. Adjusting to the main device: {device}")
  232. self.encoder_device = torch.device(device)
  233. self.bottleneck_device = torch.device(device)
  234. self.decoder_device = torch.device(device)
  235. else:
  236. self.encoder_device = torch.device("cuda:0")
  237. self.bottleneck_device = torch.device("cuda:1")
  238. self.decoder_device = torch.device("cuda:2")
  240. # --- Encoder (GPU 0) ---
  241. # Input shape: (N, input_channels, 129, 750) [N, C, H, W]
  242. self.conv1 = nn.Conv2d(input_channels, 32, kernel_size=3, stride=2, padding=1).to(self.encoder_device) # Output: (N, 32, 65, 375)
  243. self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1).to(self.encoder_device) # Output: (N, 64, 33, 188)
  244. self.conv3 = nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1).to(self.encoder_device) # Output: (N, 128, 17, 94)
  245. # Flattened size after conv3: 128 * 17 * 94 = 204224
  247. # --- Bottleneck Layers (GPU 1) ---
  248. self.flattened_size = 128 * 17 * 94
  249. self.fc_mu = nn.Linear(self.flattened_size + condition_dim, latent_dim).to(self.bottleneck_device)
  250. self.fc_logvar = nn.Linear(self.flattened_size + condition_dim, latent_dim).to(self.bottleneck_device)
  252. # --- Decoder (GPU 2) ---
  253. self.fc_decode = nn.Linear(latent_dim + condition_dim, self.flattened_size).to(self.decoder_device)
  254. # Input to deconv1: (N, 128, 17, 94)
  255. self.deconv1 = nn.ConvTranspose2d(128, 64, kernel_size=3, stride=2, padding=1, output_padding=1).to(self.decoder_device) # Output: (N, 64, 33, 188) approx
  256. self.deconv2 = nn.ConvTranspose2d(64, 32, kernel_size=3, stride=2, padding=1, output_padding=1).to(self.decoder_device) # Output: (N, 32, 65, 375) approx
  257. self.deconv3 = nn.ConvTranspose2d(32, input_channels, kernel_size=3, stride=2, padding=1, output_padding=1).to(self.decoder_device) # Output: (N, C, 129, 750) approx
  258. # The output size might slightly differ due to padding/stride, will be cropped.
  260. def encode(self, x, c):
  261. """Encodes input x conditioned by c into mu and logvar."""
  262. x = x.to(self.encoder_device)
  263. # No need to move c here if it's already on the correct device before calling encode
  264. # c = c.to(self.encoder_device) # Will be moved in the main forward pass if needed
  266. h1 = torch.relu(self.conv1(x))
  267. h2 = torch.relu(self.conv2(h1))
  268. h3 = torch.relu(self.conv3(h2)) # Using relu here is also common
  270. # Flatten and move to the bottleneck GPU
  271. h3_flat = h3.view(h3.size(0), -1).to(self.bottleneck_device)
  272. # Move condition c to the bottleneck GPU as well
  273. c_bottleneck = c.to(self.bottleneck_device)
  275. # Concatenate the flattened representation with the condition
  276. combined_bottleneck = torch.cat([h3_flat, c_bottleneck], dim=1)
  278. # Calculate mu and logvar
  279. mu = self.fc_mu(combined_bottleneck)
  280. logvar = self.fc_logvar(combined_bottleneck)
  281. return mu, logvar
  283. def reparameterize(self, mu, logvar):
  284. """Performs the reparameterization trick."""
  285. # Ensure tensors are on the correct device (bottleneck)
  286. mu = mu.to(self.bottleneck_device)
  287. logvar = logvar.to(self.bottleneck_device)
  288. std = torch.exp(0.5 * logvar)
  289. eps = torch.randn_like(std) # Creates tensor on the same device as std
  290. # Optional clamping for numerical stability
  291. # std = torch.clamp(std, min=1e-6, max=10) # Example
  292. return mu + eps * std
  294. def decode(self, z, c):
  295. """Decodes latent vector z conditioned by c into reconstruction."""
  296. # Move z and c to the decoder GPU
  297. z = z.to(self.decoder_device)
  298. c_decoder = c.to(self.decoder_device)
  300. # Concatenate latent vector and condition
  301. combined_decoder_input = torch.cat([z, c_decoder], dim=1)
  303. # Pass through the dense layer and reshape
  304. h6 = torch.relu(self.fc_decode(combined_decoder_input))
  305. # Reshape to the expected shape for the first deconvolutional layer (N, C, H, W)
  306. h6_reshaped = h6.view(h6.size(0), 128, 17, 94) # Use post-conv3 dimensions
  308. # Pass through the deconvolutional layers
  309. h9 = torch.relu(self.deconv1(h6_reshaped))
  310. h10 = torch.relu(self.deconv2(h9))
  311. # Final layer might not have activation (depends on data domain, e.g., sigmoid for [0,1])
  312. x_recon_raw = self.deconv3(h10)
  314. # Crop the output to the original input size (129 electrodes, 750 time points)
  315. # Original input dimensions were (N, 1, 129, 750)
  316. original_height, original_width = 129, 750
  317. x_recon = x_recon_raw[:, :, :original_height, :original_width]
  318. return x_recon
  320. def forward(self, x, c):
  321. """Complete forward pass of the CVAE."""
  322. # Move initial condition c to the encoder device
  323. c_enc = c.to(self.encoder_device)
  324. x_enc = x.to(self.encoder_device)
  326. # Encode
  327. mu, logvar = self.encode(x_enc, c_enc)
  329. # Reparameterize (occurs on bottleneck_device)
  330. z = self.reparameterize(mu, logvar)
  332. # Decode (requires z and c on decoder_device)
  333. c_dec = c.to(self.decoder_device)
  334. x_recon = self.decode(z, c_dec)
  336. # Ensure mu and logvar are on the correct device for loss calculation (bottleneck device)
  337. mu_final = mu.to(self.bottleneck_device)
  338. logvar_final = logvar.to(self.bottleneck_device)
  340. return x_recon, mu_final, logvar_final
  342. # VAE Loss Function
  343. def vae_loss(x_recon, x, mu, logvar, epoch, beta_factor=0.00001, annealing_midpoint=100, annealing_steepness=0.1):
  344. """
  345. Calculates the VAE loss (Reconstruction + KL Divergence).
  346. Includes a beta factor with sigmoidal annealing for the KL divergence.
  347. Assumes inputs (mu, logvar) determine the device for calculation.
  348. """
  349. # Move all tensors to the same device for loss calculation
  350. # Assuming loss is calculated on the bottleneck GPU (where mu/logvar reside)
  351. loss_device = mu.device
  352. x_recon = x_recon.to(loss_device)
  353. x = x.to(loss_device)
  354. # mu and logvar should already be on loss_device
  356. # 1. Reconstruction Loss (Mean Squared Error)
  357. # Sum over pixels/timepoints, then average over the batch
  358. recon_loss = nn.MSELoss(reduction='sum')(x_recon, x) / x.size(0)
  360. # 2. Kullback-Leibler (KL) Divergence
  361. # KL divergence between the latent distribution q(z|x,c) and the prior p(z)=N(0,I)
  362. # Summed over latent dimensions, averaged over the batch
  363. kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp(), dim=1)
  364. kl_loss = torch.mean(kl_loss) # Average over the batch
  366. # 3. Beta Factor with Sigmoidal Annealing
  367. # Beta starts near 0 and grows towards beta_factor
  368. beta = beta_factor / (1 + np.exp(-annealing_steepness * (epoch - annealing_midpoint)))
  370. # 4. Total Loss
  371. total_loss = recon_loss + beta * kl_loss
  373. # Return total loss and KL component (for monitoring)
  374. return total_loss, kl_loss
  377. # =============================================================================
  378. # Section 7: CVAE Training Preparation
  379. # =============================================================================
  380. # Description: Converts data to PyTorch tensors, creates DataLoaders for batch
  381. # handling, instantiates the CVAE model, and defines the optimizer.
  383. print("\n--- Preparing CVAE training ---")
  385. # Convert NumPy data to PyTorch tensors
  386. # Permute dimensions to be (N, C, H, W) where N=trials, C=channels(1), H=electrodes, W=time
  387. # Add channel dimension (unsqueeze(1))
  388. X_train_torch = torch.tensor(X_train, dtype=torch.float32).permute(2, 0, 1).unsqueeze(1)
  389. X_test_torch = torch.tensor(X_test, dtype=torch.float32).permute(2, 0, 1).unsqueeze(1)
  390. c_train_torch = torch.tensor(c_train, dtype=torch.float32)
  391. c_test_torch = torch.tensor(c_test, dtype=torch.float32)
  393. print(f"Shape of X_train_torch: {X_train_torch.shape}") # E.g., (N_train, 1, E, T)
  394. print(f"Shape of c_train_torch: {c_train_torch.shape}") # E.g., (N_train, NumSubjects)
  396. # Create Datasets and DataLoaders
  397. batch_size_cvae = 64
  398. train_dataset = TensorDataset(X_train_torch, c_train_torch)
  399. # Use multiple workers and pin memory for potentially faster data loading
  400. train_loader = DataLoader(train_dataset, batch_size=batch_size_cvae, shuffle=True, num_workers=4, pin_memory=True)
  402. test_dataset = TensorDataset(X_test_torch, c_test_torch)
  403. test_loader = DataLoader(test_dataset, batch_size=batch_size_cvae, shuffle=False, num_workers=4, pin_memory=True)
  405. print(f"CVAE Batch size: {batch_size_cvae}")
  406. print(f"Number of training batches: {len(train_loader)}")
  407. print(f"Number of testing batches: {len(test_loader)}")
  409. # Instantiate the CVAE model
  410. input_channels = 1 # EEG is single-channel (in this context)
  411. latent_dim = 256 # Latent space dimension
  412. condition_dim = c_train.shape[1] # Number of subjects (dimension of one-hot vector)
  414. # Instantiate the model. Internal modules will be moved to designated GPUs.
  415. model_cvae = ConditionalVAE(input_channels, condition_dim, latent_dim)
  416. # The model object itself resides on the CPU, but its parameters are registered correctly across GPUs.
  418. # Define the optimizer
  419. # Adam is a common choice
  420. learning_rate_cvae = 1e-6 # Very low learning rate for precise tunin
  421. optimizer_cvae = optim.Adam(model_cvae.parameters(), lr=learning_rate_cvae)
  422. print(f"CVAE Optimizer: Adam, Learning Rate: {learning_rate_cvae}")
  424. # Lists to store loss history
  425. train_loss_all_cvae = []
  426. val_loss_all_cvae = []
  427. kl_loss_all_cvae = [] # To monitor KL divergence
  429. # =============================================================================
  430. # Section 8: CVAE Training Loop
  431. # =============================================================================
  432. # Description: Performs the training and validation cycle for the CVAE over a
  433. # defined number of epochs. Logs training progress to a file.
  435. print("\n--- Starting CVAE training ---")
  436. epochs_cvae = 1300 # Number of epochs for training the cVAE
  438. # Annealing parameters for VAE loss (consistent with original settings)
  439. beta_factor_cvae = 0.00001 # Final beta value for KL term weight
  440. annealing_midpoint_cvae = 100 # Epoch where beta reaches half its final value
  441. annealing_steepness_cvae = 0.1 # Controls how quickly beta increases
  443. # Load state if exists (e.g., if training was interrupted)
  444. # This would require saving/loading model state_dict, optimizer state_dict, and loss lists.
  445. # For simplicity, starting from scratch here as in the original script.
  446. start_epoch = 0 # Start from epoch 0
  448. # Open log file to append results
  449. log_filename = 'training_log_cvae.txt'
  450. with open(log_filename, 'a') as f:
  451. f.write("\n--- New CVAE Training Session ---\n")
  452. f.write(f"Epochs: {epochs_cvae}, Batch Size: {batch_size_cvae}, LR: {learning_rate_cvae}, Latent Dim: {latent_dim}\n")
  453. f.write(f"Beta Factor: {beta_factor_cvae}, Annealing Midpoint: {annealing_midpoint_cvae}, Steepness: {annealing_steepness_cvae}\n")
  455. for epoch in range(start_epoch, epochs_cvae):
  456. # --- Training Phase ---
  457. model_cvae.train() # Set model to training mode
  458. train_loss_epoch = 0
  459. kl_loss_epoch_train = 0
  461. for batch_idx, (data, condition) in enumerate(train_loader):
  462. # Data and conditions will be moved to the correct devices inside the model
  463. # Move initial input to the encoder device
  464. data = data.to(model_cvae.encoder_device, non_blocking=True)
  465. condition = condition.to(model_cvae.encoder_device, non_blocking=True)
  467. # Reset optimizer gradients
  468. optimizer_cvae.zero_grad()
  470. # Forward pass: get reconstruction, mu, and logvar
  471. x_recon, mu, logvar = model_cvae(data, condition)
  473. # Calculate loss (occurs on bottleneck_device)
  474. # Move original 'data' to that device for comparison
  475. data_loss_device = data.to(mu.device, non_blocking=True)
  476. loss, kl_component = vae_loss(x_recon, data_loss_device, mu, logvar, epoch,
  477. beta_factor=beta_factor_cvae,
  478. annealing_midpoint=annealing_midpoint_cvae,
  479. annealing_steepness=annealing_steepness_cvae)
  481. # Backward pass: compute gradients
  482. loss.backward()
  484. # Update model parameters
  485. optimizer_cvae.step()
  487. # Accumulate epoch losses
  488. train_loss_epoch += loss.item()
  489. kl_loss_epoch_train += kl_component.item()
  491. # Calculate average training loss for the epoch (per sample)
  492. train_loss_avg = train_loss_epoch / len(train_loader.dataset)
  493. kl_loss_avg_train = kl_loss_epoch_train / len(train_loader) # KL average per batch
  494. train_loss_all_cvae.append(train_loss_avg)
  497. # --- Validation Phase ---
  498. model_cvae.eval() # Set model to evaluation mode
  499. val_loss_epoch = 0
  500. kl_loss_epoch_val = 0
  501. mu_all_val = [] # To collect mu values for Shapiro test
  503. with torch.no_grad(): # Disable gradient calculations during validation
  504. for batch_idx, (data, condition) in enumerate(test_loader):
  505. data = data.to(model_cvae.encoder_device, non_blocking=True)
  506. condition = condition.to(model_cvae.encoder_device, non_blocking=True)
  508. x_recon, mu, logvar = model_cvae(data, condition)
  510. data_loss_device = data.to(mu.device, non_blocking=True)
  511. loss, kl_component = vae_loss(x_recon, data_loss_device, mu, logvar, epoch,
  512. beta_factor=beta_factor_cvae,
  513. annealing_midpoint=annealing_midpoint_cvae,
  514. annealing_steepness=annealing_steepness_cvae)
  516. val_loss_epoch += loss.item()
  517. kl_loss_epoch_val += kl_component.item()
  518. mu_all_val.append(mu.cpu().numpy()) # Store mu for Shapiro test (move to CPU)
  520. # Calculate average validation loss for the epoch (per sample)
  521. val_loss_avg = val_loss_epoch / len(test_loader.dataset)
  522. kl_loss_avg_val = kl_loss_epoch_val / len(test_loader) # KL average per batch
  523. val_loss_all_cvae.append(val_loss_avg)
  524. kl_loss_all_cvae.append(kl_loss_avg_val) # Store average validation KL
  526. # Shapiro-Wilk normality test on 'mu' values from the validation set
  527. # Concatenate mus from all batches and flatten for Shapiro test
  528. mu_val_np = np.concatenate(mu_all_val, axis=0).flatten()
  529. shapiro_stat, shapiro_p = -1, -1 # Default values
  530. if len(mu_val_np) >= 3: # Shapiro requires at least 3 samples
  531. try:
  532. shapiro_stat, shapiro_p = shapiro(mu_val_np)
  533. except ValueError:
  534. print(f"Warning: Could not compute Shapiro test for epoch {epoch+1} (possibly zero variance).")
  537. # Print progress and save to log
  538. log_message = (
  539. f"Epoch [{epoch + 1}/{epochs_cvae}], "
  540. f"Train Loss: {train_loss_avg:.4f}, Val Loss: {val_loss_avg:.4f}, "
  541. f"KL Val: {kl_loss_avg_val:.3f}, Shapiro p: {shapiro_p:.3g}, "
  542. # Use last batch mu/logvar for min/max/std stats (representative)
  543. f"Mu Val (min/max/std): {mu.min():.3f}/{mu.max():.3f}/{mu.std():.3f}, "
  544. f"LogVar Val (min/max/std): {logvar.min():.3f}/{logvar.max():.3f}/{logvar.std():.3f}"
  545. )
  546. print(log_message)
  547. with open(log_filename, 'a') as f:
  548. f.write(log_message + "\n")
  550. # Free GPU memory at the end of each epoch (optional, but can help)
  551. # free_gpu_memory() # Can slow down training if called too often
  553. print("CVAE training completed.")
  555. # Save the trained model (optional but recommended)
  556. # Save state_dict to CPU for easier loading in different environments
  557. model_cvae_save_path = "MODEL_CCVAE_AnnelingBeta00001_01-100sigmoid_lr1e-6_1300epc_dict.pth"
  558. # Ensure the model's parameters are moved to CPU before saving state_dict
  559. # Or save directly, but specify map_location='cpu' when loading if needed
  560. # torch.save(model_cvae.cpu().state_dict(), model_cvae_save_path)
  561. # print(f"CVAE model state dictionary saved to {model_cvae_save_path}")
  562. # Note: Multi-GPU models might require careful handling for saving/loading. Saving state_dict is generally safer.
  565. # =============================================================================
  566. # Section 9: CVAE Post-Training Visualization and Analysis
  567. # =============================================================================
  568. # Description: Plots loss curves, generates reconstruction examples, and performs
  569. # visual analysis to evaluate the trained CVAE's performance.
  571. print("\n--- CVAE Post-Training Analysis ---")
  573. # 1. Plot training and validation loss curves
  574. plt.figure(figsize=(10, 5))
  575. plt.plot(train_loss_all_cvae, label='Train Loss (Total)')
  576. plt.plot(val_loss_all_cvae, label='Validation Loss (Total)')
  577. # Could also plot KL loss if desired
  578. # plt.plot(kl_loss_all_cvae, label='Validation KL Loss (Avg per Batch)')
  579. plt.xlabel('Epoch')
  580. plt.ylabel('Loss (Per Sample)')
  581. plt.title('CVAE Loss Curves during Training')
  582. plt.legend()
  583. plt.grid(True)
  584. plt.savefig('cvae_loss_curve.png')
  585. # plt.show() # Uncomment if running interactively
  587. # 2. Generate and visualize a reconstruction example from the test set
  588. model_cvae.eval() # Ensure the model is in evaluation mode
  589. with torch.no_grad():
  590. # Take a sample from the test set
  591. idx_sample = 0 # Use the first sample
  592. x_sample = X_test_torch[idx_sample].unsqueeze(0) # Add batch dimension
  593. c_sample = c_test_torch[idx_sample].unsqueeze(0)
  595. # Move sample to the appropriate initial device (encoder)
  596. x_sample_dev = x_sample.to(model_cvae.encoder_device)
  597. c_sample_dev = c_sample.to(model_cvae.encoder_device)
  599. # Generate reconstruction
  600. x_recon_sample, mu_sample, logvar_sample = model_cvae(x_sample_dev, c_sample_dev)
  602. # Move results to CPU for visualization
  603. x_original_np = x_sample.squeeze().cpu().numpy() # Remove batch/channel, move to CPU
  604. x_reconstructed_np = x_recon_sample.squeeze().cpu().numpy() # Remove batch/channel, move to CPU
  605. mu_sample_np = mu_sample.cpu().numpy()
  606. logvar_sample_np = logvar_sample.cpu().numpy()
  608. # Visualize original vs. reconstructed (as image and time series)
  609. plt.figure(figsize=(15, 10))
  611. # Original image
  612. plt.subplot(2, 3, 1)
  613. plt.imshow(x_original_np, aspect='auto', cmap='viridis')
  614. plt.title(f'Original EEG (Test Sample {idx_sample})')
  615. plt.xlabel('Time (points)')
  616. plt.ylabel('Electrode')
  617. plt.colorbar()
  619. # Reconstructed image
  620. plt.subplot(2, 3, 2)
  621. plt.imshow(x_reconstructed_np, aspect='auto', cmap='viridis')
  622. plt.title('CVAE Reconstruction')
  623. plt.xlabel('Time (points)')
  624. plt.ylabel('Electrode')
  625. plt.colorbar()
  627. # Difference image
  628. plt.subplot(2, 3, 3)
  629. diff = x_original_np - x_reconstructed_np
  630. vmax = np.max(np.abs(diff))
  631. plt.imshow(diff, aspect='auto', cmap='coolwarm', vmin=-vmax, vmax=vmax)
  632. plt.title('Difference (Original - Reconstructed)')
  633. plt.xlabel('Time (points)')
  634. plt.ylabel('Electrode')
  635. plt.colorbar()
  638. # Original time series (all electrodes overlaid)
  639. plt.subplot(2, 3, 4)
  640. plt.plot(x_original_np.T) # Transpose so time is the x-axis
  641. plt.title('Original EEG (Time Series)')
  642. plt.xlabel('Time (points)')
  643. plt.ylabel('Normalized Amplitude')
  645. # Reconstructed time series
  646. plt.subplot(2, 3, 5)
  647. plt.plot(x_reconstructed_np.T)
  648. plt.title('CVAE Reconstruction (Time Series)')
  649. plt.xlabel('Time (points)')
  650. plt.ylabel('Normalized Amplitude')
  652. # Latent space (mu and logvar) for this sample
  653. plt.subplot(2, 3, 6)
  654. plt.plot(mu_sample_np.flatten(), label=f'Mu (std={mu_sample_np.std():.2f})')
  655. plt.plot(logvar_sample_np.flatten(), label=f'LogVar (std={logvar_sample_np.std():.2f})')
  656. plt.title('Latent Space (mu, logvar)')
  657. plt.xlabel('Latent Dimension')
  658. plt.ylabel('Value')
  659. plt.legend()
  661. plt.tight_layout()
  662. plt.savefig('cvae_reconstruction_sample.png')
  663. # plt.show()
  665. # 3. Analysis of Subject Effects (requires generating multiple reconstructions)
  666. # Original code generated reconstructions for the first 3 subject conditions (c_test[0], c_test[1], c_test[2])
  667. # applied to *all* X_test inputs. This checks if the model can impose a subject's "style".
  669. print("Generating reconstructions conditioned by different subjects...")
  670. model_cvae.eval()
  671. num_test_samples = X_test_torch.shape[0]
  672. num_subjects_to_test = min(3, c_test.shape[1]) # Test with the first 3 subjects
  673. reconstructions_by_subject = {} # Dictionary: {subject_idx: [reconstructions]}
  675. with torch.no_grad():
  676. for subj_idx in range(num_subjects_to_test):
  677. # Create condition vector for this subject (one-hot)
  678. c_subject = torch.zeros(1, condition_dim)
  679. c_subject[0, subj_idx] = 1
  680. # Move the *target* condition to the decoder device
  681. c_subject_target = c_subject.to(model_cvae.decoder_device)
  683. subject_recons = []
  684. # Iterate over a subset of test samples for efficiency
  685. samples_to_process = min(num_test_samples, 100) # Limit to 100 samples
  686. print(f" Processing {samples_to_process} samples for subject {subj_idx+1} condition...")
  687. for i in range(samples_to_process):
  688. x_input = X_test_torch[i].unsqueeze(0).to(model_cvae.encoder_device)
  689. # Use the *original* condition associated with x_input for encoding
  690. c_input_original = c_test_torch[i].unsqueeze(0).to(model_cvae.encoder_device)
  692. # Encode to get mu, logvar using the original input and condition
  693. mu, logvar = model_cvae.encode(x_input, c_input_original)
  694. # Reparameterize to get z (on bottleneck device)
  695. z = model_cvae.reparameterize(mu, logvar)
  696. # Decode using the obtained z but with the *new target* subject condition
  697. x_recon_subj = model_cvae.decode(z, c_subject_target)
  698. subject_recons.append(x_recon_subj.squeeze().cpu().numpy()) # Collect reconstructions
  700. # Store all reconstructions for this target subject condition
  701. reconstructions_by_subject[subj_idx] = np.stack(subject_recons, axis=-1) # Shape: (E, T, N_samples)
  703. # Visualize the average ERP for each simulated subject condition
  704. plt.figure(figsize=(15, 5))
  705. plt.suptitle('Average Simulated ERP Conditioned by Subject (Using Test Set Inputs)')
  706. for subj_idx in range(num_subjects_to_test):
  707. # Average across the samples processed for this condition
  708. erp_mean = reconstructions_by_subject[subj_idx].mean(axis=2)
  709. plt.subplot(1, num_subjects_to_test, subj_idx + 1)
  710. plt.plot(erp_mean.T) # Plot time series
  711. plt.title(f'Simulated Subject {subj_idx + 1}')
  712. plt.xlabel('Time (points)')
  713. if subj_idx == 0:
  714. plt.ylabel('Normalized Amplitude')
  715. plt.tight_layout(rect=[0, 0.03, 1, 0.95]) # Adjust layout for suptitle
  716. plt.savefig('cvae_subject_conditioning_erp.png')
  717. # plt.show()
  719. # 4. Analysis of Emotional Condition Effects (requires mapping labels to conditions)
  720. print("Analyzing average reconstructions by emotional condition (Pleasant, Neutral, Unpleasant)...")
  721. # Define label ranges for each condition (based on original code comments)
  722. pics_pleasant = list(range(322, 362))
  723. pics_neutral = list(range(282, 322))
  724. pics_unpleasant = list(range(242, 282))
  726. # Need the *actual* reconstructions from the test set (not simulated ones)
  727. all_test_recons = []
  728. all_test_mu = [] # Also collect mu if needed later
  729. print(" Generating reconstructions for the entire test set...")
  730. with torch.no_grad():
  731. for data, condition in test_loader:
  732. data_dev = data.to(model_cvae.encoder_device)
  733. cond_dev = condition.to(model_cvae.encoder_device)
  734. x_recon, mu, _ = model_cvae(data_dev, cond_dev)
  735. all_test_recons.append(x_recon.cpu().numpy())
  736. all_test_mu.append(mu.cpu().numpy())
  738. all_test_recons_np = np.concatenate(all_test_recons, axis=0).squeeze() # Shape: (N_test, E, T)
  739. # y_test has shape (N_test, 1), need (N_test,) for isin
  740. y_test_flat = y_test.squeeze()
  742. # Find indices in the test set corresponding to each emotional condition
  743. indices_p = np.where(np.isin(y_test_flat, pics_pleasant))[0]
  744. indices_n = np.where(np.isin(y_test_flat, pics_neutral))[0]
  745. indices_u = np.where(np.isin(y_test_flat, pics_unpleasant))[0]
  747. # Calculate average ERP for each condition based on the reconstructions
  748. erp_p_recon = all_test_recons_np[indices_p].mean(axis=0) # Shape: (E, T)
  749. erp_n_recon = all_test_recons_np[indices_n].mean(axis=0)
  750. erp_u_recon = all_test_recons_np[indices_u].mean(axis=0)
  752. # Visualize average reconstructed ERPs by condition
  753. plt.figure(figsize=(18, 6))
  754. plt.suptitle('Average Reconstructed ERP by Emotional Condition')
  756. # Time series plot (all electrodes)
  757. plt.subplot(1, 2, 1)
  758. # Use different linestyles or colors if plotting all electrodes is too crowded
  759. plt.plot(erp_p_recon.T, label='Pleasant Recon', alpha=0.1, color='green')
  760. plt.plot(erp_n_recon.T, label='Neutral Recon', alpha=0.1, color='blue')
  761. plt.plot(erp_u_recon.T, label='Unpleasant Recon', alpha=0.1, color='red')
  762. # Create a custom legend for clarity
  763. from matplotlib.lines import Line2D
  764. custom_lines = [Line2D([0], [0], color='green', lw=2),
  765. Line2D([0], [0], color='blue', lw=2),
  766. Line2D([0], [0], color='red', lw=2)]
  767. plt.legend(custom_lines, ['Pleasant', 'Neutral', 'Unpleasant'])
  768. plt.title('All Reconstructed Signals')
  769. plt.xlabel('Time (points)')
  770. plt.ylabel('Normalized Amplitude')
  773. # Plot for a specific electrode (e.g., electrode 61 corresponds to Pz)
  774. plt.subplot(1, 2, 2)
  775. electrode_idx = 61 # Check if this index is valid (0 to E-1)
  776. if 0 <= electrode_idx < erp_p_recon.shape[0]:
  777. plt.plot(erp_p_recon[electrode_idx, :], label='Pleasant Recon', color='green')
  778. plt.plot(erp_n_recon[electrode_idx, :], label='Neutral Recon', color='blue')
  779. plt.plot(erp_u_recon[electrode_idx, :], label='Unpleasant Recon', color='red')
  780. plt.title(f'Reconstructed ERP - Electrode {electrode_idx+1}')
  781. plt.xlabel('Time (points)')
  782. plt.ylabel('Normalized Amplitude')
  783. plt.legend()
  784. plt.grid(True)
  785. else:
  786. plt.text(0.5, 0.5, f'Electrode index {electrode_idx} out of range [0, {erp_p_recon.shape[0]-1}]',
  787. horizontalalignment='center', verticalalignment='center')
  790. plt.tight_layout(rect=[0, 0.03, 1, 0.95])
  791. plt.savefig('cvae_condition_analysis_reconstructed.png')
  792. # plt.show()
  794. # Free up memory
  795. del X_train_torch, X_test_torch, c_train_torch, c_test_torch, train_loader, test_loader
  796. del all_test_recons, all_test_recons_np, all_test_mu
  797. if 'reconstructions_by_subject' in locals(): del reconstructions_by_subject
  798. free_gpu_memory()
  801. # =============================================================================
  802. # Section 10: Data Preparation for Latent Neural Network (CLIP -> CVAE)
  803. # =============================================================================
  804. # Description: Loads the saved CLIP features and creates corresponding datasets
  805. # to train a network mapping from CLIP's latent space to the CVAE's latent space.
  807. print("\n--- Preparing data for the Latent Neural Network (CLIP -> CVAE) ---")
  809. # Load previously saved CLIP image features
  810. try:
  811. # Ensure the path matches where features were saved
  812. image_features_loaded = np.load('image_features.npy')
  813. print(f"CLIP features loaded from image_features.npy, shape: {image_features_loaded.shape}")
  814. except FileNotFoundError:
  815. print("Error: File image_features.npy not found. Run Section 3 first.")
  816. exit() # Exit if features cannot be loaded
  818. # Create input datasets (CLIP features) for training and testing
  819. # Use y_train, y_test labels to find the correct CLIP features
  820. # Labels (y_train, y_test) are 1-based image numbers, subtract 1 for 0-based indexing
  821. CLIP_latent_train = image_features_loaded[y_train.squeeze() - 1]
  822. CLIP_latent_test = image_features_loaded[y_test.squeeze() - 1]
  824. print(f"Shape of CLIP_latent_train: {CLIP_latent_train.shape}") # (N_train, 512)
  825. print(f"Shape of CLIP_latent_test: {CLIP_latent_test.shape}") # (N_test, 512)
  827. # The 'targets' for this network are not directly the CVAE's mu/logvar.
  828. # Instead, the loss is calculated by comparing the *reconstructed EEG* generated
  829. # from the 'z' predicted by this network, with the original EEG.
  830. # Therefore, we need the original EEG data (X) and conditions (c) again.
  832. # Convert data to PyTorch tensors for the latent network
  833. input_clip_train = torch.tensor(CLIP_latent_train, dtype=torch.float32)
  834. input_clip_test = torch.tensor(CLIP_latent_test, dtype=torch.float32)
  836. # Need original EEG data (X) and conditions (c) again
  837. # Reload or reuse tensors if still in memory (reloading for clarity is safer)
  838. X_train_torch_latent = torch.tensor(X_train, dtype=torch.float32).permute(2, 0, 1).unsqueeze(1)
  839. X_test_torch_latent = torch.tensor(X_test, dtype=torch.float32).permute(2, 0, 1).unsqueeze(1)
  840. c_train_torch_latent = torch.tensor(c_train, dtype=torch.float32)
  841. c_test_torch_latent = torch.tensor(c_test, dtype=torch.float32)
  843. # Create DataLoaders for the latent network
  844. batch_size_latent = 512 # Based on original code
  845. # Dataset contains (CLIP_input, Original_EEG_output, Condition_input)
  846. latent_train_dataset = TensorDataset(input_clip_train, X_train_torch_latent, c_train_torch_latent)
  847. latent_train_loader = DataLoader(latent_train_dataset, batch_size=batch_size_latent, shuffle=True, num_workers=4, pin_memory=True)
  849. latent_test_dataset = TensorDataset(input_clip_test, X_test_torch_latent, c_test_torch_latent)
  850. latent_test_loader = DataLoader(latent_test_dataset, batch_size=batch_size_latent, shuffle=False, num_workers=4, pin_memory=True)
  852. print(f"DataLoader for LatentNN created. Batch size: {batch_size_latent}")
  853. print(f"Number of LatentNN training batches: {len(latent_train_loader)}")
  854. print(f"Number of LatentNN testing batches: {len(latent_test_loader)}")
  856. # Free memory (original X_train, X_test arrays no longer needed if tensors created)
  857. del X_train, X_test, y_train, c_train, s_train, y_test, c_test, s_test
  858. gc.collect()
  861. # =============================================================================
  862. # Section 11: Latent Neural Network (LatentNN) Definition
  863. # =============================================================================
  864. # Description: Defines the architecture of the neural network mapping CLIP
  865. # features and subject condition to the CVAE's latent space.
  867. print("\n--- Defining the Latent Neural Network (LatentNN) ---")
  869. # Network parameters (based on original code)
  870. input_features_clip = CLIP_latent_train.shape[1] # 512
  871. condition_features_subj = c_train_torch_latent.shape[1] # NumSubjects (e.g., 88)
  872. output_features_latent = latent_dim # 256 (CVAE's latent dimension)
  873. hidden_units_1 = 256
  874. hidden_units_2 = 512
  875. # Original code had commented-out layers (BatchNorm, Dropout), omitted here to match active version.
  877. class LatentNN(nn.Module):
  878. """
  879. Neural Network to map (CLIP features + Subject Condition) -> CVAE Latent Space (z).
  880. Designed to operate on a specific GPU (GPU 1 according to CVAE setup).
  881. """
  882. def __init__(self):
  883. super(LatentNN, self).__init__()
  884. # Device where this network will operate (GPU 1 if available, matching CVAE bottleneck)
  885. num_gpus = torch.cuda.device_count()
  886. if num_gpus >= 2:
  887. # Assumes GPU 1 corresponds to the CVAE bottleneck device
  888. self.device_latent = torch.device("cuda:1")
  889. else:
  890. print("Warning: Not enough GPUs for original assignment (LatentNN on GPU 1). Using main device.")
  891. self.device_latent = torch.device(device)
  893. print(f"LatentNN will operate on: {self.device_latent}")
  895. # Define layers and move them to the designated device
  896. self.fc1 = nn.Linear(input_features_clip + condition_features_subj, hidden_units_1).to(self.device_latent)
  897. self.fc2 = nn.Linear(hidden_units_1, hidden_units_2).to(self.device_latent)
  898. self.fc3 = nn.Linear(hidden_units_2, output_features_latent).to(self.device_latent)
  899. self.elu = nn.ELU(alpha=1.0) # ELU activation used in original code
  901. def forward(self, x_clip, c_subj):
  902. """Forward pass."""
  903. # Move inputs to this network's device
  904. x_clip = x_clip.to(self.device_latent)
  905. c_subj = c_subj.to(self.device_latent)
  907. # Concatenate CLIP features and subject condition
  908. xc_combined = torch.cat([x_clip, c_subj], dim=1)
  910. # Pass through layers
  911. hidden1 = self.elu(self.fc1(xc_combined))
  912. hidden2 = self.elu(self.fc2(hidden1))
  913. z_predicted = self.fc3(hidden2) # Linear output (predicting the latent vector z)
  915. return z_predicted
  917. # Instantiate the LatentNN model
  918. model_latent = LatentNN()
  919. # The model already moves its layers to the correct device in __init__
  921. # =============================================================================
  922. # Section 12: LatentNN Training Preparation
  923. # =============================================================================
  924. # Description: Defines the loss function (MSE on reconstructed EEG), the
  925. # optimizer, and sets the CVAE to evaluation mode (freezing its weights).
  927. print("\n--- Preparing LatentNN training ---")
  929. # Loss function: Mean Squared Error (MSE) between original EEG and EEG reconstructed
  930. # from the z predicted by LatentNN.
  931. criterion_latent = nn.MSELoss()
  933. # Optimizer for LatentNN
  934. learning_rate_latent = 1e-3 # Based on original code
  935. optimizer_latent = optim.Adam(model_latent.parameters(), lr=learning_rate_latent)
  936. print(f"LatentNN Optimizer: Adam, LR: {learning_rate_latent}")
  938. # Freeze CVAE weights: we don't want to train it further
  939. print("Freezing CVAE weights...")
  940. for param in model_cvae.parameters():
  941. param.requires_grad = False
  942. model_cvae.eval() # Set CVAE to evaluation mode (important if it uses Dropout/BatchNorm)
  944. # Lists to store LatentNN loss history
  945. train_loss_all_latent = []
  946. val_loss_all_latent = []
  948. # Parameters for Early Stopping (based on original code)
  949. patience = 30 # Number of epochs to wait for improvement before stopping
  950. epochs_no_improve = 0
  951. best_val_loss = float('inf')
  952. best_model_latent_path = 'MODEL_Latent_10epc_512batch_dict.pth' # File to save the best model
  953. # In original script a model fit in 10 epochs was the best found option.
  955. # =============================================================================
  956. # Section 13: LatentNN Training Loop
  957. # =============================================================================
  958. # Description: Performs the training and validation cycle for LatentNN, using
  959. # the frozen CVAE to decode the predicted 'z' and calculate loss on the EEG
  960. # reconstruction. Implements Early Stopping.
  962. print("\n--- Starting LatentNN training ---")
  963. epochs_latent = 1000 # Maximum number of epochs (may stop earlier due to Early Stopping)
  964. #Original script stopped at 10 epochs when evaluation data was not better.
  966. for epoch in range(epochs_latent):
  967. # --- Training Phase ---
  968. model_latent.train() # Set LatentNN to training mode
  969. epoch_loss_train = 0.0
  971. for batch_clip, batch_eeg_original, batch_condition in latent_train_loader:
  972. # batch_clip & batch_condition go to LatentNN device
  973. # batch_eeg_original goes to CVAE Decoder/Loss device
  974. # batch_condition also needed on Decoder device
  976. # Move condition to LatentNN device (and later to decoder device)
  977. batch_condition_latent = batch_condition.to(model_latent.device_latent, non_blocking=True)
  978. # Move clip features to LatentNN device
  979. batch_clip_latent = batch_clip.to(model_latent.device_latent, non_blocking=True)
  981. # Reset LatentNN gradients
  982. optimizer_latent.zero_grad()
  984. # 1. Predict z using LatentNN
  985. z_hat = model_latent(batch_clip_latent, batch_condition_latent)
  987. # 2. Decode z_hat using the frozen CVAE to get reconstructed EEG
  988. # Move predicted z_hat and condition to the CVAE decoder device
  989. z_hat_decoder = z_hat.to(model_cvae.decoder_device, non_blocking=True)
  990. batch_condition_decoder = batch_condition.to(model_cvae.decoder_device, non_blocking=True)
  991. X_hat_reconstructed = model_cvae.decode(z_hat_decoder, batch_condition_decoder)
  993. # 3. Calculate MSE loss between reconstruction and original EEG
  994. # Move original EEG to the device where loss is calculated (decoder device)
  995. batch_eeg_original_loss = batch_eeg_original.to(model_cvae.decoder_device, non_blocking=True)
  996. loss = criterion_latent(X_hat_reconstructed, batch_eeg_original_loss)
  998. # Backward pass (computes gradients only for LatentNN)
  999. loss.backward()
  1001. # Update LatentNN weights
  1002. optimizer_latent.step()
  1004. epoch_loss_train += loss.item()
  1006. # Calculate average training loss for the epoch (per sample)
  1007. avg_epoch_loss_train = epoch_loss_train / len(latent_train_loader.dataset)
  1008. train_loss_all_latent.append(avg_epoch_loss_train)
  1011. # --- Validation Phase ---
  1012. model_latent.eval() # Set LatentNN to evaluation mode
  1013. epoch_loss_val = 0.0
  1014. z_hat_last_batch = None # To print min/max/std stats
  1016. with torch.no_grad():
  1017. for batch_clip, batch_eeg_original, batch_condition in latent_test_loader:
  1018. batch_condition_latent = batch_condition.to(model_latent.device_latent, non_blocking=True)
  1019. batch_clip_latent = batch_clip.to(model_latent.device_latent, non_blocking=True)
  1021. # 1. Predict z
  1022. z_hat = model_latent(batch_clip_latent, batch_condition_latent)
  1023. z_hat_last_batch = z_hat # Store for stats
  1025. # 2. Decode z_hat
  1026. z_hat_decoder = z_hat.to(model_cvae.decoder_device, non_blocking=True)
  1027. batch_condition_decoder = batch_condition.to(model_cvae.decoder_device, non_blocking=True)
  1028. X_hat_reconstructed = model_cvae.decode(z_hat_decoder, batch_condition_decoder)
  1030. # 3. Calculate loss
  1031. batch_eeg_original_loss = batch_eeg_original.to(model_cvae.decoder_device, non_blocking=True)
  1032. loss = criterion_latent(X_hat_reconstructed, batch_eeg_original_loss)
  1034. epoch_loss_val += loss.item()
  1036. # Calculate average validation loss for the epoch (per sample)
  1037. avg_epoch_loss_val = epoch_loss_val / len(latent_test_loader.dataset)
  1038. val_loss_all_latent.append(avg_epoch_loss_val)
  1040. # Print progress
  1041. z_min, z_max, z_std = -1, -1, -1 # Default values
  1042. if z_hat_last_batch is not None:
  1043. z_min = z_hat_last_batch.min().item()
  1044. z_max = z_hat_last_batch.max().item()
  1045. z_std = z_hat_last_batch.std().item()
  1047. print(f"Epoch {epoch+1}/{epochs_latent}, Train Loss: {avg_epoch_loss_train:.4f}, Val Loss: {avg_epoch_loss_val:.4f}, "
  1048. f"z_hat Val (min/max/std): {z_min:.3f}/{z_max:.3f}/{z_std:.3f}")
  1050. # Early Stopping logic
  1051. if avg_epoch_loss_val < best_val_loss:
  1052. best_val_loss = avg_epoch_loss_val
  1053. epochs_no_improve = 0
  1054. # Save the best model found so far
  1055. torch.save(model_latent.state_dict(), best_model_latent_path)
  1056. print(f" Validation Loss improved. Saving model to {best_model_latent_path}")
  1057. else:
  1058. epochs_no_improve += 1
  1059. print(f" Validation Loss did not improve for {epochs_no_improve} epochs.")
  1061. if epochs_no_improve >= patience:
  1062. print(f"Early stopping triggered after epoch {epoch + 1}.")
  1063. break
  1065. # Optional memory clearing
  1066. # free_gpu_memory()
  1069. print("LatentNN training completed.")
  1070. # Load the best model saved by Early Stopping
  1071. if os.path.exists(best_model_latent_path):
  1072. print(f"Loading best model state from {best_model_latent_path}")
  1073. # Ensure map_location matches the device LatentNN is intended for
  1074. model_latent.load_state_dict(torch.load(best_model_latent_path, map_location=model_latent.device_latent))
  1075. else:
  1076. print(f"Warning: Best model file {best_model_latent_path} not found. Using the model's last state.")
  1078. # =============================================================================
  1079. # Section 14: LatentNN Post-Training Visualization and Analysis
  1080. # =============================================================================
  1081. # Description: Plots LatentNN loss curves, generates EEG from test CLIP features,
  1082. # and analyzes the results by emotional condition.
  1084. print("\n--- LatentNN Post-Training Analysis ---")
  1086. # 1. Plot LatentNN loss curves
  1087. plt.figure(figsize=(10, 5))
  1088. plt.plot(train_loss_all_latent, label='Train Loss (LatentNN)')
  1089. plt.plot(val_loss_all_latent, label='Validation Loss (LatentNN)')
  1090. plt.xlabel('Epoch')
  1091. plt.ylabel('MSE Loss (on EEG reconstruction, per sample)')
  1092. plt.title('LatentNN Loss Curves during Training')
  1093. plt.legend()
  1094. plt.grid(True)
  1095. plt.yscale('log') # Loss might decrease significantly, log scale can be useful
  1096. plt.savefig('latentnn_loss_curve.png')
  1097. # plt.show()
  1099. # 2. Generate EEG from test CLIP features using LatentNN + CVAE(decoder)
  1100. print("Generating EEG from test CLIP features...")
  1101. model_latent.eval()
  1102. model_cvae.eval()
  1104. # Process the test set (or a large subset as in original code)
  1105. num_samples_to_generate = min(2000, len(latent_test_dataset)) # Limit to 2000
  1106. generated_eegs = []
  1107. generated_latents = [] # Store the predicted 'z' vectors
  1109. # Use DataLoader or iterate manually. Manual iteration for clarity:
  1110. # Need the CLIP inputs and conditions for the subset
  1111. input_clip_test_subset = input_clip_test[:num_samples_to_generate]
  1112. c_test_torch_subset = c_test_torch_latent[:num_samples_to_generate]
  1113. # Also need the corresponding labels for analysis
  1114. y_test_subset = y_test[:num_samples_to_generate] # Assuming y_test numpy array is still available
  1116. print(f" Processing {num_samples_to_generate} test samples...")
  1117. with torch.no_grad():
  1118. # Process in batches if the subset is too large for GPU memory
  1119. batch_size_inference = 256 # Adjust based on GPU memory
  1120. for i in range(0, num_samples_to_generate, batch_size_inference):
  1121. # Prepare batch inputs
  1122. batch_clip = input_clip_test_subset[i:i+batch_size_inference].to(model_latent.device_latent)
  1123. batch_cond = c_test_torch_subset[i:i+batch_size_inference].to(model_latent.device_latent)
  1125. # 1. Predict z using LatentNN
  1126. z_hat_batch = model_latent(batch_clip, batch_cond)
  1127. generated_latents.append(z_hat_batch.cpu().numpy()) # Store predicted z
  1129. # 2. Decode z using CVAE
  1130. z_hat_dec = z_hat_batch.to(model_cvae.decoder_device)
  1131. batch_cond_dec = batch_cond.to(model_cvae.decoder_device) # Condition also needed for decoder
  1132. X_hat_batch = model_cvae.decode(z_hat_dec, batch_cond_dec)
  1133. generated_eegs.append(X_hat_batch.cpu().numpy()) # Store generated EEG
  1135. # Concatenate results from all batches
  1136. generated_eegs_np = np.concatenate(generated_eegs, axis=0).squeeze() # Shape: (N_gen, E, T)
  1137. generated_latents_np = np.concatenate(generated_latents, axis=0) # Shape: (N_gen, latent_dim)
  1139. print(f"Shape of generated EEGs: {generated_eegs_np.shape}")
  1140. print(f"Shape of generated latents (z_hat): {generated_latents_np.shape}")
  1143. # 3. Visualize the generated latent space (z_hat) by emotional condition
  1144. # Use the labels corresponding to the generated subset
  1145. y_test_subset_flat = y_test_subset.squeeze()
  1146. # Find indices within the subset for each condition
  1147. indices_p_gen = np.where(np.isin(y_test_subset_flat, pics_pleasant))[0]
  1148. indices_n_gen = np.where(np.isin(y_test_subset_flat, pics_neutral))[0]
  1149. indices_u_gen = np.where(np.isin(y_test_subset_flat, pics_unpleasant))[0]
  1151. # Group predicted latents by condition
  1152. latents_p = generated_latents_np[indices_p_gen]
  1153. latents_n = generated_latents_np[indices_n_gen]
  1154. latents_u = generated_latents_np[indices_u_gen]
  1155. # Concatenate for visualization (ensure order P, N, U for consistency)
  1156. concatenated_latents = np.concatenate((latents_p, latents_n, latents_u), axis=0)
  1158. plt.figure(figsize=(12, 6))
  1159. plt.subplot(1, 2, 1)
  1160. plt.imshow(concatenated_latents, aspect='auto', cmap='viridis')
  1161. plt.title('Generated Latent Space Z_hat (P, N, U ordered)')
  1162. plt.xlabel('Latent Dimension')
  1163. plt.ylabel('Samples (Ordered Pleasant-Neutral-Unpleasant)')
  1164. plt.colorbar()
  1166. plt.subplot(1, 2, 2)
  1167. plt.plot(latents_p.mean(axis=0), label='Pleasant (Mean Z_hat)')
  1168. plt.plot(latents_n.mean(axis=0), label='Neutral (Mean Z_hat)')
  1169. plt.plot(latents_u.mean(axis=0), label='Unpleasant (Mean Z_hat)')
  1170. plt.title('Average Predicted Latent Vector Z_hat by Condition')
  1171. plt.xlabel('Latent Dimension')
  1172. plt.ylabel('Average Value')
  1173. plt.legend()
  1174. plt.grid(True)
  1176. plt.tight_layout()
  1177. plt.savefig('latentnn_latent_space_analysis.png')
  1178. # plt.show()
  1181. # 4. Analysis of the generated EEGs by emotional condition
  1182. # Calculate average generated ERP for each condition
  1183. erp_p_gen = generated_eegs_np[indices_p_gen].mean(axis=0) # Shape: (E, T)
  1184. erp_n_gen = generated_eegs_np[indices_n_gen].mean(axis=0)
  1185. erp_u_gen = generated_eegs_np[indices_u_gen].mean(axis=0)
  1187. plt.figure(figsize=(18, 12))
  1188. plt.suptitle('Average Generated ERP (CLIP -> LatentNN -> CVAE) by Emotional Condition')
  1190. # Time series plot (all electrodes overlaid - might be messy)
  1191. plt.subplot(2, 2, 1)
  1192. plt.plot(erp_p_gen.T, alpha=0.1, color='green') # Pleasant generated
  1193. plt.plot(erp_n_gen.T, alpha=0.1, color='blue') # Neutral generated
  1194. plt.plot(erp_u_gen.T, alpha=0.1, color='red') # Unpleasant generated
  1195. # Custom legend for clarity
  1196. custom_lines_gen = [Line2D([0], [0], color='green', lw=2),
  1197. Line2D([0], [0], color='blue', lw=2),
  1198. Line2D([0], [0], color='red', lw=2)]
  1199. plt.legend(custom_lines_gen, ['Pleasant (Gen)', 'Neutral (Gen)', 'Unpleasant (Gen)'])
  1200. plt.title('Generated Signals (All Electrodes)')
  1201. plt.xlabel('Time (points)')
  1202. plt.ylabel('Normalized Amplitude')
  1205. # Image plots (topographical maps over time)
  1206. # Determine symmetric color limits based on max absolute amplitude
  1207. clim_max_gen = np.max(np.abs(np.stack([erp_p_gen, erp_n_gen, erp_u_gen])))
  1208. clim_gen = (-clim_max_gen, clim_max_gen)
  1210. plt.subplot(2, 2, 2)
  1211. plt.imshow(erp_p_gen, aspect='auto', cmap='viridis', vmin=clim_gen[0], vmax=clim_gen[1])
  1212. plt.title('Pleasant (Generated)')
  1213. plt.xlabel('Time (points)'), plt.ylabel('Electrode'), plt.colorbar()
  1215. plt.subplot(2, 2, 3)
  1216. plt.imshow(erp_n_gen, aspect='auto', cmap='viridis', vmin=clim_gen[0], vmax=clim_gen[1])
  1217. plt.title('Neutral (Generated)')
  1218. plt.xlabel('Time (points)'), plt.ylabel('Electrode'), plt.colorbar()
  1220. plt.subplot(2, 2, 4)
  1221. plt.imshow(erp_u_gen, aspect='auto', cmap='viridis', vmin=clim_gen[0], vmax=clim_gen[1])
  1222. plt.title('Unpleasant (Generated)')
  1223. plt.xlabel('Time (points)'), plt.ylabel('Electrode'), plt.colorbar()
  1225. plt.tight_layout(rect=[0, 0.03, 1, 0.95]) # Adjust layout for suptitle
  1226. plt.savefig('latentnn_condition_analysis_generated_eeg_maps.png')
  1227. # plt.show()
  1229. # Plot for the specific electrode (61) as in the original code
  1230. plt.figure(figsize=(8, 5))
  1231. electrode_idx = 61 # Ensure this is a valid index (0 to E-1)
  1232. if 0 <= electrode_idx < erp_p_gen.shape[0]:
  1233. plt.plot(erp_p_gen[electrode_idx, :], label='Pleasant (Generated)', color='green')
  1234. plt.plot(erp_n_gen[electrode_idx, :], label='Neutral (Generated)', color='blue')
  1235. plt.plot(erp_u_gen[electrode_idx, :], label='Unpleasant (Generated)', color='red')
  1236. plt.title(f'Generated EEG - Electrode {electrode_idx+1}')
  1237. plt.xlabel('Time (points)')
  1238. plt.ylabel('Normalized Amplitude')
  1239. plt.legend()
  1240. plt.grid(True)
  1241. else:
  1242. plt.text(0.5, 0.5, f'Electrode index {electrode_idx} out of range [0, {erp_p_gen.shape[0]-1}]',
  1243. horizontalalignment='center', verticalalignment='center')
  1245. plt.savefig('latentnn_condition_analysis_generated_eeg_electrode_61.png')
  1246. # plt.show()
  1249. print("\n--- Analysis completed ---")
  1251. # Optional: Explicitly clear memory at the end
  1252. del model_cvae, model_latent
  1253. del latent_train_dataset, latent_test_dataset, latent_train_loader, latent_test_loader
  1254. del generated_eegs_np, generated_latents_np
  1255. free_gpu_memory()

Concept2Brain_training_script.py at commit c7c8ae2, under GPL-3.0 · at the source

Overview

Authors: Alejandro Santos-Mayo1,2,3, Faith Gilbert1,2, Arash Mirifar1,2, Anna-Lena Tebbe1,2, Ruogu Fang4, Mingzhou Ding4, Andreas Keil1,2
ORCID iDs: Alejandro Santos-Mayo, Arash Mirifar, Andreas Keil
  1. Laboratory of Brain, Body, and Behavior, University of Florida, Gainesville, FL USA
  2. Department of Psychology, University of Florida, Gainesville, FL USA
  3. Center for Cognitive and Computational Neuroscience, Complutense University of Madrid, Madrid, Spain
  4. J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL USA
Institutions: Universidad Complutense de Madrid (Spain); University of Florida (United States)
Journal: Nature communications, volume 17, issue 1, article 8961
Dates: received 16 September 2025; accepted 7 July 2026; published online 23 July 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1038/s41467-026-75653-x · PMID 42637720 · PMCID PMC13503920 · OpenAlex W7170083615
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: EEG (modality), human (organism), cognitive (subfield)
Methods: Spectral & time-frequency, Smoothing, state filtering, decompositions, Machine learning, Preprocessing, Evoked potentials, Connectivity
Keywords: Perception, Neural encoding
MeSH: Artificial Intelligence*, Brain*, Models, Neurological*, Electroencephalography, Humans, Semantics (* major topic)
Topic: EEG and Brain-Computer Interfaces (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: NIMH NIH HHS (R01 MH112558)
Citations: not cited yet (Europe PMC); 88 references in the paper

Abstract

Evolving methods rooted in artificial intelligence (AI) offer new opportunities for linking human behavior and experience to brain function. Here, we introduce the Concept2Brain model, a deep network designed to generate synthetic electrophysiological responses to semantic/emotional information conveyed through pictures or text. Leveraging AI solutions like CLIP from OpenAI, the model generates a representation of a stimulus and maps it into an electrophysiological latent space. We demonstrate that this openly available resource generates synthetic neural responses that closely resemble those observed empirically. The Concept2Brain model is provided as a web service tool for creating open and reproducible EEG datasets by predicting brain responses to any semantic concept or picture. Beyond its practical applications, it also paves the way for AI-driven brain activity modeling, offering new possibilities for studying how the brain represents the world.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 8 matches between paragraphs and lines of code.

ASantosMayo/Concept2Brain

License: GPL-3.0
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: c7c8ae2e82d7b21275b35718efa76dc9f72fd347, 5 May 2025
Languages: Python (3)
Size: 5 files, 3 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file
Not found: CITATION.cff, environment file, tests, continuous integration, documentation
Tools: NumPy (3 files), pandas (3 files), Pillow (3 files), PyTorch (3 files), SciPy (3 files), Matplotlib (2 files), scikit-learn (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
5 files
At the source: github.com/ASantosMayo/Concept2Brain

Code availability

The complete source code for the Concept2Brain model architecture, training pipelines, and evaluation scripts is publicly available in a GitHub repository: https://github.com/ASantosMayo/Concept2Brain.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 3 scripts, each with its path and the digest of its content;
  • 8 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

  • osf:7dpgm, at OSF; found in the text, “Evaluating the Concept2Brain model”

Data availability

The empirical EEG datasets used for training and evaluating the Concept2Brain model, along with the generated synthetic neurophysiological responses underlying the main figures, have been deposited in the OSF repository: https://osf.io/7dpgm.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 7 authors, 2 keywords, 6 MeSH terms, 1 funder, 76 references.

Cite

This paper

Santos-Mayo, A., Gilbert, F., Mirifar, A., Tebbe, A.-L., Fang, R., Ding, M., & Keil, A. (2026). Concept2Brain: an AI model for predicting neurophysiological responses to text and pictures. Nature communications, 17(1), 8961. https://doi.org/10.1038/s41467-026-75653-x

BibTeX

@article{santosmayo2026concept2brain,
author = {Santos-Mayo, Alejandro and Gilbert, Faith and Mirifar, Arash and Tebbe, Anna-Lena and Fang, Ruogu and Ding, Mingzhou and Keil, Andreas},
title = {{Concept2Brain: an AI model for predicting neurophysiological responses to text and pictures}},
journal = {Nature communications},
year = {2026},
month = jul,
volume = {17},
number = {1},
pages = {8961},
publisher = {Nature Publishing Group},
issn = {2041-1723},
doi = {10.1038/s41467-026-75653-x},
url = {https://doi.org/10.1038/s41467-026-75653-x},
pmid = {42637720},
pmcid = {PMC13503920}
}

RIS

TY - JOUR
AU - Santos-Mayo, Alejandro
AU - Gilbert, Faith
AU - Mirifar, Arash
AU - Tebbe, Anna-Lena
AU - Fang, Ruogu
AU - Ding, Mingzhou
AU - Keil, Andreas
TI - Concept2Brain: an AI model for predicting neurophysiological responses to text and pictures
T2 - Nature communications
J2 - Nat Commun
PY - 2026
DA - 2026/07/23
VL - 17
IS - 1
SP - 8961
SN - 2041-1723
PB - Nature Publishing Group
DO - 10.1038/s41467-026-75653-x
UR - https://doi.org/10.1038/s41467-026-75653-x
LA - en
ER -

CSL-JSON

{
"id": "10.1038/s41467-026-75653-x",
"type": "article-journal",
"title": "Concept2Brain: an AI model for predicting neurophysiological responses to text and pictures",
"container-title": "Nature communications",
"author": [
{
"family": "Santos-Mayo",
"given": "Alejandro"
},
{
"family": "Gilbert",
"given": "Faith"
},
{
"family": "Mirifar",
"given": "Arash"
},
{
"family": "Tebbe",
"given": "Anna-Lena"
},
{
"family": "Fang",
"given": "Ruogu"
},
{
"family": "Ding",
"given": "Mingzhou"
},
{
"family": "Keil",
"given": "Andreas"
}
],
"container-title-short": "Nat Commun",
"volume": "17",
"issue": "1",
"page": "8961",
"DOI": "10.1038/s41467-026-75653-x",
"PMID": "42637720",
"PMCID": "PMC13503920",
"ISSN": "2041-1723",
"publisher": "Nature Publishing Group",
"URL": "https://doi.org/10.1038/s41467-026-75653-x",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
23
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.3758/s13415-026-01446-w
People, places, and things: The impact of scene animacy on emotional modulation of the early posterior negativity.
Journal: Cognitive, affective & behavioral neuroscience
In common: EEG, cognitive, 6 references
[2] doi:10.1371/journal.pone.0351872 [code]
Decoding visual object recognition from EEG signals.
Journal: PloS one
In common: Pillow, PyTorch, scikit-learn, 4 other tools, EEG, 3 references
[3] doi:10.1162/imag.a.1207 [code]
Investigating the temporal dynamics and modeling of mid-level feature representations in humans.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Pillow, PyTorch, scikit-learn, 4 other tools, EEG, cognitive, 2 references
[4] doi:10.1038/s41467-026-71267-5 [code]
Human-like cognitive generalization for large models via mental representation-guided supervision.
Journal: Nature communications
In common: Pillow, PyTorch, scikit-learn, 4 other tools, cognitive, 2 references
[5] doi:10.7554/elife.105081 [code]
Movie reconstruction from mouse visual cortex activity.
Journal: eLife
In common: Pillow, PyTorch, pandas, 3 other tools, 3 references
[6] doi:10.1038/s41597-025-05174-7 [code]
A large-scale MEG and EEG dataset for object recognition in naturalistic scenes
Journal: n/a
In common: Pillow, PyTorch, scikit-learn, 4 other tools, EEG, 2 references
[7] doi:10.1038/s41593-026-02285-1 [code]
Fixation duration on natural scenes is explained by memory encoding not processing demand.
Journal: Nature neuroscience
In common: Pillow, PyTorch, scikit-learn, 4 other tools, cognitive, 2 references
[8] doi:10.1002/hbm.70528 [code]
Explainable AI Insights Into EEG Classification and Its Alignment to Neural Correlates.
Journal: Human brain mapping
In common: Pillow, PyTorch, scikit-learn, 4 other tools, EEG, cognitive, 1 reference
[9] doi:10.1167/jov.26.8.2 [code]
Disentangling objects' contextual associations from perceptual and conceptual attributes using time-resolved neural decoding.
Journal: Journal of vision
In common: pandas, SciPy, Matplotlib, 1 other tool, EEG, cognitive, 3 references
[10] doi:10.1167/jov.26.8.1 [code]
MAME: Multidimensional adaptive metamer exploration with human perceptual feedback.
Journal: Journal of vision
In common: Pillow, PyTorch, scikit-learn, 4 other tools, cognitive, 1 reference

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.