Contrastive learning to fine-tune feature extraction models for the visual cortex.
5 blocks of code here have a permanent identifier (oscr:b:…), listed with the matches below: citable for good, with a page of its own. Select a match and press Cite to copy it. What a block identifier is.
The 5 matches
- [1] § Results ↔ code/results_utils.py, lines 739 to 824 · score 0.66 · mfs words, VWFA, FBA, FFA, OFA, OWFA · block
oscr:b:YD633ZRDC2G4M-Y· Report a mismatch - [2] § Results ↔ code/utils.py, lines 314 to 355 · score 0.65 · mfs words, VWFA, FBA, FFA, OFA, OWFA · block
oscr:b:XKSX2QNXD6PC0-S· Report a mismatch - [3] § Methodology › Implementation details ↔ code/models.py, lines 28 to 77 · score 0.59 · MLP projection head, ReLU, batch, layers, voxels · block
oscr:b:RKY1CJWTG3CJH-C· Report a mismatch - [4] § Methodology › Baseline feature extraction models ↔ code/results_utils.py, lines 22 to 84 · score 0.55 · ImageNet, channels, resized, AlexNet · block
oscr:b:MTA1ZSDDBN8BK-Y· Report a mismatch - [5] § Methodology › External dataset validation ↔ code/fit_encoding_models_nod.py, lines 212 to 271 · score 0.51 · AlexNet model, pooled model, Encoding Model, CL tuned, V1, NOD · block
oscr:b:VRZ4AYPJMENND-5· Report a mismatch
Paper
Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC
The paper is loaded when this pane is shown.
The authors' code
Python · 890 lines · 43 KB · no license · 2 matches
- from models import CLR_model, fmri_reg, get_pooled_CL_model
- from utils import get_dataloaders
- from sklearn.metrics.pairwise import cosine_similarity
- import torchextractor as tx
- from tqdm import tqdm
- from sklearn.metrics import accuracy_score
- from sklearn.preprocessing import StandardScaler
- from sklearn.linear_model import LogisticRegression
- from torchvision.models.feature_extraction import create_feature_extractor
- from torchvision.models import AlexNet_Weights
- import torchvision
- import joblib
- from torchvision.models import AlexNet_Weights
- from torchvision import transforms
- import torch
- import numpy as np
- import os
- os.environ["OMP_NUM_THREADS"] = '1'
- # Get results for image classification task
- def image_classification_results(project_dir, subj_num, hemisphere, rois, device, tuning_method='cl', dataset_name='caltech256', save=False,
- pooled=False, pooled_h_method='const', save_probs=False):
- hemisphere_abbr = 'l' if hemisphere == 'left' else 'r'
- if pooled and pooled_h_method == 'avg':
- save_path = os.path.join(project_dir, "results", hemisphere_abbr + "h_" +
- dataset_name + "_" + tuning_method + "_results_pooled_havg.joblib")
- if save_probs:
- probs_save_folder = os.path.join(
- project_dir, "classification_preds", "Pooled")
- elif pooled and pooled_h_method == 'const':
- save_path = os.path.join(project_dir, "results", hemisphere_abbr + "h_" +
- dataset_name + "_" + tuning_method + "_results_pooled_hconst.joblib")
- if save_probs:
- probs_save_folder = os.path.join(
- project_dir, "classification_preds", "Pooled")
- else:
- save_path = project_dir + "/results/Subj" + str(subj_num) + "/subj" + str(
- subj_num) + "_" + hemisphere_abbr + "h_" + dataset_name + "_" + tuning_method + "_results.joblib"
- probs_save_folder = os.path.join(
- project_dir, "classification_preds", "Subj" + str(subj_num))
- # Seed RNG, define image transforms for alexnet
- torch.manual_seed(0)
- alex_transform = transforms.Compose([
- transforms.Resize(256),
- transforms.CenterCrop(224),
- transforms.ToTensor(), # convert the images to a PyTorch tensor
- # normalize the images color channels
- transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
- ])
- # Load the Data
- if (dataset_name == 'caltech256'):
- image_dir = os.path.join(project_dir, "caltech256")
- dataset = torchvision.datasets.ImageFolder(
- root=image_dir, transform=alex_transform)
- elif (dataset_name == 'places365'):
- image_dir = os.path.join(project_dir, "places365")
- dataset = torchvision.datasets.Places365(
- root=image_dir, split='val', small=True, transform=alex_transform)
- elif (dataset_name == 'sun397'):
- image_dir = os.path.join(project_dir, "sun397")
- dataset = torchvision.datasets.SUN397(
- root=image_dir, transform=alex_transform, download=True)
- elif (dataset_name == 'imagenet'):
- image_dir = os.path.join(project_dir, "imagenet")
- dataset = torchvision.datasets.ImageNet(
- root=image_dir, split='val', transform=alex_transform)
- total_num_images = len(dataset)
- generator = torch.Generator()
- generator.manual_seed(0)
- shuffled_idxs = torch.randperm(total_num_images, generator=generator)
- train_size = int(0.85 * total_num_images)
- test_size = total_num_images - train_size
- print(train_size, test_size)
- train_idxs = shuffled_idxs[:train_size]
- test_idxs = shuffled_idxs[train_size:train_size+test_size]
- # Create train and test dataloaders
- train_dataset = torch.utils.data.Subset(dataset, train_idxs)
- test_dataset = torch.utils.data.Subset(dataset, test_idxs)
- train_dataloader = torch.utils.data.DataLoader(
- train_dataset, batch_size=256, shuffle=False)
- test_dataloader = torch.utils.data.DataLoader(
- test_dataset, batch_size=256, shuffle=False)
- # Get image features for untuned AlexNet
- alex = torch.hub.load('pytorch/vision:v0.10.0', 'alexnet',
- weights=AlexNet_Weights.IMAGENET1K_V1)
- alex.to(device)
- alex.eval()
- feature_extractor = create_feature_extractor(
- alex, return_nodes=['classifier.5']).to(device)
- del alex
- # Get untuned alexnet features
- train_features_untuned = np.zeros((train_size, 4096))
- train_labels = np.zeros(train_size)
- for batch_index, data in tqdm(enumerate(train_dataloader), total=len(train_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- ft = feature_extractor(data[0].to(device))
- # Flatten the features
- ft = torch.hstack([torch.flatten(l, start_dim=1)
- for l in ft.values()]).cpu().detach().numpy()
- train_features_untuned[low_idx:high_idx] = ft
- train_labels[low_idx:high_idx] = data[1]
- del ft
- test_features_untuned = np.zeros((test_size, 4096))
- test_labels = np.zeros(test_size)
- for batch_index, data in tqdm(enumerate(test_dataloader), total=len(test_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- ft = feature_extractor(data[0].to(device))
- # Flatten the features
- ft = torch.hstack([torch.flatten(l, start_dim=1)
- for l in ft.values()]).cpu().detach().numpy()
- test_features_untuned[low_idx:high_idx] = ft
- del ft
- test_labels[low_idx:high_idx] = data[1]
- del feature_extractor
- # Save labels
- if save_probs:
- np.save(os.path.join(project_dir, "classification_preds",
- dataset_name + "_test_labels.npy"), test_labels)
- scaler = StandardScaler()
- fit_scaler = scaler.fit(train_features_untuned)
- train_features_untuned = fit_scaler.transform(train_features_untuned)
- test_features_untuned = fit_scaler.transform(test_features_untuned)
- print("Fitting linear classifier...")
- classifier = LogisticRegression(max_iter=5000).fit(
- train_features_untuned, train_labels)
- preds = classifier.predict(test_features_untuned)
- if save_probs:
- untuned_pred_probs = classifier.predict_proba(test_features_untuned)
- untuned_pred_probs_save_path = os.path.join(
- project_dir, "classification_preds", "Untuned", "untuned_" + dataset_name + "_test_probs.npy")
- np.save(untuned_pred_probs_save_path, untuned_pred_probs)
- acc = accuracy_score(test_labels, preds) * 100
- print("Untuned", acc)
- if rois[0] == 'all' and hemisphere == 'left':
- rois = ["V1v", "V1d", "V2v", "V2d", "V3v", "V3d", "hV4", "EBA", "FBA-1", "FBA-2",
- "OFA", "FFA-1", "FFA-2", "OPA", "PPA", "RSC", "OWFA", "VWFA-1", "VWFA-2", "mfs-words"]
- elif rois[0] == 'all' and hemisphere == 'right':
- rois = ["V1v", "V1d", "V2v", "V2d", "V3v", "V3d", "hV4", "EBA", "FBA-1", "FBA-2",
- "mTL-bodies", "OFA", "FFA-1", "FFA-2", "OPA",
- "PPA", "RSC", "OWFA", "VWFA-1", "VWFA-2", "mfs-words", "mTL-words"]
- # Go through list of rois
- for roi in rois:
- print(roi)
- found_subj = False
- # Get number of voxels for this model
- _, _, _, _, num_voxels = get_dataloaders(
- project_dir, device, subj_num, hemisphere, roi, 1024, shuffle=False)
- if num_voxels < 20 and not pooled:
- print(roi, "is too small or empty")
- elif num_voxels >= 20 and not pooled:
- found_subj = True
- # If using pooled models, may need dummy fmri input data matching shape of some other subject's acitvations
- elif num_voxels >= 20 and pooled:
- found_subj = True
- elif num_voxels < 20 and pooled:
- for subj_idx in range(2, 9):
- _, _, _, _, num_voxels = get_dataloaders(
- project_dir, device, subj_idx, hemisphere, roi, 1024, shuffle=False)
- if num_voxels > 0:
- found_subj = True
- break
- if found_subj:
- if tuning_method == 'cl':
- if pooled == True:
- # Get voxel counts for ROI across all subjects with the ROI from previously created dictionary
- voxel_counts_file = os.path.join(
- project_dir, hemisphere_abbr + "h_voxel_counts_rois.joblib")
- voxel_counts = joblib.load(voxel_counts_file)
- num_voxels_subjs = np.array(voxel_counts[roi])
- num_voxels_subjs = np.where(num_voxels_subjs == 0, np.nan, num_voxels_subjs)
- avg_voxel_dim = int(np.nanmean(num_voxels_subjs))
- if pooled_h_method == 'avg':
- # Load pooled CL model
- model, _ = get_pooled_CL_model(
- num_voxels_subjs, device, avg_voxel_dim)
- model_path = os.path.join(
- project_dir, "cl_models", hemisphere_abbr + "h_" + roi + "_pooled_model_e30_havg.pt")
- elif pooled_h_method == 'const':
- h_max_dim = 5741
- # Load pooled model
- model, _ = get_pooled_CL_model(
- num_voxels_subjs, device, h_max_dim)
- model_path = os.path.join(
- project_dir, "cl_models", hemisphere_abbr + "h_" + roi + "_pooled_model_e30_hconst.pt")
- else:
- model_dir = os.path.join(project_dir, "cl_models", "Subj" + str(subj_num))
- model_path = os.path.join(model_dir, "subj" + \
- str(subj_num) + "_" + hemisphere_abbr + \
- "h_" + roi + "_model_e30.pt")
- h_dim = int(num_voxels*0.8)
- z_dim = int(num_voxels*0.2)
- model = CLR_model(num_voxels, h_dim, z_dim)
- elif tuning_method == 'reg':
- model_dir = os.path.join(project_dir, "baseline_models", "nn_reg", "Subj" + str(subj_num))
- model_path = os.path.join(model_dir, "subj" + \
- str(subj_num) + "_" + hemisphere_abbr + \
- "h_" + roi + "_reg_model_e75.pt")
- model = fmri_reg(num_voxels)
- # Some models are saved differently
- try:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu'))[0].state_dict())
- except:
- try:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu')).state_dict())
- except:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu')))
- model.to(device)
- model.eval()
- feature_extractor = tx.Extractor(
- model, ["alex.classifier.5"]).to(device)
- train_features_tuned = np.zeros((train_size, 4096))
- train_labels = np.zeros(train_size)
- for batch_index, data in tqdm(enumerate(train_dataloader), total=len(train_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- if tuning_method == 'cl':
- fmri_dummy = torch.zeros(
- (batch_size, num_voxels)).to(device)
- if pooled:
- _, alex_out_dict = feature_extractor(
- fmri_dummy, data[0], subj_num)
- else:
- _, alex_out_dict = feature_extractor(
- fmri_dummy, data[0])
- # _, alex_out_dict = feature_extractor(fmri_dummy, data[0].to(device))
- elif tuning_method == 'reg':
- _, alex_out_dict = feature_extractor(
- data[0].to(device))
- ft = alex_out_dict['alex.classifier.5'].detach().cpu().numpy()
- train_features_tuned[low_idx:high_idx] = ft
- train_labels[low_idx:high_idx] = data[1]
- del ft
- test_features_tuned = np.zeros((test_size, 4096))
- test_labels = np.zeros(test_size)
- for batch_index, data in tqdm(enumerate(test_dataloader), total=len(test_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- if tuning_method == 'cl':
- fmri_dummy = torch.zeros(
- (batch_size, num_voxels)).to(device)
- if pooled:
- _, alex_out_dict = feature_extractor(
- fmri_dummy, data[0].to(torch.float), subj_num)
- else:
- _, alex_out_dict = feature_extractor(
- fmri_dummy, data[0])
- # _, alex_out_dict = feature_extractor(fmri_dummy, data[0].to(device))
- elif tuning_method == 'reg':
- _, alex_out_dict = feature_extractor(
- data[0].to(device))
- ft = alex_out_dict['alex.classifier.5'].detach().cpu().numpy()
- test_features_tuned[low_idx:high_idx] = ft
- test_labels[low_idx:high_idx] = data[1]
- del ft
- scaler = StandardScaler()
- fit_scaler = scaler.fit(train_features_tuned)
- train_features_tuned = fit_scaler.transform(train_features_tuned)
- test_features_tuned = fit_scaler.transform(test_features_tuned)
- print("Fitting linear classifier...")
- classifier = LogisticRegression(max_iter=5000).fit(
- train_features_tuned, train_labels)
- preds = classifier.predict(test_features_tuned)
- if save_probs:
- tuned_pred_probs = classifier.predict_proba(
- test_features_tuned)
- if pooled:
- if pooled_h_method == 'avg':
- tuned_pred_probs_save_path = os.path.join(
- probs_save_folder, hemisphere_abbr + "h_" + roi + "_pooled_havg_" + dataset_name + "_test_probs.npy")
- elif pooled_h_method == 'const':
- tuned_pred_probs_save_path = os.path.join(
- probs_save_folder, hemisphere_abbr + "h_" + roi + "_pooled_hconst_" + dataset_name + "_test_probs.npy")
- else:
- if tuning_method == 'cl':
- tuned_pred_probs_save_path = os.path.join(probs_save_folder, "subj" + str(
- subj_num) + "_" + hemisphere_abbr + "h_" + roi + "_" + dataset_name + "_test_probs.npy")
- else:
- tuned_pred_probs_save_path = os.path.join(probs_save_folder, "subj" + str(
- subj_num) + "_" + hemisphere_abbr + "h_" + roi + "_" + dataset_name + "_reg_test_probs.npy")
- np.save(tuned_pred_probs_save_path, tuned_pred_probs)
- acc = accuracy_score(test_labels, preds) * 100
- # results[roi] = acc
- print(roi, acc)
- if save:
- try:
- # Load existing results, add result for roi if not already in the existing results
- existing_results = joblib.load(save_path)
- if roi not in existing_results.keys():
- existing_results[roi] = acc
- joblib.dump(existing_results, save_path)
- except:
- results = {}
- results[roi] = acc
- joblib.dump(results, save_path)
- # Compute lower bound on mutual information between CNN features and ROI response using the testing data
- def compute_mi_lower_bound(project_dir, device, subj_num, roi, hemisphere, pooled=False, h_method='const'):
- # Temperature tau used to tune CL models
- tau = 0.3
- hemisphere_abbr = 'l' if hemisphere == 'left' else 'r'
- print(roi, hemisphere)
- # Get test dataloader
- _, test_dataloader, _, test_size, num_voxels = get_dataloaders(
- project_dir, device, subj_num, hemisphere, roi, 1024, shuffle=False)
- if (num_voxels == 0):
- print("Empty ROI")
- return -1, -1, -1, -1, -1, -1, -1
- elif (num_voxels < 20):
- print("Too few voxels")
- return -1, -1, -1, -1, -1, -1, -1
- if pooled:
- # Get list of number of voxels for each subj
- present_subjs = []
- num_voxels_subjs = []
- for subj_idx in range(1, 9):
- _, _, _, _, num_voxels = get_dataloaders(
- project_dir, device, subj_idx, hemisphere, roi, batch_size=1024)
- if num_voxels != 0:
- present_subjs.append(subj_idx)
- num_voxels_subjs.append(num_voxels)
- # Get voxel counts for ROI across all subjects with the ROI from previously created dictionary
- voxel_counts_file = os.path.join(
- project_dir, hemisphere_abbr + "h_voxel_counts_rois.joblib")
- voxel_counts = joblib.load(voxel_counts_file)
- num_voxels_subjs = np.array(voxel_counts[roi])
- num_voxels_subjs = np.where(num_voxels_subjs == 0, np.nan, num_voxels_subjs)
- avg_voxel_dim = int(np.nanmean(num_voxels_subjs))
- # Get pooled model
- print("Getting CL predictions...")
- cl_model_dir = os.path.join(project_dir, "cl_models")
- if h_method == 'avg':
- h_dim = avg_voxel_dim
- # Load pooled CL model
- cl_model, _ = get_pooled_CL_model(num_voxels_subjs, device, h_dim)
- cl_model_path = os.path.join(
- cl_model_dir, hemisphere_abbr + "h_" + roi + "_pooled_model_e30_havg.pt")
- elif h_method == 'const':
- h_dim = 5741
- # Load pooled model
- cl_model, _ = get_pooled_CL_model(num_voxels_subjs, device, h_dim)
- cl_model_path = os.path.join(
- cl_model_dir, hemisphere_abbr + "h_" + roi + "_pooled_model_e30_hconst.pt")
- z_dim = int(h_dim * 0.25)
- # Some models are saved differently
- try:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu'))[0].state_dict())
- except:
- try:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu')).state_dict())
- except:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu')))
- else:
- # Load CL-tuned model
- cl_model_dir = os.path.join(project_dir, "Subj" + str(subj_num))
- cl_model_path = os.path.join(cl_model_dir, "subj" + str(subj_num) + "_" + hemisphere_abbr +
- "h_" + roi + "_model_e30.pt")
- h_dim = int(num_voxels*0.8)
- z_dim = int(num_voxels*0.2)
- cl_model = CLR_model(num_voxels, h_dim, z_dim)
- # Some models are saved differently
- try:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu'))[0].state_dict())
- except:
- try:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu')).state_dict())
- except:
- cl_model.load_state_dict(torch.load(
- cl_model_path, map_location=torch.device('cpu')))
- cl_model.to(device)
- cl_model.eval()
- # Use just 1 batch
- K = test_size
- # Get outputs (after projection head) for CL model and fMRI data
- nn_z = np.zeros((K, z_dim))
- fmri_z = np.zeros((K, z_dim))
- for batch_index, data in tqdm(enumerate(test_dataloader), total=len(test_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- if pooled:
- # Forward function for pooled CL models expects subj num = (1-based) position in list of subjects with roi present (exclusing subjects without roi),
- # so need to correct indexing
- subj_num_adjusted = present_subjs.index(subj_num) + 1
- # print(subj_num_adjusted)
- ft_fmri, ft_nn = cl_model(data[0], data[1], subj_num_adjusted)
- else:
- ft_fmri, ft_nn = cl_model(data[0], data[1])
- # Flatten the features, collect them
- ft_fmri = ft_fmri.cpu().detach().numpy()
- ft_nn = ft_nn.cpu().detach().numpy()
- fmri_z[low_idx:high_idx] = ft_fmri
- nn_z[low_idx:high_idx] = ft_nn
- critic_out = (1 / tau) * cosine_similarity(nn_z, fmri_z)
- exp_critic = np.exp(critic_out)
- lower_bound_mi = 0
- for i in range(K):
- lower_bound_mi += np.log(exp_critic[i, i] /
- ((1 / K) * np.sum(exp_critic[i, :])))
- lower_bound_mi_unscaled = (1 / K) * lower_bound_mi
- # Do post-hoc scaling for temp
- betas = np.logspace(-2, 2, 1000)
- best_beta = 1
- best_lower_bound = lower_bound_mi_unscaled
- for beta in betas:
- tau_new = tau * beta
- critic_out = (1 / tau_new) * cosine_similarity(nn_z, fmri_z)
- exp_critic = np.exp(critic_out)
- lower_bound_mi = 0
- for i in range(K):
- lower_bound_mi += np.log(exp_critic[i, i] /
- ((1 / K) * np.sum(exp_critic[i, :])))
- lower_bound_mi = (1 / K) * lower_bound_mi
- if lower_bound_mi > best_lower_bound:
- best_lower_bound = lower_bound_mi
- best_beta = beta
- print(best_beta, best_lower_bound)
- # Save results
- if pooled:
- save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(
- subj_num) + "_" + hemisphere_abbr + "h_mi_lower_bound_pooled_h" + h_method + ".joblib")
- else:
- save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_mi_lower_bound.joblib")
- try:
- results = joblib.load(save_path)
- except:
- results = {}
- results[roi] = lower_bound_mi_unscaled, np.log(
- K), best_beta, best_lower_bound
- joblib.dump(results, save_path)
- # Load test cv results for single subject untuned for all layers and subjects to create figure 2 (matrix of encoding scores)
- def load_test_cv_single_subj_untuned_all_layers(project_dir):
- num_rows = 46
- results_matrix = np.zeros((num_rows, 8)) # Average across subjects
- # Keep track of how many subjs have each ROI
- row_counters = np.zeros((num_rows))
- for subj_num in range(1, 9):
- results_folder_path = os.path.join(
- project_dir, "best_alex_out_layers_test_cv", "Subj" + str(subj_num))
- results_file = os.path.join(
- results_folder_path, "best_alex_layers_mat_untuned.npy")
- subj_results = np.load(results_file)
- for row_idx in range(num_rows):
- if subj_results[row_idx, 0] > 0:
- row_counters[row_idx] += 1
- results_matrix[row_idx, :] += subj_results[row_idx, :]
- for row_idx in range(num_rows):
- if results_matrix[row_idx, 0] > 0:
- results_matrix[row_idx, :] /= row_counters[row_idx]
- return results_matrix
- # Load test cv results for single subject (untuned, cl-tuned, reg-tuned, pooled-avg, or pooled-specific)
- # split options are all, early, higher
- def load_test_cv_single_subj_results(project_dir, subj_num, split='all', return_pooled_results=False, roi=None):
- results_folder_path = os.path.join(
- project_dir, "best_alex_out_layers_test_cv", "Subj" + str(subj_num))
- untuned_results_path = os.path.join(
- results_folder_path, "best_alphas_voxel_accs_dict_untuned.joblib")
- cl_tuned_results_path = os.path.join(
- results_folder_path, "best_alphas_voxel_accs_dict_cl_tuned.joblib")
- reg_tuned_results_path = os.path.join(
- results_folder_path, "best_alphas_voxel_accs_dict_reg_tuned.joblib")
- pooled_avg_results_path = os.path.join(
- results_folder_path, "best_alphas_voxel_accs_dict_pooled_avg.joblib")
- pooled_const_results_path = os.path.join(
- results_folder_path, "best_alphas_voxel_accs_dict_pooled_const.joblib")
- untuned_results = joblib.load(untuned_results_path)
- cl_tuned_results = joblib.load(cl_tuned_results_path)
- reg_tuned_results = joblib.load(reg_tuned_results_path)
- pooled_avg_results = joblib.load(pooled_avg_results_path)
- pooled_const_results = joblib.load(pooled_const_results_path)
- if roi is not None:
- try:
- return untuned_results[roi], reg_tuned_results[roi], cl_tuned_results[roi], pooled_avg_results[roi], pooled_const_results[roi]
- except:
- return 0, 0, 0, 0, 0
- else:
- if split == 'all':
- rois = ["V1v", "V1d", "V2v", "V2d", "V3v", "V3d", "hV4", "EBA", "FBA-1", "FBA-2",
- "mTL-bodies", "OFA", "FFA-1", "FFA-2", "mTL-faces", "aTL-faces", "OPA",
- "PPA", "RSC", "OWFA", "VWFA-1", "VWFA-2", "mfs-words", "mTL-words"]
- elif split == 'early':
- rois = ["V1v", "V1d", "V2v", "V2d", "V3v", "V3d", "hV4"]
- elif split == 'higher':
- rois = ["EBA", "FBA-1", "FBA-2", "mTL-bodies", "OFA", "FFA-1", "FFA-2", "mTL-faces", "aTL-faces", "OPA",
- "PPA", "RSC", "OWFA", "VWFA-1", "VWFA-2", "mfs-words", "mTL-words"]
- else:
- print("Unsupported Split!")
- return
- untuned_mean_acc = 0
- reg_tuned_mean_acc = 0
- cl_tuned_mean_acc = 0
- pooled_avg_mean_acc = 0
- pooled_const_mean_acc = 0
- cl_over_untuned_voxel_mean_percentage = 0
- cl_over_reg_tuned_voxel_mean_percentage = 0
- pooled_avg_over_cl_voxel_mean_percentage = 0
- pooled_const_over_cl_voxel_mean_percentage = 0
- num_rois_present = 0
- for hemi in ["lh", "rh"]:
- for roi in rois:
- try:
- untuned_hemi_roi_results = untuned_results[hemi + '_' + roi]
- reg_tuned_hemi_roi_results = reg_tuned_results[hemi + '_' + roi]
- cl_tuned_hemi_roi_results = cl_tuned_results[hemi + '_' + roi]
- pooled_avg_hemi_roi_results = pooled_avg_results[hemi + '_' + roi]
- pooled_const_hemi_roi_results = pooled_const_results[hemi + '_' + roi]
- untuned_mean_acc += untuned_hemi_roi_results.mean()
- reg_tuned_mean_acc += reg_tuned_hemi_roi_results.mean()
- cl_tuned_mean_acc += cl_tuned_hemi_roi_results.mean()
- pooled_avg_mean_acc += pooled_avg_hemi_roi_results.mean()
- pooled_const_mean_acc += pooled_const_hemi_roi_results.mean()
- cl_over_untuned_voxel_mean_percentage += np.count_nonzero(
- cl_tuned_hemi_roi_results - untuned_hemi_roi_results > 0) / cl_tuned_hemi_roi_results.shape[0]
- cl_over_reg_tuned_voxel_mean_percentage += np.count_nonzero(
- cl_tuned_hemi_roi_results - reg_tuned_hemi_roi_results > 0) / cl_tuned_hemi_roi_results.shape[0]
- pooled_avg_over_cl_voxel_mean_percentage += np.count_nonzero(
- pooled_avg_hemi_roi_results - cl_tuned_hemi_roi_results > 0) / cl_tuned_hemi_roi_results.shape[0]
- pooled_const_over_cl_voxel_mean_percentage += np.count_nonzero(
- pooled_const_hemi_roi_results - cl_tuned_hemi_roi_results > 0) / cl_tuned_hemi_roi_results.shape[0]
- num_rois_present += 1
- except:
- pass # Skip missing ROIs
- untuned_mean_acc /= num_rois_present
- reg_tuned_mean_acc /= num_rois_present
- cl_tuned_mean_acc /= num_rois_present
- pooled_avg_mean_acc /= num_rois_present
- pooled_const_mean_acc /= num_rois_present
- cl_over_untuned_voxel_mean_percentage /= num_rois_present
- cl_over_reg_tuned_voxel_mean_percentage /= num_rois_present
- pooled_avg_over_cl_voxel_mean_percentage /= num_rois_present
- pooled_const_over_cl_voxel_mean_percentage /= num_rois_present
- if return_pooled_results:
- return pooled_avg_mean_acc, pooled_const_mean_acc, pooled_avg_over_cl_voxel_mean_percentage, pooled_const_over_cl_voxel_mean_percentage
- else:
- return untuned_mean_acc, reg_tuned_mean_acc, cl_tuned_mean_acc, cl_over_untuned_voxel_mean_percentage, cl_over_reg_tuned_voxel_mean_percentage
- # Return matrix of results for each layer/roi
- def load_test_cv_single_subj_results_all_layers(project_dir, subj_num):
- results_folder_path = os.path.join(
- project_dir, "best_alex_out_layers_test_cv", "Subj" + str(subj_num))
- untuned_results_path = os.path.join(
- results_folder_path, "best_alex_layers_mat_untuned.npy")
- cl_tuned_results_path = os.path.join(
- results_folder_path, "best_alex_layers_mat_cl_tuned.npy")
- reg_tuned_results_path = os.path.join(
- results_folder_path, "best_alex_layers_mat_reg_tuned.npy")
- pooled_avg_results_path = os.path.join(
- results_folder_path, "best_alex_layers_mat_pooled_avg.npy")
- pooled_const_results_path = os.path.join(
- results_folder_path, "best_alex_layers_mat_pooled_const.npy")
- untuned_results = np.load(untuned_results_path)
- cl_tuned_results = np.load(cl_tuned_results_path)
- reg_tuned_results = np.load(reg_tuned_results_path)
- pooled_avg_results = np.load(pooled_avg_results_path)
- pooled_const_results = np.load(pooled_const_results_path)
- return untuned_results, cl_tuned_results, reg_tuned_results, pooled_avg_results, pooled_const_results
- # Generate embeddings for test images from untuned, CL, or regression-tuned models. Save corresponding NSD IDs of images.
- # Options for tuning_method are 'untuned', 'CL' or 'reg'
- # Optionally get embeddings from best layer for encoding instead of final layer
- def save_embeddings(project_dir, subj_num, hemisphere, roi, device, tuning_method='CL', use_best_intermediate_layer=False, use_other_subj_images=False, cross_subj_num=1):
- # Strip whitespace from roi to handle cases where it comes from files with trailing spaces
- roi = roi.strip()
- hemisphere_abbr = 'l' if hemisphere == 'left' else 'r'
- if tuning_method == 'CL':
- if use_best_intermediate_layer:
- if use_other_subj_images:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_best_encoding_layer_cross_subj" + str(cross_subj_num) + ".npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_best_encoding_layer_cross_subj" + str(cross_subj_num) + "_img_ids.npy")
- else:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_best_encoding_layer.npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_best_encoding_layer_img_ids.npy")
- else:
- if use_other_subj_images:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_cross_subj" + str(cross_subj_num) + ".npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_cross_subj" + str(cross_subj_num) + "_img_ids.npy")
- else:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings.npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_cl_embeddings_img_ids.npy")
- elif tuning_method == 'reg':
- if use_best_intermediate_layer:
- if use_other_subj_images:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_best_encoding_layer_cross_subj" + str(cross_subj_num) + ".npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_best_encoding_layer_cross_subj" + str(cross_subj_num) + "_img_ids.npy")
- else:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_best_encoding_layer.npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_best_encoding_layer_img_ids.npy")
- else:
- if use_other_subj_images:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_cross_subj" + str(cross_subj_num) + ".npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_cross_subj" + str(cross_subj_num) + "_img_ids.npy")
- else:
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings.npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_reg_embeddings_img_ids.npy")
- elif tuning_method == 'untuned':
- features_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_untuned_embeddings.npy")
- ids_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_untuned_embeddings_img_ids.npy")
- if use_other_subj_images:
- _, _, _, _, num_voxels = get_dataloaders(project_dir,
- device, subj_num, hemisphere, roi, batch_size=1024, use_all_data=False, shuffle=False, return_nsd_id=True)
- _, test_dataloader, _, test_size, _ = get_dataloaders(project_dir,
- device, cross_subj_num, hemisphere, roi, batch_size=1024, use_all_data=False, shuffle=False, return_nsd_id=True)
- else:
- _, test_dataloader, _, test_size, num_voxels = get_dataloaders(project_dir,
- device, subj_num, hemisphere, roi, batch_size=1024, use_all_data=False, shuffle=False, return_nsd_id=True)
- # Load tuned model
- if tuning_method == 'CL':
- model_dir = os.path.join(project_dir, "cl_models", "Subj" + str(subj_num))
- model_path = os.path.join(model_dir, "subj" + str(subj_num) + "_" + hemisphere_abbr + "h_" + roi + "_model_e30.pt")
- h_dim = int(num_voxels*0.8)
- z_dim = int(num_voxels*0.2)
- model = CLR_model(num_voxels, h_dim, z_dim)
- elif tuning_method == 'reg':
- model_dir = os.path.join(project_dir, "baseline_models", "nn_reg", "Subj" + str(subj_num))
- model_path = os.path.join(model_dir, "subj" + \
- str(subj_num) + "_" + hemisphere_abbr + \
- "h_" + roi + "_reg_model_e75.pt")
- model = fmri_reg(num_voxels)
- elif tuning_method == 'untuned':
- model = torch.hub.load('pytorch/vision:v0.10.0', 'alexnet',
- weights=AlexNet_Weights.IMAGENET1K_V1)
- # Some tuned models are saved differently
- if tuning_method == 'CL' or tuning_method == 'reg':
- try:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu'), weights_only=False)[0].state_dict())
- except:
- try:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu'), weights_only=False).state_dict())
- except:
- model.load_state_dict(torch.load(
- model_path, map_location=torch.device('cpu'), weights_only=False))
- model.to(device)
- model.eval()
- roi_names = ["V1v", "V1d", "V2v", "V2d", "V3v", "V3d", "hV4", "EBA", "FBA-1", "FBA-2",
- "mTL-bodies", "OFA", "FFA-1", "FFA-2", "mTL-faces", "OPA",
- "PPA", "RSC", "OWFA", "VWFA-1", "VWFA-2", "mfs-words", "mTL-words"]
- hemis = ["lh", "rh"]
- layers = [
- "features.2", "features.5", "features.7", "features.9",
- "features.12", "classifier.2", "classifier.5", "classifier.6"
- ]
- layer_dim_dict = {"features.2": 46656, "features.5": 32448, "features.7": 64896, "features.9": 43264,
- "features.12": 9216, "classifier.2": 4096, "classifier.5": 4096, "classifier.6": 1000}
- if tuning_method == 'CL' or tuning_method == 'reg':
- if use_best_intermediate_layer:
- _, results, _, _, _ = load_test_cv_single_subj_results_all_layers(project_dir, subj_num)
- results_dict = {}
- counter = 0
- for hemi in hemis:
- for roi_name in roi_names:
- current_roi = hemi + "_" + roi_name
- if not np.isnan(results[counter]).any():
- results_dict[current_roi] = results[counter]
- counter += 1
- best_layer = layers[np.argmax(results_dict[hemisphere_abbr + "h_" + roi])]
- print("Using layer:", best_layer)
- feature_extractor = tx.Extractor(
- model, ["alex." + best_layer]).to(device)
- output_dim = layer_dim_dict[best_layer]
- else:
- feature_extractor = tx.Extractor(
- model, ["alex.classifier.5"]).to(device)
- output_dim = 4096
- elif tuning_method == 'reg':
- if use_best_intermediate_layer:
- _, _, results, _, _ = load_test_cv_single_subj_results_all_layers(project_dir, subj_num)
- results_dict = {}
- counter = 0
- for hemi in hemis:
- for roi_name in roi_names:
- current_roi = hemi + "_" + roi_name
- if not np.isnan(results[counter]).any():
- results_dict[current_roi] = results[counter]
- counter += 1
- best_layer = layers[np.argmax(results_dict[hemisphere_abbr + "h_" + roi])]
- print("Using layer:", best_layer)
- feature_extractor = tx.Extractor(
- model, ["alex." + best_layer]).to(device)
- output_dim = layer_dim_dict[best_layer]
- else:
- feature_extractor = tx.Extractor(
- model, ["alex.classifier.5"]).to(device)
- output_dim = 4096
- elif tuning_method == 'untuned':
- feature_extractor = tx.Extractor(
- model, ["classifier.5"]).to(device)
- output_dim = 4096
- features = np.zeros((test_size, output_dim))
- ids = np.zeros(test_size)
- for batch_index, data in tqdm(enumerate(test_dataloader), total=len(test_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- # Extract features
- with torch.no_grad():
- if tuning_method == 'CL':
- fmri_dummy = torch.zeros(
- (batch_size, num_voxels)).to(device)
- _, alex_out_dict = feature_extractor(fmri_dummy, data[1])
- # _, alex_out_dict = feature_extractor(fmri_dummy, data[0].to(device))
- elif tuning_method == 'reg':
- _, alex_out_dict = feature_extractor(data[1].to(device))
- elif tuning_method == 'untuned':
- _, alex_out_dict = feature_extractor(data[1].to(device))
- if tuning_method == 'CL' or tuning_method == 'reg':
- if use_best_intermediate_layer:
- ft = alex_out_dict["alex." + best_layer].detach().cpu().numpy().reshape(high_idx-low_idx, -1)
- else:
- ft = alex_out_dict['alex.classifier.5'].detach().cpu().numpy()
- elif tuning_method == 'untuned':
- ft = alex_out_dict['classifier.5'].detach().cpu().numpy()
- features[low_idx:high_idx] = ft
- ids[low_idx:high_idx] = data[3]
- del ft
- # Save features and ids
- np.save(features_save_path, features)
- np.save(ids_save_path, ids)
- # Save fmri responses for test images. Save corresponding NSD IDs of images.
- # Use to match format of save_embeddings function.
- def save_test_fmri_responses(project_dir, subj_num, hemisphere, roi, device):
- # Strip whitespace from roi to handle cases where it comes from files with trailing spaces
- roi = roi.strip()
- hemisphere_abbr = 'l' if hemisphere == 'left' else 'r'
- fmri_responses_save_path = os.path.join(project_dir, "results", "Subj" + str(subj_num), "subj" + str(subj_num) + "_" +
- hemisphere_abbr + "h_" + roi + "_test_fmri_responses.npy")
- _, test_dataloader, _, test_size, num_voxels = get_dataloaders(project_dir,
- device, subj_num, hemisphere, roi, batch_size=1024, use_all_data=False, shuffle=False, return_nsd_id=True)
- fmri_responses = np.zeros((test_size, num_voxels))
- for batch_index, data in tqdm(enumerate(test_dataloader), total=len(test_dataloader)):
- batch_size = data[0].shape[0]
- if batch_index == 0:
- low_idx = 0
- high_idx = batch_size
- else:
- low_idx = high_idx
- high_idx += batch_size
- fmri_responses[low_idx:high_idx] = data[0]
- del data
- np.save(fmri_responses_save_path, fmri_responses)
results_utils.py at commit af1f772, no license · at the source
Overview
- Department of Electrical and Computer Engineering, University of Delaware, Newark, Delaware, United States of America
- Department of Computer and Information Sciences, University of Delaware, Newark, Delaware, United States of America
Abstract
Predicting the neural response to natural images in the visual cortex requires extracting relevant features from the images and relating those feature to the observed responses. In this work, we optimize the feature extraction in order to maximize the information shared between the image features and the neural response across voxels in a given region of interest (ROI) extracted from the BOLD signal measured by functional magnetic resonance imaging (fMRI). We adapt contrastive learning (CL) to fine-tune a convolutional neural network, which was pretrained for image classification, such that a mapping of a given image’s features are more similar to the corresponding fMRI response than to the responses to other images. We exploit the Natural Scenes Dataset as organized for the Algonauts Project, which contains the high-resolution fMRI responses of eight subjects to tens of thousands of naturalistic images. We show that CL fine-tuning creates feature extraction models that enable higher encoding accuracy in both early and higher visual ROIs as compared to the features from the pretrained network. Quantitatively, the performance is similar to a baseline approach that directly uses a regression loss at the output of the network to tune it for fMRI response encoding. We investigate inter-subject transfer of the CL fine-tuned models, including subjects from the Natural Object Dataset, another lower-resolution dataset with 9 subjects. We also pool subjects for fine-tuning, which further improves encoding performance in early ROIs. Finally, we examine the performance of the fine-tuned models on common image classification tasks, explore the landscape of ROI-specific models by applying dimensionality reduction on the Bhattacharya dissimilarity matrix created using the predictions on those tasks, and show that these landscapes match those based on representational similarity analysis. Finally, we generate images via Stable Diffusion based on vector-space prompts created by aligning the CL-tuned models embeddings for different ROIs, showing that generated images have similar embeddings to the original but that estimates of the intrinsic dimension are lower for generated versus original representations.
Reproduced under the paper's license (CC BY), from the paper cited above.
Repository
Its files are read in the Code ↔ Paper reader above, with 5 matches between paragraphs and lines of code.
alexmul1114/fmri_encoding_contrastive_learning
af1f772661f2c5789154d2f1078b077569fceba4, 18 August 2026Availability: 1 check, the latest on 27 September 2026: the link answers
- 27 September 2026: the link answers
16 files
- code/
cl_model_training.py , Python, 68 lines - code/
cl_model_training_all_ro , Python, 87 linesis.py - code/
feature_extraction_model , Jupyter, 938 lines_landscape_imagenet.ipyn b - code/
fit_encoding_models.py , Python, 476 lines - code/
fit_encoding_models_nod. , Python, 552 lines, 1 matchpy - code/
generate_results.py , Python, 124 lines - code/
generate_voxel_roi_mappi , Python, 52 linesngs.py - code/
models.py , Python, 336 lines, 1 match - code/
nod/ , Shell, 43 linesdownload_data.sh - code/
nod_utils.py , Python, 326 lines - code/
pooled_model_training.py , Python, 87 lines - code/
reg_model_training.py , Python, 72 lines - code/
reg_model_training_all_r , Python, 90 linesois.py - code/
results_utils.py , Python, 890 lines, 2 matches - code/
utils.py , Python, 1,197 lines, 1 match - README.md, Text, 22 lines
The paper's code and data availability statement is in the Data section.
Tracing map
Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.
What the map holds:
- 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
- 15 scripts, each with its path and the digest of its content;
- 5 matches between paragraphs of the paper and lines of the code (method lexical-v1);
- neither the text of the paper nor the code itself.
Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.
Data
Datasets cited
- openneuro:ds004496, at OpenNeuro; found in “Data Availability”
Data Availability
All data we use in our paper is publicly available. The Natural Scenes Dataset is available at https://
Reproduced under the paper's license (CC BY), from the paper cited above.
Versions
The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.
Version 1, 27 September 2026: the first record
Recorded: type, language, journal, volume, issue, pages, dates, 3 authors, 9 MeSH terms, 6 funders, 36 references.
Cite
This paper
Mulrooney, A., Li, Z., & Brockmeier, A. J. (2026). Contrastive learning to fine-tune feature extraction models for the visual cortex. PLoS computational biology, 22(8), e1014656. https://
BibTeX
@article{mulrooney2026co
author = {Mulrooney, Alex and Li, Zhi and Brockmeier, Austin J},
title = {{Contrastive learning to fine-tune feature extraction models for the visual cortex}},
journal = {PLoS computational biology},
year = {2026},
month = aug,
volume = {22},
number = {8},
pages = {e1014656},
publisher = {PLOS},
issn = {1553-734X},
doi = {10.1371/
url = {https://
pmid = {42607104},
pmcid = {PMC13492995}
}
RIS
TY - JOUR
AU - Mulrooney, Alex
AU - Li, Zhi
AU - Brockmeier, Austin J
TI - Contrastive learning to fine-tune feature extraction models for the visual cortex
T2 - PLoS computational biology
J2 - PLoS Comput Biol
PY - 2026
DA - 2026/
VL - 22
IS - 8
SP - e1014656
SN - 1553-734X
PB - PLOS
DO - 10.1371/
UR - https://
LA - en
ER -
CSL-JSON
{
"id": "10.1371/
"type": "article-journal",
"title": "Contrastive learning to fine-tune feature extraction models for the visual cortex",
"container-title": "PLoS computational biology",
"author": [
{
"family": "Mulrooney",
"given": "Alex"
},
{
"family": "Li",
"given": "Zhi"
},
{
"family": "Brockmeier",
"given": "Austin J"
}
],
"container-title-short":
"volume": "22",
"issue": "8",
"page": "e1014656",
"DOI": "10.1371/
"PMID": "42607104",
"PMCID": "PMC13492995",
"ISSN": "1553-734X",
"publisher": "PLOS",
"URL": "https://
"language": "en",
"issued": {
"date-parts": [
[
2026,
8,
17
]
]
}
}
The tracing map gets a citation of its own once an author has validated it and it has a DOI.
Similar papers
The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.
- [1] doi:10.1038/s41467-024-50310-3 [code]
- Modeling short visual events through the BOLD moments video fMRI dataset and metadata.Journal: Nature communicationsIn common: Pillow, NiBabel, PyTorch, 4 other tools, fMRI, 9 references
- [2] doi:10.1038/s41597-026-07248-6 [code]
- A large-scale fMRI dataset for vision-language semantic association.Journal: Scientific dataIn common: Pillow, NiBabel, PyTorch, 5 other tools, fMRI, 8 references
- [3] doi:10.1016/j.neuroimage.2024.120772 [code]
- Stacked regressions and structured variance partitioning for interpretable brain maps.Journal: NeuroImageIn common: scikit-learn, pandas, SciPy, 2 other tools, 9 references
- [4] doi:10.1038/s41467-024-53147-y [code]
- A large-scale examination of inductive biases shaping high-level visual representation in brains and machines.Journal: Nature communicationsIn common: Pillow, NiBabel, PyTorch, 5 other tools, 7 references
- [5] doi:10.1038/s42003-026-10169-0 [code]
- Shared representations in brains and models reveal a two-route cortical organization during scene perception.Journal: Communications biologyIn common: Pillow, NiBabel, PyTorch, 5 other tools, 7 references
- [6] doi:10.1038/s42256-025-01072-0 [code]
- High-level visual representations in the human brain are aligned with large language models.Journal: Nature machine intelligenceIn common: Pillow, NiBabel, PyTorch, 5 other tools, 7 references
- [7] doi:10.1016/j.neuroimage.2022.119728 [code]
- Feature-space selection with banded ridge regression.Journal: NeuroImageIn common: Pillow, PyTorch, scikit-learn, 3 other tools, 7 references
- [8] doi:10.1162/imag_a_00575 [code]
- The Voxelwise Encoding Model framework: A tutorial introduction to fitting encoding models to fMRI data.Journal: Imaging neuroscience (Cambridge, Mass.)In common: DataLad, scikit-learn, SciPy, 2 other tools, fMRI, 7 references
- [9] doi:10.1038/s41467-022-28091-4 [code]
- A self-supervised domain-general learning framework for human ventral stream representation.Journal: Nature communicationsIn common: Pillow, PyTorch, scikit-learn, 4 other tools, 7 references
- [10] doi:10.1371/journal.pcbi.1011792 [code]
- High-performing neural network models of visual cortex benefit from high latent dimensionality.Journal: PLoS computational biologyIn common: Pillow, PyTorch, scikit-learn, 4 other tools, systems, 6 references
Contribute
The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.
Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.
Claim this paper
Correct its record
Say what each link of this record is, remove the ones that are not the paper's, add the ones that are missing. The correction becomes a new version of the record, in its Versions section.
Validate its tracing map
You validate the map as this page shows it: 1 repository of the authors' code, each at its verified commit and with its license, 15 scripts, and 5 matches between paragraphs and code (see the Code and Map sections). It then receives a DOI on Zenodo, with you (your ORCID iD) and OSCR as its creators; the code itself is not deposited.
The map's fingerprint: sha256:9b02a989610e16e9…
Add the badge to the code's README
If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.
Markdown
<!-- oscr-badge:start -->
[](https://openscicode.org/badge/doi/10.1371/journal.pcbi.1014656)
<!-- oscr-badge:end -->reStructuredText
.. oscr-badge:start
.. image:: https://openscicode.org/badge/doi/10.1371/journal.pcbi.1014656.svg
:target: https://openscicode.org/badge/doi/10.1371/journal.pcbi.1014656
:alt: OSCR
.. oscr-badge:endHTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.1371/journal.pcbi.1014656"><img src="https://openscicode.org/badge/doi/10.1371/journal.pcbi.1014656.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->On GitHub, open the README, choose Edit (the pencil), paste the snippet under the title, then "Commit changes".
If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.
Request its removal
To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).
Discussion, reproductions, activity
Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.
Report a mismatch between the code and the paper · Report a code error (signed in)
Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.
Report a reproduction failure (signed in)
Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.
Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.
