OSCR

A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies

Code ↔ Paper

20 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 20 matches · 2 of them tie a paragraph to a whole file, not to given lines: weak matches, whose lines are not tinted
  1. [1] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ Experiments/_Composite_score_generation/Save_Composite_Score_Brain.ipynb, lines 18–34 · score 0.95 · CelloScope, SpatialDecon, STDeconvolve, Cell DART, SpiceMix, DestVI
  2. [2] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ Experiments/_Deconvolution_Metrics_Calculation/Evaluation_Brain_Category.ipynb, lines 29–43 · score 0.95 · CelloScope, SpatialDecon, STDeconvolve, Cell DART, SpiceMix, DestVI
  3. [3] § Results › Benchmarking of spatial domain detection methods ↔ Experiments/_Domain_Detection_Metrics_Calculation/all_datasets_metric_computation.ipynb, lines 185–194 · score 0.93 · DR SC, DeepST, SpaSRL, SpatialPCA, BayesCafe, BayesSpace
  4. [4] § Results › Benchmarking of spatial domain detection methods ↔ Experiments/_Domain_Detection_Metrics_Calculation/Metrics.py, lines 351–421 · score 0.93 · DR SC, DeepST, SpaSRL, SpatialPCA, BayesCafe, BayesSpace
  5. [5] § Results › Benchmarking of spatial domain detection methods ↔ Experiments/_Domain_Detection_Metrics_Calculation/all_datasets_metric_computation.ipynb, lines 289–313 · score 0.91 · mouse breast cancer, osmFISH, chicken heart, prostate cancer, liver cancer, kidney cancer
  6. [6] § Results › Benchmarking of spatial domain detection methods ↔ Experiments/_Domain_Detection_Metrics_Calculation/Metrics.py, lines 944–1089 · score 0.86 · SpaSRL, SpatialPCA, BayesCafe, BayesSpace, SpaceFlow, GraphST
  7. [7] § Methods › Overview of SynthST ↔ SynthST/Simulator_CTP/SynthST/model.py, lines 56–99 · score 0.86 · cross entropy, weight decay, adjacency matrix, proportion matrix, L2, encoder
  8. [8] § Results › Benchmarking of spatial domain detection methods ↔ Experiments/_Domain_Detection_Metrics_Calculation/Metrics.py, lines 351–421 · score 0.84 · DeepST, BayesSpace, SpaceFlow, prostate cancer, GraphST, PRECAST
  9. [9] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ Experiments/_Domain_Detection_Metrics_Calculation/Metrics.py, lines 107–248 · score 0.78 · chicken heart, prostate cancer, liver cancer, kidney cancer, breast cancer, ground truth
  10. [10] § Methods › Evaluation metrics for domain detection ↔ Experiments/_Domain_Detection_Metrics_Calculation/Metrics.py, lines 423–497 · score 0.78 · Normalized Mutual, Adjusted Rand, Completeness Score, Homogeneity, ground truth, ARI
  11. [11] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ Experiments/_Composite_score_generation/plotting_files.py, lines 1–39 · score 0.72 · spatial Pearson, Spatial Metrics, composite score, Rare Cell, Cosine, Lee
  12. [12] § Methods › Overview of SynthST ↔ SynthST/Simulator_CTP/SynthST/SynthST.py, the whole file · a weak match · score 0.68 · weight decay, SynthST, GAT, optimization, inferred, loss
  13. [13] § Results › Cell2location, RCTD and SONAR achieve the highest accuracy in cell-type deconvolution ↔ Experiments/_Composite_score_generation/Save_Composite_Score_Brain.ipynb, lines 18–34 · score 0.67 · SpatialDWLS, DestVI, Cell2location, Redeconve, Composite Score, SSIM
  14. [14] § Results › Cell2location, RCTD and SONAR achieve the highest accuracy in cell-type deconvolution ↔ Experiments/_Deconvolution_Metrics_Calculation/Evaluation_Brain_Category.ipynb, lines 29–43 · score 0.66 · SpatialDWLS, DestVI, Cell2location, Redeconve, SSIM, Geary
  15. [15] § Extended Data ↔ SynthST/Simulator_CTP/SynthST/model.py, lines 56–99 · score 0.64 · cross entropy, adjacency matrix, proportion matrix, reconstructs, decoder, loss
  16. [16] § Methods › Overview of SynthST ↔ SynthST/Simulator_gene_expression/Simulate_gene_expression_notebook.ipynb, lines 58–86 · score 0.62 · SynthST, gene expression, Cell2location, signature, abundance, simulated
  17. [17] § Methods › Composite score metrics for cell-type deconvolution ↔ Experiments/_Composite_score_generation/plotting_files.py, lines 58–89 · score 0.58 · Spatial Score, Composite Rare, Composite Score, SSIM, Correlation, Moran
  18. [18] § Methods › Composite score metrics for cell-type deconvolution ↔ Experiments/_Deconvolution_Metrics_Calculation/metrics.py, lines 261–279 · score 0.55 · Recall curve, Precision, threshold, ground truth, AUPR, metric
  19. [19] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ SynthST/Simulator_gene_expression/Simulate_gene_expression_notebook.ipynb, lines 58–86 · score 0.52 · simulated gene expression, SynthST, signature, single cell, matrix, spots
  20. [20] § Results › Overview of spatial cell-type deconvolution benchmarking ↔ benchmarking_domain_detection/CCST/run_CCST.py, the whole file · a weak match · score 0.52 · single cell resolution, deep graph, gene expression, MERFISH, training, ground truth

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,193 lines · 47 KB · MIT · 5 matches

  1. # ----------------------- Imports -----------------------
  4. import os
  5. import re
  6. import numpy as np
  7. import pandas as pd
  8. import scanpy as sc
  9. import squidpy as sq
  10. import anndata as an
  11. import seaborn as sns
  12. import matplotlib.pyplot as plt
  13. from PIL import Image
  14. from matplotlib import font_manager as fm
  15. from scipy.spatial import distance_matrix
  16. from scipy.spatial.distance import squareform, pdist
  17. from sklearn.metrics import (
  18. adjusted_rand_score,
  19. normalized_mutual_info_score,
  20. silhouette_score,
  21. homogeneity_score,
  22. completeness_score
  23. )
  24. from sklearn.preprocessing import StandardScaler
  26. # ----------------------- Utility -----------------------
  27. def fx_1NN(i,location_in):
  28. location_in = np.array(location_in)
  29. dist_array = distance_matrix(location_in[i,:][None,:],location_in)[0,:]
  30. dist_array[i] = np.inf
  31. return np.min(dist_array)
  34. def fx_kNN(i,location_in,k,cluster_in):
  36. location_in = np.array(location_in)
  37. cluster_in = np.array(cluster_in)
  40. dist_array = distance_matrix(location_in[i,:][None,:],location_in)[0,:]
  41. dist_array[i] = np.inf
  42. ind = np.argsort(dist_array)[:k]
  43. cluster_use = np.array(cluster_in)
  44. if np.sum(cluster_use[ind]!=cluster_in[i])>(k/2):
  45. return 0
  46. else:
  47. return 1
  49. def _compute_CHAOS(clusterlabel, location):
  51. clusterlabel = np.array(clusterlabel)
  52. location = np.array(location)
  53. matched_location = StandardScaler().fit_transform(location)
  55. clusterlabel_unique = np.unique(clusterlabel)
  56. dist_val = np.zeros(len(clusterlabel_unique))
  57. count = 0
  58. for k in clusterlabel_unique:
  59. location_cluster = matched_location[clusterlabel==k,:]
  60. if len(location_cluster)<=2:
  61. continue
  62. n_location_cluster = len(location_cluster)
  63. results = [fx_1NN(i,location_cluster) for i in range(n_location_cluster)]
  64. dist_val[count] = np.sum(results)
  65. count = count + 1
  67. chaos = np.sum(dist_val)/len(clusterlabel)
  68. return np.exp(-0.5*chaos)
  70. def _compute_PAS(clusterlabel,location):
  72. clusterlabel = np.array(clusterlabel)
  73. location = np.array(location)
  74. matched_location = location
  75. results = [fx_kNN(i,matched_location,k=10,cluster_in=clusterlabel) for i in range(matched_location.shape[0])]
  76. return np.sum(results)/len(clusterlabel)
  78. def compute_ASW(pred,spatial_coords):
  79. distance_matrix = squareform(pdist(spatial_coords))
  80. sil = silhouette_score(X=distance_matrix, labels=pred, metric='precomputed')
  81. return (sil + 1)/2
  83. def LISI(coords, meta, label, perplexity=30, nn_eps=0):
  84. import rpy2.robjects as robjects
  85. from rpy2.robjects import pandas2ri
  86. pandas2ri.activate()
  87. from rpy2.robjects.packages import importr
  88. importr("lisi")
  89. if not isinstance(coords, pd.DataFrame):
  90. coords = pd.DataFrame(coords)
  91. if not isinstance(meta, pd.DataFrame):
  92. meta = pd.DataFrame(meta)
  93. meta = meta.loc[:, [label]]
  94. meta[label] = meta[label].astype(str)
  96. coords = robjects.conversion.py2rpy(coords)
  97. meta = robjects.conversion.py2rpy(meta)
  98. as_matrix = robjects.r["as.matrix"]
  99. lisi = robjects.r["compute_lisi"](as_matrix(coords), meta, label, perplexity, nn_eps)
  100. if isinstance(lisi, pd.DataFrame):
  101. lisi = lisi.values
  102. elif isinstance(lisi, np.recarray):
  103. lisi = [item[0] for item in lisi]
  105. return lisi
  107. # ----------------------- Import functions -----------------------
  108. def import_dataset(dataset_name, paths, mode = "evaluation"):
  109. """
  110. Load dataset given its name and a dictionary of file paths.
  112. Parameters
  113. ----------
  114. dataset_name : str
  115. Name of the dataset (used for branching logic).
  116. paths : dict
  117. Dictionary of file paths needed for that dataset.
  118. Must contain at least 'st_path'.
  119. Can contain 'gnd_path' if ground truth is from a CSV/TSV file.
  121. Returns
  122. -------
  123. gnd : pd.Series or pd.DataFrame
  124. Ground truth clusters.
  125. locs : pd.DataFrame
  126. Spatial coordinates.
  127. """
  128. # Load main AnnData object
  129. ST = an.read_h5ad(paths["st_path"])
  131. if dataset_name.startswith("DLPFC"):
  132. gnd = pd.read_csv(paths["gnd_path"], sep="\t")['layer_guess_reordered']
  133. gnd.index.name = "Index"
  134. locs = pd.DataFrame({
  135. "array_row": ST.obs.array_row,
  136. "array_col": ST.obs.array_col
  137. })
  139. elif dataset_name.startswith("embryo"):
  140. gnd = pd.DataFrame(ST.obs['annotation']).rename(columns={"annotation": "Cluster"})
  141. gnd.index.name = "Index"
  142. locs = pd.DataFrame({
  143. "array_row": ST.obsm["spatial"][:, 0],
  144. "array_col": ST.obsm["spatial"][:, 1]
  145. }, index=ST.obs_names)
  147. elif dataset_name == "mouse_breast_cancer":
  148. gnd = pd.DataFrame(ST.obs['ground_truth']).rename(columns={"annotation": "Cluster"})
  149. gnd.index.name = "Index"
  150. locs = pd.DataFrame({
  151. "array_row": ST.obs.array_row,
  152. "array_col": ST.obs.array_col
  153. })
  155. elif dataset_name.startswith("MERFISH_brain"):
  156. gnd = pd.DataFrame(ST.obs['ground_truth']).rename(columns={"annotation": "Cluster"})
  157. gnd.index.name = "Index"
  158. locs = pd.DataFrame({
  159. "array_row": ST.obsm["spatial"][:, 0],
  160. "array_col": ST.obsm["spatial"][:, 1]
  161. }, index=ST.obs_names)
  163. elif dataset_name == "simulated_breast_atlas":
  164. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  165. gnd.index.name = "Index"
  166. locs = pd.DataFrame({
  167. "array_row": ST.obsm["spatial"][:, 0],
  168. "array_col": ST.obsm["spatial"][:, 1]
  169. }, index=ST.obs_names)
  171. elif dataset_name == "osmFISH":
  172. gnd = pd.DataFrame(ST.obs['ground_truth']).rename(columns={"annotation": "Cluster"})
  173. gnd.index.name = "Index"
  174. locs = pd.DataFrame({
  175. "array_row": ST.obsm["spatial"][:, 0],
  176. "array_col": ST.obsm["spatial"][:, 1]
  177. }, index=ST.obs_names)
  179. elif dataset_name.startswith("simulated_kidney_cancer"):
  180. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  181. gnd.index.name = "Index"
  182. locs = pd.DataFrame({
  183. "array_row": ST.obs.new_x,
  184. "array_col": ST.obs.new_y
  185. })
  187. elif dataset_name.startswith("simulated_breast_cancer"):
  188. ST.obs_names_make_unique()
  189. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  190. gnd.index.name = "Index"
  191. locs = pd.DataFrame({
  192. "array_row": ST.obs.new_x,
  193. "array_col": ST.obs.new_y
  194. })
  196. elif dataset_name.startswith("simulated_liver_cancer"):
  197. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  198. gnd.index.name = "Index"
  199. locs = pd.DataFrame({
  200. "array_row": ST.obs.new_x,
  201. "array_col": ST.obs.new_y
  202. })
  204. elif dataset_name.startswith("simulated_intestine"):
  205. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  206. gnd.index.name = "Index"
  207. locs = pd.DataFrame({
  208. "array_row": ST.obs.new_x,
  209. "array_col": ST.obs.new_y
  210. })
  212. elif dataset_name == "simulated_chicken_heart":
  213. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  214. gnd.index.name = "Index"
  215. locs = pd.DataFrame({
  216. "array_row": ST.obs.array_row,
  217. "array_col": ST.obs.array_col
  218. })
  220. elif dataset_name == "simulated_prostate_cancer":
  221. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names)['Ground Truth']
  222. gnd.index.name = "Index"
  223. locs = pd.DataFrame({
  224. "array_row": ST.obs.x_new,
  225. "array_col": ST.obs.y_new
  226. })
  228. elif dataset_name == "simulated_cerebellum":
  229. gnd = pd.read_csv(paths["gnd_path"]).set_index(ST.obs_names.astype(int))['Ground Truth']
  230. gnd.index.name = "Index"
  231. locs = pd.DataFrame({
  232. "array_row": ST.obs.xcoord,
  233. "array_col": ST.obs.ycoord
  234. }).set_index(ST.obs_names.astype(int))
  236. else:
  237. raise ValueError(f"Incorrect input: {dataset_name}")
  239. if mode == "plot":
  240. locs = pd.DataFrame({
  241. "array_row": ST.obsm["spatial"][:, 0],
  242. "array_col": ST.obsm["spatial"][:, 1]
  243. }).set_index(ST.obs_names)
  244. if dataset_name == "simulated_cerebellum":
  245. locs.index = locs.index.astype(int)
  246. locs.index.name = "Index"
  248. return gnd, locs
  251. def load_prediction(dataset_name, method_name, path):
  252. """
  253. Load output file given its name and file path.
  255. Parameters
  256. ----------
  257. dataset_name : str
  258. Name of the dataset (used for branching logic).
  259. method_name : str
  260. Name of the method (used for branching logic).
  261. path : str
  262. Path to output file
  263. Returns
  264. -------
  265. pred : predicted clusters
  266. """
  268. if os.path.exists(path):
  270. if dataset_name.startswith("simulated_breast_cancer") and method_name in ["BASS","PRECAST", "BayesSpace" , "DR_SC"]:
  271. pred = pd.read_csv(path)
  272. pred = pred.set_index(pred.columns[0])
  273. if method_name == "PRECAST":
  274. pred = pred['cluster']
  275. pred.index = pred.index.str.replace(r'_\d+$', '', regex=True)
  277. elif(method_name == "banksy"):
  278. pred = pd.read_csv(path)
  279. pred = pred.set_index(pred.columns[0])
  280. columns_to_grab = [col for col in pred.columns if col.startswith("clust_M1_lam0.")]
  281. pred = pred[columns_to_grab]
  283. elif method_name == "giotto":
  284. pred = pd.read_csv(path)
  285. if "leiden_clus" in pred.columns:
  286. columns_to_grab = ["leiden_clus"]
  287. pred = pred.set_index(pred.columns[1])
  288. else:
  289. columns_to_grab = ["cluster"]
  290. pred = pred.set_index(pred.columns[0])
  291. pred = pred[columns_to_grab]
  293. elif method_name == "PRECAST" or method_name == "BayesCafe":
  294. pred = pd.read_csv(path)
  295. pred = pred.set_index(pred.columns[0])
  296. columns_to_grab = ["cluster"]
  297. pred = pred[columns_to_grab]
  299. elif method_name == "SpaceFlow":
  300. pred = pd.read_csv(path)
  301. pred = pred.set_index(pred.columns[1])
  302. columns_to_grab = ["Predicted_cell_label"]
  303. pred = pred[columns_to_grab]
  305. else:
  306. pred = pd.read_csv(path)
  307. pred = pred.set_index(pred.columns[0])
  309. else:
  310. print(f"path doesn't exist : {path}")
  311. print(f"{method_name} output missing for {dataset_name}")
  312. pred = None
  314. return pred
  317. # ----------------------- Metric Computationn -----------------------
  320. def compute_metrics(dataset_names, dataset_paths, method_names, pred_paths, error_log_file, output_dir, metric_names="all"):
  321. """
  322. Compute metric values given dataset and method names and paths
  324. Parameters
  325. ----------
  326. dataset_names : list
  327. List of datasets to compute on
  328. Options for dataset names present at bottom of this file
  329. dataset_paths: dictionary of dictionaries
  330. Dictionaries mapping dataset names to dictionaries with st_path and gnd_path needed for importing dataset
  331. method_names: list
  332. List of methods to compute on
  333. Options for method names present at bottom of this file
  334. pred_paths : dictionary of dictionaries
  335. Dictionary mapping dataset names and method names to path of output file
  336. metric_names : list or str
  337. List of metrics to compute or "all"
  338. error_log_file : str
  339. Path to error log file
  340. output_dir : str
  341. Path to save final metric compute files to
  342. Returns
  343. -------
  344. pred : predicted clusters
  345. """
  346. # Available metrics
  347. available_metrics = ["ARI", "NMI", "CHAOS", "PAS", "ASW", "HOM", "COM"]
  348. if metric_names == "all":
  349. metric_names = available_metrics
  351. # Initialize pivoted DataFrame for each metric
  352. method_names_dict = {
  353. "SCANIT": "SCANIT",
  354. "CCST": "CCST",
  355. "DeepST": "DeepST",
  356. "GraphST": "GraphST",
  357. "PROST": "PROST",
  358. "SpaSRL": "SpaSRL",
  359. "STAGATE": "STAGATE",
  360. "SpatialPCA": "SpatialPCA",
  361. "banksy": "Banksy",
  362. "giotto": "Giotto",
  363. "DR_SC": "DR.SC",
  364. "ISC_MEB": "ISC.MEB",
  365. "BayesSpace": "BayesSpace",
  366. "PRECAST": "PRECAST",
  367. "BayesCafe": "BayesCafe",
  368. "BASS": "BASS",
  369. "SpaceFlow" : "SpaceFlow",
  370. "IRIS" : "IRIS"
  371. }
  373. dataset_names_dict = {
  374. "DLPFC151507": "DLPFC 151507",
  375. "DLPFC151508": "DLPFC 151508",
  376. "DLPFC151509": "DLPFC 151509",
  377. "DLPFC151510": "DLPFC 151510",
  378. "DLPFC151669": "DLPFC 151669",
  379. "DLPFC151670": "DLPFC 151670",
  380. "DLPFC151671": "DLPFC 151671",
  381. "DLPFC151672": "DLPFC 151672",
  382. "DLPFC151673": "DLPFC 151673",
  383. "DLPFC151674": "DLPFC 151674",
  384. "DLPFC151675": "DLPFC 151675",
  385. "DLPFC151676": "DLPFC 151676",
  386. "embryo9.5": "Embryo 9.5",
  387. "embryo14.5": "Embryo 14.5",
  388. "mouse_breast_cancer": "Mouse Breast Cancer",
  389. "MERFISH_brain0.04": "MERFISH Brain 0.04",
  390. "MERFISH_brain0.09": "MERFISH Brain 0.09",
  391. "MERFISH_brain0.14": "MERFISH Brain 0.14",
  392. "MERFISH_brain0.19": "MERFISH Brain 0.19",
  393. "MERFISH_brain0.24": "MERFISH Brain 0.24",
  394. "osmFISH": "osmFISH",
  395. "simulated_kidney_cancer410": "Kidney Cancer 410",
  396. "simulated_kidney_cancer411": "Kidney Cancer 411",
  397. "simulated_kidney_cancer506": "Kidney Cancer 506",
  398. "simulated_breast_cancerER+_CID4290": "Breast Cancer ER+ CID 4290",
  399. "simulated_breast_cancerTNBC_CID44971": "Breast Cancer TNBC CID 44971",
  400. "simulated_liver_cancerHCC-1L": "Liver Cancer HCC-1L",
  401. "simulated_liver_cancerHCC-2L": "Liver Cancer HCC-2L",
  402. "simulated_liver_cancerHCC-3L": "Liver Cancer HCC-3L",
  403. "simulated_liver_cancerHCC-4L": "Liver Cancer HCC-4L",
  404. "simulated_breast_atlas": "Breast Atlas",
  405. "simulated_intestineA1": "Intestine A1",
  406. "simulated_intestineA2": "Intestine A2",
  407. "simulated_chicken_heart": "Chicken Heart",
  408. "simulated_prostate_cancer" : "Prostate Cancer",
  409. "simulated_cerebellum" : "Cerebellum"
  410. }
  414. metric_results = {
  415. metric: pd.DataFrame(
  416. index=list(method_names_dict.values()), columns=list(dataset_names_dict.values()), dtype=float
  417. ) for metric in metric_names
  418. }
  420. os.makedirs(output_dir, exist_ok=True)
  421. print(f"saving to {output_dir}")
  423. # Process each dataset
  424. for dataset_name in dataset_names:
  425. try:
  426. # Load dataset (ground truth and spatial coordinates)
  427. gnd, locs = import_dataset(dataset_name , dataset_paths[dataset_name])
  428. gnd = gnd.dropna()
  429. except Exception as e:
  430. with open(error_log_file, "a") as log:
  431. log.write(f"Dataset loading error: {dataset_name} - {e}\n")
  432. continue
  434. for method_name in method_names:
  435. try:
  436. # Load predictions
  437. print(f"Running on {dataset_name}_{method_name}")
  438. pred = load_prediction(dataset_name, method_name, pred_paths[dataset_name][method_name])
  440. # If predictions are missing, skip to the next method
  441. if pred is None:
  442. continue
  443. pred = pred[~pred.index.duplicated(keep='first')]
  445. # Ensure first columns are indices
  446. intersect_idx = gnd.index.intersection(pred.index)
  448. # Filter gnd and pred based on the intersection
  449. gnd_filtered = gnd.loc[intersect_idx]
  450. pred_filtered = pred.loc[intersect_idx]
  452. # Replace pred and gnd for further computation
  453. gnd_values = gnd_filtered.values.flatten()
  454. pred_values = pred_filtered.iloc[:, 0].values
  456. # Placeholder for spatial coordinates
  457. spatial_coords = locs.loc[intersect_idx].values
  459. # Compute metrics and populate the pivoted DataFrame
  460. for metric in metric_names:
  461. try:
  462. if metric == "ARI":
  463. value = adjusted_rand_score(gnd_values, pred_values)
  464. elif metric == "NMI":
  465. value = normalized_mutual_info_score(gnd_values, pred_values)
  466. elif metric == "CHAOS":
  467. value = _compute_CHAOS(pred_values, spatial_coords)
  468. elif metric == "PAS":
  469. value = _compute_PAS(pred_values, spatial_coords)
  470. elif metric == "ASW":
  471. value = compute_ASW(pred_values, spatial_coords)
  472. elif metric == "HOM":
  473. value = homogeneity_score(gnd_values, pred_values)
  474. elif metric == "COM":
  475. value = completeness_score(gnd_values, pred_values)
  476. else:
  477. continue
  479. # Populate the DataFrame
  480. metric_results[metric].loc[method_names_dict[method_name], dataset_names_dict[dataset_name]] = value
  482. except Exception as e:
  483. with open(error_log_file, "a") as log:
  484. log.write(f"Metric error: Dataset={dataset_name}, Method={method_name}, Metric={metric} - {e}\n")
  485. except Exception as e:
  486. with open(error_log_file, "a") as log:
  487. log.write(f"Prediction loading error: Dataset={dataset_name}, Method={method_name} - {e}\n")
  489. # Save each metric result as a CSV
  490. for metric, df in metric_results.items():
  491. try:
  492. output_file = os.path.join(output_dir, f"{metric}_results.csv")
  493. df.to_csv(output_file)
  494. print(f"Saved {metric} results to {output_file}")
  495. except Exception as e:
  496. with open(error_log_file, "a") as log:
  497. log.write(f"Metric saving error: Metric={metric} - {e}\n")
  499. def update_metrics(dataset_names, dataset_paths, method_names, pred_paths, error_log_file , output_dir,metric_names="all"):
  501. """
  502. Modify existing output files with new metric values given dataset and method names and paths
  504. Parameters
  505. ----------
  506. dataset_names : list
  507. List of datasets to compute on
  508. Options for dataset names present at bottom of this file
  509. dataset_paths: dictionary of dictionaries
  510. Dictionaries mapping dataset names to dictionaries with st_path and gnd_path needed for importing dataset
  511. method_names: list
  512. List of methods to compute on
  513. Options for method names present at bottom of this file
  514. pred_paths : dictionary of dictionaries
  515. Dictionary mapping dataset names and method names to path of output file
  516. metric_names : list or str
  517. List of metrics to compute or "all"
  518. error_log_file : str
  519. Path to error log file
  520. output_dir : str
  521. Path to save final metric compute files to
  522. Returns
  523. -------
  524. pred : predicted clusters
  525. """
  526. # Available metrics
  527. available_metrics = ["ARI", "NMI", "CHAOS", "PAS", "ASW", "HOM", "COM"]
  528. if metric_names == "all":
  529. metric_names = available_metrics
  530. method_names_dict = {
  531. "SCANIT": "SCANIT",
  532. "CCST": "CCST",
  533. "DeepST": "DeepST",
  534. "GraphST": "GraphST",
  535. "PROST": "PROST",
  536. "SpaSRL": "SpaSRL",
  537. "STAGATE": "STAGATE",
  538. "SpatialPCA": "SpatialPCA",
  539. "banksy": "Banksy",
  540. "giotto": "Giotto",
  541. "DR_SC": "DR.SC",
  542. "ISC_MEB": "ISC.MEB",
  543. "BayesSpace": "BayesSpace",
  544. "PRECAST": "PRECAST",
  545. "BayesCafe": "BayesCafe",
  546. "BASS": "BASS",
  547. "SpaceFlow":"SpaceFlow",
  548. "IRIS":"IRIS"
  549. }
  551. dataset_names_dict = {
  552. "DLPFC151507": "DLPFC 151507",
  553. "DLPFC151508": "DLPFC 151508",
  554. "DLPFC151509": "DLPFC 151509",
  555. "DLPFC151510": "DLPFC 151510",
  556. "DLPFC151669": "DLPFC 151669",
  557. "DLPFC151670": "DLPFC 151670",
  558. "DLPFC151671": "DLPFC 151671",
  559. "DLPFC151672": "DLPFC 151672",
  560. "DLPFC151673": "DLPFC 151673",
  561. "DLPFC151674": "DLPFC 151674",
  562. "DLPFC151675": "DLPFC 151675",
  563. "DLPFC151676": "DLPFC 151676",
  564. "embryo9.5": "Embryo 9.5",
  565. "embryo14.5": "Embryo 14.5",
  566. "mouse_breast_cancer": "Mouse Breast Cancer",
  567. "MERFISH_brain0.04": "MERFISH Brain 0.04",
  568. "MERFISH_brain0.09": "MERFISH Brain 0.09",
  569. "MERFISH_brain0.14": "MERFISH Brain 0.14",
  570. "MERFISH_brain0.19": "MERFISH Brain 0.19",
  571. "MERFISH_brain0.24": "MERFISH Brain 0.24",
  572. "osmFISH": "osmFISH",
  573. "simulated_kidney_cancer410": "Kidney Cancer 410",
  574. "simulated_kidney_cancer411": "Kidney Cancer 411",
  575. "simulated_kidney_cancer506": "Kidney Cancer 506",
  576. "simulated_breast_cancerER+_CID4290": "Breast Cancer ER+ CID 4290",
  577. "simulated_breast_cancerTNBC_CID44971": "Breast Cancer TNBC CID 44971",
  578. "simulated_liver_cancerHCC-1L": "Liver Cancer HCC-1L",
  579. "simulated_liver_cancerHCC-2L": "Liver Cancer HCC-2L",
  580. "simulated_liver_cancerHCC-3L": "Liver Cancer HCC-3L",
  581. "simulated_liver_cancerHCC-4L": "Liver Cancer HCC-4L",
  582. "simulated_breast_atlas": "Breast Atlas",
  583. "simulated_intestineA1": "Intestine A1",
  584. "simulated_intestineA2": "Intestine A2",
  585. "simulated_chicken_heart": "Chicken Heart",
  586. "simulated_prostate_cancer" : "Prostate Cancer",
  587. "simulated_cerebellum" : "Cerebellum"
  588. }
  589. # Initialize pivoted DataFrame for each metric
  590. metric_results = {}
  591. for metric in metric_names:
  592. path = f"{output_dir}/{metric}_results.csv"
  593. try:
  594. df = pd.read_csv(path,skip_blank_lines=True)
  595. df.set_index(df.columns[0], inplace=True, drop=True) # Set the first column as index
  596. df.index.name = None # Optional: remove index name
  597. metric_results[metric] = df
  598. except FileNotFoundError:
  599. # Create a new DataFrame if not already present
  600. metric_results[metric] = pd.DataFrame(index=method_names_dict.values(), columns=dataset_names_dict.values())
  601. print(f"Created new DataFrame for metric {metric}")
  604. os.makedirs(output_dir, exist_ok=True)
  606. # Process each dataset
  607. for dataset_name in dataset_names:
  608. try:
  609. # Load dataset (ground truth and spatial coordinates)
  610. gnd, locs = import_dataset(dataset_name , dataset_paths[dataset_name])
  611. gnd = gnd.dropna()
  613. except Exception as e:
  614. with open(error_log_file, "a") as log:
  615. log.write(f"Dataset loading error: {dataset_name} - {e}\n")
  616. continue
  618. for method_name in method_names:
  619. try:
  621. method_display_name = method_names_dict[method_name]
  622. # Load predictions
  623. print(f"Running on {dataset_name}_{method_name}")
  624. pred = load_prediction(dataset_name, method_name , pred_paths[dataset_name][method_name])
  627. # If predictions are missing, skip to the next method
  628. if pred is None:
  629. continue
  630. pred = pred[~pred.index.duplicated(keep='first')]
  632. # Ensure first columns are indices
  633. intersect_idx = gnd.index.intersection(pred.index)
  635. # Filter gnd and pred based on the intersection
  636. gnd_filtered = gnd.loc[intersect_idx]
  637. pred_filtered = pred.loc[intersect_idx]
  639. # Replace pred and gnd for further computation
  640. gnd_values = gnd_filtered.values.flatten()
  642. pred_values = pred_filtered.iloc[:,0].values
  643. # Placeholder for spatial coordinates
  644. spatial_coords = locs.loc[intersect_idx].values
  646. # Compute metrics and populate the pivoted DataFrame
  647. for metric in metric_names:
  648. try:
  649. if metric == "ARI":
  650. value = adjusted_rand_score(gnd_values, pred_values)
  651. elif metric == "NMI":
  652. value = normalized_mutual_info_score(gnd_values, pred_values)
  653. elif metric == "CHAOS":
  654. value = _compute_CHAOS(pred_values, spatial_coords)
  655. elif metric == "PAS":
  656. value = _compute_PAS(pred_values, spatial_coords)
  657. elif metric == "ASW":
  658. value = compute_ASW(pred_values, spatial_coords)
  659. elif metric == "HOM":
  660. value = homogeneity_score(gnd_values, pred_values)
  661. elif metric == "COM":
  662. value = completeness_score(gnd_values, pred_values)
  663. else:
  664. continue
  666. # Populate the DataFrame
  667. if method_display_name not in metric_results[metric].index:
  668. metric_results[metric].loc[method_display_name] = np.nan
  669. metric_results[metric].loc[method_names_dict[method_name], dataset_names_dict[dataset_name]] = value
  670. print(f"set value of {metric} to {value}")
  672. except Exception as e:
  673. with open(error_log_file, "a") as log:
  674. log.write(f"Metric error: Dataset={dataset_name}, Method={method_name}, Metric={metric} - {e}\n")
  676. except Exception as e:
  677. with open(error_log_file, "a") as log:
  678. log.write(f"Prediction loading error: Dataset={dataset_name}, Method={method_name} - {e}\n")
  680. # Save each metric result as a CSV
  682. for metric, df in metric_results.items():
  683. try:
  684. output_file = os.path.join(output_dir, f"{metric}_results.csv")
  685. df = df.iloc[:len(method_names_dict)] # Dynamic cutoff (good)
  686. df.to_csv(output_file)
  687. print(f"Saved {metric} results to {output_file}")
  688. except Exception as e:
  689. with open(error_log_file, "a") as log:
  690. log.write(f"Metric saving error: Metric={metric} - {e}\n")
  692. # ----------------------- Visualization Functions -----------------------
  695. def plot_stitched_heatmaps(metric_files, dataset_names, dataset_type, output_file):
  697. """
  698. Produce stitched heatmaps for all metrics
  700. Parameters
  701. ----------
  702. metric_files : dictionary
  703. Mapping of metric names to output files
  704. dataset_names: list
  705. Names of datasets to include in heatmap
  706. dataset_type: str
  707. Represents name of selected group of datasets (for example : simulated/real/DLPFC)
  708. Used only for naming
  709. output_file : str
  710. Path to save output image to
  711. """
  713. num_metrics = len(metric_files) + 1
  714. individual_plot_height = 8 # Height of each individual heatmap
  715. total_height = individual_plot_height * num_metrics # Total height for all metrics
  716. fig, axes = plt.subplots(4, 2, figsize=(10, total_height)) # Adjust figure size dynamically
  717. axes = axes.flatten()
  719. for ax, (metric, file_path) in zip(axes, metric_files.items()):
  720. try:
  721. data = pd.read_csv(file_path, index_col=0)
  722. data_subset = data[dataset_names]
  724. sns.heatmap(
  725. data_subset,
  726. cmap="coolwarm",
  727. cbar_kws={'label': metric},
  728. annot=True, # Add annotations
  729. fmt=".2f", # Format for annotations
  730. annot_kws={"size": 6 ,"weight": "bold" }, # Small font size for annotations
  731. ax=ax
  732. )
  733. ax.set_title(f"{dataset_type} Dataset Heatmap - {metric}" , fontweight = "bold")
  734. ax.set_xlabel("Datasets" , fontweight = "bold")
  735. ax.set_ylabel("Methods" , fontweight = "bold")
  737. labels = ax.get_xticklabels()
  738. bold_font = fm.FontProperties(weight='bold')
  739. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  741. y_labels = ax.get_yticklabels()
  742. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  744. except Exception as e:
  745. ax.set_visible(False) # Hide axes if there's an error
  746. print(f"Error processing metric {metric}: {e}")
  748. try:
  749. metrics_to_average = ["NMI", "ARI", "ASW", "HOM", "COM"]
  750. metric_dfs = []
  751. for metric in metrics_to_average:
  752. if metric in metric_files:
  753. data = pd.read_csv(metric_files[metric], index_col=0)
  754. data_subset = data[dataset_names]
  755. metric_dfs.append(data_subset)
  757. # Compute composite score
  758. composite_data = sum(metric_dfs) / len(metrics_to_average)
  760. sns.heatmap(
  761. composite_data,
  762. cmap="coolwarm",
  763. cbar_kws={'label': "composite"},
  764. annot=True, # Add annotations
  765. fmt=".2f", # Format for annotations
  766. annot_kws={"size": 6 ,"weight": "bold" }, # Small font size for annotations
  767. ax=axes[-1]
  768. )
  769. ax.set_title(f"{dataset_type} Dataset Heatmap - Composite" , fontweight = "bold")
  770. ax.set_xlabel("Datasets" , fontweight = "bold")
  771. ax.set_ylabel("Methods" , fontweight = "bold")
  773. labels = ax.get_xticklabels()
  774. bold_font = fm.FontProperties(weight='bold')
  775. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  777. y_labels = ax.get_yticklabels()
  778. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  780. except Exception as e:
  781. axes[-1].set_visible(False) # Hide composite score plot if there's an error
  782. print(f"Error computing composite score: {e}")
  786. plt.tight_layout() # Ensure spacing between plots
  787. plt.savefig(output_file, bbox_inches="tight", dpi = 200)
  788. plt.close()
  790. def plot_individual_heatmaps(metric_files, dataset_names, dataset_type, output_dir):
  792. """
  793. Produce individual heatmaps for metrics of choice
  795. Parameters
  796. ----------
  797. metric_files : dictionary
  798. Mapping of metric names to output files
  799. dataset_names: list
  800. Names of datasets to include in heatmap
  801. dataset_type: str
  802. Represents name of selected group of datasets (for example : simulated/real/DLPFC)
  803. Used only for naming
  804. output_file : str
  805. Path to save output image to
  806. """
  807. os.makedirs(output_dir, exist_ok=True)
  808. metrics = ["ARI", "NMI", "CHAOS", "PAS", "ASW", "HOM", "COM"]
  809. metric_dfs = []
  811. for metric, file_path in metric_files.items():
  812. try:
  813. data = pd.read_csv(file_path, index_col=0)
  814. data_subset = data[dataset_names]
  816. plt.figure(figsize=(8, 6))
  817. ax = sns.heatmap(
  818. data_subset,
  819. cmap="coolwarm",
  820. cbar_kws={'label': metric},
  821. annot=True,
  822. fmt=".2f",
  823. annot_kws={"size": 6, "weight": "bold"}
  824. )
  825. ax.set_title(f"{dataset_type} Dataset Heatmap - {metric}", fontweight="bold")
  826. ax.set_xlabel("Datasets", fontweight="bold")
  827. ax.set_ylabel("Methods", fontweight="bold")
  829. labels = ax.get_xticklabels()
  830. bold_font = fm.FontProperties(weight='bold')
  831. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  833. y_labels = ax.get_yticklabels()
  834. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  836. output_path = os.path.join(output_dir, f"{dataset_type}_Heatmap_{metric}.png")
  837. plt.tight_layout()
  838. plt.savefig(output_path, bbox_inches="tight")
  839. plt.close()
  841. if metric in metrics:
  842. metric_dfs.append(data_subset)
  844. except Exception as e:
  845. print(f"Error processing metric {metric}: {e}")
  848. try:
  849. if metric_dfs:
  850. composite_data_accuracy = sum(metric_dfs[0:4]) / 4
  851. composite_data_accuracy.to_csv(os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite_accuracy.csv"))
  852. plt.figure(figsize=(8, 6))
  853. ax = sns.heatmap(
  854. composite_data_accuracy,
  855. cmap="coolwarm",
  856. cbar_kws={'label': "composite - accuracy"},
  857. annot=True,
  858. fmt=".2f",
  859. annot_kws={"size": 6, "weight": "bold"}
  860. )
  861. ax.set_title(f"{dataset_type} Dataset Heatmap - Composite (accuracy)", fontweight="bold")
  862. ax.set_xlabel("Datasets", fontweight="bold")
  863. ax.set_ylabel("Methods", fontweight="bold")
  865. labels = ax.get_xticklabels()
  866. bold_font = fm.FontProperties(weight='bold')
  867. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  869. y_labels = ax.get_yticklabels()
  870. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  872. output_path = os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite_accuracy.png")
  873. plt.tight_layout()
  874. plt.savefig(output_path, bbox_inches="tight", dpi=200)
  875. plt.close()
  876. except Exception as e:
  877. print(f"Error computing composite accuracy score: {e}")
  879. try:
  880. if metric_dfs:
  881. composite_data_consistency = sum(metric_dfs[4:7]) / 3
  882. composite_data_consistency.to_csv(os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite_consistency.csv"))
  884. plt.figure(figsize=(8, 6))
  885. ax = sns.heatmap(
  886. composite_data_consistency,
  887. cmap="coolwarm",
  888. cbar_kws={'label': "composite - consistency"},
  889. annot=True,
  890. fmt=".2f",
  891. annot_kws={"size": 6, "weight": "bold"}
  892. )
  893. ax.set_title(f"{dataset_type} Dataset Heatmap - Composite (consistency)", fontweight="bold")
  894. ax.set_xlabel("Datasets", fontweight="bold")
  895. ax.set_ylabel("Methods", fontweight="bold")
  897. labels = ax.get_xticklabels()
  898. bold_font = fm.FontProperties(weight='bold')
  899. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  901. y_labels = ax.get_yticklabels()
  902. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  904. output_path = os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite_consistency.png")
  905. plt.tight_layout()
  906. plt.savefig(output_path, bbox_inches="tight", dpi=200)
  907. plt.close()
  908. except Exception as e:
  909. print(f"Error computing composite consistency score: {e}")
  911. try:
  912. if metric_dfs:
  913. composite_data = (composite_data_accuracy + composite_data_consistency)/2
  914. composite_data.to_csv(os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite.csv"))
  916. plt.figure(figsize=(8, 6))
  917. ax = sns.heatmap(
  918. composite_data,
  919. cmap="coolwarm",
  920. cbar_kws={'label': "composite"},
  921. annot=True,
  922. fmt=".2f",
  923. annot_kws={"size": 6, "weight": "bold"}
  924. )
  925. ax.set_title(f"{dataset_type} Dataset Heatmap - Composite", fontweight="bold")
  926. ax.set_xlabel("Datasets", fontweight="bold")
  927. ax.set_ylabel("Methods", fontweight="bold")
  929. labels = ax.get_xticklabels()
  930. bold_font = fm.FontProperties(weight='bold')
  931. ax.set_xticklabels(labels, rotation=45, ha="right", fontproperties=bold_font)
  933. y_labels = ax.get_yticklabels()
  934. ax.set_yticklabels(y_labels, fontproperties=bold_font)
  936. output_path = os.path.join(output_dir, f"{dataset_type}_Heatmap_Composite.png")
  937. plt.tight_layout()
  938. plt.savefig(output_path, bbox_inches="tight", dpi=200)
  939. plt.close()
  940. except Exception as e:
  941. print(f"Error computing composite score: {e}")
  944. def make_embedding_plots(dataset_name,dataset_paths,pred_paths,save_dir, method_names = ["ground_truth","SCANIT", "CCST" , "DeepST" , "GraphST" , "PROST" , "SpaSRL" , "STAGATE","SpatialPCA" , "banksy" , "giotto" , "DR_SC" , "ISC_MEB" , "BayesSpace" , "PRECAST" , "BayesCafe" , "BASS"],stitch=True):
  945. """
  946. Produce embedding plots for select datasets and method outputs
  948. Parameters
  949. ----------
  950. dataset_name : str
  951. Name of dataset to create plot for
  952. dataset_paths : dict
  953. Dictionary of file paths needed for that dataset.
  954. Must contain at least 'st_path'.
  955. Can contain 'gnd_path' if ground truth is from a CSV/TSV file.
  956. pred_paths: dict of dicts
  957. Dictionary of output file paths for those datasets and names
  958. saved as a dataset x method dictionary
  959. save_dir : str
  960. Path to save output images to
  961. """
  963. method_names_dict = {
  964. "SCANIT": "SCANIT",
  965. "CCST": "CCST",
  966. "DeepST": "DeepST",
  967. "GraphST": "GraphST",
  968. "PROST": "PROST",
  969. "SpaSRL": "SpaSRL",
  970. "STAGATE": "STAGATE",
  971. "SpatialPCA": "SpatialPCA",
  972. "banksy": "Banksy",
  973. "giotto": "Giotto",
  974. "DR_SC": "DR.SC",
  975. "ISC_MEB": "ISC.MEB",
  976. "BayesSpace": "BayesSpace",
  977. "PRECAST": "PRECAST",
  978. "BayesCafe": "BayesCafe",
  979. "BASS": "BASS",
  980. "SpaceFlow":"SpaceFlow",
  981. "IRIS" : "IRIS"
  982. }
  984. dataset_names_dict = {
  985. "DLPFC151507": "DLPFC 151507",
  986. "DLPFC151508": "DLPFC 151508",
  987. "DLPFC151509": "DLPFC 151509",
  988. "DLPFC151510": "DLPFC 151510",
  989. "DLPFC151669": "DLPFC 151669",
  990. "DLPFC151670": "DLPFC 151670",
  991. "DLPFC151671": "DLPFC 151671",
  992. "DLPFC151672": "DLPFC 151672",
  993. "DLPFC151673": "DLPFC 151673",
  994. "DLPFC151674": "DLPFC 151674",
  995. "DLPFC151675": "DLPFC 151675",
  996. "DLPFC151676": "DLPFC 151676",
  997. "embryo9.5": "Embryo 9.5",
  998. "embryo14.5": "Embryo 14.5",
  999. "mouse_breast_cancer": "Mouse Breast Cancer",
  1000. "MERFISH_brain0.04": "MERFISH Brain 0.04",
  1001. "MERFISH_brain0.09": "MERFISH Brain 0.09",
  1002. "MERFISH_brain0.14": "MERFISH Brain 0.14",
  1003. "MERFISH_brain0.19": "MERFISH Brain 0.19",
  1004. "MERFISH_brain0.24": "MERFISH Brain 0.24",
  1005. "osmFISH": "osmFISH",
  1006. "simulated_kidney_cancer410": "Kidney Cancer 410",
  1007. "simulated_kidney_cancer411": "Kidney Cancer 411",
  1008. "simulated_kidney_cancer506": "Kidney Cancer 506",
  1009. "simulated_breast_cancerER+_CID4290": "Breast Cancer ER+ CID 4290",
  1010. "simulated_breast_cancerTNBC_CID44971": "Breast Cancer TNBC CID 44971",
  1011. "simulated_liver_cancerHCC-1L": "Liver Cancer HCC-1L",
  1012. "simulated_liver_cancerHCC-2L": "Liver Cancer HCC-2L",
  1013. "simulated_liver_cancerHCC-3L": "Liver Cancer HCC-3L",
  1014. "simulated_liver_cancerHCC-4L": "Liver Cancer HCC-4L",
  1015. "simulated_breast_atlas": "Breast Atlas",
  1016. "simulated_intestineA1": "Intestine A1",
  1017. "simulated_intestineA2": "Intestine A2",
  1018. "simulated_chicken_heart": "Chicken Heart",
  1019. "simulated_prostate_cancer" : "Prostate Cancer",
  1020. "simulated_cerebellum" : "Cerebellum"
  1021. }
  1025. gnd, locs = import_dataset(dataset_name, dataset_paths[dataset_name], mode = "plot")
  1027. for method_name in method_names:
  1028. if method_name == "ground_truth":
  1029. gnd = gnd.squeeze()
  1030. adata = an.AnnData(obs=pd.DataFrame({"cluster": gnd.astype(str)}))
  1031. adata.obsm["spatial"] = np.array(locs)
  1032. else:
  1033. pred = load_prediction(dataset_name,method_name, pred_paths[dataset_name][method_name])
  1034. if pred is None:
  1035. print(f"output for {dataset_name} {method_name} not present")
  1036. continue
  1037. pred = pred[~pred.index.duplicated(keep='first')]
  1038. intersect_idx = gnd.index.intersection(pred.index)
  1039. pred_filtered = pred.loc[intersect_idx]
  1041. # Handle both DataFrame and Series cases safely
  1042. if isinstance(pred_filtered, pd.DataFrame):
  1043. if pred_filtered.shape[1] > 1:
  1044. # If multiple columns (e.g. banksy), take the first or warn
  1045. print(f"Warning: {method_name} prediction for {dataset_name} has multiple columns; using the first one.")
  1046. pred_final = pred_filtered.iloc[:, 0]
  1047. else:
  1048. pred_final = pred_filtered
  1050. pred_final = pred_final.squeeze() # Ensure Series, not DataFrame
  1051. pred_final.index = intersect_idx # Ensure correct indexing
  1052. # pred_values = pred_filtered.iloc[:, 0].values
  1053. spatial_coords = locs.loc[intersect_idx].values
  1054. adata = an.AnnData(obs=pd.DataFrame({"cluster": pred_final.astype(str)}))
  1055. adata.obsm['spatial'] = np.array(spatial_coords)
  1058. # Ensure the directory exists
  1059. if not os.path.exists(save_dir):
  1060. os.makedirs(save_dir)
  1061. print(f"Created directory: {save_dir}")
  1063. # Generate and save the plot
  1064. plt.rcParams['font.weight'] = 'bold' # Bold for all text
  1065. plt.rcParams['axes.titleweight'] = 'bold' # Bold for titles
  1066. plt.rcParams['axes.labelweight'] = 'bold'
  1067. fig = sc.pl.embedding(
  1068. adata,
  1069. basis="spatial", # Use the 'spatial' embedding
  1070. color="cluster", # Color points by the 'cluster' column
  1071. title=f"{dataset_names_dict[dataset_name]} - {"Ground Truth" if method_name == "ground_truth" else method_names_dict[method_name]}",
  1072. size=100,
  1073. alpha=1,
  1074. cmap="coolwarm",
  1075. show = True,
  1076. return_fig=True
  1077. # save=f"{dataset_name}_{method_name}.png" # File will be saved in the default directory by Scanpy
  1078. )
  1081. # Move the file to the desired directory
  1082. current_dir = os.getcwd()
  1083. plot_path = os.path.join(current_dir, f"figures/spatial{dataset_name}.png")
  1084. if not os.path.exists(os.path.join(save_dir, f"{dataset_name}")):
  1085. os.makedirs(os.path.join(save_dir, f"{dataset_name}"))
  1086. new_path = os.path.join(save_dir, f"{dataset_name}/{method_name}.png")
  1087. fig.savefig(new_path, dpi=200, bbox_inches="tight", facecolor="white")
  1088. print(f"figure saved to {new_path}")
  1089. plt.close(fig)
  1092. # if os.path.exists(plot_path):
  1093. # os.rename(plot_path, new_path)
  1094. # print(f"Moved plot to: {new_path}")
  1095. # else:
  1096. # print(f"Plot file not found at: {plot_path}")
  1100. # Helper functions to stitch embedding plots together
  1101. def get_all_files(folder_path):
  1102. file_names = [f for f in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, f))]
  1103. png_files = [f for f in file_names if f.lower().endswith('.png')]
  1104. png_files.sort()
  1106. ground_truth = 'ground_truth.png' # Replace with the actual name or pattern for the ground truth file
  1107. if ground_truth in png_files:
  1108. png_files.remove(ground_truth)
  1109. png_files.insert(0, ground_truth)
  1110. return png_files
  1112. def save_grid_image(grid_rows, grid_cols, folder_path, dataset_name, save_path, width, height):
  1113. png_files = get_all_files(folder_path)
  1114. image_paths = [os.path.join(folder_path, p) for p in png_files]
  1116. images = [Image.open(img) for img in image_paths]
  1117. fig, axes = plt.subplots(grid_rows, grid_cols, figsize=(width, height), facecolor="white")
  1118. fig.subplots_adjust(wspace=0.0, hspace=0.0) # Reduced spacing between rows and columns
  1119. axes = axes.flatten()
  1120. for ax, img in zip(axes, images + [None] * (grid_rows * grid_cols - len(images))):
  1121. if img is not None:
  1122. ax.imshow(img)
  1123. ax.axis("off") # Remove axes
  1125. output_filename = dataset_name + "_grid_image.png"
  1126. plt.savefig(os.path.join(save_path, output_filename), dpi=200, bbox_inches="tight", facecolor="white")
  1127. plt.close()
  1128. print(f"Grid saved as {output_filename} with 200 DPI.")
  1130. # ----------------------- Composite Score Computation (over data subsets) -----------------------
  1133. def return_custom_score(metric_files, dataset_names):
  1134. metric_dfs = []
  1136. for metric, file_path in metric_files.items():
  1137. try:
  1138. data = pd.read_csv(file_path, index_col=0)
  1139. data_subset = data[dataset_names]
  1140. metric_dfs.append(data_subset)
  1142. except Exception as e:
  1143. print(f"Error processing metric {metric}: {e}")
  1146. try:
  1147. if metric_dfs:
  1148. composite_data_accuracy = sum(metric_dfs[0:4]) / 4
  1150. except Exception as e:
  1151. print(f"Error computing composite accuracy score: {e}")
  1153. try:
  1154. if metric_dfs:
  1155. composite_data_consistency = sum(metric_dfs[4:7]) / 3
  1158. except Exception as e:
  1159. print(f"Error computing composite consistency score: {e}")
  1161. try:
  1162. if metric_dfs:
  1163. composite_data = (composite_data_accuracy + composite_data_consistency)/2
  1165. except Exception as e:
  1166. print(f"Error computing composite score: {e}")
  1168. composite_data = np.sum(composite_data, axis = 1)/len(dataset_names)
  1169. top_5_rows = (
  1170. composite_data # Convert to Series with MultiIndex (row, column)
  1171. .sort_values(ascending=False) # Sort in descending order
  1172. .head(5) # Get top 5
  1173. .index.get_level_values(0) # Extract row names
  1174. .tolist() # Convert to list
  1175. )
  1178. return(composite_data, top_5_rows)
  1181. # ----------------------- Useful Variables (comment out to use) -----------------------
  1182. # dataset_names = ["DLPFC151507" , "DLPFC151508" , "DLPFC151509","DLPFC151510","DLPFC151669","DLPFC151670","DLPFC151671","DLPFC151672","DLPFC151673","DLPFC151674","DLPFC151675","DLPFC151676","embryo9.5" , "embryo14.5" , "mouse_breast_cancer", "MERFISH_brain0.04", "MERFISH_brain0.09", "MERFISH_brain0.14", "MERFISH_brain0.19", "MERFISH_brain0.24","osmFISH" , "simulated_kidney_cancer410" , "simulated_kidney_cancer411" , "simulated_kidney_cancer506" , "simulated_breast_cancerER+_CID4290" , "simulated_breast_cancerTNBC_CID44971" , "simulated_liver_cancerHCC-1L", "simulated_liver_cancerHCC-2L", "simulated_liver_cancerHCC-3L", "simulated_liver_cancerHCC-4L" , "simulated_breast_atlas" , "simulated_intestineA1", "simulated_intestineA2","simulated_chicken_heart", "simulated_prostate_cancer", "simulated_cerebellum"]
  1184. # DLPFC_dataset_names = ["DLPFC151507" , "DLPFC151508" , "DLPFC151509","DLPFC151510","DLPFC151669","DLPFC151670","DLPFC151671","DLPFC151672","DLPFC151673","DLPFC151674","DLPFC151675","DLPFC151676"]
  1186. # real_dataset_names = ["embryo9.5" , "embryo14.5" , "mouse_breast_cancer", "MERFISH_brain0.04", "MERFISH_brain0.09", "MERFISH_brain0.14", "MERFISH_brain0.19", "MERFISH_brain0.24","osmFISH"]
  1188. # simulated_dataset_names = ["simulated_kidney_cancer410" , "simulated_kidney_cancer411" , "simulated_kidney_cancer506" , "simulated_breast_cancerER+_CID4290" , "simulated_breast_cancerTNBC_CID44971" , "simulated_liver_cancerHCC-1L", "simulated_liver_cancerHCC-2L", "simulated_liver_cancerHCC-3L", "simulated_liver_cancerHCC-4L" , "simulated_breast_atlas" , "simulated_intestineA1", "simulated_intestineA2","simulated_chicken_heart", "simulated_prostate_cancer", "simulated_cerebellum"]
  1190. # method_names = ["SCANIT", "CCST" , "DeepST" , "GraphST" , "PROST" , "SpaSRL" , "STAGATE","SpatialPCA" , "banksy" , "giotto" , "DR_SC" , "ISC_MEB" , "BayesSpace" , "PRECAST" , "BayesCafe" , "BASS","SpaceFlow", "IRIS"]

Metrics.py at commit d3b2bec, under MIT · at the source

Overview

Authors: Ajita Shree1, V Aditya2, Tanush Kumar2, Hamim Zafar1,3,4
ORCID iDs: Hamim Zafar
  1. Department of Computer Science and Engineering, Indian Institute of Technology Kanpur
  2. Department of Mathematics and Statistics, Indian Institute of Technology Kanpur
  3. Department of Biological Sciences and Bioengineering, Indian Institute of Technology Kanpur
  4. Mehta Family Centre for Engineering in Medicine, Indian Institute of Technology Kanpur
Institutions: Indian Institute of Technology Kanpur (India)
Journal: Research Square (preprint)
Dates: published online 10 June 2026
Type: Preprint
License: CC BY
Identifiers: DOI 10.21203/rs.3.rs-9676637/v1 · OpenAlex W7164197983
Open access: green, a free copy (OpenAlex)
Status: code verified
Categories: genetics / omics (modality), methods / tools (subfield)
Methods: Connectivity, Machine learning
Keywords: Spatial transcriptomics, Spatial deconvolution, Spatial domain detection, Bivariate metrics
Topic: Single-cell and spatial transcriptomics (Molecular Biology, Biochemistry, Genetics and Molecular Biology), according to OpenAlex
Citations: not cited yet (Europe PMC); 107 references in the paper

Abstract

Spatial transcriptomic technologies enable high-resolution characterization of gene expression patterns and reconstruction of cellular architecture within tissue contexts. Two key computational problems have emerged for analyzing these datasets: spatial deconvolution, for disentangling celltype compositions at spatial locations, and spatial domain detection, for identifying spatially coherent regions within a tissue section. Although numerous methods have been developed for each task, a comprehensive and unified benchmarking study spanning diverse tissue types, spatial resolutions, and technological platforms remains lacking, hindering informed method selection by end users and impeding future methodological advancements. Here, we present spDDB (https://github.com/Zafar-Lab/spDDB), a comprehensive benchmarking framework for spatial deconvolution and domain detection methods across a large and diverse collection of datasets spanning multiple tissues, technologies, and biological conditions. We evaluated 21 deconvolution methods, including seven recently-developed approaches, across 37 datasets curated from brain, cancer, and organ tissues encompassing four distinct technologies. To enable rigorous evaluation, we introduced SynthST, a deep graph attention autoencoder-based simulator that generates realistic spatial cell-type distributions from spatial transcriptomic data, and employed a suite of spatial bivariate metrics including a novel bivariate Geary’s C metric, alongside rare cell-type, and cell-shape characterization metrics, for multidimensional performance assessment. While Cell2location, RCTD and SONAR emerged as top-performing methods for spatial deconvolution across tissue types, deconvolution performance varied substantially based on tissue architecture, spatial technology, dataset scale, and cell type diversity. For domain detection, we benchmarked 18 methods across 36 datasets spanning six spatial technologies, identifying PROST, BASS, and SpaceFlow as the leading approaches, while revealing notable limitations of existing methods in handling large-scale datasets. Finally, we provide practical guidelines to assist end users in selecting optimal methods for both tasks across diverse experimental settings.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 20 matches between paragraphs and lines of code.

Zafar-Lab/spDDB

License: MIT
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: d3b2bec027726acb47fe5a57b0ea8bb1dffea380, 7 June 2026
Languages: Jupyter (60), Python (31), R (21)
Size: 164 files, 112 scripts
Software Heritage: not archived
Found in: “Code availability”
Holds: README, license file, environment (docs/requirements.txt), documentation, 35 notebooks
Not found: CITATION.cff, tests, continuous integration
Tools: pandas (81 files), NumPy (76 files), Matplotlib (66 files), Scanpy (62 files), seaborn (44 files), scikit-learn (43 files), SciPy (38 files), PyTorch (29 files), anndata (24 files), TensorFlow (12 files), Seurat (9 files), SingleCellExperiment (6 files), Squidpy (5 files), ggplot2 (4 files), UMAP (4 files), OpenCV (3 files), Pillow (3 files), PyTorch Geometric (3 files), rpy2 (3 files), tidyverse (3 files), h5py (2 files), cowplot (1 file), data.table (1 file), reticulate (1 file), scikit-image (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
114 files
At the source: github.com/Zafar-Lab/spDDB

Code availability

Our benchmarking pipeline for both the tasks is provided as reproducible pipeline at https://github.com/Zafar-Lab/spDDB. The simulator code is available at https://github.com/Zafar-Lab/spDDB/tree/main/SynthST and code for all the evaluation experiments at https://github.com/Zafar-Lab/spDDB/tree/main/Experiments.

Reproduced under the paper's license (CC BY), from the paper cited above.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 112 scripts, each with its path and the digest of its content;
  • 20 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data availability

All datasets used in the study are publicly available. The simulated spatial gene expression datasets generated for spatial deconvolution and domain detection benchmarking have been deposited on figshare and are accessible through the project website: https://zafar-lab.github.io/spDDB_datasets.github.io/. Detailed information about all datasets used in this study is provided in Supplementary Tables 2 and 11.

For the spatial cell-type deconvolution task, the spatial and single cell gene expression datasets used to generate simulated spatial gene expression datasets are as follows. The spatial and scRNA dataset for DLPFC are available at ref (46) and (47) respectively, Mouse Brain at ref (39), Hippocampus at ref (24), Cerebellum at ref (24), Visium HD Mouse brain at ref (45) (https://www.10xgenomics.com/datasets, scRNA data at ref (39)), Kidney Cancer at ref (49), Breast Cancer at ref (50), Liver Cancer at ref (51) and (52) respectively, Prostate Cancer at ref (53), Visium HD Lung Cancer at ref (45) (https://www.10xgenomics.com/datasets)and (54), Developmental Lung at ref (56), Human Breast Atlas at ref (50), Intestine at ref (58), Mouse Liver Atlas at ref (59), Human Liver Atlas at ref (59), Kidney Atlas at ref (60) and Chicken heart at ref (57). For the deconvolution task on the simulation strategy 2 datasets, MERFISH preoptic hypothalamic brain dataset is available at ref (48), MERFISH Ileum is available at ref (61), MERFISH Breast Cancer at ref (55), and MERFISH Lung Cancer at ref (55).

For the domain detection task, Tthe spatial and single cell gene expression datasets for DLPFC is available at ref(46) (spatial:https://research.libd.org/spatialLIBD/ (107)) and (47) respectively, Mouse embryo at ref (88)https://db.cngb.org/stomics/mosta/ and (89) respectively, Mouse breast cancer at ref (3) and (90) respectively, osmFISH Mouse cortex at ref (91) (http://sdmbench.drai.cn/ (20)), and (92) respectively, and MERFISH brain at ref (48) (http://sdmbench.drai.cn/ (20)). All other data supporting the findings of this study are available within the article and its supplementary files. Any additional requests for information can be directed to, and will be fulfilled by, the corresponding author.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, journal, dates, 4 authors, 4 keywords, 92 references.

Cite

This paper

Shree, A., Aditya, V., Kumar, T., & Zafar, H. (2026). A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies. Research Square (preprint). https://doi.org/10.21203/rs.3.rs-9676637/v1

BibTeX

@article{shree2026comprehensive,
author = {Shree, Ajita and Aditya, V and Kumar, Tanush and Zafar, Hamim},
title = {{A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies}},
journal = {Research Square (preprint)},
year = {2026},
month = jun,
publisher = {Research Square},
issn = {2693-5015},
doi = {10.21203/rs.3.rs-9676637/v1},
url = {https://doi.org/10.21203/rs.3.rs-9676637/v1}
}

RIS

TY - JOUR
AU - Shree, Ajita
AU - Aditya, V
AU - Kumar, Tanush
AU - Zafar, Hamim
TI - A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies
T2 - Research Square (preprint)
J2 - Res Sq
PY - 2026
DA - 2026/06/10
SN - 2693-5015
PB - Research Square
DO - 10.21203/rs.3.rs-9676637/v1
UR - https://doi.org/10.21203/rs.3.rs-9676637/v1
ER -

CSL-JSON

{
"id": "10.21203/rs.3.rs-9676637/v1",
"type": "article",
"title": "A Comprehensive Benchmarking of Spatial Deconvolution and Domain Detection Methods across Diverse Tissues and Spatial Transcriptomic Technologies",
"container-title": "Research Square (preprint)",
"author": [
{
"family": "Shree",
"given": "Ajita"
},
{
"family": "Aditya",
"given": "V"
},
{
"family": "Kumar",
"given": "Tanush"
},
{
"family": "Zafar",
"given": "Hamim"
}
],
"container-title-short": "Res Sq",
"DOI": "10.21203/rs.3.rs-9676637/v1",
"ISSN": "2693-5015",
"publisher": "Research Square",
"URL": "https://doi.org/10.21203/rs.3.rs-9676637/v1",
"issued": {
"date-parts": [
[
2026,
6,
10
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41592-026-03194-8 [code]
Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.
Journal: Nature methods
In common: Squidpy, rpy2, PyTorch Geometric, 17 other tools, methods / tools, genetics / omics, 14 references
[2] doi:10.1002/advs.77003 [code]
SemanticST: A Scalable Multi-Contextual Graph Learning Framework for Uncovering Spatial Niches and Robust Multi-Sample Integration in Spatial Transcriptomics.
Journal: Advanced science (Weinheim, Baden-Wurttemberg, Germany)
In common: rpy2, PyTorch Geometric, anndata, 10 other tools, methods / tools, genetics / omics, 10 references
[3] doi:10.1016/j.isci.2026.117206 [code]
ReliST: A model-agnostic risk layer for spatial transcriptomics deconvolution.
Journal: iScience
In common: anndata, Scanpy, Pillow, 7 other tools, genetics / omics, 14 references
[4] doi:10.1093/bioinformatics/btag540 [code]
Deciphering spatial heterogeneity by multimodal spatial transcriptomics modelling with SpatialModal.
Journal: Bioinformatics (Oxford, England)
In common: Squidpy, rpy2, PyTorch Geometric, 11 other tools, methods / tools, genetics / omics, 6 references
[5] doi:10.1016/j.xcrm.2026.102766 [code]
A longitudinal single-cell and spatial multiomic atlas of pediatric high-grade glioma.
Journal: Cell reports. Medicine
In common: rpy2, SingleCellExperiment, UMAP, 18 other tools, genetics / omics
[6] doi:10.1093/nar/gkag706 [code]
scDifformer: diffusion-based post-training for virtual cell modeling across large-scale single-cell data.
Journal: Nucleic acids research
In common: Squidpy, PyTorch Geometric, UMAP, 14 other tools, 3 references
[7] doi:10.1186/s13073-026-01704-z [code]
Gene expression profiling enables refined parcellation of cortical layers in the heterogeneous human cerebral cortex.
Journal: Genome medicine
In common: SingleCellExperiment, UMAP, anndata, 14 other tools, genetics / omics, 4 references
[8] doi:10.1093/bib/bbag404 [code]
Navigating cell maps by deep learning integration of single-cell and spatially resolved transcriptomics.
Journal: Briefings in bioinformatics
In common: PyTorch Geometric, anndata, Scanpy, 10 other tools, genetics / omics, 7 references
[9] doi:10.1093/bib/bbag298 [code]
Empowering multifaceted analysis of spatial transcriptomics data with RGAST.
Journal: Briefings in bioinformatics
In common: rpy2, PyTorch Geometric, anndata, 8 other tools, methods / tools, genetics / omics, 8 references
[10] doi:10.1093/bioinformatics/btag578 [code]
NicheDeSig: niche-aware deconvolution and adaptive signature analysis for spatial transcriptomics.
Journal: Bioinformatics (Oxford, England)
In common: anndata, Scanpy, PyTorch, 4 other tools, methods / tools, genetics / omics, 12 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.