Brain metastases converge on shared geometric architecture and transcriptomic landscape yet remain distinct from gliomas.

Code ↔ Paper

17 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 17 matches · 2 of them tie a paragraph to a whole file, not to given lines: weak matches, whose lines are not tinted

[1] § STAR★Methods › Method details › Supervised machine learning model for discriminating brain-mets and glioma ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 292–345 · score 0.84 · cross validation, repeated stratified, confusion matrices, permutation importance, balanced accuracy, subsets
[2] § STAR★Methods › Method details › Transcriptional signatures associated with geometric complexity of tumor regions ↔ Rcode/1_TCGA_download_GitHub.R, the whole file · a weak match · score 0.79 · RNA seq, TCGA GBM, TCGA LGG, gene expression, TPM, profiles
[3] § STAR★Methods › Method details › Fractal dimension and lacunarity measurement for tumor subcomponents ↔ Python/1_FD3D_Calculation_GitHub_1902026.ipynb, lines 87–134 · score 0.71 · bounding rectangle, fractal dimension, binned, contour, slice, algorithm
[4] § STAR★Methods › Quantification and statistical analysis ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 970–1013 · score 0.71 · confusion matrices, ROC AUC, balanced accuracy, predicting, ML, XGB
[5] § STAR★Methods › Quantification and statistical analysis ↔ Rcode/5_Brain-Mets_all_cutoff and survival.R, lines 315–399 · score 0.70 · Cox proportional hazards, Kaplan Meier, fractal dimension, cutoff, fitted, survival
[6] § Results › Fractal complexity, lacunarity, and fractional volume of tumor subcomponents across brain-metastases ↔ Python/2_Lac3D_Modified_Calculation_Github_19022025.ipynb, lines 68–126 · score 0.63 · convex hull, Bounding box, crop, contour, max, Lac
[7] § STAR★Methods › Method details › Transcriptional signatures associated with geometric complexity of tumor regions ↔ Rcode/3_Spearman_Corelation_Geometry_Molecular_GitHub_19022026.R, lines 1–42 · score 0.60 · gene expression, correlated genes, TPM, protein, Spearman, GBM
[8] § STAR★Methods › Method details › Evaluation of effects of manual and automated segmented tumor subcomponents on fractality and lacunarity estimates ↔ Python/6_Manual_auto_Check_passing_bablok_Github_200202026.ipynb, lines 39–116 · score 0.60 · Passing Bablok regression, slopes
[9] § Results › Integrating geometric measures and fractional volumetry with machine learning for brain-metastases vs. glioma differentiation ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 241–246 · score 0.59 · UPENN gliomas, UCSF gliomas, UCSF BMSR, cohorts, Mets, brain
[10] § Results › Comparison of fractal dimensions derived from manual and automated tumor subcomponent masks ↔ Python/6_Manual_auto_Check_passing_bablok_Github_200202026.ipynb, lines 39–116 · score 0.59 · Passing Bablok regression, intercept, slope, component
[11] § Results › Integrating geometric measures and fractional volumetry with machine learning for brain-metastases vs. glioma differentiation ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 970–1013 · score 0.59 · cost sensitive, confusion matrices, balanced accuracy, XGB, RF, KNN
[12] § STAR★Methods › Method details › Molecular profiling in gliomas and brain metastases ↔ Python/7_AUCell_Score_for_Hallmark_pathwatys_Variance_Github_20022026.ipynb, lines 324–339 · score 0.58 · pathway activity, AUCell, variance, scores, Hallmark
[13] § STAR★Methods › Experimental model and study participant details ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 222–226 · score 0.58 · BraTS, Africa cohorts, UPENN, Brain Mets, TCGA, UCSF
[14] § STAR★Methods › Quantification and statistical analysis ↔ Python/5_Survival_For_Brain_Mets_GitHub_19022026.ipynb, lines 305–335 · score 0.58 · Cox proportional hazards, CPH, fitted, age, survival, model
[15] § STAR★Methods › Method details › Molecular profiling in gliomas and brain metastases ↔ Rcode/1_TCGA_download_GitHub.R, the whole file · a weak match · score 0.56 · RNA seq, gene expression, GBM, TCGA, profiling, LGG
[16] § STAR★Methods › Method details › Fractal dimension and lacunarity measurement for tumor subcomponents ↔ Python/2_Lac3D_Modified_Calculation_Github_19022025.ipynb, lines 68–126 · score 0.56 · bounding boxes, binned, contour, slice, lacunarity, mask
[17] § STAR★Methods › Method details › Supervised machine learning model for discriminating brain-mets and glioma ↔ Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, lines 1709–1745 · score 0.51 · random forest, ML, neighbors, XGB, RF, KNN

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The paper is loaded by your browser, with JavaScript: read it at doi.org or on Europe PMC.

The authors' code

Jupyter notebook · 1,997 lines · 78 KB · no license · 6 matches

# %%
##### ML Model For Distinguishing Brain Mets and Glioma Using FD3d and Fractional volume ####
# %%
import warnings
from IPython.display import display
import logging ### Version 0.5.1.2
warnings.filterwarnings('ignore')
logging.getLogger().setLevel(logging.ERROR)
# %%
import numpy as np ### Version 1.26.4
import pandas as pd ### Version 2.2.3
import matplotlib.pyplot as plt
import seaborn as sns ### Version 0.13.2
from statannot import add_stat_annotation
from statsmodels.formula.api import ols
from matplotlib import rc, rcParams
# import statsmodels.api as sm
import os
from scipy.ndimage import affine_transform
from scipy.stats import median_test
from scipy import stats
from scipy.stats import kruskal
import scikit_posthocs as sp
import math
from scipy.stats import shapiro
# %%
# print(sklearn.__version__)
# %%
# this error via Jupyter Notebook installed alongside Anaconda navigation, you must install the imbalanced-learn library via the Conda package manager.
# Therefore, do the following:
# pip uninstall imblearn --yes
# conda install -c conda-forge imbalanced-learn
# %%
import imblearn ### Version 0.12.4
# %%
from imblearn.pipeline import Pipeline
from sklearn.model_selection import RepeatedStratifiedKFold
# %%
import statannot ### Version 0.2.3
import itertools
from statannot import add_stat_annotation
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
# from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier #, AdaBoostClassifier
# from sklearn.naive_bayes import GaussianNB
# from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.preprocessing import label_binarize
from sklearn.metrics import roc_curve, auc
from sklearn.multiclass import OneVsRestClassifier
import warnings
from matplotlib.colors import ListedColormap
from sklearn import svm
from sklearn.metrics import RocCurveDisplay, auc
from sklearn.metrics import confusion_matrix
from xgboost import XGBClassifier
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import StratifiedKFold, GridSearchCV
from sklearn.metrics import accuracy_score, balanced_accuracy_score, roc_auc_score, recall_score, precision_score
# %%
# !pip install shap
# %%
from sklearn.utils.class_weight import compute_sample_weight, compute_class_weight
from sklearn.inspection import permutation_importance
import shap
import pickle
# %%
### for changing the working directory
os.chdir(r'Path/to/working/diectory')
os.getcwd()
# %%
#### The input CSV file with FD lac and fractional volume Ratio and the of all the patients and the Tumor status
df_combine= pd.read_excel("All_FD_LAC_Vol_Mets_.xlsx") ### add the file containing tumor Fractaldimension FD and Lac
df_combine
# %%
Columns_names=df_combine.columns.values.tolist()
# print("COLUMNS Name :-", Columns_names)
# %%
df_combine['Cohort'].value_counts()
# %%
df=df_combine[['ID', 'New_ID', 'Tumor_type', 'Grade', 'Age_years_at_diagnosis',
'Gender', 'Survival_months', 'Vital_status_1_dead', 'Primary', 'Cohort',
'Volume_ET_ml', 'Volume_NET_ml', 'Volume_ED_ml', 'Volume_WT_ml',
'ncr_net_mean3dfd', 'et_mean3dfd', 'ed_mean3dfd',
'ncr_net_meanlac3d', 'et_meanlac3d', 'ed_meanlac3d',
'Ratio_ET_WT', 'Ratio_NET_WT', 'Ratio_ED_WT']]
df
# %%
##### List for different combination of features
# for only FD combination
lst1= [['et_mean3dfd'], ['ed_mean3dfd'],['ncr_net_mean3dfd'],
['et_mean3dfd', 'ed_mean3dfd'],['ncr_net_mean3dfd', 'et_mean3dfd'],['ncr_net_mean3dfd', 'ed_mean3dfd'],
['et_mean3dfd', 'ncr_net_mean3dfd', 'ed_mean3dfd']]
# for only Ratio combination
lst2=[['Ratio_ET_WT'], ['Ratio_NET_WT'], ['Ratio_ED_WT'],
['Ratio_ET_WT', 'Ratio_NET_WT'], ['Ratio_ET_WT', 'Ratio_ED_WT'], ['Ratio_NET_WT', 'Ratio_ED_WT'],
['Ratio_ET_WT', 'Ratio_NET_WT', 'Ratio_ED_WT']]
# for only Lacunarity Combination
lst3= [['et_meanlac3d'], ['ed_meanlac3d'],['ncr_net_meanlac3d'],
['et_meanlac3d', 'ed_meanlac3d'],['ncr_net_meanlac3d', 'et_meanlac3d'],['ncr_net_meanlac3d', 'ed_meanlac3d'],
['et_meanlac3d', 'ncr_net_meanlac3d', 'ed_meanlac3d']]
# for only FD and Lacunarity Combination
lst4= [['et_mean3dfd','et_meanlac3d'],['ed_mean3dfd','ed_meanlac3d'], ['ncr_net_mean3dfd','ncr_net_meanlac3d'],
['et_mean3dfd', 'ed_mean3dfd','et_meanlac3d', 'ed_meanlac3d'],
['ncr_net_mean3dfd', 'et_mean3dfd','ncr_net_meanlac3d', 'et_meanlac3d'],
['ncr_net_mean3dfd', 'ed_mean3dfd','ncr_net_meanlac3d', 'ed_meanlac3d'],
['et_mean3dfd', 'ncr_net_mean3dfd', 'ed_mean3dfd','et_meanlac3d', 'ncr_net_meanlac3d', 'ed_meanlac3d']]
# for only FD and ratio Combination
lst5= [['et_mean3dfd','Ratio_ET_WT'], ['ed_mean3dfd','Ratio_ED_WT'],['ncr_net_mean3dfd', 'Ratio_NET_WT'],
['ncr_net_mean3dfd', 'et_mean3dfd','Ratio_ET_WT', 'Ratio_NET_WT'],
['ncr_net_mean3dfd', 'ed_mean3dfd','Ratio_NET_WT', 'Ratio_ED_WT'],
['et_mean3dfd', 'ed_mean3dfd','Ratio_ET_WT', 'Ratio_ED_WT'],
['et_mean3dfd', 'ncr_net_mean3dfd', 'ed_mean3dfd','Ratio_ET_WT', 'Ratio_NET_WT', 'Ratio_ED_WT']]
# %%
#### for ploting
plt.rcParams["figure.figsize"] = (6, 6)
rcParams['xtick.major.width'] = 2
rcParams['xtick.major.size'] = 12
rcParams['ytick.major.width'] = 2
rcParams['ytick.major.size'] = 10
rcParams['xtick.labelsize'] = 12
rcParams['ytick.labelsize'] = 12
plt.rcParams["axes.linewidth"] = 2
plt.rcParams['xtick.labelsize'] = 24 # X-axis tick labels
plt.rcParams['ytick.labelsize'] = 24 # Y-axis tick labels
plt.rcParams['legend.fontsize'] = 14 # Global legend font size
# %%
########## Function for Ploting ROC curve ###############
def plot_roc_and_metrics(model, X, y, classifier_name, cv, color=None):
tprs = []
aucs = []
mean_fpr = np.linspace(0, 1, 100)
for train_idx, test_idx in cv.split(X, y):
model.fit(X[train_idx], y[train_idx])
y_scores = model.predict_proba(X[test_idx])[:, 1]
fpr, tpr, _ = roc_curve(y[test_idx], y_scores)
interp_tpr = np.interp(mean_fpr, fpr, tpr)
interp_tpr[0] = 0.0
tprs.append(interp_tpr)
aucs.append(roc_auc_score(y[test_idx], y_scores)) # Exact sklearn AUC
mean_tpr = np.mean(tprs, axis=0)
mean_auc = np.mean(aucs)
std_auc = np.std(aucs)
std_tpr = np.std(tprs, axis=0)
lower_tpr = np.maximum(mean_tpr - std_tpr, 0)
upper_tpr = np.minimum(mean_tpr + std_tpr, 1)
line, =plt.plot(mean_fpr, mean_tpr, label=f"{classifier_name} (Mean AUC: {mean_auc:.2f} ± {std_auc:.2f})",
color=color)
plt.fill_between(mean_fpr, lower_tpr, upper_tpr,
alpha=0.2, color=line.get_color())
# Add chance line
plt.plot([0, 1], [0, 1], linestyle="--", color="black", lw=1, label=None)
plt.xticks(np.arange(0, 1.1, 0.2))
plt.yticks(np.arange(0, 1.1, 0.2))
plt.xlim([0, 1])
plt.ylim([0, 1])
return mean_auc, std_auc
# %%
df_UBM= df[df['Cohort'].isin(["UCSF_BMSR"])] #### For UCSF-BMSR Brain Mets and TCGA cohort
df_G = df[df['Cohort'].isin(["TCGA","UCSF","UPENN","Africa"])] #### For UCSF-BMSR Brain Mets and UCSF cohort
# %%
df_UBM1= df_UBM[df_UBM['ID'].str.endswith("A")]
df_UBM1
# %%
df_1= pd.concat([df_UBM1,df_G ], ignore_index= True)
df_1
# %%
df_BM_G = df[df['Cohort'].isin(["Brain_Mets", "TCGA"])] #### For Pre-treat-Brain Mets and TCGA cohort
df_BM_UCSF = df[df['Cohort'].isin(["Brain_Mets", "UCSF"])] #### For Pre-treat-Brain Mets and UCSF cohort
df_BM_UPENN = df[df['Cohort'].isin(["Brain_Mets", "UPENN"])] #### For Pre-treat-Brain Mets and UPENN cohort
df_BM_AFG = df[df['Cohort'].isin(["Brain_Mets", "Africa"])] #### For Pre-treat-Brain Mets and BraTS Africa cohort
# %%
df_BM_G = df[df['Cohort'].isin(["Brain_Mets", "TCGA"])] #### For Pre-treat-Brain Mets and TCGA cohort
df_BM_UCSF = df[df['Cohort'].isin(["Brain_Mets", "UCSF"])] #### For Pre-treat-Brain Mets and UCSF cohort
df_BM_UPENN = df[df['Cohort'].isin(["Brain_Mets", "UPENN"])] #### For Pre-treat-Brain Mets and UPENN cohort
df_BM_AFG = df[df['Cohort'].isin(["Brain_Mets", "Africa"])] #### For Pre-treat-Brain Mets and BraTS Africa cohort
# %%
############# For Cross cohort Validation between Pretreat Brain Mets and Glioma
# df_Cohort = df_BM_G ### Uncomment for BM and TCGA Glioma
# df_Cohort = df_BM_UCSF ### Uncomment for BM and UCSF Glioma
# df_Cohort = df_BM_UPENN ### Uncomment for BM and UPENN Glioma
# df_Cohort = df_BM_AFG ### Uncomment for BM and AFG Glioma
# %%
############# For Cross cohort Validation Between UCSF-BMSR Brain-Mets and Glioma
df_Cohort = df_UBM1_G ### Uncomment for UCSF-BMSR and TCGA Glioma
# df_Cohort = df_UBM1_UCSF ### Uncomment for UCSF-BMSR and UCSF Glioma
# df_Cohort = df_UBM1_UPENN ### Uncomment for UCSF-BMSR and UPENN Glioma
# df_Cohort = df_UBM1_AFG ### Uncomment for UCSF-BMSR and AFG Glioma
# %% [markdown]
# # Baseline Code
# %%
###################### Baseline code with repeated stratified K-fold ######################
name_classifier = {
svm.SVC(random_state=12, probability=True): 'SVM',
RandomForestClassifier(n_estimators=10, random_state=12): 'RF',
KNeighborsClassifier(10): 'KNN',
XGBClassifier(random_state=12, use_label_encoder=False, eval_metric='logloss',verbosity=0): 'XGB'
}
# ------------------------ Classifier Pipelines ------------------------
param_grids = {'SVM': {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'rbf'],
'gamma': [1, 0.1, 0.01, 0.001] },
'RF': {'n_estimators': [10, 50, 100, 200],'max_depth': [None, 10, 20, 30],
'min_samples_split': [2, 5, 10]},
'KNN': {'n_neighbors': [3, 5, 10, 15], 'weights': ['uniform', 'distance'],
'p': [1, 2] },
'XGB': {'n_estimators': [10, 50, 100, 200],'learning_rate': [0.01, 0.1, 0.2,0.3],
'max_depth': [3,4,5,6], 'subsample': [0.8, 0.9,1.0], 'min_child_weight': [1, 3, 5],
'reg_alpha': [0, 0.1, 0.5,1], # L1 regularization
'reg_lambda': [1, 2, 5, 10]}}
# ------------------------ Parameter Grids ------------------------
# Function to calculate average confusion matrix
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
# ------------------------ Storage for results ------------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
# ------------------------ Training + Evaluation ------------------------
for i in lst5: # Loop through features ### change the list for different feature combination
print(i)
feature_key = str(i)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=i)
X = df3[i].values
y = (df3['Tumor_type'] == 'Mets').values.astype(int)
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=2, random_state=12)
for j, k in name_classifier.items():
grid_search = GridSearchCV(
estimator=j,
param_grid=param_grids[k],
scoring='roc_auc',
cv=5,
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, sensitivity_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
sensitivity_list.append(recall_score(y_test, y_pred) * 100)
prec_list.append(precision_score(y_test, y_pred) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
##-----------------------------------------------------------------------------------
## <<< MODIFICATION: Calculate Permutation Importance for BOTH metrics >>>
# 1. Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, j)] = {
'features': i,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
##-------------------------------- Store aggregated metrics-----------------------------------------------
# Save per-fold metrics
metrics_storage[feature_key][k] = {
'accuracy_model': acc_list,
'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list,
'sensitivity_model': sensitivity_list,
'precision_model': prec_list
}
# Summary results
result_label = f"{i}"
result2.append([
result_label, k, f"Best {k}: {best_params}",
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(sensitivity_list), 2), round(np.std(sensitivity_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][k] = confusion_avg
print(f"confusion matrix for feature {i} and {k} ML model is:\n", confusion_avg)
# --- ROC Plotting (Now uses the metrics from the CV loop) ---
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=k,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#--------------------------------------------------------------
# Save ROC curve
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/Basline_model_CV_Importance"
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/Basline_model_CV_Importance" ### Uncumment for USSF-BMSR cohort
os.makedirs(os.path.join(output_dir, "Figures/ROC_curve"), exist_ok=True)
plot_filename = f"{output_dir }/Figures/ROC_curve/ROC_curve{i}_R1_Aug2025_ROC_curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False, fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"**********Finished processing feature set: {i}*****************")
# ------------------------ Save metrics ------------------------
os.makedirs(os.path.join(output_dir, "per_fold_metrics"), exist_ok=True)
df_results = pd.DataFrame(result2, columns=["Feature Set","Classifier", "Classifier with best parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy",
"SD_balanced_Acc", "Mean_AUC_Score", "SD_AUC_score",
"Mean_Sensitivity", "SD_Sensitivity", "Mean_Precision",
"SD_Precision", "Number of Subjects"])
df_results.to_csv(f"{output_dir}/Baseline_models_summary_results.csv", index=False)
# Save per-fold metrics
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir }/per_fold_metrics/BS_{metric}.csv", index=False)
#------------------------------------------------------
# Define the list of metrics
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
# Initialize a list to collect final rows
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
# Determine the number of folds dynamically from any metric
num_folds = len(next(iter(values.values())))
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
# Fill in each metric value for this fold
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
# Create DataFrame from the collected rows
df_final = pd.DataFrame(final_rows)
# Save the DataFrame to CSV ## change the folder accordingly
output_file = f"{output_dir}/per_fold_metrics/ALL_SP_Baseline_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
# Save the confusion_avg_dict for later use
with open(f"{output_dir}/per_fold_metrics/Base_confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved confusion_avg_dict.pkl")
### ----------------------------------------------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %%
# %% [markdown]
# # Baseline with standard scaler¶
# %%
#########################################################################################
##### Modified code including Permutation importance
##### Basline model with Feature Importance (Cross-Validated)
########################################################################################
################ Baseline with standard scaler ##############
name_classifier = {
'SVM': Pipeline([
('scaler', StandardScaler()),
('clf', svm.SVC(random_state=12, probability=True))
]),
'RF': Pipeline([('scaler', StandardScaler()),
('clf', RandomForestClassifier(n_estimators=10, random_state=12))
]),
'KNN': Pipeline([
('scaler', StandardScaler()),
('clf', KNeighborsClassifier())
]),
'XGB': Pipeline([ ('scaler', StandardScaler()),
('clf', XGBClassifier(random_state=12, use_label_encoder=False, eval_metric='logloss',verbosity=0)) # you may calculate actual imbalance ratio
])
}
# ----------------------------- Parameter Grids --------------------------
param_grids = {
'SVM': {
'clf__C': [0.1, 1, 10, 100],
'clf__kernel': ['linear', 'rbf'],
'clf__gamma': [1, 0.1, 0.01, 0.001]
},
'RF': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__max_depth': [None, 10, 20, 30],
'clf__min_samples_split': [2, 5, 10]
},
'KNN': {
'clf__n_neighbors': [3, 5, 10, 15],
'clf__weights': ['uniform', 'distance'],
'clf__p': [1, 2]
},
'XGB': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__learning_rate': [0.01, 0.1, 0.2,0.3],
'clf__max_depth': [3,4,5,6],
'clf__subsample': [0.8, 0.9, 1.0],
'clf__min_child_weight': [1,3,5],
'clf__reg_alpha': [0, 0.1, 0.5,1],
'clf__reg_lambda': [1, 2, 5, 10]
}
}
###---------------- Confusion matrices Function -----------------------------
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
#----------------------Storage for results------------------------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
#------------------------Training + Evaluation--------------------------------
for features in lst5:
print(f"\nEvaluating feature set: {features}")
feature_key = str(features)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=features)
X = df3[features].values
y = (df3['Tumor_type'] == 'Mets').values.astype(int)
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=12)
for clf_name, pipeline in name_classifier.items():
print(f"\nClassifier: {clf_name}")
grid_search = GridSearchCV(
estimator=pipeline,
param_grid=param_grids[clf_name],
scoring='roc_auc',
cv=5,
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, sensitivity_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
sensitivity_list.append(recall_score(y_test, y_pred) * 100)
prec_list.append(precision_score(y_test, y_pred) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
####---------------------------------------------------------------------------------
## <<< MODIFICATION: Calculate Permutation Importance >>>
# Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, clf_name)] = {
'features': features,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
###---------------------------Store aggregated metrics for reporting----------------------
metrics_storage[feature_key][clf_name] = {
'accuracy_model': acc_list,
'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list,
'sensitivity_model': sensitivity_list,
'precision_model': prec_list
}
result2.append([
feature_key, clf_name, str(best_params),
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(sensitivity_list), 2), round(np.std(sensitivity_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][clf_name] = confusion_avg
print(f"Avg confusion matrix for {features} and {clf_name} ML model:\n{confusion_avg}")
# --- ROC Plotting (Now uses the metrics from the CV loop) ---
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=clf_name,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#-----------------------------------------------------------------
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/Basline_Standard_model_CV_Importance" #Change the output dir accordingly
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/Basline_Standard_model_CV_Importance"
os.makedirs(f"{output_dir}/Figures/ROC_curve", exist_ok=True)
plot_filename = f"{output_dir}/Figures/ROC_curve/{features}_R1_Aug2025_ROC_curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False, fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"**********Finished processing feature set: {features}*****************")
# ------------------------ Save all results ------------------------------
# ------------------------ Save metrics -----------------------------------
# output_dir = "ML_Results/TCGA/FD_only_Check/Cost_Sensitive_model_CV_Importance"
os.makedirs(f"{output_dir}/per_fold_metrics", exist_ok=True)
df_results = pd.DataFrame(result2, columns=[
"Feature Set","Classifier", "Best Parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy", "SD_Bal_Acc",
"Mean_AUC_Score", "SD_AUC", "Mean_Recall", "SD_Recall",
"Mean_Precision", "SD_Precision", "N_Samples"
])
df_results.to_csv(f"{output_dir}/BS_model_summary_results.csv", index=False)
# Save per-fold metrics
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir}/per_fold_metrics/BS_{metric}.csv", index=False)
# Define the list of metrics
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
# Initialize a list to collect final rows
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
# Determine the number of folds dynamically from any metric
num_folds = len(next(iter(values.values())))
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
# Fill in each metric value for this fold
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
# Create DataFrame from the collected rows
df_final = pd.DataFrame(final_rows)
# Save the DataFrame to CSV
output_file = f"{output_dir}/per_fold_metrics/ALL_SPlot_BaselineStandard_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
# -------------------------------------------------
# Save the confusion_avg_dict for later use
with open(f"{output_dir}/per_fold_metrics/BS_confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved BS_confusion_avg_dict.pkl")
### ----------------------------------------------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/BS_permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %%
# %% [markdown]
# # Cost- sensitive moels
# %%
from collections import Counter
# %%
def plot_roc_and_metrics(model, X, y, classifier_name, cv, color=None):
tprs = []
aucs = []
mean_fpr = np.linspace(0, 1, 100)
for train_idx, test_idx in cv.split(X, y):
model.fit(X[train_idx], y[train_idx])
y_scores = model.predict_proba(X[test_idx])[:, 1]
fpr, tpr, _ = roc_curve(y[test_idx], y_scores)
interp_tpr = np.interp(mean_fpr, fpr, tpr)
interp_tpr[0] = 0.0
tprs.append(interp_tpr)
aucs.append(roc_auc_score(y[test_idx], y_scores)) # Exact sklearn AUC
mean_tpr = np.mean(tprs, axis=0)
mean_auc = np.mean(aucs)
std_auc = np.std(aucs)
std_tpr = np.std(tprs, axis=0)
lower_tpr = np.maximum(mean_tpr - std_tpr, 0)
upper_tpr = np.minimum(mean_tpr + std_tpr, 1)
line, =plt.plot(mean_fpr, mean_tpr, label=f"{classifier_name} (Mean AUC: {mean_auc:.2f}±{std_auc:.2f})",
color=color)
plt.fill_between(mean_fpr, lower_tpr, upper_tpr,
alpha=0.2, color=line.get_color())
# Add chance line
plt.plot([0, 1], [0, 1], linestyle="--", color="grey", lw=1, label=None)
plt.xticks(np.arange(0, 1.1, 0.2), fontsize=28)
plt.yticks(np.arange(0, 1.1, 0.2), fontsize=28)
plt.xlim([0, 1])
plt.ylim([0, 1])
return mean_auc, std_auc
# %%
#########################################################################################
##### Modified code including Permutation importance
##### Costsensitive Model with Feature Importance (Cross-Validated)
#########################################################################################
# ------------------------ Standard Imports ------------------------
# ------------------------ Custom KNN with cost-sensitive learning ------------------------
class CostSensitiveKNN(KNeighborsClassifier):
def __init__(self, n_neighbors=5, weights='uniform', p=2, class_weight=None):
super().__init__(n_neighbors=n_neighbors, weights=weights, p=p)
self.class_weight = class_weight
def predict(self, X):
neigh_ind = self.kneighbors(X, return_distance=False)
predictions = []
for neighbors in neigh_ind:
neighbor_labels = self._y[neighbors]
votes = {}
for label in np.unique(self._y): # Check against all possible labels
count = np.sum(neighbor_labels == label)
weight = self.class_weight.get(label, 1.0) if self.class_weight else 1.0
votes[label] = count * weight
predictions.append(max(votes, key=votes.get))
return np.array(predictions)
def fit(self, X, y):
self._y = np.array(y)
# The actual fitting is done by the parent class
return super().fit(X, y)
# ------------------------ Classifier Pipelines ------------------------
name_classifier = {
'SVM': Pipeline([
('scaler', StandardScaler()),
('clf', svm.SVC(probability=True, class_weight='balanced', random_state=12))
]),
'RF': Pipeline([
('scaler', StandardScaler()),
('clf', RandomForestClassifier(class_weight='balanced', random_state=12))
]),
'KNN': Pipeline([
('scaler', StandardScaler()),
('clf', CostSensitiveKNN())
]),
'XGB': Pipeline([
('scaler', StandardScaler()),
('clf', XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=12, verbosity=0))
])
}
# ------------------------ Parameter Grids ------------------------
param_grids = {
'SVM': {
'clf__C': [0.1, 1, 10,100],
'clf__kernel': ['linear', 'rbf'],
'clf__gamma': ['scale', 0.1, 0.01,0.001]
},
'RF': {
'clf__n_estimators': [10,50, 100,200],
'clf__max_depth': [None, 10, 20,30],
'clf__min_samples_split': [2, 5,10]
},
'KNN': {
'clf__n_neighbors': [3, 5, 10, 15],
'clf__weights': ['uniform', 'distance'],
'clf__p': [1, 2]
},
'XGB': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__learning_rate': [0.01, 0.1, 0.2, 0.3],
'clf__max_depth': [3,4,5,6],
'clf__subsample': [0.8, 0.9, 1.0],
'clf__min_child_weight': [1,3,5],
'clf__reg_alpha': [0, 0.1, 0.5,1],
'clf__reg_lambda': [1, 2, 5, 10]
}
}
######################################
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
# ------------------------ Storage for results ------------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
# ------------------------ Training + Evaluation ------------------------
for features in lst5:
print(f"\nEvaluating feature set: {features}")
feature_key = str(features)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=features)
X = df3[features].values
y = (df3['Tumor_type'] == 'Mets').astype(int).values
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=12)
for clf_name, pipeline in name_classifier.items():
print(f"\nClassifier: {clf_name}")
grid_search = GridSearchCV(
estimator=pipeline,
param_grid=param_grids.get(clf_name, {}), # Use .get for safety
scoring='roc_auc',
cv=5, # Use a smaller CV for faster grid search
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, recall_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
sample_weight = compute_sample_weight(class_weight='balanced', y=y_train)
# --- Cost-sensitive model fitting FOR THIS FOLD ---
if clf_name == 'KNN':
class_weights_array = compute_class_weight(class_weight='balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = dict(zip(np.unique(y_train), class_weights_array))
best_model.named_steps['clf'].class_weight = class_weights_dict
best_model.fit(X_train, y_train)
elif clf_name == 'XGB':
counter = Counter(y_train)
scale_pos_weight = counter[0] / counter[1] if counter[1] > 0 else 1
best_model.named_steps['clf'].set_params(scale_pos_weight=scale_pos_weight)
best_model.fit(X_train, y_train) # XGB uses scale_pos_weight internally
else: # For RF, SVM which have class_weight='balanced'
# best_model.fit(X_train, y_train)
try:
best_model.fit(X_train, y_train, clf__sample_weight=sample_weight)
except TypeError:
best_model.fit(X_train, y_train)
# --- Calculate performance metrics on test set ---
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
recall_list.append(recall_score(y_test, y_pred, zero_division=0) * 100)
prec_list.append(precision_score(y_test, y_pred, zero_division=0) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
##---------------------------------------------------------------------------------
## <<< MODIFICATION: Calculate Permutation Importance for BOTH metrics >>>
# Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, clf_name)] = {
'features': features,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
# ------------------- Store aggregated metrics ---------------------------------
metrics_storage[feature_key][clf_name] = {
'accuracy_model': acc_list, 'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list, 'sensitivity_model': recall_list, 'precision_model': prec_list
}
result2.append([
feature_key, clf_name, str(best_params),
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(recall_list), 2), round(np.std(recall_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][clf_name] = confusion_avg
print(f"Avg confusion matrix for {clf_name}:\n{confusion_avg}")
# --- ROC Plotting (Now uses the metrics from the CV loop) ---
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=clf_name,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#--------------------------------------------------------------
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/Cost_Sensitive_model_CV_Importance"
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/Cost_Sensitive_model_CV_Importance"
os.makedirs(os.path.join(output_dir, "Figures/ROC_curve"), exist_ok=True)
plot_filename = f"{output_dir }/Figures/ROC_curve/{features}__ROC_curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False, fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"**********Finished processing feature set: {features}*****************")
# ------------------------ Save all results ------------------------
# ------------------------ Save metrics ------------------------
os.makedirs(os.path.join(output_dir, "per_fold_metrics"), exist_ok=True)
df_results = pd.DataFrame(result2, columns=[
"Feature Set","Classifier", "Best Parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy", "SD_Bal_Acc",
"Mean_AUC_Score", "SD_AUC", "Mean_Recall", "SD_Recall",
"Mean_Precision", "SD_Precision", "N_Samples"
])
df_results.to_csv(f"{output_dir}/CS_model_summary_results.csv", index=False)
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir}/per_fold_metrics/{metric}.csv", index=False)
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
num_folds = len(next(iter(values.values())))
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
df_final = pd.DataFrame(final_rows)
output_file = f"{output_dir}/per_fold_metrics/ALL_SPlot_CS_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
###-----------------------------------------------------------------------------
#### for Confusion avg dict
with open(f"{output_dir}/per_fold_metrics/CS_confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved CS_confusion_avg_dict.pkl")
### ----------------------------------------------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/CS_permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %% [markdown]
# # Oversampling using SMOTE
# %%
from imblearn.over_sampling import SMOTE
# %%
###################### Over sampling Using SMOTE ######################################
from imblearn.over_sampling import SMOTE
################################################################################
##### Modified code including Permutation importance
##### Oversampling with Feature Importance (Cross-Validated)
################ Undersampling with standard scaler ###########################
################ Oversampling with standard scaler ##############
name_classifier = {
'SVM': Pipeline([
('scaler', StandardScaler()),
('smote', SMOTE(random_state=12)),
('clf', svm.SVC(random_state=12, probability=True))
]),
'RF': Pipeline([('scaler', StandardScaler()),
('smote', SMOTE(random_state=12)),
('clf', RandomForestClassifier(n_estimators=10, random_state=12))
]),
'KNN': Pipeline([
('scaler', StandardScaler()),
('smote', SMOTE(random_state=12)),
('clf', KNeighborsClassifier())
]),
'XGB': Pipeline([ ('scaler', StandardScaler()),
('smote', SMOTE(random_state=12)),
('clf', XGBClassifier(random_state=12, use_label_encoder=False, eval_metric='logloss',verbosity=0)) # you may calculate actual imbalance ratio
])
}
# ------------------------ Classifier Pipelines ------------------------
# Define parameter grids
param_grids = {
'SVM': {
'clf__C': [0.1, 1, 10, 100],
'clf__kernel': ['linear', 'rbf'],
'clf__gamma': [1, 0.1, 0.01, 0.001]
},
'RF': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__max_depth': [None, 10, 20, 30],
'clf__min_samples_split': [2, 5, 10]
},
'KNN': {
'clf__n_neighbors': [3, 5, 10, 15],
'clf__weights': ['uniform', 'distance'],
'clf__p': [1, 2]
},
'XGB': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__learning_rate': [0.01, 0.1, 0.2,0.3],
'clf__max_depth': [3,4,5,6],
'clf__subsample': [0.8, 0.9, 1.0],
'clf__min_child_weight': [1,3,5],
'clf__reg_alpha': [0, 0.1, 0.5,1],
'clf__reg_lambda': [1, 2, 5, 10]
}
}
##-------------------Fubction For Confusion Matrix --------------------------
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
##------------------------Storage for results --------------------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
##-----------------------Training + Evaluation-------------------------------
for features in lst5: # Loop through features ### change the list for different feature combination
print(f"\nEvaluating feature set: {features}")
feature_key = str(features)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=features)
X = df3[features].values
y = (df3['Tumor_type'] == 'Mets').values.astype(int)
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=12)
for clf_name, pipeline in name_classifier.items():
print(f"\nClassifier: {clf_name}")
grid_search = GridSearchCV(
estimator=pipeline,
param_grid=param_grids[clf_name],
scoring='roc_auc',
cv=5,
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, sensitivity_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
sensitivity_list.append(recall_score(y_test, y_pred) * 100)
prec_list.append(precision_score(y_test, y_pred) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
## <<< MODIFICATION: Calculate Permutation Importance for BOTH metrics >>>
# Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, clf_name)] = {
'features': features,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
# ------------------- Store aggregated metrics for reporting ---------------------------------
metrics_storage[feature_key][clf_name] = {
'accuracy_model': acc_list,
'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list,
'sensitivity_model': sensitivity_list,
'precision_model': prec_list
}
result2.append([
feature_key, clf_name, str(best_params),
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(sensitivity_list), 2), round(np.std(sensitivity_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][clf_name] = confusion_avg
print(f"Avg confusion matrix for {features} and {clf_name} ML model:\n{confusion_avg}")
# --- ROC Plotting (Now uses the metrics from the CV loop) ---
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=clf_name,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#--------------------------------------------------------
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/OverSampling_SMOTE_model_CV_Importance"
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/OverSampling_SMOTE_model_CV_Importance"
os.makedirs(f"{output_dir}/Figures/ROC_curve", exist_ok=True)
plot_filename = f"{output_dir}/Figures/ROC_curve/{features}__curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False, fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"**********Finished processing feature set: {features}*****************")
# ------------------------ Save all results ------------------------
# ------------------------ Save metrics -----------------------------
os.makedirs(os.path.join(output_dir, "per_fold_metrics"), exist_ok=True)
df_results = pd.DataFrame(result2, columns=[
"Feature Set","Classifier", "Best Parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy", "SD_Bal_Acc",
"Mean_AUC_Score", "SD_AUC", "Mean_Recall", "SD_Recall",
"Mean_Precision", "SD_Precision", "N_Samples"
])
df_results.to_csv(f"{output_dir}/OS_model_summary_results.csv", index=False)
# Save per-fold metrics
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir}/per_fold_metrics/BS_{metric}.csv", index=False)
#----------------------------------------------------
# Define the list of metrics
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
# Initialize a list to collect final rows
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
# Determine the number of folds dynamically from any metric
num_folds = len(next(iter(values.values())))
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
# Fill in each metric value for this fold
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
# Create DataFrame from the collected rows
df_final = pd.DataFrame(final_rows)
# Save the DataFrame to CSV
output_file = f"{output_dir}/per_fold_metrics/OS_ALL_SP_Baseline_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
#-----------------------------------------------------------
# Save the confusion_avg_dict for later use
with open(f"{output_dir}/per_fold_metrics/OS_confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved OS_confusion_avg_dict.pkl")
### ----------------------------------------------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/OS_permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %% [markdown]
# # Undersampling Using NearMiss
# %%
# from imblearn.under_sampling import NearMiss
# %%
############################ Under Sampling Using NearMiss #######################
from imblearn.under_sampling import NearMiss
################################################################################
##### Modified code including Permutation importance
##### Udersampling with Feature Importance (Cross-Validated)
################ Undersampling with standard scaler ###########################
name_classifier = {
'SVM': Pipeline([
('scaler', StandardScaler()),
('undersample', NearMiss(version=1)),
('clf', svm.SVC(random_state=12, probability=True))
]),
'RF': Pipeline([('scaler', StandardScaler()),
('undersample', NearMiss(version=1)),
('clf', RandomForestClassifier(n_estimators=10, random_state=12))
]),
'KNN': Pipeline([
('scaler', StandardScaler()),
('undersample', NearMiss(version=1)),
('clf', KNeighborsClassifier())
]),
'XGB': Pipeline([ ('scaler', StandardScaler()),
('undersample', NearMiss(version=1)),
('clf', XGBClassifier(random_state=12, use_label_encoder=False, eval_metric='logloss',verbosity=0)) # you may calculate actual imbalance ratio
])
}
# Define parameter grids ###clf__ is added before the parametes because using pipline
param_grids = {
'SVM': {
'clf__C': [0.1, 1, 10, 100],
'clf__kernel': ['linear', 'rbf'],
'clf__gamma': [1, 0.1, 0.01, 0.001]
},
'RF': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__max_depth': [None, 10, 20, 30],
'clf__min_samples_split': [2, 5, 10]
},
'KNN': {
'clf__n_neighbors': [3, 5, 10, 15],
'clf__weights': ['uniform', 'distance'],
'clf__p': [1, 2]
},
'XGB': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__learning_rate': [0.01, 0.1, 0.2,0.3],
'clf__max_depth': [3,4,5,6],
'clf__subsample': [0.8, 0.9, 1.0],
'clf__min_child_weight': [1,3,5],
'clf__reg_alpha': [0, 0.1, 0.5,1],
'clf__reg_lambda': [1, 2, 5, 10]
}
}
# Confusion matrices function
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
#-------------------------Storage for results------------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
# ------------------------ Training + Evaluation ------------------------
for features in lst5: # Loop through features ### change the list for different feature combination
print(f"\nEvaluating feature set: {features}")
feature_key = str(features)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=features)
X = df3[features].values
y = (df3['Tumor_type'] == 'Mets').values.astype(int)
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=12)
for clf_name, pipeline in name_classifier.items():
print(f"\nClassifier: {clf_name}")
grid_search = GridSearchCV(
estimator=pipeline,
param_grid=param_grids[clf_name],
scoring='roc_auc',
cv=5,
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, sensitivity_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
sensitivity_list.append(recall_score(y_test, y_pred) * 100)
prec_list.append(precision_score(y_test, y_pred) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
###------------------------------Modify here--------------------
## <<< MODIFICATION: Calculate Permutation Importance for BOTH metrics >>>
# Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, clf_name)] = {
'features': features,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
##-----------------------------Store Metrics ------------------------
metrics_storage[feature_key][clf_name] = {
'accuracy_model': acc_list,
'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list,
'sensitivity_model': sensitivity_list,
'precision_model': prec_list
}
result2.append([
feature_key, clf_name, str(best_params),
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(sensitivity_list), 2), round(np.std(sensitivity_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][clf_name] = confusion_avg
print(f"Avg confusion matrix for {features} and {clf_name} ML model:\n{confusion_avg}")
# ---------------------Save ROC curve-----------------------------------
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=clf_name,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#----------------------------------------------------------------------
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/UnderSampling_CV_Importance" ### change the directory where you want to store the files
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/UnderSampling_CV_Importance"
os.makedirs(os.path.join(output_dir, "Figures/ROC_curve"), exist_ok=True)
plot_filename = f"{output_dir}/Figures/ROC_curve/{features}__ROC_curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False, fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"********** Finished processing feature set: {features} *****************")
#---------------------- Save results----------------------------------------------------
os.makedirs(os.path.join(output_dir, "per_fold_metrics"), exist_ok=True)
df_results = pd.DataFrame(result2, columns=[
"Feature Set","Classifier", "Best Parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy", "SD_Bal_Acc",
"Mean_AUC_Score", "SD_AUC", "Mean_Recall", "SD_Recall",
"Mean_Precision", "SD_Precision", "N_Samples"
])
df_results.to_csv(f"{output_dir}/US_model_summary_results.csv", index=False)
# Save per-fold metrics
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir}/per_fold_metrics/BS_{metric}.csv", index=False)
# Define the list of metrics
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
# Initialize a list to collect final rows
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
num_folds = len(next(iter(values.values()))) # Determine the number of folds dynamically from any metric
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
# Fill in each metric value for this fold
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
# Create DataFrame from the collected rows
df_final = pd.DataFrame(final_rows)
# Save the DataFrame to CSV
output_file = f"{output_dir}/per_fold_metrics/ALL_SP_Baseline_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
# Save the confusion_avg_dict for later use
with open(f"{output_dir}/per_fold_metrics/confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved confusion_avg_dict.pkl")
##---------------------------Save Importance Score ---------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/US_permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %% [markdown]
# # Combination of Oversampling and Undersampling
# %%
############################ Oversampling + Undersampling #####################################
#########################################################################################
##### Modified code including Permutation importance
##### Oversampling + Undersampling Model with Feature Importance (Cross-Validated)
########################################################################
### Combination of Oversampling and Undersampling with standard scaler ##############
name_classifier = {
'SVM': Pipeline([
('scaler', StandardScaler()),
('smote', SMOTE(random_state=12,sampling_strategy=0.85)),
('undersample', NearMiss(version=1, sampling_strategy=1)),
('clf', svm.SVC(random_state=12, probability=True))
]),
'RF': Pipeline([('scaler', StandardScaler()),
('smote', SMOTE(random_state=12,sampling_strategy=0.85)),
('undersample', NearMiss(version=1, sampling_strategy=1)),
('clf', RandomForestClassifier(n_estimators=10, random_state=12))
]),
'KNN': Pipeline([
('scaler', StandardScaler()),
('smote', SMOTE(random_state=12, sampling_strategy=0.85)),
('undersample', NearMiss(version=1, sampling_strategy=1)),
('clf', KNeighborsClassifier())
]),
'XGB': Pipeline([ ('scaler', StandardScaler()),
('smote', SMOTE(random_state=12, sampling_strategy=0.85)),
('undersample', NearMiss(version=1, sampling_strategy=1)),
('clf', XGBClassifier(random_state=12, use_label_encoder=False, eval_metric='logloss',verbosity=0)) # you may calculate actual imbalance ratio
])
}
# ------------------------ Parameter Grids ------------------------
param_grids = {
'SVM': {
'clf__C': [0.1, 1, 10, 100],
'clf__kernel': ['linear', 'rbf'],
'clf__gamma': [1, 0.1, 0.01, 0.001]
},
'RF': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__max_depth': [None, 10, 20, 30],
'clf__min_samples_split': [2, 5, 10]
},
'KNN': {
'clf__n_neighbors': [3, 5, 10, 15],
'clf__weights': ['uniform', 'distance'],
'clf__p': [1, 2]
},
'XGB': {
'clf__n_estimators': [10, 50, 100, 200],
'clf__learning_rate': [0.01, 0.1, 0.2,0.3],
'clf__max_depth': [3,4,5,6],
'clf__subsample': [0.8, 0.9, 1.0],
'clf__min_child_weight': [1,3,5],
'clf__reg_alpha': [0, 0.1, 0.5,1],
'clf__reg_lambda': [1, 2, 5, 10]
}
}
# Example feature combinations
def avg_confusion_calculate(confusion_accuracy):
no_of_splits = len(confusion_accuracy)
rows, columns = confusion_accuracy[0].shape
avg_confusion_matrix = np.zeros((rows, columns))
for i in range(no_of_splits):
avg_confusion_matrix += confusion_accuracy[i]
avg_confusion_matrix /= no_of_splits
return avg_confusion_matrix
###--------------------Storage for results----------------------
results, result2 = [], []
confusion_avg_dict = {}
metrics_storage = {}
perm_importance_dict = {}
#-----------------------------------------------------
for features in lst5:
print(f"\nEvaluating feature set: {features}")
feature_key = str(features)
confusion_avg_dict[feature_key] = {}
metrics_storage[feature_key] = {}
df3 = df_Cohort.dropna(subset=features)
X = df3[features].values
y = (df3['Tumor_type'] == 'Mets').values.astype(int)
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=12)
for clf_name, pipeline in name_classifier.items():
print(f"\nClassifier: {clf_name}")
grid_search = GridSearchCV(
estimator=pipeline,
param_grid=param_grids[clf_name],
scoring='roc_auc',
cv=5,
n_jobs=-1
)
grid_search.fit(X, y)
best_model = grid_search.best_estimator_
best_params = grid_search.best_params_
# ---------------- Cross-validation for metrics AND feature importance ----------------
acc_list, bal_acc_list, auc_list, sensitivity_list, prec_list = [], [], [], [], []
conf_norm_list, conf_raw_list = [], []
## <<< MODIFICATION: Initialize lists for importance scores from each fold >>>
perm_importance_auc_list = []
for train_idx, test_idx in rskf.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
best_model.fit(X_train, y_train)
y_pred = best_model.predict(X_test)
y_prob = best_model.predict_proba(X_test)[:, 1]
acc_list.append(accuracy_score(y_test, y_pred) * 100)
bal_acc_list.append(balanced_accuracy_score(y_test, y_pred) * 100)
auc_list.append(roc_auc_score(y_test, y_prob))
sensitivity_list.append(recall_score(y_test, y_pred) * 100)
prec_list.append(precision_score(y_test, y_pred) * 100)
cm = confusion_matrix(y_test, y_pred)
if cm.sum(axis=1).min() == 0: # Avoid division by zero if a class is missing in a small test fold
conf_norm_list.append(cm.astype('float'))
else:
conf_norm_list.append(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis])
conf_raw_list.append(cm)
## <<< MODIFICATION: Calculate Permutation Importance for BOTH metrics >>>
# Based on ROC AUC
perm_result_auc = permutation_importance(
best_model, X_test, y_test, scoring='roc_auc',
n_repeats=5, random_state=42, n_jobs=1)
perm_importance_auc_list.append(perm_result_auc.importances_mean)
## <<< MODIFICATION: Aggregate importance scores AFTER the CV loop >>>
# Permutation Importance (based on AUC)
perm_importance_dict[(feature_key, clf_name)] = {
'features': features,
'importances_mean': np.mean(perm_importance_auc_list, axis=0),
'importances_std': np.std(perm_importance_auc_list, axis=0)
}
# ------------------- Store aggregated metrics for reporting ---------------------------------
metrics_storage[feature_key][clf_name] = {
'accuracy_model': acc_list,
'balanced_accuracy_model': bal_acc_list,
'roc_auc': auc_list,
'sensitivity_model': sensitivity_list,
'precision_model': prec_list
}
result2.append([
feature_key, clf_name, str(best_params),
round(np.mean(acc_list), 2), round(np.std(acc_list), 2),
round(np.mean(bal_acc_list), 2), round(np.std(bal_acc_list), 2),
round(np.mean(auc_list), 2), round(np.std(auc_list), 2),
round(np.mean(sensitivity_list), 2), round(np.std(sensitivity_list), 2),
round(np.mean(prec_list), 2), round(np.std(prec_list), 2),
len(df3)
])
confusion_avg = avg_confusion_calculate(conf_norm_list)
confusion_avg_dict[feature_key][clf_name] = confusion_avg
print(f"Avg confusion matrix for {features} and {clf_name} ML model:\n{confusion_avg}")
# --- ROC Plotting (Now uses the metrics from the CV loop) ---
mean_auc, std_auc = plot_roc_and_metrics(
model=best_model,
X=X,
y=y,
classifier_name=clf_name,
cv=rskf, # SAME splits as metrics calculation
color=None
)
#--------------------------------------------------------
output_dir = "ML_Results/TCGA_BM-pretreats/FD_Ratio/Oversampling_Undersampling_model_CV_Importance"
# output_dir = "ML_Results/Firstvisit_A_TCGA_UCSF-BMSR/FD_Ratio/Oversampling_Undersampling_model_CV_Importance"
os.makedirs(f"{output_dir}/Figures/ROC_curve", exist_ok=True)
plot_filename = f"{output_dir}/Figures/ROC_curve/OUS_{features}__ROC_curve.png"
plt.xlabel("")
plt.ylabel("")
plt.legend(loc="lower right", frameon=False,fontsize=18)
plt.savefig(plot_filename, dpi=300, bbox_inches='tight')
plt.show()
plt.close()
print(f"**********Finished processing feature set: {features}*****************")
# ------------------------ Save all results ------------------------
# ------------------------ Save metrics ------------------------
os.makedirs(f"{output_dir}/per_fold_metrics", exist_ok=True)
df_results = pd.DataFrame(result2, columns=[
"Feature Set","Classifier", "Best Parameters",
"Mean_Accuracy", "SD_Acc", "Mean_Balanced_Accuracy", "SD_Bal_Acc",
"Mean_AUC_Score", "SD_AUC", "Mean_Recall", "SD_Recall",
"Mean_Precision", "SD_Precision", "N_Samples"])
df_results.to_csv(f"{output_dir}/US_model_summary_results.csv", index=False)
# Save per-fold metrics
for metric in ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']:
rows = []
for feat, clf_dict in metrics_storage.items():
for clf, metric_dict in clf_dict.items():
row = [feat, clf] + metric_dict[metric]
rows.append(row)
if rows:
cols = ['Feature Set', 'Classifier'] + [f'Fold_{i+1}' for i in range(len(rows[0])-2)]
pd.DataFrame(rows, columns=cols).to_csv(f"{output_dir}/per_fold_metrics/OUS_{metric}.csv", index=False)
#-------------------------------------------------
# Define the list of metrics
metric_names = ['accuracy_model', 'balanced_accuracy_model', 'roc_auc', 'sensitivity_model', 'precision_model']
# Initialize a list to collect final rows
final_rows = []
# Loop through the features and classifiers, and gather the metric data
for feature, classifiers in metrics_storage.items():
for clf, values in classifiers.items():
# Determine the number of folds dynamically from any metric
num_folds = len(next(iter(values.values())))
for fold_idx in range(num_folds):
row = {
'Feature Set': feature,
'Classifier': clf,
'Fold': fold_idx + 1
}
# Fill in each metric value for this fold
for metric in metric_names:
metric_values = values.get(metric, [None] * num_folds)
row[metric] = metric_values[fold_idx]
final_rows.append(row)
# Create DataFrame from the collected rows
df_final = pd.DataFrame(final_rows)
# Save the DataFrame to CSV
output_file = f"{output_dir}/per_fold_metrics/OUS_ALL_SP_Baseline_combined_metrics.csv"
df_final.to_csv(output_file, index=False)
print(f"Saved: {output_file}")
#-------------------------------------------------
# Save the confusion_avg_dict for later use
with open(f"{output_dir}/per_fold_metrics/OUS_confusion_avg_dict.pkl", 'wb') as f:
pickle.dump(confusion_avg_dict, f)
print("Saved OVS_confusion_avg_dict.pkl")
### ----------------------------------------------------------------------------
# Save Permutation Importance (AUC)
perm_auc_df = []
for (feat_set, clf), vals in perm_importance_dict.items():
for f, imp, std in zip(vals['features'], vals['importances_mean'], vals['importances_std']):
perm_auc_df.append([feat_set, clf, f, imp, std])
pd.DataFrame(perm_auc_df, columns=['Feature Set', 'Classifier', 'Feature', 'Mean Importance', 'Std']).to_csv(
f"{output_dir}/OUS_permutation_importance_auc_cv.csv", index=False)
print("Saved permutation importance (ROC AUC).")
# %%

4_ML_modified_with_feature_importance_Github_19022026.ipynb at commit a86a96c, no license · at the source

Overview

Authors: Neha Yadav¹, Jayshaan Baibhav¹, Seemadri Subhadarshini², Saroj Kumar Das Majumdar³, Mohit K. Jolly², Vivek Tiwari¹

ORCID iDs: Vivek Tiwari

Institutions: Indian Institute of Science Education and Research Berhampur (India); Indian Institute of Science Bangalore (India); All India Institute of Medical Sciences Bhubaneswar (India)

Journal: iScience, volume 29, issue 4, article 115329

Dates: received 16 July 2025; accepted 9 March 2026; published online 11 March 2026

Type: Research article · Language: English

License: CC BY-NC

Identifiers: DOI 10.1016/j.isci.2026.115329 · PMID 42006316 · PMCID PMC13091536 · OpenAlex W7135032288

Open access: gold, a free copy (OpenAlex)

Status: code verified

Categories: genetics / omics (modality), other condition (population), clinical / translational (subfield)

Methods: Statistics, Smoothing, state filtering, decompositions, Machine learning, Connectivity, Complexity

Keywords: health sciences, oncology, medical physics, magnetic resonance imaging

Topic: Glioma Diagnosis and Treatment (Genetics, Medicine), according to OpenAlex

Funding: IISER Berhampur Seed Fund (5/13/2022/NCD/III); Indian Council of Medical Research; SERB-SRG (SRG/2022/002253); WoCoN; Param Hansa Centre for Computational Oncology; IISc Bangalore

Citations: not cited yet (Europe PMC); 59 references in the paper

Research resources: Python 3 RRID:SCR_008394

Abstract

Brain metastases (BMs) and gliomas arise from distinct biological origins yet frequently present with overlapping radiological features, particularly within peritumoral regions. By quantifying fractal dimension (FD3D) and lacunarity (Lac3D) across enhancing, non-enhancing, and edematous tumor subcomponents, we demonstrate that BMs from diverse primary cancers converge on a conserved structural phenotype within the brain-microenvironment, whereas gliomas retain distinct subcomponent complexity and geometric profiles. Enhancing subcomponents of BMs exhibited intermediate structural complexity between low- and high-grade gliomas (LGGs and HGGs), while non-enhancing and edematous compartments were structurally smoother. The integration of subcomponent complexity and volumetric fractions enabled accurate discrimination across independent imaging cohorts, and edema fractality uniquely predicted survival in BMs. Independent transcriptomic analyses revealed pathway-level convergence consistent with brain microenvironment-driven remodeling, yet molecular programs remained distinct from gliomas. These findings establish a quantitative neuroimaging-based geometric framework with clinical utility in neuro-oncology settings, reducing reliance on immediate biopsy for diagnostic and prognostic stratification.

Reproduced under the paper's license (CC BY-NC), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 17 matches between paragraphs and lines of code.

nibr-lab/Brain_Mets_FD_Lac_Manuscript

License: none: the authors keep all their rights

State: the link answers, verified on 30 September 2026

Evidence: files inventoried

Commit: a86a96cc00504d7f9f1898c94fe761cd9cefe9dd, 20 February 2026

Languages: Jupyter (7), R (5)

Size: 12 files, 12 scripts

Software Heritage: not archived

Found in: “Data and code availability”

Holds: 7 notebooks

Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation

Tools: NumPy (7 files), pandas (7 files), Matplotlib (5 files), SciPy (5 files), seaborn (4 files), tidyverse (4 files), ggplot2 (3 files), NiBabel (3 files), statannotations (3 files), statsmodels (3 files), ggpubr (2 files), OpenCV (2 files), scikit-posthocs (2 files), clusterProfiler (1 file), DESeq2 (1 file), emmeans (1 file), imbalanced-learn (1 file), pheatmap (1 file), scikit-learn (1 file), SHAP (1 file), SimpleITK (1 file), survival (1 file), XGBoost (1 file)

Availability: 1 check, the latest on 30 September 2026: the link answers

30 September 2026: the link answers

12 files

Python/1_FD3D_Calculation_GitHub_1902026.ipynb, Jupyter, 356 lines, 1 match
Python/2_Lac3D_Modified_Calculation_Github_19022025.ipynb, Jupyter, 369 lines, 2 matches
Python/3_Calculating_Volume_tumor_subcomponent_Github_19022026.ipynb, Jupyter, 149 lines
Python/4_ML_modified_with_feature_importance_Github_19022026.ipynb, Jupyter, 1,997 lines, 6 matches
Python/5_Survival_For_Brain_Mets_GitHub_19022026.ipynb, Jupyter, 607 lines, 1 match
Python/6_Manual_auto_Check_passing_bablok_Github_200202026.ipynb, Jupyter, 164 lines, 2 matches
Python/7_AUCell_Score_for_Hallmark_pathwatys_Variance_Github_20022026.ipynb, Jupyter, 346 lines, 1 match
Rcode/1_TCGA_download_GitHub.R, R, 71 lines, 2 matches
Rcode/2_DESEQ_HGG_vs_BM-M_Github.R, R, 71 lines
Rcode/3_Spearman_Corelation_Geometry_Molecular_GitHub_19022026.R, R, 202 lines, 1 match
Rcode/4_WT_ED_Corelated_genes_GitHub_19022026.R, R, 351 lines
Rcode/5_Brain-Mets_all_cutoff and survival.R, R, 399 lines, 1 match

At the source: github.com/nibr-lab/Brain_Mets_FD_Lac_Manuscript

nibr671lab/Brain_Mets_FD_Lac_Manuscript).•Any

License: none: the authors keep all their rights

State: the link is dead, verified on 30 September 2026

Evidence: found in the paper

Software Heritage: not archived

Found in: “Data and code availability”

Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation

Availability: 1 check, the latest on 30 September 2026: the link is dead

30 September 2026: the link is dead

At the source: github.com/nibr671lab/Brain_Mets_FD_Lac_Manuscript).•Any

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

2 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
12 scripts, each with its path and the digest of its content;
17 matches between paragraphs of the paper and lines of the code (method lexical-v1);
neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

ncbi.nlm.nih.gov/projects/gap, at NCBI; found in “Data and code availability”
portal.gdc.cancer.gov, at portal.gdc.cancer.gov; found in “Data and code availability”

Data and code availability

• The imaging data utilized in this study were sourced from The Cancer Imaging Archive (TCIA) portal (https://www.cancerimagingarchive.net/) and UCSF (https://imagingdatasets.ucsf.edu/dataset/1). The tumor subcomponent masks were obtained using the information published by Bakas et al. and others. • Transcriptomic data for gliomas (TCGA-LGG and TCGA-GBM) were accessed via the Genomic Data Commons (https://portal.gdc.cancer.gov/; dbGaP Study Accession: phs000178 (https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs000178)), while transcriptomic data for brain metastases were obtained from publicly available datasets by Zhang et al. (https://doi.org/10.1016/j.xcrm.2023.100932) and Biermann et al. (https://doi.org/10.1016/j.cell.2022.06.007) through the NCBI Gene Expression Omnibus (GEO). • All datasets used are publicly accessible and fully deidentified. • The Python and R codes used for fractal dimension and lacunarity calculations from tumor masks, as well as for other analyses, are available in our publicly accessible GitHub repository (https://github.com/nibr671lab/Brain_Mets_FD_Lac_Manuscript (https://github.com/nibr-lab/Brain_Mets_FD_Lac_Manuscript)). • Any additional information required to reanalyze the data reported in this paper is available from the lead author upon request.

Reproduced under the paper's license (CC BY-NC), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 30 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 6 authors, 4 keywords, 6 funders, 56 references, 1 RRID.

Cite

This paper

Yadav, N., Baibhav, J., Subhadarshini, S., Das Majumdar, S. K., Jolly, M. K., & Tiwari, V. (2026). Brain metastases converge on shared geometric architecture and transcriptomic landscape yet remain distinct from gliomas. iScience, 29(4), 115329. https://doi.org/10.1016/j.isci.2026.115329

BibTeX

@article{yadav2026brain, author = {Yadav, Neha and Baibhav, Jayshaan and Subhadarshini, Seemadri and Das Majumdar, Saroj Kumar and Jolly, Mohit K. and Tiwari, Vivek}, title = {{Brain metastases converge on shared geometric architecture and transcriptomic landscape yet remain distinct from gliomas}}, journal = {iScience}, year = {2026}, month = mar, volume = {29}, number = {4}, pages = {115329}, publisher = {Elsevier}, issn = {2589-0042}, doi = {10.1016/j.isci.2026.115329}, url = {https://doi.org/10.1016/j.isci.2026.115329}, pmid = {42006316}, pmcid = {PMC13091536} }

RIS

TY - JOUR AU - Yadav, Neha AU - Baibhav, Jayshaan AU - Subhadarshini, Seemadri AU - Das Majumdar, Saroj Kumar AU - Jolly, Mohit K. AU - Tiwari, Vivek TI - Brain metastases converge on shared geometric architecture and transcriptomic landscape yet remain distinct from gliomas T2 - iScience J2 - iScience PY - 2026 DA - 2026/03/11 VL - 29 IS - 4 SP - 115329 SN - 2589-0042 PB - Elsevier DO - 10.1016/j.isci.2026.115329 UR - https://doi.org/10.1016/j.isci.2026.115329 LA - en ER -

CSL-JSON

{ "id": "10.1016/j.isci.2026.115329", "type": "article-journal", "title": "Brain metastases converge on shared geometric architecture and transcriptomic landscape yet remain distinct from gliomas", "container-title": "iScience", "author": [ { "family": "Yadav", "given": "Neha" }, { "family": "Baibhav", "given": "Jayshaan" }, { "family": "Subhadarshini", "given": "Seemadri" }, { "family": "Das Majumdar", "given": "Saroj Kumar" }, { "family": "Jolly", "given": "Mohit K." }, { "family": "Tiwari", "given": "Vivek" } ], "container-title-short": "iScience", "volume": "29", "issue": "4", "page": "115329", "DOI": "10.1016/j.isci.2026.115329", "PMID": "42006316", "PMCID": "PMC13091536", "ISSN": "2589-0042", "publisher": "Elsevier", "URL": "https://doi.org/10.1016/j.isci.2026.115329", "language": "en", "issued": { "date-parts": [ [ 2026, 3, 11 ] ] } }

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s42003-026-10957-8 [code]: Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: imbalanced-learn, survival, SHAP, 15 other tools
[2] doi:10.1016/j.nicl.2026.104012 [code]: Structural-functional multilayer brain network properties and outcome of combined repetitive transcranial magnetic stimulation and psychotherapy for obsessive-compulsive disorder.
Journal: NeuroImage. Clinical
In common: imbalanced-learn, survival, SHAP, 12 other tools, other condition
[3] doi:10.1038/s41467-026-72598-z [code]: Functional impact of genetic background on variable expressivity in neurodevelopmental disorders.
Journal: Nature communications
In common: DESeq2, clusterProfiler, pheatmap, 9 other tools, ncbi.nlm.nih.gov/projects/gap, other condition
[4] doi:10.1093/bioinformatics/btag592 [code]: Network-based stratification of allele-specific expression reveals patient subgroups in Huntington's disease.
Journal: Bioinformatics (Oxford, England)
In common: survival, DESeq2, clusterProfiler, 11 other tools, genetics / omics, other condition
[5] doi:10.1016/j.xcrm.2026.102766 [code]: A longitudinal single-cell and spatial multiomic atlas of pediatric high-grade glioma.
Journal: Cell reports. Medicine
In common: DESeq2, clusterProfiler, pheatmap, 10 other tools, genetics / omics, other condition, 2 references
[6] doi:10.1073/pnas.2516601123 [code]: Unveiling the glymphatic system's role in brain aging: A comprehensive biomarker and modifiable intervention target.
Journal: Proceedings of the National Academy of Sciences of the United States of America
In common: survival, SHAP, XGBoost, 10 other tools, clinical / translational
[7] doi:10.1038/s41586-026-10629-x [code]: Whole-genome duplication shaped cell-type evolution in the vertebrate brain.
Journal: Nature
In common: DESeq2, clusterProfiler, pheatmap, 10 other tools, genetics / omics, 1 reference
[8] doi:10.1016/j.cell.2026.05.026 [code]: The critical role of the endogenous immune compartment after CAR T cell therapy in recurrent GBM.
Journal: Cell
In common: statannotations, survival, clusterProfiler, 9 other tools, genetics / omics, other condition
[9] doi:10.1002/imt2.70163 [code]: Spatial multi-omics unveils sphingolipid metabolic reprogramming within the retinal pathological niche.
Journal: iMeta
In common: DESeq2, clusterProfiler, pheatmap, 10 other tools, genetics / omics
[10] doi:10.1016/j.cell.2026.05.047 [code]: An emergent disease-associated motor neuron state precedes cell death in ALS.
Journal: Cell
In common: statannotations, DESeq2, pheatmap, 9 other tools, genetics / omics, other condition

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.