OSCR

A benchmark dataset and interpretable deep learning framework for drug-induced developmental neurotoxicity prediction.

Code ↔ Paper

13 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 13 matches · 1 of them tie a paragraph to a whole file, not to given lines: a weak match, whose lines are not tinted
  1. [1] § Materials and methods › Model construction and training strategy › Model training and evaluation ↔ 3-DNN_MACCS_modeling/DNN-test_selection.py, lines 33–67 · score 0.70 · monitoring validation, model selection, cross validation, weights, optimal, epochs
  2. [2] § Materials and methods › Performance metrics and statistical analysis ↔ 3-DNN_MACCS_modeling/DNN-independent validation.py, lines 208–293 · score 0.69 · F1 score, Matthew, TN, TP, FN, precision
  3. [3] § Materials and methods › Performance metrics and statistical analysis ↔ 3-DNN_MACCS_modeling/DNN-test_selection.py, lines 560–599 · score 0.67 · predicted toxic, evaluation metrics, F1 score, precision, recall, accuracy
  4. [4] § Materials and methods › Model construction and training strategy › Model training and evaluation ↔ 3-DNN_MACCS_modeling/DNN-training-5-fold-CV.py, lines 34–65 · score 0.65 · fold cross validation, network architectures, overfitting, epochs, training, Model
  5. [5] § Results and discussion › Analysis of the training stability and generalization ability of the DNN_MACCS model ↔ 3-DNN_MACCS_modeling/DNN-test_selection.py, lines 889–968 · score 0.64 · developmental neurotoxicity prediction, validation AUC, F1 score, dnn maccs, overfitting, configuration
  6. [6] § Materials and methods › Model interpretability analysis ↔ 4-applicability_domain_SHAP_analysis/SHAP-analysisl.py, lines 1056–1121 · score 0.62 · SHapley, exPlanations, Model interpretability, Additive, global, transparency
  7. [7] § Materials and methods › Benchmark dataset preparation › Data partitioning of the benchmark dataset ↔ 1-Benchmark_Dataset_Preparation/plot_umap_chemical_space.py, the whole file · a weak match · score 0.59 · chemical space, DNT positive, UMAP, benchmark, SMILES, fingerprints
  8. [8] § Results and discussion › Strengths and limitations ↔ 4-applicability_domain_SHAP_analysis/Application_Domain_UMAP_tSNE.py, lines 429–462 · score 0.58 · broader chemical, applicability domain, experimentally validated, model generalizability, chemical space, expand
  9. [9] § Results and discussion › Analysis of the training stability and generalization ability of the DNN_MACCS model ↔ 3-DNN_MACCS_modeling/DNN-test_selection.py, lines 889–968 · score 0.56 · potential overfitting, best validation, dnn maccs, loss, epochs, accuracy
  10. [10] § Materials and methods › Benchmark dataset preparation › Preprocessing of positive samples ↔ 1-Benchmark_Dataset_Preparation/DNTREF_curation.py, lines 25–46 · score 0.54 · heavy atoms, mixtures, parent, curation, SMILES
  11. [11] § Materials and methods › Chemical space visualization and applicability domain analysis ↔ 4-applicability_domain_SHAP_analysis/Application_Domain_UMAP_tSNE.py, lines 198–201 · score 0.52 · Dimensionality reduction, UMAP, Neighbor, SNE, domain
  12. [12] § Materials and methods › Benchmark dataset preparation ↔ 3-DNN_MACCS_modeling/DNN-test_selection.py, lines 1014–1134 · score 0.52 · model training, developmental neurotoxicity prediction, pipeline, selection, preprocessing
  13. [13] § Materials and methods › Benchmark dataset preparation › Preprocessing of presumed negative samples and construction of a balanced benchmark dataset ↔ 1-Benchmark_Dataset_Preparation/hard_negative_matching.py, lines 54–90 · score 0.50 · hard negative matching, greedy, MW, log, benchmark

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,147 lines · 51 KB · no license · 5 matches

  1. """
  2. Developmental Neurotoxicity Prediction DNN QSAR Model
  3. Author: DeepSeek Assistant
  4. Function: Train DNN model using validation accuracy for model selection
  5. Input files: DNT_benchmark_random_train.csv, DNT_benchmark_random_val.csv
  6. Output: Trained model, performance evaluation results, visualization charts
  7. """
  9. import numpy as np
  10. import pandas as pd
  11. import matplotlib
  12. matplotlib.use('TkAgg') # Set backend to avoid GUI issues
  13. import matplotlib.pyplot as plt
  14. import seaborn as sns
  15. from sklearn.metrics import (roc_auc_score, accuracy_score, precision_score,
  16. recall_score, f1_score, confusion_matrix,
  17. roc_curve, classification_report)
  18. from sklearn.preprocessing import StandardScaler
  19. from rdkit import Chem
  20. from rdkit.Chem import MACCSkeys
  21. import tensorflow as tf
  22. from tensorflow import keras
  23. from tensorflow.keras import layers, models, optimizers, regularizers, callbacks
  24. import warnings
  25. warnings.filterwarnings('ignore')
  26. import os
  28. # Set random seeds for reproducibility
  29. SEED = 42
  30. np.random.seed(SEED)
  31. tf.random.set_seed(SEED)
  33. # ==================== Configuration ====================
  34. class Config:
  35. """Model configuration parameters"""
  36. # Optimal parameters (based on previous cross-validation results)
  37. OPTIMAL_PARAMS = {
  38. 'layer_sizes': [256, 128, 64],
  39. 'dropout_rate': 0.4,
  40. 'learning_rate': 0.0005,
  41. 'l2_reg': 0.001,
  42. 'batch_size': 32,
  43. 'epochs': 200 # Slightly reduced for practical training time
  44. }
  46. # Training parameters
  47. REDUCE_LR_PATIENCE = 20
  48. REDUCE_LR_FACTOR = 0.5
  49. MIN_LR = 1e-6
  51. # Early stopping parameters
  52. EARLY_STOPPING_MONITOR = 'val_accuracy' # Monitor validation accuracy
  53. EARLY_STOPPING_PATIENCE = 40 # Stop if no improvement for 40 epochs
  54. EARLY_STOPPING_MIN_DELTA = 0.001 # Minimum change to qualify as improvement
  55. EARLY_STOPPING_MODE = 'max' # Maximize validation accuracy
  56. EARLY_STOPPING_RESTORE_BEST_WEIGHTS = True # Restore best weights when stopped
  58. # Model selection
  59. MODEL_SELECTION_METRIC = 'val_accuracy' # Use validation accuracy for model selection
  60. MODEL_SELECTION_MODE = 'max' # Maximize validation accuracy
  62. # Visualization settings
  63. FIGURE_DPI = 300
  65. # File paths
  66. TRAIN_FILE = 'DNT_benchmark_random_train-4_split_train.csv'
  67. VAL_FILE = 'DNT_benchmark_random_train-4_split_test.csv'
  69. # ==================== Data Preprocessing ====================
  70. class DataProcessor:
  71. """Data preprocessing class"""
  73. @staticmethod
  74. def load_data(train_path, val_path):
  75. """Load training and validation datasets"""
  76. try:
  77. train_df = pd.read_csv(train_path)
  78. val_df = pd.read_csv(val_path)
  80. print("=" * 60)
  81. print("Data Loading Complete")
  82. print("=" * 60)
  83. print(f"Training set size: {len(train_df)}")
  84. print(f"Validation set size: {len(val_df)}")
  85. print(f"\nTraining set class distribution:")
  86. print(train_df['label'].value_counts())
  87. print(f"\nValidation set class distribution:")
  88. print(val_df['label'].value_counts())
  90. if 'source' in train_df.columns:
  91. print(f"\nTraining set source distribution:")
  92. print(train_df['source'].value_counts())
  94. return train_df, val_df
  95. except Exception as e:
  96. print(f"Failed to load data: {e}")
  97. raise
  99. @staticmethod
  100. def smiles_to_maccs(smiles):
  101. """Convert SMILES to MACCS fingerprints"""
  102. try:
  103. mol = Chem.MolFromSmiles(smiles)
  104. if mol is not None:
  105. fp = MACCSkeys.GenMACCSKeys(mol)
  106. return np.array(fp)
  107. else:
  108. return np.zeros(167)
  109. except:
  110. return np.zeros(167)
  112. def prepare_features(self, train_df, val_df):
  113. """Prepare feature matrices and label vectors"""
  114. print("\nGenerating MACCS fingerprint features...")
  116. # Generate MACCS fingerprints
  117. X_train = np.array([self.smiles_to_maccs(s) for s in train_df['smiles']])
  118. y_train = train_df['label'].values
  119. X_val = np.array([self.smiles_to_maccs(s) for s in val_df['smiles']])
  120. y_val = val_df['label'].values
  122. # Check feature dimensions
  123. print(f"Training set feature dimension: {X_train.shape}")
  124. print(f"Validation set feature dimension: {X_val.shape}")
  126. # Data standardization
  127. scaler = StandardScaler()
  128. X_train_scaled = scaler.fit_transform(X_train)
  129. X_val_scaled = scaler.transform(X_val)
  131. print("Feature standardization complete")
  133. return X_train_scaled, y_train, X_val_scaled, y_val, scaler
  135. # ==================== DNN Model Builder ====================
  136. class DNNModelBuilder:
  137. """DNN model builder"""
  139. @staticmethod
  140. def create_model(input_dim=167,
  141. layer_sizes=[256, 128, 64],
  142. dropout_rate=0.3,
  143. learning_rate=0.001,
  144. l2_reg=0.001):
  145. """Create DNN model architecture"""
  146. model = models.Sequential()
  148. # Input layer and first hidden layer
  149. model.add(layers.Dense(layer_sizes[0], input_dim=input_dim,
  150. activation='relu',
  151. kernel_regularizer=regularizers.l2(l2_reg),
  152. name=f"dense_input_{layer_sizes[0]}"))
  153. model.add(layers.BatchNormalization(name=f"bn_1"))
  154. model.add(layers.Dropout(dropout_rate, name=f"dropout_1"))
  156. # Add subsequent hidden layers
  157. for i, layer_size in enumerate(layer_sizes[1:], start=2):
  158. model.add(layers.Dense(layer_size, activation='relu',
  159. kernel_regularizer=regularizers.l2(l2_reg),
  160. name=f"dense_hidden_{i}_{layer_size}"))
  161. model.add(layers.BatchNormalization(name=f"bn_{i}"))
  162. model.add(layers.Dropout(dropout_rate, name=f"dropout_{i}"))
  164. # Output layer (binary classification)
  165. model.add(layers.Dense(1, activation='sigmoid', name="output"))
  167. # Compile model
  168. optimizer = optimizers.Adam(learning_rate=learning_rate)
  169. model.compile(loss='binary_crossentropy',
  170. optimizer=optimizer,
  171. metrics=['accuracy',
  172. keras.metrics.AUC(name='auc'),
  173. keras.metrics.Precision(name='precision'),
  174. keras.metrics.Recall(name='recall')])
  176. return model
  178. # ==================== Model Trainer ====================
  179. class ModelTrainer:
  180. """Model trainer with validation accuracy selection and early stopping"""
  182. def __init__(self, config):
  183. self.config = config
  184. self.best_epoch = 0
  185. self.best_val_accuracy = 0.0
  186. self.best_model_path = None
  187. self.early_stopped_epoch = None
  188. self.stopped_reason = None
  190. def train_final_model(self, X_train, y_train, X_val, y_val):
  191. """Train final model using optimal parameters with validation accuracy selection and early stopping"""
  192. print("\n" + "=" * 60)
  193. print("Training Final Model with Optimal Parameters")
  194. print("=" * 60)
  195. print("Model Selection Strategy: Using validation accuracy to select best model")
  196. print("Early Stopping: Enabled with patience =", self.config.EARLY_STOPPING_PATIENCE)
  197. print("=" * 60)
  199. # Get optimal parameters
  200. optimal_params = self.config.OPTIMAL_PARAMS
  202. # Separate model building parameters and training parameters
  203. model_build_params = {
  204. 'layer_sizes': optimal_params['layer_sizes'],
  205. 'dropout_rate': optimal_params['dropout_rate'],
  206. 'learning_rate': optimal_params['learning_rate'],
  207. 'l2_reg': optimal_params['l2_reg']
  208. }
  210. # Create final model
  211. final_model = DNNModelBuilder.create_model(
  212. input_dim=X_train.shape[1],
  213. **model_build_params
  214. )
  216. # Print model architecture
  217. print("\nModel Architecture:")
  218. final_model.summary()
  220. # Create directory for saved models
  221. os.makedirs('checkpoints', exist_ok=True)
  222. self.best_model_path = 'checkpoints/best_model_val_acc.h5'
  224. # Training callbacks - focus on validation accuracy with early stopping
  225. callbacks_list = [
  226. # Early stopping callback
  227. callbacks.EarlyStopping(
  228. monitor=self.config.EARLY_STOPPING_MONITOR,
  229. patience=self.config.EARLY_STOPPING_PATIENCE,
  230. min_delta=self.config.EARLY_STOPPING_MIN_DELTA,
  231. mode=self.config.EARLY_STOPPING_MODE,
  232. restore_best_weights=self.config.EARLY_STOPPING_RESTORE_BEST_WEIGHTS,
  233. verbose=1
  234. ),
  235. # Learning rate reduction
  236. callbacks.ReduceLROnPlateau(
  237. monitor='val_loss',
  238. factor=self.config.REDUCE_LR_FACTOR,
  239. patience=self.config.REDUCE_LR_PATIENCE,
  240. min_lr=self.config.MIN_LR,
  241. verbose=1
  242. ),
  243. # Model checkpoint - save best model based on validation accuracy
  244. callbacks.ModelCheckpoint(
  245. self.best_model_path,
  246. monitor=self.config.MODEL_SELECTION_METRIC,
  247. mode=self.config.MODEL_SELECTION_MODE,
  248. save_best_only=True,
  249. save_weights_only=False,
  250. verbose=1
  251. ),
  252. callbacks.CSVLogger('training_log.csv'),
  253. # Custom callback to track best epoch
  254. callbacks.LambdaCallback(
  255. on_epoch_end=lambda epoch, logs: self._update_best_epoch(epoch, logs)
  256. ),
  257. # Custom callback to track early stopping
  258. callbacks.LambdaCallback(
  259. on_epoch_end=lambda epoch, logs: self._check_early_stopping(epoch, logs)
  260. )
  261. ]
  263. # Train model
  264. print(f"\nTraining Configuration:")
  265. print(f" Network Architecture: {model_build_params['layer_sizes']}")
  266. print(f" Dropout Rate: {model_build_params['dropout_rate']}")
  267. print(f" Learning Rate: {model_build_params['learning_rate']}")
  268. print(f" L2 Regularization: {model_build_params['l2_reg']}")
  269. print(f" Batch Size: {optimal_params['batch_size']}")
  270. print(f" Maximum Epochs: {optimal_params['epochs']}")
  271. print(f" Early Stopping Patience: {self.config.EARLY_STOPPING_PATIENCE}")
  272. print(f" Model Selection Metric: {self.config.MODEL_SELECTION_METRIC}")
  273. print(f" Best Model Saved To: {self.best_model_path}")
  275. history = final_model.fit(
  276. X_train, y_train,
  277. validation_data=(X_val, y_val),
  278. epochs=optimal_params['epochs'],
  279. batch_size=optimal_params['batch_size'],
  280. callbacks=callbacks_list,
  281. verbose=1
  282. )
  284. # Check if early stopping occurred
  285. if hasattr(final_model, 'history') and len(final_model.history.history['loss']) < optimal_params['epochs']:
  286. self.early_stopped_epoch = len(final_model.history.history['loss'])
  287. self.stopped_reason = f"Early stopping triggered at epoch {self.early_stopped_epoch}"
  288. print(f"\n⚠ Early stopping was triggered at epoch {self.early_stopped_epoch}")
  289. print(f" Best validation accuracy: {self.best_val_accuracy:.4f}")
  291. # Load the best model based on validation accuracy
  292. print(f"\nLoading best model from epoch {self.best_epoch + 1}")
  293. print(f"Best validation accuracy: {self.best_val_accuracy:.4f}")
  295. # Create a new model instance and load the best weights
  296. best_model = DNNModelBuilder.create_model(
  297. input_dim=X_train.shape[1],
  298. **model_build_params
  299. )
  300. best_model.load_weights(self.best_model_path)
  302. return best_model, history, final_model
  304. def _update_best_epoch(self, epoch, logs):
  305. """Update the best epoch based on validation accuracy"""
  306. if logs is not None and 'val_accuracy' in logs:
  307. current_val_acc = logs['val_accuracy']
  308. if current_val_acc > self.best_val_accuracy:
  309. self.best_val_accuracy = current_val_acc
  310. self.best_epoch = epoch
  311. print(f"\nNew best validation accuracy: {current_val_acc:.4f} at epoch {epoch + 1}")
  313. def _check_early_stopping(self, epoch, logs):
  314. """Check if early stopping criteria are met"""
  315. # This is handled by Keras EarlyStopping callback
  316. pass
  318. @staticmethod
  319. def plot_training_history(history, params, best_epoch=None, best_val_acc=None):
  320. """Plot training history with best epoch marked and y-axis starting from 0"""
  321. plt.figure(figsize=(14, 5))
  323. # Subplot 1: Loss (y-axis starts from 0)
  324. plt.subplot(1, 3, 1)
  325. plt.plot(history.history['loss'], label='Training Loss')
  326. plt.plot(history.history['val_loss'], label='Validation Loss')
  328. # Mark best epoch if provided
  329. if best_epoch is not None:
  330. plt.axvline(x=best_epoch, color='r', linestyle='--', alpha=0.7,
  331. label=f'Best Epoch: {best_epoch + 1}')
  332. plt.plot(best_epoch, history.history['val_loss'][best_epoch],
  333. 'ro', markersize=8)
  335. plt.title('Model Loss')
  336. plt.xlabel('Epochs')
  337. plt.ylabel('Loss')
  338. plt.legend()
  339. plt.grid(True, alpha=0.3)
  340. plt.ylim(bottom=0) # Set y-axis to start from 0
  342. # Subplot 2: AUC (y-axis starts from 0)
  343. plt.subplot(1, 3, 2)
  344. plt.plot(history.history['auc'], label='Training AUC')
  345. plt.plot(history.history['val_auc'], label='Validation AUC')
  347. # Mark best epoch if provided
  348. if best_epoch is not None:
  349. plt.axvline(x=best_epoch, color='r', linestyle='--', alpha=0.7)
  350. plt.plot(best_epoch, history.history['val_auc'][best_epoch],
  351. 'ro', markersize=8)
  353. plt.title('Model AUC')
  354. plt.xlabel('Epochs')
  355. plt.ylabel('AUC')
  356. plt.legend()
  357. plt.grid(True, alpha=0.3)
  358. plt.ylim(0, 1) # Set y-axis from 0 to 1
  360. # Subplot 3: Accuracy (y-axis starts from 0)
  361. plt.subplot(1, 3, 3)
  362. plt.plot(history.history['accuracy'], label='Training Accuracy')
  363. plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
  365. # Mark best epoch if provided
  366. if best_epoch is not None:
  367. plt.axvline(x=best_epoch, color='r', linestyle='--', alpha=0.7)
  368. plt.plot(best_epoch, history.history['val_accuracy'][best_epoch],
  369. 'ro', markersize=8, label=f'Best: {best_val_acc:.4f}')
  371. plt.title('Model Accuracy')
  372. plt.xlabel('Epochs')
  373. plt.ylabel('Accuracy')
  374. plt.legend()
  375. plt.grid(True, alpha=0.3)
  376. plt.ylim(0, 1) # Set y-axis from 0 to 1
  378. title_suffix = ""
  379. if best_epoch is not None:
  380. title_suffix = f" | Best Epoch: {best_epoch + 1}"
  382. plt.suptitle(f"Training History - Network Architecture: {params['layer_sizes']}{title_suffix}",
  383. fontsize=14)
  384. plt.tight_layout()
  385. plt.savefig('training_history_val_acc.png', dpi=Config.FIGURE_DPI, bbox_inches='tight')
  386. plt.show()
  388. # ==================== Model Evaluator ====================
  389. class ModelEvaluator:
  390. """Model evaluator"""
  392. @staticmethod
  393. def evaluate_model(model, X_train, y_train, X_val, y_val, model_name="Final Model", val_df=None):
  394. """Evaluate model performance and output statistical results"""
  395. print("\n" + "=" * 60)
  396. print(f"Model Performance Evaluation - {model_name}")
  397. print("=" * 60)
  399. # Training set predictions
  400. y_train_pred = model.predict(X_train, verbose=0)
  401. y_train_pred_class = (y_train_pred > 0.5).astype(int)
  402. y_train_pred = y_train_pred.flatten()
  404. # Validation set predictions
  405. y_val_pred = model.predict(X_val, verbose=0)
  406. y_val_pred_class = (y_val_pred > 0.5).astype(int)
  407. y_val_pred = y_val_pred.flatten()
  409. # Calculate metrics
  410. train_metrics = ModelEvaluator._calculate_metrics(y_train, y_train_pred, y_train_pred_class)
  411. val_metrics = ModelEvaluator._calculate_metrics(y_val, y_val_pred, y_val_pred_class)
  413. # Output performance metrics
  414. metrics_df = ModelEvaluator._create_metrics_table(train_metrics, val_metrics)
  416. # Output classification report
  417. ModelEvaluator._print_classification_report(y_val, y_val_pred_class)
  419. # Save validation set predictions if val_df is provided
  420. if val_df is not None:
  421. ModelEvaluator._save_validation_predictions(val_df, y_val, y_val_pred, y_val_pred_class, model_name)
  423. return metrics_df, y_train_pred, y_val_pred, y_val_pred_class
  425. @staticmethod
  426. def _calculate_metrics(y_true, y_pred_prob, y_pred_class):
  427. """Calculate all evaluation metrics"""
  428. return {
  429. 'Accuracy': accuracy_score(y_true, y_pred_class),
  430. 'Precision': precision_score(y_true, y_pred_class, zero_division=0),
  431. 'Recall': recall_score(y_true, y_pred_class, zero_division=0),
  432. 'F1-Score': f1_score(y_true, y_pred_class, zero_division=0),
  433. 'AUC': roc_auc_score(y_true, y_pred_prob),
  434. 'Specificity': ModelEvaluator._calculate_specificity(y_true, y_pred_class)
  435. }
  437. @staticmethod
  438. def _calculate_specificity(y_true, y_pred):
  439. """Calculate specificity"""
  440. tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
  441. return tn / (tn + fp) if (tn + fp) > 0 else 0
  443. @staticmethod
  444. def _create_metrics_table(train_metrics, val_metrics):
  445. """Create performance metrics table"""
  446. metrics_df = pd.DataFrame({
  447. 'Training Set': train_metrics,
  448. 'Validation Set': val_metrics
  449. })
  451. print("\nPerformance Metrics Comparison:")
  452. print(metrics_df.round(4))
  454. # Calculate differences
  455. metrics_df['Difference (Validation-Training)'] = metrics_df['Validation Set'] - metrics_df['Training Set']
  456. print("\nTraining vs Validation Set Differences:")
  457. print(metrics_df['Difference (Validation-Training)'].round(4))
  459. return metrics_df
  461. @staticmethod
  462. def _print_classification_report(y_true, y_pred):
  463. """Print classification report"""
  464. print("\nValidation Set Classification Report:")
  465. print(classification_report(y_true, y_pred,
  466. target_names=['Non-toxic (Class 0)', 'Toxic (Class 1)']))
  468. @staticmethod
  469. def _save_validation_predictions(val_df, y_true, y_pred_prob, y_pred_class, model_name):
  470. """Save validation set predictions to CSV file"""
  471. try:
  472. # Create a DataFrame with predictions
  473. predictions_df = pd.DataFrame({
  474. 'smiles': val_df['smiles'].values,
  475. 'true_label': y_true,
  476. 'predicted_probability': y_pred_prob,
  477. 'predicted_label': y_pred_class,
  478. 'correct_prediction': (y_true == y_pred_class).astype(int)
  479. })
  481. # Add prediction confidence categories
  482. predictions_df['prediction_confidence'] = pd.cut(
  483. predictions_df['predicted_probability'],
  484. bins=[0, 0.3, 0.7, 1.0],
  485. labels=['Low', 'Medium', 'High'],
  486. include_lowest=True
  487. )
  489. # Sort by prediction probability (descending)
  490. predictions_df = predictions_df.sort_values('predicted_probability', ascending=False)
  492. # Create model-specific filename
  493. model_name_clean = model_name.lower().replace(' ', '_').replace('-', '_')
  494. filename = f'validation_predictions_{model_name_clean}.csv'
  496. # Save to CSV
  497. predictions_df.to_csv(filename, index=False)
  499. print(f"\n✓ Validation set predictions saved to: {filename}")
  500. print(f" - Total molecules: {len(predictions_df)}")
  501. print(f" - Correct predictions: {predictions_df['correct_prediction'].sum()} ({predictions_df['correct_prediction'].mean():.2%})")
  503. # Print summary statistics
  504. print(f"\n Prediction Statistics:")
  505. print(f" - Mean predicted probability: {predictions_df['predicted_probability'].mean():.4f}")
  506. print(f" - Std predicted probability: {predictions_df['predicted_probability'].std():.4f}")
  507. print(f" - Min predicted probability: {predictions_df['predicted_probability'].min():.4f}")
  508. print(f" - Max predicted probability: {predictions_df['predicted_probability'].max():.4f}")
  510. # Count by confidence level
  511. confidence_counts = predictions_df['prediction_confidence'].value_counts().sort_index()
  512. print(f"\n Confidence Level Distribution:")
  513. for conf_level, count in confidence_counts.items():
  514. percentage = count / len(predictions_df) * 100
  515. print(f" - {conf_level}: {count} molecules ({percentage:.1f}%)")
  517. except Exception as e:
  518. print(f"⚠ Warning: Failed to save validation predictions: {e}")
  520. @staticmethod
  521. def plot_comparison_visualizations(y_train, y_train_pred_prob_best, y_train_pred_prob_final,
  522. y_val, y_val_pred_prob_best, y_val_pred_prob_final,
  523. y_val_true, y_val_pred_class_best,
  524. train_metrics_best, val_metrics_best,
  525. train_metrics_final=None, val_metrics_final=None):
  526. """Plot comparison visualizations between best and final models"""
  527. fig = plt.figure(figsize=(16, 10))
  529. # Subplot 1: ROC Curves Comparison
  530. ax1 = plt.subplot(2, 3, 1)
  532. # Calculate ROC curves for best model
  533. fpr_train_best, tpr_train_best, _ = roc_curve(y_train, y_train_pred_prob_best)
  534. fpr_val_best, tpr_val_best, _ = roc_curve(y_val, y_val_pred_prob_best)
  536. ax1.plot(fpr_train_best, tpr_train_best, 'b-',
  537. label=f'Training (Best) AUC = {train_metrics_best["AUC"]:.3f}')
  538. ax1.plot(fpr_val_best, tpr_val_best, 'r-',
  539. label=f'Validation (Best) AUC = {val_metrics_best["AUC"]:.3f}')
  541. # Calculate ROC curves for final model if provided
  542. if y_train_pred_prob_final is not None and y_val_pred_prob_final is not None:
  543. fpr_train_final, tpr_train_final, _ = roc_curve(y_train, y_train_pred_prob_final)
  544. fpr_val_final, tpr_val_final, _ = roc_curve(y_val, y_val_pred_prob_final)
  546. ax1.plot(fpr_train_final, tpr_train_final, 'b--', alpha=0.7,
  547. label=f'Training (Final) AUC = {train_metrics_final["AUC"]:.3f}')
  548. ax1.plot(fpr_val_final, tpr_val_final, 'r--', alpha=0.7,
  549. label=f'Validation (Final) AUC = {val_metrics_final["AUC"]:.3f}')
  551. ax1.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
  552. ax1.set_xlim([0.0, 1.0])
  553. ax1.set_ylim([0.0, 1.05])
  554. ax1.set_xlabel('False Positive Rate (FPR)')
  555. ax1.set_ylabel('True Positive Rate (TPR)')
  556. ax1.set_title('ROC Curve Comparison')
  557. ax1.legend(loc="lower right")
  558. ax1.grid(True, alpha=0.3)
  560. # Subplot 2: Confusion Matrix for Best Model
  561. ax2 = plt.subplot(2, 3, 2)
  562. cm_best = confusion_matrix(y_val_true, y_val_pred_class_best)
  563. sns.heatmap(cm_best, annot=True, fmt='d', cmap='Blues', ax=ax2,
  564. xticklabels=['Predicted Non-toxic', 'Predicted Toxic'],
  565. yticklabels=['Actual Non-toxic', 'Actual Toxic'])
  566. ax2.set_title('Confusion Matrix (Best Model)')
  567. ax2.set_ylabel('True Label')
  568. ax2.set_xlabel('Predicted Label')
  570. # Subplot 3: Metrics Comparison Bar Chart
  571. ax3 = plt.subplot(2, 3, 3)
  572. metrics_to_plot = ['Accuracy', 'Precision', 'Recall', 'F1-Score', 'AUC']
  573. x = np.arange(len(metrics_to_plot))
  574. width = 0.35
  576. train_values_best = [train_metrics_best[m] for m in metrics_to_plot]
  577. val_values_best = [val_metrics_best[m] for m in metrics_to_plot]
  579. bars1 = ax3.bar(x - width/2, train_values_best, width, label='Training (Best)', alpha=0.8, color='blue')
  580. bars2 = ax3.bar(x + width/2, val_values_best, width, label='Validation (Best)', alpha=0.8, color='red')
  582. # Add final model metrics if provided
  583. if train_metrics_final is not None and val_metrics_final is not None:
  584. train_values_final = [train_metrics_final[m] for m in metrics_to_plot]
  585. val_values_final = [val_metrics_final[m] for m in metrics_to_plot]
  587. bars3 = ax3.bar(x - width/2, train_values_final, width, label='Training (Final)',
  588. alpha=0.5, color='lightblue', hatch='//')
  589. bars4 = ax3.bar(x + width/2, val_values_final, width, label='Validation (Final)',
  590. alpha=0.5, color='lightcoral', hatch='\\\\')
  592. ax3.set_xlabel('Evaluation Metrics')
  593. ax3.set_ylabel('Score')
  594. ax3.set_title('Performance Metrics Comparison')
  595. ax3.set_xticks(x)
  596. ax3.set_xticklabels(metrics_to_plot, rotation=45)
  597. ax3.legend()
  598. ax3.grid(True, alpha=0.3, axis='y')
  599. ax3.set_ylim(0, 1) # Set y-axis from 0 to 1
  601. # Subplot 4: Validation Accuracy Comparison
  602. ax4 = plt.subplot(2, 3, 4)
  603. comparison_data = {
  604. 'Best Model': val_metrics_best['Accuracy'],
  605. }
  607. if val_metrics_final is not None:
  608. comparison_data['Final Epoch Model'] = val_metrics_final['Accuracy']
  610. models = list(comparison_data.keys())
  611. acc_values = list(comparison_data.values())
  613. bars = ax4.bar(models, acc_values, color=['green', 'orange'][:len(models)])
  614. ax4.set_ylabel('Validation Accuracy')
  615. ax4.set_title('Validation Accuracy Comparison')
  616. ax4.set_ylim([0, 1.0])
  618. # Add value labels on bars
  619. for i, (bar, val) in enumerate(zip(bars, acc_values)):
  620. height = bar.get_height()
  621. ax4.text(bar.get_x() + bar.get_width()/2., height + 0.01,
  622. f'{val:.4f}', ha='center', va='bottom')
  624. ax4.grid(True, alpha=0.3, axis='y')
  626. # Subplot 5: AUC Comparison
  627. ax5 = plt.subplot(2, 3, 5)
  628. auc_comparison_data = {
  629. 'Best Model': val_metrics_best['AUC'],
  630. }
  632. if val_metrics_final is not None:
  633. auc_comparison_data['Final Epoch Model'] = val_metrics_final['AUC']
  635. models_auc = list(auc_comparison_data.keys())
  636. auc_values = list(auc_comparison_data.values())
  638. bars_auc = ax5.bar(models_auc, auc_values, color=['green', 'orange'][:len(models_auc)])
  639. ax5.set_ylabel('AUC Score')
  640. ax5.set_title('AUC Score Comparison')
  641. ax5.set_ylim([0, 1.0])
  643. # Add value labels on bars
  644. for i, (bar, val) in enumerate(zip(bars_auc, auc_values)):
  645. height = bar.get_height()
  646. ax5.text(bar.get_x() + bar.get_width()/2., height + 0.01,
  647. f'{val:.4f}', ha='center', va='bottom')
  649. ax5.grid(True, alpha=0.3, axis='y')
  651. # Subplot 6: F1-Score Comparison
  652. ax6 = plt.subplot(2, 3, 6)
  653. f1_comparison_data = {
  654. 'Best Model': val_metrics_best['F1-Score'],
  655. }
  657. if val_metrics_final is not None:
  658. f1_comparison_data['Final Epoch Model'] = val_metrics_final['F1-Score']
  660. models_f1 = list(f1_comparison_data.keys())
  661. f1_values = list(f1_comparison_data.values())
  663. bars_f1 = ax6.bar(models_f1, f1_values, color=['green', 'orange'][:len(models_f1)])
  664. ax6.set_ylabel('F1-Score')
  665. ax6.set_title('F1-Score Comparison')
  666. ax6.set_ylim([0, 1.0])
  668. # Add value labels on bars
  669. for i, (bar, val) in enumerate(zip(bars_f1, f1_values)):
  670. height = bar.get_height()
  671. ax6.text(bar.get_x() + bar.get_width()/2., height + 0.01,
  672. f'{val:.4f}', ha='center', va='bottom')
  674. ax6.grid(True, alpha=0.3, axis='y')
  676. plt.suptitle('Model Performance Comparison: Best vs Final Epoch Model', fontsize=16)
  677. plt.tight_layout()
  678. plt.savefig('model_comparison.png', dpi=Config.FIGURE_DPI, bbox_inches='tight')
  679. plt.show()
  681. # ==================== Results Saver ====================
  682. class ResultSaver:
  683. """Results saver"""
  685. @staticmethod
  686. def save_results(best_model, scaler, best_metrics_df, history, best_epoch, best_val_acc,
  687. final_model=None, final_metrics_df=None, val_df=None,
  688. y_val=None, y_val_pred_best=None, y_val_pred_final=None):
  689. """Save all results"""
  690. print("\n" + "=" * 60)
  691. print("Saving Model and Results")
  692. print("=" * 60)
  694. # Create results directory
  695. os.makedirs('results_val_acc', exist_ok=True)
  697. # Get optimal parameters
  698. optimal_params = Config.OPTIMAL_PARAMS
  700. # 1. Save best model
  701. best_model.save('results_val_acc/best_model_val_acc.h5')
  702. print("✓ Best model saved: results_val_acc/best_model_val_acc.h5")
  704. # Save final model if provided
  705. if final_model is not None:
  706. final_model.save('results_val_acc/final_epoch_model.h5')
  707. print("✓ Final epoch model saved: results_val_acc/final_epoch_model.h5")
  709. # 2. Save scaler
  710. import joblib
  711. joblib.dump(scaler, 'results_val_acc/scaler.pkl')
  712. print("✓ Scaler saved: results_val_acc/scaler.pkl")
  714. # 3. Save validation set predictions for best model
  715. if val_df is not None and y_val is not None and y_val_pred_best is not None:
  716. ResultSaver._save_detailed_predictions(val_df, y_val, y_val_pred_best,
  717. 'best_model', optimal_params)
  719. # 4. Save validation set predictions for final model (optional)
  720. if val_df is not None and y_val is not None and y_val_pred_final is not None:
  721. ResultSaver._save_detailed_predictions(val_df, y_val, y_val_pred_final,
  722. 'final_epoch_model', optimal_params)
  724. # 5. Save optimal parameters and training info
  725. with open('results_val_acc/training_info.txt', 'w') as f:
  726. f.write("Optimal Model Parameters (based on cross-validation results):\n")
  727. f.write("=" * 50 + "\n")
  728. for param, value in optimal_params.items():
  729. f.write(f"{param}: {value}\n")
  731. f.write("\nModel Selection Information:\n")
  732. f.write("=" * 50 + "\n")
  733. f.write(f"Model Selection Metric: {Config.MODEL_SELECTION_METRIC}\n")
  734. f.write(f"Best Epoch: {best_epoch + 1}\n")
  735. f.write(f"Best Validation Accuracy: {best_val_acc:.4f}\n")
  736. f.write(f"Total Epochs Trained: {len(history.history['loss'])}\n")
  738. print("✓ Training information saved: results_val_acc/training_info.txt")
  740. # 6. Save performance metrics
  741. best_metrics_df.to_csv('results_val_acc/best_model_performance.csv')
  742. print("✓ Best model performance metrics saved: results_val_acc/best_model_performance.csv")
  744. if final_metrics_df is not None:
  745. final_metrics_df.to_csv('results_val_acc/final_epoch_performance.csv')
  746. print("✓ Final epoch performance metrics saved: results_val_acc/final_epoch_performance.csv")
  748. # 7. Save training history
  749. history_df = pd.DataFrame(history.history)
  750. history_df.to_csv('results_val_acc/training_history.csv')
  751. print("✓ Training history saved: results_val_acc/training_history.csv")
  753. # 8. Save configuration summary
  754. with open('results_val_acc/config_summary.txt', 'w') as f:
  755. f.write("Model Configuration Summary\n")
  756. f.write("=" * 60 + "\n\n")
  757. f.write("Overfitting Prevention Measures:\n")
  758. f.write(f" - L2 Regularization: {optimal_params['l2_reg']}\n")
  759. f.write(f" - Dropout Rate: {optimal_params['dropout_rate']}\n")
  760. f.write(f" - Learning Rate Reduction Factor: {Config.REDUCE_LR_FACTOR}\n")
  761. f.write(f" - Learning Rate Reduction Patience: {Config.REDUCE_LR_PATIENCE}\n")
  762. f.write(f" - Early Stopping Patience: {Config.EARLY_STOPPING_PATIENCE}\n\n")
  764. f.write("Network Architecture:\n")
  765. f.write(f" - Layer Structure: {optimal_params['layer_sizes']}\n")
  766. f.write(f" - Total Layers: {len(optimal_params['layer_sizes'])} (hidden) + 1 (output)\n\n")
  768. f.write("Training Parameters:\n")
  769. f.write(f" - Learning Rate: {optimal_params['learning_rate']}\n")
  770. f.write(f" - Batch Size: {optimal_params['batch_size']}\n")
  771. f.write(f" - Maximum Epochs: {optimal_params['epochs']}\n")
  772. f.write(f" - Actual Epochs Trained: {len(history.history['loss'])}\n")
  773. f.write(f" - Model Selection: Based on {Config.MODEL_SELECTION_METRIC}\n")
  774. f.write(f" - Best Model Epoch: {best_epoch + 1}\n\n")
  776. f.write("Dataset Information:\n")
  777. f.write(f" - Training Set Size: 2179\n")
  778. f.write(f" - Validation Set Size: 545\n")
  779. f.write(f" - Feature Dimension: 167 (MACCS fingerprints)\n")
  781. print("✓ Configuration summary saved: results_val_acc/config_summary.txt")
  783. # 9. Generate final report
  784. ResultSaver._generate_final_report(best_metrics_df, optimal_params, history,
  785. best_epoch, best_val_acc, final_metrics_df)
  787. @staticmethod
  788. def _save_detailed_predictions(val_df, y_true, y_pred_prob, model_type, optimal_params):
  789. """Save detailed validation set predictions for a specific model"""
  790. try:
  791. # Calculate predicted class
  792. y_pred_class = (y_pred_prob > 0.5).astype(int)
  794. # Create detailed predictions DataFrame
  795. predictions_df = pd.DataFrame({
  796. 'smiles': val_df['smiles'].values,
  797. 'true_label': y_true,
  798. 'predicted_probability': y_pred_prob,
  799. 'predicted_label': y_pred_class,
  800. 'prediction_correct': (y_true == y_pred_class).astype(int),
  801. 'prediction_error': np.abs(y_true - y_pred_prob)
  802. })
  804. # Add additional information if available in original dataframe
  805. for col in val_df.columns:
  806. if col not in ['smiles', 'label'] and col not in predictions_df.columns:
  807. predictions_df[col] = val_df[col].values
  809. # Add prediction confidence level
  810. predictions_df['confidence_level'] = pd.cut(
  811. predictions_df['predicted_probability'],
  812. bins=[0, 0.3, 0.7, 1.0],
  813. labels=['Low (0-0.3)', 'Medium (0.3-0.7)', 'High (0.7-1.0)'],
  814. include_lowest=True
  815. )
  817. # Calculate prediction performance metrics for each confidence level
  818. confidence_stats = []
  819. for level in ['Low (0-0.3)', 'Medium (0.3-0.7)', 'High (0.7-1.0)']:
  820. subset = predictions_df[predictions_df['confidence_level'] == level]
  821. if len(subset) > 0:
  822. accuracy = subset['prediction_correct'].mean()
  823. confidence_stats.append({
  824. 'confidence_level': level,
  825. 'n_molecules': len(subset),
  826. 'accuracy': accuracy,
  827. 'avg_probability': subset['predicted_probability'].mean()
  828. })
  830. # Sort by prediction probability (descending)
  831. predictions_df = predictions_df.sort_values('predicted_probability', ascending=False)
  833. # Save predictions to CSV
  834. filename = f'results_val_acc/{model_type}_validation_predictions.csv'
  835. predictions_df.to_csv(filename, index=False)
  837. # Save confidence level statistics
  838. if confidence_stats:
  839. confidence_df = pd.DataFrame(confidence_stats)
  840. confidence_filename = f'results_val_acc/{model_type}_confidence_statistics.csv'
  841. confidence_df.to_csv(confidence_filename, index=False)
  842. print(f"✓ {model_type} confidence statistics saved: {confidence_filename}")
  844. # Generate summary file
  845. summary_filename = f'results_val_acc/{model_type}_predictions_summary.txt'
  846. with open(summary_filename, 'w') as f:
  847. f.write(f"{'=' * 60}\n")
  848. f.write(f"{model_type.replace('_', ' ').title()} - Validation Set Predictions Summary\n")
  849. f.write(f"{'=' * 60}\n\n")
  851. f.write(f"Model Parameters:\n")
  852. f.write(f" - Network Architecture: {optimal_params['layer_sizes']}\n")
  853. f.write(f" - Dropout Rate: {optimal_params['dropout_rate']}\n")
  854. f.write(f" - Learning Rate: {optimal_params['learning_rate']}\n")
  855. f.write(f" - L2 Regularization: {optimal_params['l2_reg']}\n\n")
  857. f.write(f"Overall Performance:\n")
  858. f.write(f" - Total Molecules: {len(predictions_df)}\n")
  859. f.write(f" - Correct Predictions: {predictions_df['prediction_correct'].sum()} ({predictions_df['prediction_correct'].mean():.2%})\n")
  860. f.write(f" - Mean Predicted Probability: {predictions_df['predicted_probability'].mean():.4f}\n")
  861. f.write(f" - Std Predicted Probability: {predictions_df['predicted_probability'].std():.4f}\n\n")
  863. f.write(f"Confidence Level Distribution:\n")
  864. for stats in confidence_stats:
  865. f.write(f" - {stats['confidence_level']}: {stats['n_molecules']} molecules, Accuracy: {stats['accuracy']:.2%}, Avg Probability: {stats['avg_probability']:.4f}\n")
  867. f.write(f"\nPrediction Probability Statistics:\n")
  868. f.write(f" - Min: {predictions_df['predicted_probability'].min():.4f}\n")
  869. f.write(f" - 25th Percentile: {predictions_df['predicted_probability'].quantile(0.25):.4f}\n")
  870. f.write(f" - Median: {predictions_df['predicted_probability'].median():.4f}\n")
  871. f.write(f" - 75th Percentile: {predictions_df['predicted_probability'].quantile(0.75):.4f}\n")
  872. f.write(f" - Max: {predictions_df['predicted_probability'].max():.4f}\n\n")
  874. f.write(f"Misclassified Molecules (Top 10 by Error):\n")
  875. misclassified = predictions_df[predictions_df['prediction_correct'] == 0]
  876. if len(misclassified) > 0:
  877. top_misclassified = misclassified.nlargest(10, 'prediction_error')
  878. for idx, row in top_misclassified.iterrows():
  879. f.write(f" - SMILES: {row['smiles']}, True: {row['true_label']}, Pred: {row['predicted_label']} (Prob: {row['predicted_probability']:.4f})\n")
  880. else:
  881. f.write(f" - No misclassified molecules!\n")
  883. print(f"✓ {model_type} validation predictions saved: {filename}")
  884. print(f"✓ {model_type} predictions summary saved: {summary_filename}")
  886. except Exception as e:
  887. print(f"⚠ Warning: Failed to save {model_type} predictions: {e}")
  889. @staticmethod
  890. def _generate_final_report(best_metrics_df, optimal_params, history,
  891. best_epoch, best_val_acc, final_metrics_df=None):
  892. """Generate final report"""
  893. actual_epochs = len(history.history['loss'])
  895. report = f"""
  896. {'=' * 60}
  897. Developmental Neurotoxicity Prediction DNN Model - Training Complete Report
  898. {'=' * 60}
  900. Model Configuration:
  901. ----------
  902. Network Architecture: {optimal_params['layer_sizes']}
  903. Dropout Rate: {optimal_params['dropout_rate']}
  904. Learning Rate: {optimal_params['learning_rate']}
  905. L2 Regularization: {optimal_params['l2_reg']}
  906. Batch Size: {optimal_params['batch_size']}
  907. Maximum Epochs: {optimal_params['epochs']}
  908. Actual Epochs Trained: {actual_epochs}
  909. Model Selection: Based on {Config.MODEL_SELECTION_METRIC}
  910. Best Model Epoch: {best_epoch + 1}
  912. Performance Summary (Best Model):
  913. ----------
  914. Training Set AUC: {best_metrics_df.loc['AUC', 'Training Set']:.4f}
  915. Validation Set AUC: {best_metrics_df.loc['AUC', 'Validation Set']:.4f}
  917. Training Set Accuracy: {best_metrics_df.loc['Accuracy', 'Training Set']:.4f}
  918. Validation Set Accuracy: {best_metrics_df.loc['Accuracy', 'Validation Set']:.4f}
  919. (Best Validation Accuracy: {best_val_acc:.4f})
  921. Training Set F1-Score: {best_metrics_df.loc['F1-Score', 'Training Set']:.4f}
  922. Validation Set F1-Score: {best_metrics_df.loc['F1-Score', 'Validation Set']:.4f}
  924. Overfitting Assessment (Best Model):
  925. -----------------
  926. AUC Difference (Validation-Training): {best_metrics_df.loc['AUC', 'Difference (Validation-Training)']:.4f}
  927. F1-Score Difference (Validation-Training): {best_metrics_df.loc['F1-Score', 'Difference (Validation-Training)']:.4f}
  928. """
  930. if final_metrics_df is not None:
  931. report += f"""
  932. Performance Comparison (Final Epoch vs Best Model):
  933. -----------------
  934. Validation Accuracy:
  935. - Final Epoch: {final_metrics_df.loc['Accuracy', 'Validation Set']:.4f}
  936. - Best Model: {best_metrics_df.loc['Accuracy', 'Validation Set']:.4f}
  937. - Improvement: {best_metrics_df.loc['Accuracy', 'Validation Set'] - final_metrics_df.loc['Accuracy', 'Validation Set']:.4f}
  939. Validation AUC:
  940. - Final Epoch: {final_metrics_df.loc['AUC', 'Validation Set']:.4f}
  941. - Best Model: {best_metrics_df.loc['AUC', 'Validation Set']:.4f}
  942. - Improvement: {best_metrics_df.loc['AUC', 'Validation Set'] - final_metrics_df.loc['AUC', 'Validation Set']:.4f}
  943. """
  945. report += f"""
  947. Evaluation Results:
  948. ---------
  949. {'Excellent Validation Performance (AUC > 0.85)' if best_metrics_df.loc['AUC', 'Validation Set'] > 0.85 else 'Moderate Validation Performance'}
  950. {'No Significant Overfitting (|AUC Difference| < 0.1)' if abs(best_metrics_df.loc['AUC', 'Difference (Validation-Training)']) < 0.1 else 'Potential Overfitting'}
  952. Prediction Files Generated:
  953. -----------------
  954. 1. best_model_validation_predictions.csv - Detailed predictions for each validation molecule
  955. 2. best_model_confidence_statistics.csv - Performance by confidence level
  956. 3. best_model_predictions_summary.txt - Summary statistics
  958. Note: The model was selected based on validation accuracy. The best model (epoch {best_epoch + 1})
  959. has been saved and will be used for all subsequent analysis.
  961. All results have been saved to the 'results_val_acc' directory.
  962. {'=' * 60}
  963. """
  965. with open('results_val_acc/final_report.txt', 'w') as f:
  966. f.write(report)
  968. print("\n" + report)
  970. # ==================== Main Program ====================
  971. def check_dependencies():
  972. """Check if all required dependencies are installed"""
  973. required_packages = {
  974. 'numpy': 'np',
  975. 'pandas': 'pd',
  976. 'matplotlib': 'plt',
  977. 'seaborn': 'sns',
  978. 'sklearn': 'sklearn',
  979. 'rdkit': 'Chem',
  980. 'tensorflow': 'tf'
  981. }
  983. missing_packages = []
  984. for package, import_name in required_packages.items():
  985. try:
  986. if package == 'rdkit':
  987. __import__('rdkit.Chem')
  988. else:
  989. __import__(package)
  990. except ImportError:
  991. missing_packages.append(package)
  993. if missing_packages:
  994. print("Missing required dependencies:")
  995. for package in missing_packages:
  996. print(f" - {package}")
  997. print("\nPlease install using the following commands:")
  998. print("pip install numpy pandas matplotlib seaborn scikit-learn tensorflow")
  999. print("conda install -c conda-forge rdkit # or use conda to install rdkit")
  1000. return False
  1002. return True
  1004. def setup_matplotlib():
  1005. """Setup matplotlib style"""
  1006. try:
  1007. # Set seaborn style
  1008. sns.set_style("whitegrid")
  1009. sns.set_palette("husl")
  1010. print("✓ Using seaborn style")
  1011. except Exception as e:
  1012. print(f"⚠ Style setup failed: {e}, using default style")
  1014. def main():
  1015. """Main function: Execute complete model training pipeline"""
  1016. print("=" * 60)
  1017. print("Developmental Neurotoxicity Prediction - DNN QSAR Model")
  1018. print("=" * 60)
  1019. print("Model Selection Strategy: Using validation accuracy to select best model")
  1020. print("Training with Early Stopping: Enabled")
  1021. print("=" * 60)
  1023. # Check dependencies
  1024. if not check_dependencies():
  1025. return
  1027. # Setup matplotlib style
  1028. setup_matplotlib()
  1030. # Initialize components
  1031. config = Config()
  1032. data_processor = DataProcessor()
  1033. model_trainer = ModelTrainer(config)
  1034. model_evaluator = ModelEvaluator()
  1035. result_saver = ResultSaver()
  1037. try:
  1038. # Step 1: Load and preprocess data
  1039. print("\n[Step 1/6] Loading and preprocessing data...")
  1040. train_df, val_df = data_processor.load_data(
  1041. config.TRAIN_FILE, config.VAL_FILE
  1042. )
  1044. X_train, y_train, X_val, y_val, scaler = data_processor.prepare_features(
  1045. train_df, val_df
  1046. )
  1048. # Step 2: Display optimal parameters information
  1049. print("\n[Step 2/6] Configuring model with optimal parameters...")
  1050. optimal_params = config.OPTIMAL_PARAMS
  1051. print(f"\nOptimal Parameter Configuration:")
  1052. print(f" Network Architecture: {optimal_params['layer_sizes']}")
  1053. print(f" Dropout Rate: {optimal_params['dropout_rate']}")
  1054. print(f" Learning Rate: {optimal_params['learning_rate']}")
  1055. print(f" L2 Regularization: {optimal_params['l2_reg']}")
  1056. print(f" Batch Size: {optimal_params['batch_size']}")
  1057. print(f" Maximum Epochs: {optimal_params['epochs']}")
  1058. print(f" Early Stopping Patience: {config.EARLY_STOPPING_PATIENCE}")
  1059. print(f" Model Selection Metric: {config.MODEL_SELECTION_METRIC}")
  1061. # Step 3: Train model with validation accuracy selection and early stopping
  1062. print("\n[Step 3/6] Training model with validation accuracy selection and early stopping...")
  1063. best_model, history, final_model = model_trainer.train_final_model(
  1064. X_train, y_train, X_val, y_val
  1065. )
  1067. # Step 4: Plot training history with best epoch marked and y-axis starting from 0
  1068. print("\n[Step 4/6] Plotting training history (y-axis starts from 0)...")
  1069. model_trainer.plot_training_history(history, optimal_params,
  1070. model_trainer.best_epoch, model_trainer.best_val_accuracy)
  1072. # Step 5: Evaluate both best model and final epoch model
  1073. print("\n[Step 5/6] Evaluating model performance...")
  1075. # Evaluate best model
  1076. print("\n" + "=" * 50)
  1077. print("Evaluating Best Model (Selected by Validation Accuracy)")
  1078. print("=" * 50)
  1079. best_metrics_df, y_train_pred_best, y_val_pred_best, y_val_pred_class_best = model_evaluator.evaluate_model(
  1080. best_model, X_train, y_train, X_val, y_val, "Best Model", val_df
  1081. )
  1083. # Evaluate final epoch model
  1084. print("\n" + "=" * 50)
  1085. print("Evaluating Final Epoch Model")
  1086. print("=" * 50)
  1087. final_metrics_df, y_train_pred_final, y_val_pred_final, y_val_pred_class_final = model_evaluator.evaluate_model(
  1088. final_model, X_train, y_train, X_val, y_val, "Final Epoch Model", val_df
  1089. )
  1091. # Step 6: Plot comparison visualizations
  1092. print("\n[Step 6/6] Plotting performance comparison...")
  1093. model_evaluator.plot_comparison_visualizations(
  1094. y_train, y_train_pred_best, y_train_pred_final,
  1095. y_val, y_val_pred_best, y_val_pred_final,
  1096. y_val, y_val_pred_class_best,
  1097. best_metrics_df.loc[:, 'Training Set'].to_dict(),
  1098. best_metrics_df.loc[:, 'Validation Set'].to_dict(),
  1099. final_metrics_df.loc[:, 'Training Set'].to_dict(),
  1100. final_metrics_df.loc[:, 'Validation Set'].to_dict()
  1101. )
  1103. # Save all results
  1104. result_saver.save_results(
  1105. best_model, scaler, best_metrics_df, history,
  1106. model_trainer.best_epoch, model_trainer.best_val_accuracy,
  1107. final_model, final_metrics_df, val_df, y_val, y_val_pred_best, y_val_pred_final
  1108. )
  1110. print("\n" + "=" * 60)
  1111. print("🎉 Model training and evaluation complete!")
  1112. print("=" * 60)
  1113. print(f"\nSummary:")
  1114. print(f" - Best model saved from epoch: {model_trainer.best_epoch + 1}")
  1115. print(f" - Best validation accuracy: {model_trainer.best_val_accuracy:.4f}")
  1116. print(f" - Final epoch validation accuracy: {final_metrics_df.loc['Accuracy', 'Validation Set']:.4f}")
  1117. print(f" - Improvement by model selection: {model_trainer.best_val_accuracy - final_metrics_df.loc['Accuracy', 'Validation Set']:.4f}")
  1118. if model_trainer.early_stopped_epoch:
  1119. print(f" - Early stopping triggered at epoch: {model_trainer.early_stopped_epoch}")
  1120. print(f"\nPrediction Files Generated:")
  1121. print(f" - results_val_acc/best_model_validation_predictions.csv")
  1122. print(f" - results_val_acc/best_model_confidence_statistics.csv")
  1123. print(f" - results_val_acc/best_model_predictions_summary.txt")
  1124. print("=" * 60)
  1126. except FileNotFoundError as e:
  1127. print(f"\n❌ Error: File not found - {e}")
  1128. print("Please ensure the following files exist in the current directory:")
  1129. print(f" - {config.TRAIN_FILE}")
  1130. print(f" - {config.VAL_FILE}")
  1131. except Exception as e:
  1132. print(f"\n❌ Error: {e}")
  1133. import traceback
  1134. traceback.print_exc()
  1136. if __name__ == "__main__":
  1137. # Check required files
  1138. if not os.path.exists(Config.TRAIN_FILE):
  1139. print(f"Error: Training file {Config.TRAIN_FILE} does not exist")
  1140. print(f"Please ensure the following files exist in the current directory:")
  1141. print(f"1. {Config.TRAIN_FILE}")
  1142. print(f"2. {Config.VAL_FILE}")
  1143. elif not os.path.exists(Config.VAL_FILE):
  1144. print(f"Error: Validation file {Config.VAL_FILE} does not exist")
  1145. else:
  1146. # Run main program
  1147. main()

DNN-test_selection.py at commit 514073b, no license · at the source

Overview

Authors: Hongting Ma1, Wenhui Zhang1, Fengxi Liu1, Rong Ni1, Xue Wang1, Yanying Sun1, Xuelin Sun2, Xiao Li1,3
ORCID iDs: Xiao Li
  1. Shandong Key Laboratory of Digital Diagnosis and Treatment of Thoracic Oncology, Shandong Engineering Research Center of Precision Diagnosis and Treatment Technology for Neuro-Oncology, Department of Clinical Pharmacy, The First Affiliated Hospital of Shandong First Medical University, Shandong Provincial Qianfoshan Hospital Jinan 250014 China
  2. Department of Pharmacy, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences Beijing 100730 China
  3. State Key Laboratory of Neurology and Oncology Drug Development, The First Affiliated Hospital of Shandong First Medical University, Shandong Provincial Qianfoshan Hospital Jinan 250014 China
Institutions: Shandong Provincial QianFoShan Hospital (China); Shandong First Medical University (China); Chinese Academy of Medical Sciences & Peking Union Medical College (China); Beijing Hospital (China)
Journal: RSC advances, volume 16, issue 36, pages 37450-37464
Dates: received 19 April 2026; accepted 24 June 2026; published online 6 July 2026
Type: Research article · Language: English
License: CC BY-NC
Identifiers: DOI 10.1039/d6ra03343a · PMID 42440932 · PMCID PMC13334446 · OpenAlex W7167493370
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: methods / tools (subfield)
Methods: Statistics, Smoothing, state filtering, decompositions, Machine learning
Journal subjects: Chemistry
Topic: Anesthesia and Neurotoxicity Research (Developmental Neuroscience, Neuroscience), according to OpenAlex
Funding: National Natural Science Foundation of China (National Science Foundation of China) (81803433, 82504560, 82504561); NHC (WKZX2024CX501220); Shandong First Medical University (Unassigned); Shandong Academy of Medical Sciences (604D25004)
Citations: not cited yet (Europe PMC); 48 references in the paper

Abstract

Developmental neurotoxicity (DNT) represents a critical yet underevaluated toxicity endpoint within current chemical safety assessment frameworks, particularly in the context of drug development. In the present study, we constructed a standardized benchmark dataset comprising 2724 structurally curated compounds, including 1362 DNT-positive chemicals derived from the Developmental Neurotox Reference List (DNTREF) of the U.S. Environmental Protection Agency (EPA) and 1362 property-matched presumed negative compounds (non-neuroactive drug-like compounds) selected from the ChEMBL database. Using this benchmark dataset, we systematically evaluated seven representative quantitative structure activity relationship (QSAR) modeling paradigms, covering traditional machine learning, deep neural networks, molecular language models, and graph neural networks. Across all models, the area under the receiver operating characteristic curve (AUC) values ranged from 0.85 to 0.90 on an independent test set. Among these, a deep neural network based on MACCS fingerprints achieved the highest sensitivity in identifying DNT-positive compounds, supporting its utility in early-stage drug safety screening where minimizing false negatives is critical. Furthermore, SHAP-based interpretability analysis revealed key structural motifs associated with DNT, highlighting the roles of hydrophobic aromatic frameworks, electrophilic reactivity, and polar functional groups in modulating predicted neurodevelopmental risk. Collectively, this study provides a reproducible benchmark dataset and an interpretable QSAR framework that supports early DNT risk prioritization and offers actionable structural insights for medicinal chemistry optimization in drug discovery.

Reproduced under the paper's license (CC BY-NC), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 13 matches between paragraphs and lines of code.

lixiao1688/DNT_Benchmark_Dataset_Model

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 514073b77308152898322ed1e5380f12c0f7b426, 27 January 2026
Languages: Python (8)
Size: 16 files, 8 scripts
Software Heritage: not archived
Found in: “Data availability”
Holds: README
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: pandas (8 files), RDKit (8 files), NumPy (7 files), Matplotlib (6 files), scikit-learn (5 files), seaborn (5 files), Keras (4 files), TensorFlow (4 files), UMAP (2 files), SciPy (1 file), SHAP (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
9 files
At the source: github.com/lixiao1688/DNT_Benchmark_Dataset_Model.git

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 8 scripts, each with its path and the digest of its content;
  • 13 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data and model availability

The developmental neurotoxicity benchmark dataset constructed in this study includes standardized SMILES representations of all compounds, corresponding binary toxicity labels, and the fixed training, validation, and independent test set partitions used for model development and evaluation. These data are deposited in a public repository to ensure full reproducibility of the reported results. The dataset is expected to serve as a valuable resource for the computational toxicology and drug discovery communities, enabling further methodological development and benchmarking.

In addition, the scripts for MACCS fingerprint generation, the trained weights of the optimal DNN_MACCS model, and all scripts used for model training, inference, and SHAP-based interpretability analysis will be made publicly available.

All scripts for AD analysis and chemical space visualization using UMAP and t-SNE will also be released to support transparent assessment of model scope and result interpretation.

Reproduced under the paper's license (CC BY-NC), from the paper cited above.

Data availability

The data utilized in this study were derived from the DNTREF from U.S. EPA, and ChEMBL database. Complete data (including processed datasets, and all scripts necessary to reproduce the study) is available on GitHub at https://github.com/lixiao1688/DNT_Benchmark_Dataset_Model.git.

Supplementary information (SI): detailed data processing for benchmark dataset; detailed network architectures, hyperparameter settings, and training procedures; a summary of predictive performance for all models on validation set; unified chemical standardization process for data collection, structure preprocessing, and sample selection; training history and model comparison of the DNN_MACCS model. See DOI: https://doi.org/10.1039/d6ra03343a.

Reproduced under the paper's license (CC BY-NC), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 8 authors, 4 funders, 46 references.

Cite

This paper

Ma, H., Zhang, W., Liu, F., Ni, R., Wang, X., Sun, Y., Sun, X., & Li, X. (2026). A benchmark dataset and interpretable deep learning framework for drug-induced developmental neurotoxicity prediction. RSC advances, 16(36), 37450-37464. https://doi.org/10.1039/d6ra03343a

BibTeX

@article{ma2026benchmark,
author = {Ma, Hongting and Zhang, Wenhui and Liu, Fengxi and Ni, Rong and Wang, Xue and Sun, Yanying and Sun, Xuelin and Li, Xiao},
title = {{A benchmark dataset and interpretable deep learning framework for drug-induced developmental neurotoxicity prediction}},
journal = {RSC advances},
year = {2026},
month = jul,
volume = {16},
number = {36},
pages = {37450--37464},
publisher = {Royal Society of Chemistry},
issn = {2046-2069},
doi = {10.1039/d6ra03343a},
url = {https://doi.org/10.1039/d6ra03343a},
pmid = {42440932},
pmcid = {PMC13334446}
}

RIS

TY - JOUR
AU - Ma, Hongting
AU - Zhang, Wenhui
AU - Liu, Fengxi
AU - Ni, Rong
AU - Wang, Xue
AU - Sun, Yanying
AU - Sun, Xuelin
AU - Li, Xiao
TI - A benchmark dataset and interpretable deep learning framework for drug-induced developmental neurotoxicity prediction
T2 - RSC advances
J2 - RSC Adv
PY - 2026
DA - 2026/07/06
VL - 16
IS - 36
SP - 37450
EP - 37464
SN - 2046-2069
PB - Royal Society of Chemistry
DO - 10.1039/d6ra03343a
UR - https://doi.org/10.1039/d6ra03343a
LA - en
ER -

CSL-JSON

{
"id": "10.1039/d6ra03343a",
"type": "article-journal",
"title": "A benchmark dataset and interpretable deep learning framework for drug-induced developmental neurotoxicity prediction",
"container-title": "RSC advances",
"author": [
{
"family": "Ma",
"given": "Hongting"
},
{
"family": "Zhang",
"given": "Wenhui"
},
{
"family": "Liu",
"given": "Fengxi"
},
{
"family": "Ni",
"given": "Rong"
},
{
"family": "Wang",
"given": "Xue"
},
{
"family": "Sun",
"given": "Yanying"
},
{
"family": "Sun",
"given": "Xuelin"
},
{
"family": "Li",
"given": "Xiao"
}
],
"container-title-short": "RSC Adv",
"volume": "16",
"issue": "36",
"page": "37450-37464",
"DOI": "10.1039/d6ra03343a",
"PMID": "42440932",
"PMCID": "PMC13334446",
"ISSN": "2046-2069",
"publisher": "Royal Society of Chemistry",
"URL": "https://doi.org/10.1039/d6ra03343a",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
6
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s42003-026-10957-8 [code]
Brain defence by the extracellular matrix protein Cochlin.
Journal: Communications biology
In common: RDKit, SHAP, Keras, 8 other tools
[2] doi:10.1093/nar/gkag706 [code]
scDifformer: diffusion-based post-training for virtual cell modeling across large-scale single-cell data.
Journal: Nucleic acids research
In common: RDKit, Keras, UMAP, 7 other tools, 1 reference
[3] doi:10.1038/s41598-026-48613-0 [code]
An snRNA-seq aging clock for the fruit fly head sheds light on sex-biased aging.
Journal: Scientific reports
In common: SHAP, Keras, TensorFlow, 6 other tools, 2 references
[4] doi:10.1126/sciadv.aed3650 [code]
Truthful visualizations for mass spectrometry imaging enable high-spatial-resolution interactive &lt;i&gt;m/z&lt;/i&gt; mapping and exploration.
Journal: Science advances
In common: SHAP, Keras, UMAP, 6 other tools, methods / tools
[5] doi:10.1177/13872877261453512 [code]
Fluorescence spectroscopy and machine learning methods for detection of Alzheimer's disease from circulating white blood cells.
Journal: Journal of Alzheimer's disease : JAD
In common: Keras, UMAP, TensorFlow, 6 other tools, methods / tools
[6] doi:10.1038/s41592-026-03057-2 [code]
CREsted: modeling genomic and synthetic cell-type-specific enhancers across tissues and species.
Journal: Nature methods
In common: Keras, UMAP, TensorFlow, 6 other tools, methods / tools
[7] doi:10.3390/s26175327 [code]
Subject Identity Confounds qEEG Emotion Recognition on DEAP and DREAMER.
Journal: Sensors (Basel, Switzerland)
In common: SHAP, Keras, TensorFlow, 6 other tools
[8] doi:10.1371/journal.pcbi.1014615 [code]
Toward reliable machine learning models for neural circuit inference: A diagnostic study of CNNs on spike trains.
Journal: PLoS computational biology
In common: SHAP, Keras, TensorFlow, 6 other tools
[9] doi:10.1038/s41467-026-75700-7 [code]
Gene regulatory innovations from transposable elements in primate cerebellum development.
Journal: Nature communications
In common: SHAP, Keras, TensorFlow, 6 other tools
[10] doi:10.1038/s41598-026-47047-y [code]
Imaging-genetics-based dementia risk prediction using deep survival neural networks in the Rotterdam Study.
Journal: Scientific reports
In common: SHAP, Keras, TensorFlow, 6 other tools

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.