OSCR

A three-component dynamical index of consciousness-related neural organisation.

Code ↔ Paper

25 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 25 matches
  1. [1] § Generative model and data generation › Simulation of EEG-like signals ↔ Ugail-Howard-Conciousness-Index_State_Simulator.ipynb, lines 321–410 · score 0.80 · 13–30 Hz, 8–13 Hz, 30–80 Hz, mixing, 1–4 Hz, 4–8 Hz
  2. [2] § Generative model and data generation › Simulation of EEG-like signals ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 356–445 · score 0.80 · 13–30 Hz, 8–13 Hz, 30–80 Hz, mixing, 1–4 Hz, 4–8 Hz
  3. [3] § Mathematical framework › Organised cross-frequency complexity ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1239–1304 · score 0.80 · shift surrogates, Tort MI, theta phase, gamma amplitude, Sleep EDF, bins
  4. [4] § Generative model and data generation › Simulated states ↔ Ugail-Howard-Conciousness-Index_State_Simulator.ipynb, lines 416–492 · score 0.78 · reduced alpha, Dreaming states, sleep states, Minimally conscious states, REM sleep, delta
  5. [5] § Generative model and data generation › Simulated states ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 451–527 · score 0.78 · reduced alpha, Dreaming states, sleep states, Minimally conscious states, REM sleep, delta
  6. [6] § Mathematical framework › Organised cross-frequency complexity ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 192–260 · score 0.75 · phase bins, uniform reference, amplitude guard, Tort MI, Phase amplitude, divergence
  7. [7] § Mathematical framework › Scale-free temporal organisation ( ) ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 263–361 · score 0.73 · DFA scaling exponent, range normalised triangular, scale free temporal, fallback, tuning, Kuramoto
  8. [8] § Results › Validation of using real EEG ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1340–1385 · score 0.70 · Friedman omnibus, pairwise Wilcoxon, rank biserial, Bonferroni correction, Validation
  9. [9] § Mathematical framework › Scale-free temporal organisation ( ) ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 124–189 · score 0.69 · DFA scaling exponent, cumulative sum, windows, stationary, detrended, persistent
  10. [10] § Generative model and data generation › Simulated states ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 402–461 · score 0.66 · cross channel phase, slow drift, theta gamma, coupling, noise, PAC
  11. [11] § Results › Monte carlo distributions across states ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 625–678 · score 0.65 · Kruskal Wallis omnibus, adjacent state, task engaged, minimally conscious, Cliff, Dreaming
  12. [12] § Mathematical framework › Organised cross-frequency complexity ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 192–260 · score 0.64 · binary sequence, Lempel Ziv complexity, amplitude envelope
  13. [13] § Results › Validation of using real EEG ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1786–1871 · score 0.64 · Empirical benchmarking heatmap, spectral slope, alpha power, Sleep EDF, LZC, row
  14. [14] § Generative model and data generation › Simulation of EEG-like signals ↔ Ugail-Howard-Conciousness-Index_State_Simulator.ipynb, lines 321–410 · score 0.64 · 0.5–4 Hz, 30–80 Hz, ictal, Hilbert, filtering, scored
  15. [15] § Results › Validation of using real EEG ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1340–1385 · score 0.63 · Friedman omnibus, Pairwise Wilcoxon, rank biserial, Bonferroni, Validation
  16. [16] § Generative model and data generation › Simulation of EEG-like signals ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 356–445 · score 0.63 · 0.5–4 Hz, 30–80 Hz, ictal, Hilbert, filtering, scored
  17. [17] § Results › Ablation study ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 263–361 · score 0.61 · DFA scaling exponents, organisation score, scale free temporal, phase amplitude, cross frequency, Kuramoto
  18. [18] § Results › Validation of using real EEG ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1786–1871 · score 0.60 · single metric, spectral slope, alpha power, LZC, benchmark, bootstrap
  19. [19] § Results › Validation of using real EEG ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1511–1635 · score 0.58 · Stratified epoch sampling, capped, night, Sleep EDF, real EEG, validation
  20. [20] § Mathematical framework › Scale-free temporal organisation ( ) ↔ Ugail-Howard-Consciousness-Index.ipynb, lines 124–189 · score 0.58 · detrended fluctuation, long range correlations, sum, temporal, DFA
  21. [21] § Results › Sensitivity analysis ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1035–1174 · score 0.57 · Hyperparameter sensitivity, adjacent state pairs, pair AUC, sweeps, worst
  22. [22] § Results › Component discriminability in synthetic data ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1035–1174 · score 0.57 · state discrimination, adjacent state pair, Task engaged, Minimally Conscious, hierarchy, AUC
  23. [23] § Results › Monte carlo convergence ↔ Ugail-Howard-Consciousness-Index_Validation_updated.ipynb, lines 1177–1236 · score 0.56 · Monte Carlo convergence, median deviation, maximum deviation
  24. [24] § Mathematical framework › Scale-free temporal organisation ( ) ↔ Ugail-Howard-Conciousness-Index_State_Simulator.ipynb, lines 117–199 · score 0.56 · DFA scaling exponent, cumulative sum, stationary, persistent, signal
  25. [25] § Generative model and data generation › Simulated states ↔ Ugail-Howard-Conciousness-Index_State_Simulator.ipynb, lines 416–492 · score 0.50 · Task engaged states, prominent, moderate, beta, coupling, simulate

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Jupyter notebook · 1,872 lines · 78 KB · no license · 13 matches

  1. # %% [markdown]
  2. # # Ugail–Howard Consciousness Dynamics — Experiments & Empirical Validation
  3. # ## Monte Carlo, ablation, sensitivity, Sleep-EDF validation
  4. #
  5. # **Authors:** Hassan Ugail & Newton Howard
  6. #
  7. # Cite: H. Ugail and N. Howard, “Quantifying the dynamics of consciousness using
  8. # hierarchical integration, organised complexity and metastability,” arXiv preprint
  9. # arXiv:2512.10972, Dec. 2025.
  10. #
  11. #
  12. #
  13. # ---
  14. #
  15. # This notebook is the **full experimental pipeline** for the Ugail–Howard
  16. # framework. It is self-contained (no external library beyond pip packages)
  17. # and reproduces every result reported in the manuscript's main text and
  18. # appendix:
  19. #
  20. # ### Contents
  21. # 1. **Section A — Simulated Monte Carlo (30 runs/state, 9 states).**
  22. # - Per-state distributions of $\Psi$.
  23. # - Kruskal–Wallis omnibus + pairwise Mann–Whitney adjacent-state tests.
  24. # - ROC / AUC for conscious vs non-conscious separation (with bootstrap CIs).
  25. # - $\Psi$ vs single-component baselines (H_eff, D, M alone) and simple
  26. # baselines (LZC, spectral slope) — component-necessity check.
  27. # - Pairwise adjacent-state AUC matrix (Fig 6 in the manuscript).
  28. #
  29. # 2. **Section B — Ablation study.**
  30. # - Baseline (wake), No PAC, No Metastability, No Fractal.
  31. # - Each condition ablates exactly one component of $\Psi$.
  32. #
  33. # 3. **Section C — Hyperparameter sensitivity.**
  34. # - Three-panel sweep: $H_{\mathrm{opt}}$, $w_H$, $\lambda$.
  35. # - Metric: mean and minimum adjacent-pair AUC (orientation-corrected).
  36. #
  37. # 4. **Section D — Within-state Monte Carlo convergence.**
  38. # - How quickly each state's running mean converges to its long-run value.
  39. #
  40. # 5. **Section E — Publication figures (1–6) for the synthetic pipeline.**
  41. #
  42. # 6. **Section F — Sleep-EDF empirical validation (real EEG).**
  43. # - Downloads / reads the Sleep-EDF Cassette dataset via MNE-Python.
  44. # - Wake / N2 / REM extraction, per-subject H_opt calibration.
  45. # - Subject-level Friedman omnibus + Wilcoxon post-hoc (Bonferroni-corrected).
  46. # - Benchmarking $\Psi$ against LZC, spectral slope, and alpha power.
  47. # - Publication Figures 7 (subject-level $\Psi$) and 8 (AUC heatmap).
  48. #
  49. #
  50. #
  51. # ### Runtime notes
  52. # - Sections A–E are pure CPU and take ~3–8 minutes on a modest laptop.
  53. # - Section F downloads ~100 MB the first time it is run (Sleep-EDF Cassette
  54. # EDF files via MNE). If `DATA_DIR` below already contains the files, the
  55. # download is skipped. Set `N_SUBJ` lower for a quick sanity-check run.
  56. #
  57. # Sleep-EDF data sample used is available at:
  58. # https://drive.google.com/drive/folders/1zRKqwlYfLGn6OTwL4cj6ceRMvfvWjiUY?usp=sharing
  59. # %%
  60. # Install dependencies — required on a fresh Colab / local kernel.
  61. # Safe to re-run; pip will no-op packages that are already installed.
  62. # MNE is required to read the Sleep-EDF EDF files (Section E).
  63. !pip install -q numpy scipy matplotlib pandas networkx scikit-learn mne
  64. print("Dependencies installed.")
  65. # %%
  66. # ── Imports & global configuration ─────────────────────────────────────────
  67. import os, warnings, itertools
  68. import numpy as np
  69. import pandas as pd
  70. import matplotlib.pyplot as plt
  71. import matplotlib.patches as mpatches
  72. import networkx as nx
  73. from scipy.signal import butter, filtfilt, hilbert, welch
  74. from scipy.stats import kruskal, mannwhitneyu, wilcoxon, friedmanchisquare, rankdata
  75. from sklearn.metrics import roc_auc_score, roc_curve
  76. warnings.filterwarnings("ignore", category=RuntimeWarning)
  77. np.random.seed(42)
  78. CFG = {
  79. "fs": 250,
  80. "H_opt": 0.35,
  81. "sigma_H": 0.12,
  82. "w_H": 0.40,
  83. "w_D": 0.35,
  84. "w_M": 0.25,
  85. "lam": 1.0,
  86. "GAMMA_HIGH_SYN": 80,
  87. "GAMMA_HIGH_REAL": 45,
  88. }
  89. # Publication rcParams
  90. plt.rcParams.update({
  91. "font.family": "DejaVu Sans",
  92. "font.size": 10,
  93. "axes.titlesize": 11,
  94. "axes.titleweight": "bold",
  95. "axes.labelsize": 10,
  96. "xtick.labelsize": 9,
  97. "ytick.labelsize": 9,
  98. "axes.spines.top": False,
  99. "axes.spines.right": False,
  100. "axes.grid": True,
  101. "grid.alpha": 0.18,
  102. "grid.linewidth": 0.5,
  103. "legend.fontsize": 9,
  104. "legend.frameon": False,
  105. "figure.dpi": 150,
  106. })
  107. # Global composite weights (used by every downstream cell)
  108. w_H, w_D, w_M, lam = CFG["w_H"], CFG["w_D"], CFG["w_M"], CFG["lam"]
  109. H_opt = CFG["H_opt"]
  110. print(f"Imports loaded. NumPy {np.__version__}.")
  111. print(f"Weights: w_H={w_H}, w_D={w_D}, w_M={w_M}, lam={lam}")
  112. print(f"H_opt = {H_opt} (synthetic-domain prior)")
  113. # %%
  114. # ── Basic signal utilities ─────────────────────────────────────────────────
  115. def bandpass_filter(data, fs, low, high, order=4):
  116. """Zero-phase Butterworth bandpass filter along the last axis."""
  117. nyq = fs / 2.0
  118. b, a = butter(order, [low / nyq, high / nyq], btype="band")
  119. return filtfilt(b, a, data, axis=-1)
  120. def generate_pink_noise(n_channels, n_samples):
  121. """Approximate 1/f pink noise by cumulative-summed white noise."""
  122. white = np.random.randn(n_channels, n_samples)
  123. pink = np.cumsum(white, axis=-1)
  124. pink -= pink.mean(axis=-1, keepdims=True)
  125. pink /= pink.std(axis=-1, keepdims=True) + 1e-9
  126. return pink
  127. def format_label(key: str) -> str:
  128. """Convert internal identifiers like 'no_pac' -> 'No PAC'."""
  129. words = key.replace("_", " ").split()
  130. return " ".join(w if w.isupper() else w.capitalize() for w in words)
  131. print("Signal utilities loaded.")
  132. # %%
  133. # ── Core metrics: DFA, Lempel-Ziv, Tort MI ─────────────────────────────────
  134. # Returns DFA scaling exponent alpha (not Hurst). For cumulative-sum signals:
  135. # alpha in (0.5, 1) = persistent, alpha > 1 = non-stationary fBm-like.
  136. def dfa_hurst(x, min_win=16, max_win=None, n_win=10):
  137. """DFA scaling exponent alpha of a 1-D signal."""
  138. x = np.asarray(x)
  139. N = x.size
  140. if max_win is None:
  141. max_win = N // 4
  142. y = np.cumsum(x - x.mean())
  143. s_vals = np.unique(
  144. np.logspace(np.log10(min_win), np.log10(max_win), n_win, dtype=int)
  145. )
  146. F = []
  147. for s in s_vals:
  148. if s < 4:
  149. continue
  150. n_seg = N // s
  151. if n_seg < 2:
  152. continue
  153. rms = []
  154. for i in range(n_seg):
  155. seg = y[i * s : (i + 1) * s]
  156. t = np.arange(s)
  157. p = np.polyfit(t, seg, 1)
  158. rms.append(np.sqrt(np.mean((seg - np.polyval(p, t)) ** 2)))
  159. if rms:
  160. F.append(np.mean(rms))
  161. F = np.array(F)
  162. if len(F) < 2:
  163. return 0.5
  164. s_use = s_vals[: len(F)]
  165. return float(np.polyfit(np.log(s_use), np.log(F), 1)[0])
  166. def lempel_ziv_complexity(binary_seq):
  167. """Normalised LZ76 complexity of a binary sequence."""
  168. s = "".join(str(int(b)) for b in binary_seq)
  169. i, c, l = 0, 1, 1
  170. n = len(s)
  171. while True:
  172. if i + l > n:
  173. c += 1
  174. break
  175. sub = s[i : i + l]
  176. if sub in s[:i]:
  177. l += 1
  178. else:
  179. i += l
  180. c += 1
  181. l = 1
  182. if i + l > n:
  183. break
  184. return c / n
  185. def mutual_information_phase_amp(phase, amp, n_bins=36):
  186. """Tort (2010) Modulation Index with amplitude guard (Aru et al. 2015).
  187. Guards against numerical artefacts from near-zero amplitude signals:
  188. when mean |amp| < 1e-3 on a z-scored signal, the KL divergence is
  189. dominated by rounding noise rather than genuine PAC. We return 0.
  190. """
  191. phase = np.asarray(phase)
  192. amp = np.asarray(amp)
  193. if float(amp.mean()) < 1e-3:
  194. return 0.0
  195. bins = np.linspace(-np.pi, np.pi, n_bins + 1)
  196. amp_profile = np.zeros(n_bins)
  197. for i in range(n_bins):
  198. mask = (phase >= bins[i]) & (phase < bins[i + 1])
  199. amp_profile[i] = amp[mask].mean() if mask.sum() > 0 else 0.0
  200. total = amp_profile.sum()
  201. if total < 1e-12:
  202. return 0.0
  203. p = amp_profile / total
  204. q = np.ones(n_bins) / n_bins
  205. return float(np.sum(p * np.log(p / q + 1e-12)))
  206. print("Core metrics loaded: DFA alpha, Lempel-Ziv, Tort MI (with amplitude guard).")
  207. # %%
  208. # ── compute_metrics pipeline (H_eff, D, M, Psi) ────────────────────────────
  209. def compute_metrics(X, fs=250, Hopt=0.35, sigma_H=0.12, lam=1.0, **kwargs):
  210. """Compute H_raw, H_eff, D, M, Psi on a multichannel signal.
  211. H_eff formula: range-normalised triangular penalty.
  212. H_eff = max(0, 1 − |alpha − H_opt| / alpha_range)
  213. where alpha_range defaults to 3*sigma_H if not passed as a kwarg.
  214. """
  215. C, T = X.shape
  216. H_vals = np.array([dfa_hurst(X[ch]) for ch in range(C)])
  217. H_raw = float(H_vals.mean())
  218. _alpha_range = kwargs.get("alpha_range", 3.0 * sigma_H)
  219. Heff = float(max(0.0, 1.0 - abs(H_raw - Hopt) / _alpha_range))
  220. theta = bandpass_filter(X, fs, 4, 8)
  221. gamma_high = kwargs.get("gamma_high", 80)
  222. gamma = bandpass_filter(X, fs, 30, gamma_high)
  223. theta_phase = np.angle(hilbert(theta, axis=-1))
  224. gamma_amp = np.abs(hilbert(gamma, axis=-1))
  225. I_phiA = float(np.mean([
  226. mutual_information_phase_amp(theta_phase[ch], gamma_amp[ch])
  227. for ch in range(C)]))
  228. broadband = bandpass_filter(X, fs, 1, 40)
  229. amp_bb = np.abs(hilbert(broadband, axis=-1))
  230. LZ_vals = []
  231. for ch in range(C):
  232. env = amp_bb[ch]
  233. binary = (env > np.median(env)).astype(int)
  234. LZ_vals.append(lempel_ziv_complexity(binary))
  235. LZ = float(np.mean(LZ_vals))
  236. D = I_phiA * (1.0 + lam * LZ)
  237. alpha_band = bandpass_filter(X, fs, 8, 13)
  238. alpha_phase = np.angle(hilbert(alpha_band, axis=-1))
  239. R_t = np.abs(np.mean(np.exp(1j * alpha_phase), axis=0))
  240. M = float(np.std(R_t))
  241. Psi = 0.40 * Heff + 0.35 * D + 0.25 * M
  242. return {"H_raw": H_raw, "Heff": Heff, "D": D, "M": M, "Psi": Psi}
  243. def compute_all_metrics(X, fs, gamma_high=80):
  244. """Raw-component wrapper for ensemble analyses."""
  245. m = compute_metrics(X, fs=fs, Hopt=0.35, sigma_H=0.12,
  246. lam=1.0, gamma_high=gamma_high)
  247. return {"H": m["H_raw"], "D": m["D"], "M": m["M"]}
  248. print("compute_metrics and compute_all_metrics ready.")
  249. # %%
  250. # ── EEG-like state simulator: 9 canonical states + REM sleep ───────────────
  251. # 'Conscious' renamed to 'Task-engaged' for consistency with the manuscript.
  252. def generate_consciousness_state_signal(state="wake", n_channels=20,
  253. duration=12, fs=250):
  254. """EEG-like multichannel signal for 9 conscious states + REM sleep.
  255. Supported states:
  256. wake, task_engaged, sleep, anesthesia, psychedelic,
  257. seizure, minimally_conscious, non_conscious,
  258. dreaming (alias of psychedelic), rem_sleep
  259. Returns
  260. -------
  261. signal : array, shape (n_channels, duration*fs), z-scored per channel.
  262. Notes
  263. -----
  264. Seizure and non_conscious signals are constructed in the frequency
  265. domain and tiled across channels — producing D ~ 0 and M ~ 0 by
  266. construction, matching the clinical signatures of GTC seizures and
  267. chronic UWS respectively.
  268. """
  269. n_samples = int(duration * fs)
  270. params = {
  271. "wake": {
  272. "bands": {"delta":0.3,"theta":0.4,"alpha":0.8,"beta":0.6,"gamma":0.5},
  273. "coupling":0.6,"local_sync":0.3,"global_sync":0.5,"pac_strength":0.4,
  274. },
  275. # Task-engaged — higher PAC, lower global sync than resting wake
  276. "task_engaged": {
  277. "bands": {"delta":0.2,"theta":0.5,"alpha":0.7,"beta":0.7,"gamma":0.6},
  278. "coupling":0.55,"local_sync":0.50,"global_sync":0.35,"pac_strength":0.55,
  279. },
  280. "sleep": {
  281. "bands": {"delta":0.95,"theta":0.7,"alpha":0.2,"beta":0.1,"gamma":0.05},
  282. "coupling":0.4,"local_sync":0.7,"global_sync":0.3,"pac_strength":0.05,
  283. },
  284. "anesthesia": {
  285. "bands": {"delta":0.9,"theta":0.5,"alpha":0.5,"beta":0.05,"gamma":0.02},
  286. "coupling":0.10,"local_sync":0.82,"global_sync":0.08,"pac_strength":0.005,
  287. },
  288. "psychedelic": {
  289. "bands": {"delta":0.2,"theta":0.4,"alpha":0.6,"beta":0.8,"gamma":1.0},
  290. "coupling":0.7,"local_sync":0.2,"global_sync":0.4,"pac_strength":0.7,
  291. },
  292. # Seizure — generalised tonic-clonic; delta-dominated 1/f^2 tiled across ch
  293. "seizure": {
  294. "bands": {"delta":0.0,"theta":0.0,"alpha":0.0,"beta":0.0,"gamma":0.0},
  295. "coupling":0.0,"local_sync":0.0,"global_sync":1.0,"pac_strength":0.0,
  296. },
  297. # Minimally Conscious — all three components favour MCS > Anaesthesia
  298. "minimally_conscious": {
  299. "bands": {"delta":0.55,"theta":0.35,"alpha":0.25,"beta":0.05,"gamma":0.0},
  300. "coupling":0.32,"local_sync":0.45,"global_sync":0.12,"pac_strength":0.060,
  301. },
  302. # Non-conscious (UWS) — super-Brownian 1/f^2 over 0.3–5.5 Hz
  303. "non_conscious": {
  304. "bands": {"delta":0.0,"theta":0.0,"alpha":0.0,"beta":0.0,"gamma":0.0},
  305. "coupling":0.0,"local_sync":0.0,"global_sync":0.0,"pac_strength":0.0,
  306. },
  307. }
  308. # REM sleep handled separately (see _sim_rem in the next cell)
  309. if state == "rem_sleep":
  310. raise ValueError("Use _sim_rem() for REM sleep — not handled here.")
  311. if state == "dreaming":
  312. state = "psychedelic"
  313. if state not in params:
  314. raise ValueError(f"Unknown state '{state}', choose from {list(params.keys())}")
  315. p = params[state]
  316. band_edges = {"delta":(1,4),"theta":(4,8),"alpha":(8,13),
  317. "beta":(13,30),"gamma":(30,80)}
  318. base = generate_pink_noise(n_channels, n_samples)
  319. signal = np.zeros_like(base)
  320. # 1) Band-limited global + local mixture ─────────────────────────────
  321. for band_name, weight in p["bands"].items():
  322. if weight <= 0:
  323. continue
  324. low, high = band_edges[band_name]
  325. global_src = bandpass_filter(
  326. np.random.randn(1, n_samples), fs, low, high)[0]
  327. global_src -= global_src.mean()
  328. global_src /= global_src.std() + 1e-9
  329. for ch in range(n_channels):
  330. local = bandpass_filter(base[ch:ch+1, :], fs, low, high)[0]
  331. local -= local.mean()
  332. local /= local.std() + 1e-9
  333. mix = (1.0 - p["global_sync"]) * local + p["global_sync"] * global_src
  334. signal[ch] += weight * mix
  335. # 2) Theta–gamma PAC injection ───────────────────────────────────────
  336. if p["pac_strength"] > 0:
  337. theta_low, theta_high = band_edges["theta"]
  338. gamma_low, gamma_high = band_edges["gamma"]
  339. theta = bandpass_filter(signal, fs, theta_low, theta_high)
  340. theta_phase = np.angle(hilbert(theta, axis=-1))
  341. gamma_noise = np.random.randn(n_channels, n_samples) * 0.5
  342. gamma_filtered = bandpass_filter(gamma_noise, fs, gamma_low, gamma_high)
  343. pac_mod = (1.0 + p["pac_strength"] * np.sin(theta_phase)) / 2.0
  344. signal += pac_mod * gamma_filtered * 3.0
  345. # 3a) Seizure — generative 1/f^2 in 0.5–4 Hz, tiled (D=0, M=0) ──────
  346. if state == "seizure":
  347. _n = n_samples
  348. _f = np.fft.rfftfreq(_n, 1.0 / fs)
  349. _f[0] = 1e-9
  350. _brown = np.where((_f >= 0.5) & (_f <= 4.0), 1.0 / _f ** 2, 0.0)
  351. _white = np.where(_f >= 4.0, 0.03, 0.0)
  352. _spec = (_brown + _white) * (
  353. np.random.randn(len(_f)) + 1j * np.random.randn(len(_f)))
  354. _ictal = np.fft.irfft(_spec, n=_n)
  355. _ictal -= _ictal.mean(); _ictal /= _ictal.std() + 1e-9
  356. signal = np.tile(_ictal, (n_channels, 1))
  357. # 3b) Non-conscious (UWS) — super-Brownian 1/f^2 in 0.3–5.5 Hz ──────
  358. if state == "non_conscious":
  359. _n = n_samples
  360. _f = np.fft.rfftfreq(_n, 1.0 / fs)
  361. _f[0] = 1e-9
  362. _brown = np.where((_f >= 0.3) & (_f <= 5.5), 1.0 / _f ** 2, 0.0)
  363. _white = np.where(_f >= 5.5, 0.02, 0.0)
  364. _spec = (_brown + _white) * (
  365. np.random.randn(len(_f)) + 1j * np.random.randn(len(_f)))
  366. _uws = np.fft.irfft(_spec, n=_n)
  367. _uws -= _uws.mean(); _uws /= _uws.std() + 1e-9
  368. signal = np.tile(_uws, (n_channels, 1))
  369. # 4) Network-based phase coupling (skipped for tiled signals) ───────
  370. if n_channels > 1 and p["coupling"] > 0 and state not in ("seizure", "non_conscious"):
  371. if p["local_sync"] > 0.5:
  372. G = nx.watts_strogatz_graph(n_channels, k=min(4, n_channels-1), p=0.05)
  373. else:
  374. G = nx.watts_strogatz_graph(n_channels, k=min(4, n_channels-1), p=0.3)
  375. A = nx.to_numpy_array(G)
  376. A = A / (A.sum(axis=1, keepdims=True) + 1e-9)
  377. analytic = hilbert(signal, axis=-1)
  378. amp = np.abs(analytic)
  379. phase = np.angle(analytic)
  380. for t_idx in range(1, n_samples):
  381. phase_diff = (phase[:, t_idx-1][:, None] - phase[:, t_idx-1][None, :])
  382. coupling_term = (A * np.sin(phase_diff)).sum(axis=1)
  383. phase[:, t_idx] += p["coupling"] * 0.02 * coupling_term
  384. signal = amp * np.cos(phase)
  385. # 5) Per-channel z-score ─────────────────────────────────────────────
  386. signal -= signal.mean(axis=-1, keepdims=True)
  387. signal /= signal.std(axis=-1, keepdims=True) + 1e-9
  388. return signal
  389. print("generate_consciousness_state_signal() ready for 8 of 9 states (REM next).")
  390. # %%
  391. # ── REM sleep simulator + convenience wrappers ─────────────────────────────
  392. def _sim_rem(n_channels=16, T=3000, fs=250):
  393. """Direct REM-sleep simulator, calibrated for the W > REM > N2 ordering.
  394. alpha_dfa ~ 0.90 (between Wake ~0.69 and NREM ~0.02 in the
  395. range-normalised triangular scoring). Biologically: theta-prominent,
  396. reduced alpha, moderate delta, lower PAC than dreaming, moderate
  397. Kuramoto coupling.
  398. """
  399. bands = {"delta":0.55,"theta":0.60,"alpha":0.40,"beta":0.35,"gamma":0.28}
  400. band_freqs = {"delta":(1,4),"theta":(4,8),"alpha":(8,13),
  401. "beta":(13,30),"gamma":(30,80)}
  402. X = generate_pink_noise(n_channels, T) * 0.2
  403. t = np.arange(T) / fs
  404. for band, (lo, hi) in band_freqs.items():
  405. fc = (lo + hi) / 2
  406. w = bands[band]
  407. gosc = np.sin(2*np.pi*fc*t + np.random.uniform(0, 2*np.pi))
  408. for ch in range(n_channels):
  409. X[ch] += w * (0.6*gosc
  410. + 0.4*np.sin(2*np.pi*fc*t + np.random.uniform(0, 2*np.pi)))
  411. # Moderate PAC (less than dreaming)
  412. pac = 0.20
  413. theta = bandpass_filter(X, fs, 4, 8)
  414. th_ph = np.angle(hilbert(theta, axis=-1))
  415. for ch in range(n_channels):
  416. gn = np.random.randn(T) * 0.3
  417. gf = bandpass_filter(gn[np.newaxis, :], fs, 30, 80)[0]
  418. X[ch] += (1 + pac*np.sin(th_ph[ch])) / 2 * gf * 2.0
  419. # Moderate slow drift
  420. X += 0.30 * np.sin(2*np.pi*0.5*t)[np.newaxis, :]
  421. X -= X.mean(axis=-1, keepdims=True)
  422. X /= (X.std(axis=-1, keepdims=True) + 1e-9)
  423. return X
  424. # ── Convenience wrappers (uniform signature) ──────────────────────────────
  425. def simulate_state_fast(n_channels=16, T=3000, fs=250, state="wake"):
  426. if state == "rem_sleep":
  427. return _sim_rem(n_channels=n_channels, T=T, fs=fs)
  428. return generate_consciousness_state_signal(
  429. state=state, n_channels=n_channels, duration=T/fs, fs=fs)
  430. def simulate_task_engaged_state(n_channels=16, T=3000, fs=250):
  431. return simulate_state_fast(n_channels, T, fs, state="task_engaged")
  432. def simulate_non_conscious_state(n_channels=16, T=3000, fs=250):
  433. return simulate_state_fast(n_channels, T, fs, state="non_conscious")
  434. def simulate_minimally_conscious_state(n_channels=16, T=3000, fs=250):
  435. return simulate_state_fast(n_channels, T, fs, state="minimally_conscious")
  436. def simulate_dreaming_state(n_channels=16, T=3000, fs=250):
  437. # Dreaming is modelled with the psychedelic parameter set
  438. return simulate_state_fast(n_channels, T, fs, state="psychedelic")
  439. def simulate_rem_sleep_state(n_channels=16, T=3000, fs=250):
  440. return _sim_rem(n_channels=n_channels, T=T, fs=fs)
  441. # Display-label map (manuscript conventions)
  442. STATE_DISPLAY = {
  443. "wake": "Wake",
  444. "task_engaged": "Task-engaged",
  445. "dreaming": "Dreaming",
  446. "sleep": "Sleep (NREM-like)",
  447. "minimally_conscious": "Minimally Conscious",
  448. "anesthesia": "Anaesthesia",
  449. "non_conscious": "Non-conscious",
  450. "seizure": "Seizure",
  451. "psychedelic": "Psychedelic",
  452. "rem_sleep": "REM sleep",
  453. }
  454. print("Simulator wrappers ready:", list(STATE_DISPLAY.keys()))
  455. # %%
  456. # ── Population helpers (normalisation) ─────────────────────────────────────
  457. # These mirror the core notebook so the simulator output can be scored
  458. # directly in this notebook without needing the experiments pipeline.
  459. def heff_from_population(H_values, H_opt=0.35):
  460. """Range-normalised triangular H_eff across a population of alpha values."""
  461. H_values = np.asarray(H_values, dtype=float)
  462. alpha_range = float(H_values.max() - H_values.min())
  463. if alpha_range < 1e-9:
  464. return np.zeros_like(H_values)
  465. return np.maximum(0.0, 1.0 - np.abs(H_values - H_opt) / alpha_range)
  466. def minmax_normalise(values):
  467. """Min-max scale to [0, 1]."""
  468. v = np.asarray(values, dtype=float)
  469. v_min, v_max = v.min(), v.max()
  470. return (v - v_min) / (v_max - v_min + 1e-12) if v_max > v_min else np.zeros_like(v)
  471. def psi_from_components(H_eff, D, M, w_H=0.40, w_D=0.35, w_M=0.25):
  472. """Composite Psi from already-normalised components."""
  473. return w_H * np.asarray(H_eff) + w_D * np.asarray(D) + w_M * np.asarray(M)
  474. print("Population-level helpers ready.")
  475. # %% [markdown]
  476. # ## Section A — Simulated Monte Carlo (30 runs × 9 states)
  477. #
  478. # The next cell runs 30 Monte Carlo realisations for each of the nine
  479. # canonical states and stores all three components + composite Ψ in a single
  480. # `mc_df` DataFrame. The `NORM_REF` dict — min/max of each component across
  481. # the 270-run ensemble — is saved for use in the ablation and empirical
  482. # cells so that normalisation is **consistent** across all downstream
  483. # analyses (no within-subset re-fitting).
  484. #
  485. # Runtime: ~1–2 minutes on a modern laptop.
  486. # %%
  487. # ── Monte Carlo: 30 runs per state × 9 states ──────────────────────────────
  488. N_RUNS_PER_STATE = 30 # set to 30 for paper-accurate; 10 for quick runs
  489. print("="*60)
  490. print(f"Monte Carlo: {N_RUNS_PER_STATE} runs per state x 9 states")
  491. print("="*60)
  492. mc_states = ["wake", "task_engaged", "psychedelic", "dreaming",
  493. "sleep", "minimally_conscious", "anesthesia",
  494. "non_conscious", "seizure"]
  495. mc_rows = []
  496. for st in mc_states:
  497. print(f" State '{st}' ", end="", flush=True)
  498. for _ in range(N_RUNS_PER_STATE):
  499. X = simulate_state_fast(n_channels=16, T=3000, fs=CFG["fs"], state=st)
  500. m = compute_all_metrics(X, CFG["fs"], gamma_high=CFG["GAMMA_HIGH_SYN"])
  501. m["state_raw"] = st
  502. mc_rows.append(m)
  503. print("done")
  504. mc_df = pd.DataFrame(mc_rows)
  505. mc_df["state"] = mc_df["state_raw"].map(STATE_DISPLAY)
  506. # Range-normalised triangular H_eff across the full MC ensemble
  507. _alpha_range_mc = float(mc_df["H"].max() - mc_df["H"].min())
  508. print(f"\n alpha_range (MC ensemble) = {_alpha_range_mc:.3f}")
  509. mc_df["H_eff"] = mc_df["H"].apply(
  510. lambda a: max(0.0, 1.0 - abs(a - H_opt) / _alpha_range_mc))
  511. # Min-max normalise H_eff, D, M to [0, 1] on the ensemble
  512. for src, dst in [("H_eff", "H_eff_n"), ("D", "D_n"), ("M", "M_n")]:
  513. v = mc_df[src].values
  514. v_min, v_max = float(v.min()), float(v.max())
  515. mc_df[dst] = (v - v_min) / (v_max - v_min) if v_max > v_min else 0.0
  516. mc_df["PSI_mc"] = w_H*mc_df["H_eff_n"] + w_D*mc_df["D_n"] + w_M*mc_df["M_n"]
  517. # Save normalisation reference so that downstream cells (ablation,
  518. # empirical validation) normalise against the SAME ensemble, not against
  519. # a within-subset window. This is important for the ablation study to
  520. # read correctly.
  521. NORM_REF = {col: (float(mc_df[col].min()), float(mc_df[col].max()))
  522. for col in ["H_eff", "D", "M"]}
  523. print("\nNORM_REF:")
  524. for k, (lo, hi) in NORM_REF.items():
  525. print(f" {k:6s}: [{lo:.4f}, {hi:.4f}]")
  526. print(f"\nMonte Carlo summary — first 5 rows:")
  527. print(mc_df[["state", "H", "D", "M", "H_eff", "PSI_mc"]].head().round(3))
  528. print(f"\nTotal runs: {len(mc_df)} ({N_RUNS_PER_STATE} x {len(mc_states)} states)")
  529. # %%
  530. # ── Kruskal-Wallis + Mann-Whitney adjacent pairs + ROC / AUC ──────────────
  531. STATE_ORDER = ["Psychedelic", "Wake", "Task-engaged", "Dreaming",
  532. "Sleep (NREM-like)", "Minimally Conscious",
  533. "Anaesthesia", "Non-conscious", "Seizure"]
  534. # Per-state Psi arrays, in STATE_ORDER
  535. mc_groups = [mc_df.loc[mc_df["state"] == s, "PSI_mc"].values
  536. for s in STATE_ORDER]
  537. # Kruskal-Wallis omnibus
  538. H_kw, p_kw = kruskal(*mc_groups)
  539. print(f"Kruskal-Wallis (Psi across 9 states): H={H_kw:.2f}, p={p_kw:.2e}")
  540. # Pairwise Mann-Whitney U (adjacent states in Psi hierarchy)
  541. print("\nPairwise Mann-Whitney U (adjacent states, Cliff's delta effect size):")
  542. for s1, s2 in zip(STATE_ORDER[:-1], STATE_ORDER[1:]):
  543. a = mc_df.loc[mc_df["state"] == s1, "PSI_mc"].values
  544. b = mc_df.loc[mc_df["state"] == s2, "PSI_mc"].values
  545. if len(a) < 2 or len(b) < 2: continue
  546. U, p = mannwhitneyu(a, b, alternative="two-sided")
  547. cliff = (2 * U / (len(a) * len(b))) - 1
  548. print(f" {s1:22s} vs {s2:22s} p={p:.3e} Cliff delta={cliff:+.2f}")
  549. # ROC: conscious vs non-conscious
  550. CONSCIOUS = {"Wake", "Task-engaged", "Psychedelic", "Dreaming"}
  551. NONCONSCIOUS = {"Sleep (NREM-like)", "Minimally Conscious",
  552. "Anaesthesia", "Non-conscious", "Seizure"}
  553. rc = mc_df[mc_df["state"].isin(CONSCIOUS)].copy()
  554. rnc = mc_df[mc_df["state"].isin(NONCONSCIOUS)].copy()
  555. roc_df = pd.concat([rc, rnc], ignore_index=True)
  556. y_true = np.concatenate([np.ones(len(rc)), np.zeros(len(rnc))])
  557. scores = roc_df["PSI_mc"].values
  558. auc = roc_auc_score(y_true, scores)
  559. fpr, tpr, _ = roc_curve(y_true, scores)
  560. # Bootstrap 95% CI (2000 resamples)
  561. rng = np.random.default_rng(42)
  562. boot = []
  563. for _ in range(2000):
  564. idx = rng.integers(0, len(y_true), len(y_true))
  565. if len(np.unique(y_true[idx])) < 2: continue
  566. boot.append(roc_auc_score(y_true[idx], scores[idx]))
  567. ci_lo, ci_hi = np.percentile(boot, [2.5, 97.5])
  568. print("\n" + "="*60)
  569. print("ROC ANALYSIS: CONSCIOUS vs NON-CONSCIOUS (Psi)")
  570. print("="*60)
  571. print(f" N_conscious = {int(y_true.sum())}")
  572. print(f" N_non_conscious = {int(len(y_true) - y_true.sum())}")
  573. print(f" AUC = {auc:.3f} 95% CI (bootstrap) = [{ci_lo:.3f}, {ci_hi:.3f}]")
  574. # %%
  575. # ── Plot A: per-state Psi distributions + ROC curve + histogram ───────────
  576. PALETTE = {"Psychedelic":"#2166AC","Wake":"#4393C3","Task-engaged":"#74ADD1",
  577. "Dreaming":"#ABD9E9","Sleep (NREM-like)":"#FEE090",
  578. "Minimally Conscious":"#FDAE61","Anaesthesia":"#B2182B",
  579. "Non-conscious":"#800026","Seizure":"#3B0005"}
  580. colors_order = [PALETTE[s] for s in STATE_ORDER]
  581. fig, axes = plt.subplots(1, 3, figsize=(16, 4.6), constrained_layout=True)
  582. # Panel 1: Box-plot
  583. box_data = [mc_df.loc[mc_df["state"] == s, "PSI_mc"].values for s in STATE_ORDER]
  584. bp = axes[0].boxplot(box_data, patch_artist=True, showmeans=True,
  585. meanprops=dict(marker="^", markerfacecolor="k",
  586. markersize=5, markeredgewidth=0),
  587. medianprops=dict(color="white", linewidth=2),
  588. flierprops=dict(marker="o", markersize=3, alpha=0.4))
  589. for patch, c in zip(bp["boxes"], colors_order):
  590. patch.set_facecolor(c); patch.set_alpha(0.85)
  591. axes[0].set_xticklabels(STATE_ORDER, rotation=22, ha="right", fontsize=8.5)
  592. axes[0].set_ylabel(r"$\Psi$ (MC distribution)")
  593. axes[0].set_title(f"Psi per state (KW H={H_kw:.1f}, p={p_kw:.1e})")
  594. # Panel 2: Histogram
  595. axes[1].hist(rnc["PSI_mc"], bins=20, alpha=0.7, color="#D6604D",
  596. density=True, label="Non-conscious")
  597. axes[1].hist(rc["PSI_mc"], bins=20, alpha=0.7, color="#4393C3",
  598. density=True, label="Conscious")
  599. axes[1].set_xlabel(r"$\Psi$")
  600. axes[1].set_ylabel("Density")
  601. axes[1].set_title("Psi distributions")
  602. axes[1].legend(loc="upper left", fontsize=9)
  603. # Panel 3: ROC
  604. axes[2].plot(fpr, tpr, color="#2166AC", lw=2.0,
  605. label=f"Psi (AUC = {auc:.3f})")
  606. axes[2].plot([0, 1], [0, 1], "k--", lw=0.9, label="Chance")
  607. axes[2].set_xlabel("False Positive Rate")
  608. axes[2].set_ylabel("True Positive Rate")
  609. axes[2].set_title(f"ROC [95% CI {ci_lo:.3f}, {ci_hi:.3f}]")
  610. axes[2].set_aspect("equal")
  611. axes[2].legend(loc="lower right", fontsize=9)
  612. plt.show()
  613. # %%
  614. # ── Benchmarking: individual components + simple baselines ────────────────
  615. # AUC for conscious vs non-conscious using each component / baseline alone.
  616. def _lzc_broadband(X, fs):
  617. """Normalised LZC of broadband (1-40 Hz) amplitude envelope."""
  618. broad = filtfilt(*butter(4, [1/(fs/2), 40/(fs/2)], btype="band"), X, axis=-1)
  619. env = np.abs(hilbert(broad, axis=-1))
  620. out = []
  621. for ch in range(X.shape[0]):
  622. e = env[ch]; ez = (e - np.median(e)) / (np.std(e) + 1e-9)
  623. s = "".join("1" if v > 0 else "0" for v in ez)
  624. n = len(s); i, k, l, c = 0, 1, 1, 1
  625. while True:
  626. if i + k > n: c += 1; break
  627. if s[:l].find(s[i:i+k]) == -1:
  628. c += 1; l += k; i = l; k = 1
  629. if l + 1 > n: break
  630. else:
  631. k += 1
  632. if i + k > n: c += 1; break
  633. out.append(c / (n / (np.log2(n) + 1e-9) + 1e-9))
  634. return float(np.mean(out))
  635. def _spectral_slope(X, fs):
  636. """Aperiodic spectral slope (2-45 Hz log-log slope)."""
  637. slopes = []
  638. for ch in range(X.shape[0]):
  639. f, psd = welch(X[ch], fs=fs, nperseg=min(512, X.shape[1] // 2))
  640. mask = (f >= 2) & (f <= 45)
  641. if mask.sum() < 3: continue
  642. slopes.append(np.polyfit(np.log10(f[mask]), np.log10(psd[mask] + 1e-12), 1)[0])
  643. return float(np.mean(slopes)) if slopes else np.nan
  644. def _alpha_power(X, fs):
  645. """Relative alpha (8-13 Hz) band power."""
  646. vals = []
  647. for ch in range(X.shape[0]):
  648. f, psd = welch(X[ch], fs=fs, nperseg=min(512, X.shape[1] // 2))
  649. tot = np.trapezoid(psd[(f >= 1) & (f <= 45)], f[(f >= 1) & (f <= 45)])
  650. alph = np.trapezoid(psd[(f >= 8) & (f <= 13)], f[(f >= 8) & (f <= 13)])
  651. vals.append(alph / (tot + 1e-12))
  652. return float(np.mean(vals))
  653. # Aligned rows-labels construction (conscious rows first, then non-conscious)
  654. CONSCIOUS_MC = {"wake", "task_engaged", "psychedelic"}
  655. NONCONSCIOUS_MC = {"anesthesia", "non_conscious"}
  656. rows_c = mc_df[mc_df["state_raw"].isin(CONSCIOUS_MC)].copy()
  657. rows_nc = mc_df[mc_df["state_raw"].isin(NONCONSCIOUS_MC)].copy()
  658. rows_bench = pd.concat([rows_c, rows_nc], ignore_index=True)
  659. y_bench = np.concatenate([np.ones(len(rows_c)), np.zeros(len(rows_nc))])
  660. rng_bm = np.random.default_rng(99)
  661. def boot_auc(y, scores, n=1000):
  662. aucs = []
  663. for _ in range(n):
  664. idx = rng_bm.integers(0, len(y), len(y))
  665. if len(np.unique(y[idx])) < 2: continue
  666. aucs.append(roc_auc_score(y[idx], scores[idx]))
  667. a = roc_auc_score(y, scores)
  668. lo, hi = (np.percentile(aucs, [2.5, 97.5]) if aucs else (np.nan, np.nan))
  669. return a, lo, hi
  670. print("="*60)
  671. print("COMPONENT-NECESSITY BENCHMARKING")
  672. print("="*60)
  673. print(f"Conscious states : {sorted(CONSCIOUS_MC)} (n={len(rows_c)})")
  674. print(f"Non-conscious states : {sorted(NONCONSCIOUS_MC)} (n={len(rows_nc)})")
  675. print()
  676. print("AUC (conscious vs non-conscious):")
  677. for label, col in [("H_eff alone", "H_eff"),
  678. ("D alone", "D"),
  679. ("M alone", "M"),
  680. ("Psi (composite)", "PSI_mc")]:
  681. a, lo, hi = boot_auc(y_bench, rows_bench[col].values)
  682. print(f" {label:20s} AUC = {a:.3f} 95% CI [{lo:.3f}, {hi:.3f}]")
  683. # Add simple baselines — compute LZC / slope / alpha for every MC row
  684. # (fresh simulations needed because mc_df only stored H/D/M)
  685. print("\nComputing simple baselines (LZC, spectral slope, alpha power)...")
  686. print(" (~1 min — 270 additional metric calls on the same ensemble)")
  687. baseline_rows = []
  688. for st in mc_states:
  689. for _ in range(N_RUNS_PER_STATE):
  690. X = simulate_state_fast(n_channels=16, T=3000, fs=CFG["fs"], state=st)
  691. baseline_rows.append({
  692. "state_raw": st,
  693. "LZC": _lzc_broadband(X, CFG["fs"]),
  694. "spec_slope": _spectral_slope(X, CFG["fs"]),
  695. "alpha_power": _alpha_power(X, CFG["fs"]),
  696. })
  697. baseline_df = pd.DataFrame(baseline_rows)
  698. b_c = baseline_df[baseline_df["state_raw"].isin(CONSCIOUS_MC)]
  699. b_nc = baseline_df[baseline_df["state_raw"].isin(NONCONSCIOUS_MC)]
  700. b_rows = pd.concat([b_c, b_nc], ignore_index=True)
  701. y_b = np.concatenate([np.ones(len(b_c)), np.zeros(len(b_nc))])
  702. for label, col in [("LZC broadband", "LZC"),
  703. ("Spectral slope", "spec_slope"),
  704. ("Alpha power", "alpha_power")]:
  705. a, lo, hi = boot_auc(y_b, b_rows[col].values)
  706. print(f" {label:20s} AUC = {a:.3f} 95% CI [{lo:.3f}, {hi:.3f}]")
  707. # %%
  708. # ── Fig 6: Pairwise adjacent-state AUC heatmap (synthetic MC) ──────────────
  709. adj_pairs = [
  710. ("Psychedelic", "Wake"),
  711. ("Wake", "Task-engaged"),
  712. ("Task-engaged", "Dreaming"),
  713. ("Dreaming", "Sleep (NREM-like)"),
  714. ("Sleep (NREM-like)", "Minimally Conscious"),
  715. ("Minimally Conscious", "Anaesthesia"),
  716. ("Anaesthesia", "Non-conscious"),
  717. ("Non-conscious", "Seizure"),
  718. ]
  719. metrics_fig6 = [("H_eff_n", r"$H_{\mathrm{eff}}$"),
  720. ("D_n", r"$D$"),
  721. ("M_n", r"$M$"),
  722. ("PSI_mc", r"$\Psi$")]
  723. row_labels = [f"{a} \u2192 {b}" for a, b in adj_pairs]
  724. auc_mat = np.full((len(adj_pairs), len(metrics_fig6)), np.nan)
  725. for pi, (s1, s2) in enumerate(adj_pairs):
  726. r1 = mc_df[mc_df["state"] == s1]
  727. r2 = mc_df[mc_df["state"] == s2]
  728. if len(r1) < 2 or len(r2) < 2:
  729. continue
  730. yp = np.concatenate([np.ones(len(r1)), np.zeros(len(r2))])
  731. rb = pd.concat([r1, r2], ignore_index=True)
  732. for mi, (col, _) in enumerate(metrics_fig6):
  733. try:
  734. auc_mat[pi, mi] = roc_auc_score(yp, rb[col].values)
  735. except Exception:
  736. pass
  737. fig6, ax6 = plt.subplots(figsize=(5.8, 6.4), constrained_layout=True)
  738. im = ax6.imshow(auc_mat, vmin=0.0, vmax=1.0, cmap="RdYlGn", aspect="auto")
  739. ax6.set_xticks(range(len(metrics_fig6)))
  740. ax6.set_xticklabels([m[1] for m in metrics_fig6], fontsize=11)
  741. ax6.set_yticks(range(len(adj_pairs)))
  742. ax6.set_yticklabels(row_labels, fontsize=9)
  743. for pi in range(len(adj_pairs)):
  744. row = auc_mat[pi]
  745. best_idx = int(np.nanargmax(row)) if not np.all(np.isnan(row)) else -1
  746. for mi in range(len(metrics_fig6)):
  747. v = auc_mat[pi, mi]
  748. if np.isnan(v): continue
  749. ax6.text(mi, pi, f"{v:.2f}", ha="center", va="center",
  750. fontsize=9, fontweight="bold" if mi == best_idx else "normal",
  751. color="white" if (v > 0.85 or v < 0.15) else "black")
  752. cb = plt.colorbar(im, ax=ax6, label="AUC", fraction=0.04, pad=0.02, shrink=0.85)
  753. cb.ax.tick_params(labelsize=8)
  754. ax6.set_title("Pairwise adjacent-state AUC (synthetic MC)\n"
  755. "bold = argmax per row",
  756. fontsize=10)
  757. plt.show()
  758. plt.close(fig6)
  759. # %% [markdown]
  760. # ## Section B — Ablation study
  761. #
  762. # We now verify that each of the three components contributes independently
  763. # to $\Psi$. We start from the Wake simulator (which gives $H_{\mathrm{eff}}$
  764. # near its maximum and non-zero $D$ and $M$), then ablate exactly one
  765. # component at a time:
  766. #
  767. # - **No PAC** — remove $\gamma$-band content, so $I_{\varphi,A} \to 0$ and $D \to 0$.
  768. # - **No Metastability** — tile channel 0 across every channel, so $R(t) \to 1$ and $M \to 0$.
  769. # - **No Fractal** — set $H_{\mathrm{raw}} = 1.5$, pushing $\alpha_{\mathrm{dfa}}$ far from $H_{\mathrm{opt}}$ so $H_{\mathrm{eff}} = 0$.
  770. #
  771. # Normalisation uses `NORM_REF` from the Monte Carlo run above, so the bars
  772. # reflect the SAME scale as the manuscript figures.
  773. # %%
  774. # ── Ablation study: Baseline / No PAC / No Metastability / No Fractal ────
  775. def run_ablation_tests(C=16, T=7500, fs=250,
  776. Hopt=0.35, sigma_H=0.12, lam=1.0, n_runs=10):
  777. """Ablation study with simulate_state_fast('wake') as the baseline.
  778. Each ablation removes exactly one component; the others are unchanged.
  779. """
  780. condition_ids = ["baseline", "no_pac", "no_metastability", "no_fractal"]
  781. all_results = {cid: [] for cid in condition_ids}
  782. for _ in range(n_runs):
  783. # Baseline — full wake signal
  784. X = simulate_state_fast(state="wake", n_channels=C, T=T, fs=fs)
  785. m = compute_metrics(X, fs=fs, Hopt=Hopt, sigma_H=sigma_H, lam=lam)
  786. all_results["baseline"].append(m)
  787. # No PAC — remove gamma band (no phase-amplitude coupling structure)
  788. X_np = simulate_state_fast(state="wake", n_channels=C, T=T, fs=fs)
  789. b, a = butter(4, [30/(fs/2), 80/(fs/2)], btype="band")
  790. gamma_component = filtfilt(b, a, X_np, axis=-1)
  791. X_np = X_np - gamma_component
  792. m_np = compute_metrics(X_np, fs=fs, Hopt=Hopt, sigma_H=sigma_H, lam=lam)
  793. all_results["no_pac"].append(m_np)
  794. # No Metastability — tile channel 0 across every channel (R(t) = 1)
  795. X_nm = simulate_state_fast(state="wake", n_channels=C, T=T, fs=fs)
  796. X_nm_sync = np.tile(X_nm[0:1, :], (C, 1))
  797. m_nm = compute_metrics(X_nm_sync, fs=fs, Hopt=Hopt,
  798. sigma_H=sigma_H, lam=lam)
  799. all_results["no_metastability"].append(m_nm)
  800. # No Fractal — keep D and M as baseline, force H_eff = 0
  801. X_nf = simulate_state_fast(state="wake", n_channels=C, T=T, fs=fs)
  802. m_nf = compute_metrics(X_nf, fs=fs, Hopt=Hopt, sigma_H=sigma_H, lam=lam)
  803. m_nf["H_raw"] = 1.5
  804. m_nf["Heff"] = 0.0
  805. m_nf["Psi"] = w_H*0.0 + w_D*m_nf["D"] + w_M*m_nf["M"]
  806. all_results["no_fractal"].append(m_nf)
  807. summary = {}
  808. for cid in condition_ids:
  809. H = np.array([r["Heff"] for r in all_results[cid]])
  810. D = np.array([r["D"] for r in all_results[cid]])
  811. M = np.array([r["M"] for r in all_results[cid]])
  812. P = np.array([r["Psi"] for r in all_results[cid]])
  813. summary[cid] = {
  814. "Heff_mean": H.mean(), "Heff_std": H.std(),
  815. "D_mean": D.mean(), "D_std": D.std(),
  816. "M_mean": M.mean(), "M_std": M.std(),
  817. "Psi_mean": P.mean(), "Psi_std": P.std(),
  818. }
  819. return summary
  820. print("Running ablation study (10 runs per condition × 4 conditions)...")
  821. ablation_summary = run_ablation_tests(
  822. C=16, T=7500, fs=CFG["fs"], Hopt=CFG["H_opt"],
  823. sigma_H=CFG["sigma_H"], lam=CFG["lam"], n_runs=10,
  824. )
  825. # Print summary table
  826. print("\nAblation summary (mean ± SD):")
  827. print(f"{'Condition':<20s} {'H_eff':>14s} {'D':>14s} {'M':>14s} {'Psi':>14s}")
  828. for cid in ["baseline", "no_pac", "no_metastability", "no_fractal"]:
  829. s = ablation_summary[cid]
  830. print(f"{format_label(cid):<20s} "
  831. f"{s['Heff_mean']:>6.3f}±{s['Heff_std']:<5.3f} "
  832. f"{s['D_mean']:>6.3f}±{s['D_std']:<5.3f} "
  833. f"{s['M_mean']:>6.3f}±{s['M_std']:<5.3f} "
  834. f"{s['Psi_mean']:>6.3f}±{s['Psi_std']:<5.3f}")
  835. # %%
  836. # ── Ablation plot: normalised by NORM_REF for comparability with main figs ──
  837. def plot_ablation_results(summary, norm_ref=None):
  838. condition_ids = ["baseline", "no_pac", "no_metastability", "no_fractal"]
  839. labels = [format_label(cid) for cid in condition_ids]
  840. titles = [r"Integration ($H_\mathrm{eff}$)",
  841. r"Organised Cross-frequency Complexity ($D$)",
  842. r"Metastability ($M$)",
  843. r"Composite Index ($\Psi$)"]
  844. if norm_ref is not None:
  845. # Normalise to NORM_REF so ablated components sit visibly at 0
  846. for cid in condition_ids:
  847. h_n = float(np.clip(
  848. (summary[cid]["Heff_mean"] - norm_ref["H_eff"][0]) /
  849. (norm_ref["H_eff"][1] - norm_ref["H_eff"][0] + 1e-12), 0, 1))
  850. d_n = float(np.clip(
  851. (summary[cid]["D_mean"] - norm_ref["D"][0]) /
  852. (norm_ref["D"][1] - norm_ref["D"][0] + 1e-12), 0, 1))
  853. m_n = float(np.clip(
  854. (summary[cid]["M_mean"] - norm_ref["M"][0]) /
  855. (norm_ref["M"][1] - norm_ref["M"][0] + 1e-12), 0, 1))
  856. summary[cid]["Heff_n"] = h_n
  857. summary[cid]["D_n"] = d_n
  858. summary[cid]["M_n"] = m_n
  859. summary[cid]["Psi_norm"] = w_H*h_n + w_D*d_n + w_M*m_n
  860. plot_keys = ["Heff_n", "D_n", "M_n", "Psi_norm"]
  861. ylbl = "Normalised value (0–1)"
  862. else:
  863. plot_keys = ["Heff_mean", "D_mean", "M_mean", "Psi_mean"]
  864. ylbl = "Value"
  865. fig, axes = plt.subplots(1, 4, figsize=(15, 3.5), constrained_layout=True)
  866. x = np.arange(len(condition_ids))
  867. for ax, pk, title in zip(axes, plot_keys, titles):
  868. vals = [summary[cid][pk] for cid in condition_ids]
  869. ax.bar(x, vals, width=0.66, color="#4393C3",
  870. alpha=0.85, edgecolor="white", linewidth=0.4)
  871. for xi, v in enumerate(vals):
  872. if v > 0.005:
  873. ax.text(xi, v + 0.015, f"{v:.2f}",
  874. ha="center", va="bottom", fontsize=7.5)
  875. ax.set_xticks(x)
  876. ax.set_xticklabels(labels, rotation=25, ha="right", fontsize=8)
  877. ax.set_ylim(0, max(vals) * 1.20 + 0.02)
  878. ax.set_title(title, fontsize=9)
  879. ax.set_ylabel(ylbl, fontsize=8)
  880. return fig
  881. _fig_abl = plot_ablation_results(ablation_summary, norm_ref=NORM_REF)
  882. plt.show()
  883. plt.close(_fig_abl)
  884. # %% [markdown]
  885. # ## Section C — Hyperparameter sensitivity analysis
  886. #
  887. # Three sweeps over the tuning parameters, measured by the **adjacent-pair
  888. # AUC** metric (mean and minimum across 7 adjacent state pairs in the
  889. # hierarchy, orientation-corrected so direction flips don't penalise):
  890. #
  891. # 1. $H_{\mathrm{opt}}$ — the DFA α midpoint of the triangular tuning.
  892. # 2. $w_H$ — the composite weight on $H_{\mathrm{eff}}$ (with $w_D$:$w_M$
  893. # ratio held constant at the nominal 0.35 : 0.25).
  894. # 3. $\lambda$ — the weight on LZ inside $D = I_{\varphi,A}\,(1+\lambda\,\mathrm{LZ})$.
  895. #
  896. # The first two sweeps are fast (recompute $\Psi$ from the stored MC
  897. # ensemble). The $\lambda$ sweep requires fresh simulations because the LZ
  898. # weight is baked in at metric-computation time. Runtime: ~1–2 minutes.
  899. # %%
  900. # ── Sensitivity analysis: 3 panels (H_opt, w_H, lambda) ───────────────────
  901. ADJ_PAIRS_SENS = [
  902. ("Wake", "Task-engaged"),
  903. ("Task-engaged", "Dreaming"),
  904. ("Dreaming", "Sleep (NREM-like)"),
  905. ("Sleep (NREM-like)", "Minimally Conscious"),
  906. ("Minimally Conscious", "Anaesthesia"),
  907. ("Anaesthesia", "Non-conscious"),
  908. ("Non-conscious", "Seizure"),
  909. ]
  910. _STATES_LAM = ["wake","task_engaged","psychedelic","dreaming",
  911. "sleep","minimally_conscious","anesthesia",
  912. "non_conscious","seizure"]
  913. _LABEL_MAP = STATE_DISPLAY # reuse the display map
  914. def adj_auc_summary(df_in, psi_col):
  915. """Mean and minimum orientation-corrected AUC across adjacent pairs."""
  916. aucs = []
  917. for s1, s2 in ADJ_PAIRS_SENS:
  918. r1 = df_in[df_in["state"] == s1]
  919. r2 = df_in[df_in["state"] == s2]
  920. if len(r1) < 2 or len(r2) < 2:
  921. continue
  922. y = np.concatenate([np.ones(len(r1)), np.zeros(len(r2))])
  923. vals = pd.concat([r1[[psi_col]], r2[[psi_col]]], ignore_index=True)[psi_col].values
  924. try:
  925. a = float(roc_auc_score(y, vals))
  926. aucs.append(max(a, 1.0 - a)) # orientation-corrected
  927. except Exception:
  928. pass
  929. if not aucs:
  930. return np.nan, np.nan
  931. return float(np.mean(aucs)), float(np.min(aucs))
  932. def recompute_psi(df_src, Hopt_new, wH, wD, wM):
  933. """Recompute Psi from stored D_n, M_n with a new H_opt and weights."""
  934. df = df_src.copy()
  935. df["_H_eff"] = df["H"].apply(
  936. lambda a: max(0.0, 1.0 - abs(a - Hopt_new) / _alpha_range_mc))
  937. hmin, hmax = df["_H_eff"].min(), df["_H_eff"].max()
  938. df["_H_n"] = (df["_H_eff"] - hmin) / (hmax - hmin + 1e-9)
  939. df["_PSI"] = wH*df["_H_n"] + wD*df["D_n"] + wM*df["M_n"]
  940. return df
  941. # ── Panel 1: H_opt sweep (fast) ──────────────────────────────────────────
  942. print("Sweeping H_opt...")
  943. h_opts = np.round(np.arange(0.20, 0.76, 0.05), 2)
  944. mean_h, min_h = [], []
  945. for h in h_opts:
  946. df_s = recompute_psi(mc_df, h, w_H, w_D, w_M)
  947. m, mn = adj_auc_summary(df_s, "_PSI")
  948. mean_h.append(m); min_h.append(mn)
  949. # ── Panel 2: w_H sweep (fast) ────────────────────────────────────────────
  950. print("Sweeping w_H...")
  951. wH_vals = np.round(np.arange(0.20, 0.61, 0.05), 2)
  952. _D_ratio = w_D / (w_D + w_M); _M_ratio = w_M / (w_D + w_M)
  953. mean_w, min_w = [], []
  954. for wh in wH_vals:
  955. rem = 1.0 - wh
  956. df_s = recompute_psi(mc_df, H_opt, wh, rem*_D_ratio, rem*_M_ratio)
  957. m, mn = adj_auc_summary(df_s, "_PSI")
  958. mean_w.append(m); min_w.append(mn)
  959. # ── Panel 3: lambda sweep (slow; fresh simulations needed) ───────────────
  960. print("Sweeping lambda (requires fresh simulations)...")
  961. lambda_vals = np.round(np.arange(0.4, 2.01, 0.2), 1)
  962. N_QUICK = 8 # lighter than MC to keep runtime manageable
  963. mean_l, min_l = [], []
  964. for lv in lambda_vals:
  965. rows = []
  966. for st in _STATES_LAM:
  967. for _ in range(N_QUICK):
  968. X = simulate_state_fast(n_channels=16, T=3000, fs=250, state=st)
  969. m = compute_metrics(X, fs=250, Hopt=H_opt, sigma_H=CFG["sigma_H"],
  970. lam=float(lv), alpha_range=_alpha_range_mc)
  971. rows.append({"state": _LABEL_MAP[st], "Psi": m["Psi"]})
  972. df_lam = pd.DataFrame(rows)
  973. m, mn = adj_auc_summary(df_lam, "Psi")
  974. mean_l.append(m); min_l.append(mn)
  975. print(f" lambda={lv}: mean AUC={m:.3f}, min AUC={mn:.3f}")
  976. # ── Plot ─────────────────────────────────────────────────────────────────
  977. def _panel(ax, xs, m_vals, n_vals, cur_x, xlabel, title, legend_label, show_ylabel=False):
  978. ax.plot(xs, m_vals, "o-", color="#2166AC", lw=2, ms=5,
  979. label="Mean adj.-pair AUC")
  980. ax.plot(xs, n_vals, "s--", color="#4DAF4A", lw=1.8, ms=4,
  981. label="Min adj.-pair AUC")
  982. ax.axvline(cur_x, color="#D6604D", ls="--", lw=1.5, label=legend_label)
  983. ax.set_xlabel(xlabel, fontsize=12)
  984. if show_ylabel:
  985. ax.set_ylabel("AUC (adjacent state pairs)", fontsize=11)
  986. ax.set_ylim(0.48, 1.02)
  987. ax.set_title(title, fontsize=11, fontweight="bold")
  988. ax.grid(alpha=0.3)
  989. figB, axsB = plt.subplots(1, 3, figsize=(13.5, 5.6))
  990. figB.subplots_adjust(bottom=0.22, top=0.82, wspace=0.30)
  991. _panel(axsB[0], h_opts, mean_h, min_h, H_opt,
  992. r"$H_{\mathrm{opt}}$", r"Sensitivity to $H_{\mathrm{opt}}$",
  993. f"Current = {H_opt}", show_ylabel=True)
  994. _panel(axsB[1], wH_vals, mean_w, min_w, w_H,
  995. r"$w_H$ (scale-free weight)", r"Sensitivity to $w_H$",
  996. f"Current = {w_H}")
  997. _panel(axsB[2], lambda_vals, mean_l, min_l, lam,
  998. r"$\lambda$", r"Sensitivity to $\lambda$",
  999. f"Current = {lam}")
  1000. handles, labels = axsB[0].get_legend_handles_labels()
  1001. figB.legend(handles, labels, loc="lower center",
  1002. bbox_to_anchor=(0.5, -0.02), ncol=3, fontsize=9,
  1003. frameon=True, framealpha=0.9)
  1004. figB.suptitle("Hyperparameter sensitivity: adjacent-state discrimination\n"
  1005. "Solid = mean AUC across 7 adjacent pairs; dashed = worst-case pair",
  1006. fontsize=10.5, fontweight="bold", y=0.97)
  1007. plt.show()
  1008. plt.close(figB)
  1009. # %% [markdown]
  1010. # ## Section D — Within-state Monte Carlo convergence
  1011. #
  1012. # This diagnostic shows how quickly each state's running mean approaches its
  1013. # long-run mean as the number of MC runs grows. We use within-state
  1014. # convergence (not pooled) because pooled convergence conflates between-state
  1015. # variance with MC noise. The `n = 30` guide line marks the choice used in
  1016. # the paper and in Section A above.
  1017. # %%
  1018. # ── Fig : within-state MC convergence ────────────────────────────────────
  1019. print("Running 50 within-state MC runs per state for convergence diagnostic...")
  1020. N_CONV = 50
  1021. state_runs = {}
  1022. _key = {v: k for k, v in STATE_DISPLAY.items()}
  1023. for state in STATE_ORDER:
  1024. sk = _key[state]
  1025. vals = []
  1026. for _ in range(N_CONV):
  1027. try:
  1028. X = simulate_state_fast(n_channels=16, T=5000, fs=250, state=sk)
  1029. m = compute_metrics(X, fs=250, Hopt=H_opt,
  1030. sigma_H=CFG["sigma_H"], lam=lam)
  1031. vals.append(m["Psi"])
  1032. except Exception:
  1033. pass
  1034. state_runs[state] = np.array(vals, dtype=float)
  1035. print(f" {state}: {len(vals)} / {N_CONV} runs succeeded")
  1036. max_len = min((len(v) for v in state_runs.values() if len(v) > 0), default=0)
  1037. if max_len >= 2:
  1038. xs = np.arange(1, max_len + 1)
  1039. dev_mat = []
  1040. for state in STATE_ORDER:
  1041. v = state_runs[state][:max_len]
  1042. if len(v) == 0: continue
  1043. final_mean = float(np.mean(v))
  1044. running_m = np.array([np.mean(v[:n]) for n in xs])
  1045. dev_mat.append(np.abs(running_m - final_mean))
  1046. dev_mat = np.array(dev_mat)
  1047. median_dev = np.median(dev_mat, axis=0)
  1048. max_dev = np.max(dev_mat, axis=0)
  1049. figC, axC = plt.subplots(figsize=(8.5, 4.6), constrained_layout=True)
  1050. axC.plot(xs, median_dev, color="#2166AC", lw=2, label="Median deviation")
  1051. axC.plot(xs, max_dev, color="#D6604D", lw=1.8, ls="--", label="Maximum deviation")
  1052. axC.axvline(30, color="#555555", ls=":", lw=1.8, label="n = 30 per state")
  1053. axC.set_xlabel("Monte Carlo runs per state", fontsize=11)
  1054. axC.set_ylabel(r"Absolute deviation in $\Psi$ mean", fontsize=11)
  1055. axC.set_title("Within-state Monte Carlo convergence\n"
  1056. "Deviation of running state mean from long-run mean",
  1057. fontsize=11, fontweight="bold")
  1058. axC.grid(alpha=0.3)
  1059. axC.legend(fontsize=9)
  1060. plt.show()
  1061. plt.close(figC)
  1062. else:
  1063. print("Insufficient runs for convergence plot.")
  1064. # %% [markdown]
  1065. # ## Section E — Sleep-EDF empirical validation
  1066. #
  1067. # The Sleep-EDF Cassette EEG data is read **from the local `DATA_DIR`** —
  1068. # no network download, no MNE fetcher. Point `DATA_DIR` at the folder
  1069. # containing your pre-downloaded `*-PSG.edf` and `*-Hypnogram.edf` files.
  1070. #
  1071. # Pipeline:
  1072. # - Extract 30-second Wake / N2 / REM epochs per subject (stratified across the night).
  1073. # - Compute Ψ and the benchmark metrics on each epoch.
  1074. # - Per-subject $H_{\mathrm{opt}}$ calibration by KDE mode of that subject's own Wake α_{\mathrm{dfa}} — a domain-adaptation of the tuning prior, not a fit to the test data.
  1075. # - **Friedman** omnibus across the three stages (paired over subjects).
  1076. # - **Wilcoxon signed-rank** with Bonferroni correction for post-hoc pairs.
  1077. # - **Subject-level AUC** with 95% bootstrap CIs (resampling over subjects).
  1078. # - **Figure 7** — subject-level Ψ per stage with significance brackets.
  1079. # - **Figure 8** — AUC benchmarking heatmap (Ψ vs single-component and baseline metrics).
  1080. #
  1081. # `N_SUBJ = 30` reproduces the manuscript values; lower it for a quick
  1082. # sanity-check run.
  1083. #
  1084. # **Heads-up on two-channel limitations.** Sleep-EDF has 2 EEG channels, so
  1085. # $D$ and $M$ collapse toward zero — Ψ in real EEG is effectively driven by
  1086. # $H_{\mathrm{eff}}$. The per-stage component-contribution printout makes
  1087. # this transparent.
  1088. # %%
  1089. # ── Sleep-EDF empirical validation helpers ────────────────────────────────
  1090. # These are the shared helpers used by the main Sleep-EDF pipeline.
  1091. # Each operates on a DataFrame of epoch-level metrics with columns including
  1092. # subject, stage, H, D, M, H_eff, PSI, and optionally LZC / spec_slope / alpha_power.
  1093. def compute_pac_surrogate(X, fs, n_surrogates=50, gamma_high=45):
  1094. """Time-shift surrogate test for PAC significance.
  1095. Shifts gamma amplitude by a random time offset relative to theta phase.
  1096. Returns (obs_mi, surr_mean, surr_sd, z_score).
  1097. z > 2 indicates PAC significantly above chance.
  1098. """
  1099. theta = filtfilt(*butter(4, [4/(fs/2), 8/(fs/2)], btype="band"), X, axis=-1)
  1100. gamma = filtfilt(*butter(4, [30/(fs/2), gamma_high/(fs/2)], btype="band"),
  1101. X, axis=-1)
  1102. theta_phase = np.angle(hilbert(theta, axis=-1))
  1103. gamma_amp = np.abs(hilbert(gamma, axis=-1))
  1104. def tort_mi(ph, amp, n_bins=36):
  1105. bins = np.linspace(-np.pi, np.pi, n_bins + 1)
  1106. ap = np.array([amp[(ph >= bins[i]) & (ph < bins[i+1])].mean()
  1107. if ((ph >= bins[i]) & (ph < bins[i+1])).sum() > 0 else 0.0
  1108. for i in range(n_bins)])
  1109. ap /= ap.sum() + 1e-12
  1110. q = np.ones(n_bins) / n_bins
  1111. return float(np.sum(ap * np.log(ap / q + 1e-12)))
  1112. obs_mi = float(np.mean([tort_mi(theta_phase[ch], gamma_amp[ch])
  1113. for ch in range(X.shape[0])]))
  1114. T = X.shape[1]
  1115. surr = []
  1116. for _ in range(n_surrogates):
  1117. shift = np.random.randint(T // 4, 3 * T // 4)
  1118. gamma_shifted = np.roll(gamma_amp, shift, axis=-1)
  1119. surr.append(np.mean([tort_mi(theta_phase[ch], gamma_shifted[ch])
  1120. for ch in range(X.shape[0])]))
  1121. surr_mean = float(np.mean(surr))
  1122. surr_sd = float(np.std(surr) + 1e-12)
  1123. return obs_mi, surr_mean, surr_sd, (obs_mi - surr_mean) / surr_sd
  1124. def wake_mode_kde(wake_alpha, grid_n=512):
  1125. """KDE-based mode of the within-subject Wake alpha_dfa distribution.
  1126. Robust to bimodal clinical Wake distributions where the median can fall
  1127. between clean eyes-closed alpha and drowsy transitional epochs. Falls
  1128. back to median for small n or numerical failures.
  1129. """
  1130. wake_alpha = np.asarray(wake_alpha, dtype=float)
  1131. wake_alpha = wake_alpha[np.isfinite(wake_alpha)]
  1132. if len(wake_alpha) < 3:
  1133. return float(np.median(wake_alpha)) if len(wake_alpha) else float("nan")
  1134. from scipy.stats import gaussian_kde
  1135. try:
  1136. kde = gaussian_kde(wake_alpha, bw_method="scott")
  1137. lo, hi = wake_alpha.min(), wake_alpha.max()
  1138. pad = (hi - lo) * 0.05 if hi > lo else 0.05
  1139. grid = np.linspace(lo - pad, hi + pad, grid_n)
  1140. return float(grid[np.argmax(kde(grid))])
  1141. except Exception:
  1142. return float(np.median(wake_alpha))
  1143. def rank_biserial(v1, v2):
  1144. """Matched-pairs rank-biserial correlation for paired Wilcoxon."""
  1145. d = np.asarray(v1) - np.asarray(v2)
  1146. d = d[d != 0]
  1147. if len(d) == 0:
  1148. return 0.0
  1149. r = rankdata(np.abs(d))
  1150. Wp = r[d > 0].sum(); Wn = r[d < 0].sum()
  1151. return float((Wp - Wn) / (Wp + Wn)) if (Wp + Wn) > 0 else 0.0
  1152. def paired_stats_table(piv, stages, pairs, label_fn=None, indent=" "):
  1153. """Friedman omnibus + Wilcoxon post-hoc with Bonferroni correction.
  1154. Returns dict {friedman_chi2, friedman_p, friedman_n, pairs:{...}}.
  1155. """
  1156. if label_fn is None:
  1157. label_fn = str
  1158. out = {"friedman_chi2": float("nan"), "friedman_p": float("nan"),
  1159. "friedman_n": 0, "pairs": {}}
  1160. stages_present = [s for s in stages if s in piv.columns]
  1161. complete = piv[stages_present].dropna() if stages_present else piv.iloc[:0]
  1162. n_c = len(complete)
  1163. out["friedman_n"] = n_c
  1164. if n_c >= 3 and len(stages_present) >= 3:
  1165. chi2, p = friedmanchisquare(*[complete[s].values for s in stages_present])
  1166. out["friedman_chi2"] = float(chi2); out["friedman_p"] = float(p)
  1167. print(f"{indent}Friedman (paired, n={n_c} complete-case subjects, "
  1168. f"K={len(stages_present)}): chi2={chi2:.2f}, p={p:.4f}")
  1169. else:
  1170. print(f"{indent}Friedman: insufficient data (n={n_c})")
  1171. if not pairs:
  1172. return out
  1173. bonf = 0.05 / len(pairs)
  1174. print(f"{indent}Pairwise Wilcoxon (Bonferroni alpha={bonf:.4f}):")
  1175. for s1, s2 in pairs:
  1176. if s1 not in piv.columns or s2 not in piv.columns:
  1177. continue
  1178. sh = piv[[s1, s2]].dropna()
  1179. if len(sh) < 4:
  1180. print(f"{indent} {label_fn(s1):>8s} vs {label_fn(s2):<8s}:"
  1181. f" insufficient (n={len(sh)})")
  1182. continue
  1183. v1, v2 = sh[s1].values, sh[s2].values
  1184. _, p = wilcoxon(v1, v2)
  1185. r_rb = rank_biserial(v1, v2)
  1186. sig = ("***" if p < 0.001 else "**" if p < 0.01
  1187. else "*" if p < 0.05 else "ns")
  1188. bonf_sig = p < bonf
  1189. out["pairs"][(s1, s2)] = {"p": float(p), "r_rb": r_rb, "sig": sig,
  1190. "n": len(sh), "bonf_sig": bonf_sig,
  1191. "bonf_thresh": bonf}
  1192. tag = "(Bonf.sig)" if bonf_sig else ""
  1193. print(f"{indent} {label_fn(s1):>8s} vs {label_fn(s2):<8s}:"
  1194. f" p={p:.4f} {sig:>3s} r_rb={r_rb:+.2f} n={len(sh)} {tag}")
  1195. return out
  1196. def boot_auc_paired(v1, v2, n_boot=2000, seed=42):
  1197. """Bootstrap 95% CI of AUC, resampling paired subjects."""
  1198. rng = np.random.default_rng(seed)
  1199. n = len(v1); boots = []
  1200. for _ in range(n_boot):
  1201. idx = rng.integers(0, n, n)
  1202. v1b, v2b = v1[idx], v2[idx]
  1203. yb = np.concatenate([np.ones(n), np.zeros(n)])
  1204. vals = np.concatenate([v1b, v2b])
  1205. try:
  1206. boots.append(roc_auc_score(yb, vals))
  1207. except ValueError:
  1208. pass
  1209. if not boots:
  1210. return float("nan"), float("nan")
  1211. return (float(np.percentile(boots, 2.5)),
  1212. float(np.percentile(boots, 97.5)))
  1213. def subject_auc_benchmark(df, pairs, bench_cols, subject_col="subject",
  1214. stage_col="stage", n_boot=2000, seed=42,
  1215. do_print=True, indent=" "):
  1216. """Subject-level AUC benchmarking across paired stages."""
  1217. subj_piv = {}
  1218. for _lbl, col in bench_cols:
  1219. if col in df.columns:
  1220. subj_piv[col] = (df.groupby([subject_col, stage_col])[col]
  1221. .mean().unstack(stage_col))
  1222. results = {}
  1223. for s1, s2 in pairs:
  1224. if do_print:
  1225. print(f"\n{indent}{s1} vs {s2}:")
  1226. for label, col in bench_cols:
  1227. if col not in subj_piv: continue
  1228. pv = subj_piv[col]
  1229. if s1 not in pv.columns or s2 not in pv.columns: continue
  1230. sh = pv[[s1, s2]].dropna()
  1231. if len(sh) < 4:
  1232. if do_print:
  1233. print(f"{indent} {label:22s} insufficient (n={len(sh)})")
  1234. continue
  1235. v1, v2 = sh[s1].values, sh[s2].values
  1236. y = np.concatenate([np.ones(len(v1)), np.zeros(len(v2))])
  1237. vals = np.concatenate([v1, v2])
  1238. try:
  1239. a = roc_auc_score(y, vals)
  1240. lo, hi = boot_auc_paired(v1, v2, n_boot=n_boot, seed=seed)
  1241. results[(s1, s2, col)] = (a, lo, hi)
  1242. if do_print:
  1243. print(f"{indent} {label:22s} AUC = {a:.3f} "
  1244. f"[{lo:.3f}, {hi:.3f}] (n={len(v1)} subjects)")
  1245. except Exception as e:
  1246. if do_print:
  1247. print(f"{indent} {label:22s} {e}")
  1248. return results
  1249. print("Sleep-EDF helpers loaded: PAC surrogate, KDE mode, paired stats, bootstrap AUC.")
  1250. # %%
  1251. # ── Sleep-EDF main pipeline (LOCAL DATA ONLY — no download) ────────────────
  1252. # Point DATA_DIR at a folder containing *-PSG.edf and *-Hypnogram.edf files.
  1253. import glob
  1254. N_SUBJ = 30 # number of subjects (set lower for quick runs)
  1255. N_EPOCHS = 15 # epochs per stage per subject
  1256. EPOCH_DUR_SEC = 30 # 30-s epochs (AASM standard)
  1257. REAL_FS = 100 # Sleep-EDF Cassette native sampling rate
  1258. GAMMA_CAP = 45 # Hz, below 100 Hz Nyquist
  1259. # ────────────────────────────────────────────────────────────────────────
  1260. # DATA PATH — adjust this one line for your environment
  1261. # ────────────────────────────────────────────────────────────────────────
  1262. # On Google Colab, first run:
  1263. #from google.colab import drive; drive.mount('/content/drive')
  1264. # DATA_DIR = "./sleep_edf_local"
  1265. #Sleep-EDF data sample used is available at: https://drive.google.com/drive/folders/1zRKqwlYfLGn6OTwL4cj6ceRMvfvWjiUY?usp=sharing
  1266. DATA_DIR = "/content/drive/MyDrive/...."
  1267. STAGES = {"Sleep stage W": "W", "Sleep stage 2": "N2", "Sleep stage R": "R"}
  1268. def load_local_pairs(n_subj):
  1269. """Match *-PSG.edf with *-Hypnogram.edf files by 6-char prefix (SC4ssN).
  1270. Sleep-EDF pairs the two file types by subject + night prefix; the
  1271. scorer suffix on the hypnogram varies (e.g. SC4001E0-PSG.edf pairs
  1272. with SC4001EC-Hypnogram.edf).
  1273. """
  1274. if not os.path.isdir(DATA_DIR):
  1275. raise FileNotFoundError(
  1276. f"DATA_DIR not found: {DATA_DIR}\n"
  1277. "Update DATA_DIR at the top of this cell to your local path "
  1278. "containing *-PSG.edf and *-Hypnogram.edf files."
  1279. )
  1280. psg_files = sorted(glob.glob(os.path.join(DATA_DIR, "*PSG.edf")))
  1281. if not psg_files:
  1282. # Try a recursive search in case files are in subfolders
  1283. psg_files = sorted(glob.glob(os.path.join(DATA_DIR, "**", "*PSG.edf"),
  1284. recursive=True))
  1285. hyp_files = glob.glob(os.path.join(DATA_DIR, "*Hypnogram.edf"))
  1286. if not hyp_files:
  1287. hyp_files = glob.glob(os.path.join(DATA_DIR, "**", "*Hypnogram.edf"),
  1288. recursive=True)
  1289. if not psg_files:
  1290. raise FileNotFoundError(f"No *-PSG.edf files in {DATA_DIR}")
  1291. pairs = []
  1292. for p in psg_files:
  1293. prefix = os.path.basename(p)[:6]
  1294. matches = [h for h in hyp_files if os.path.basename(h).startswith(prefix)]
  1295. if matches:
  1296. pairs.append((p, matches[0]))
  1297. if len(pairs) >= n_subj:
  1298. break
  1299. return pairs
  1300. def run_sleep_edf_validation():
  1301. """Full Sleep-EDF empirical validation. Populates real_df + H_opt_per_subj."""
  1302. try:
  1303. import mne
  1304. except ImportError as _err:
  1305. raise RuntimeError(
  1306. "The 'mne' package is required to read Sleep-EDF EDF files.\n"
  1307. " Run the first cell of this notebook to install all dependencies "
  1308. "(it calls `!pip install -q ... mne`), or install manually with:\n"
  1309. " pip install mne\n"
  1310. f" (Original error: {_err})"
  1311. ) from _err
  1312. mne.set_log_level("WARNING")
  1313. print("=" * 60)
  1314. print("SLEEP-EDF EMPIRICAL VALIDATION (local data)")
  1315. print("=" * 60)
  1316. print(f"\nDATA_DIR = {DATA_DIR}")
  1317. print(f"\n[1] Matching up to {N_SUBJ} PSG/Hypnogram pairs...")
  1318. pairs = load_local_pairs(N_SUBJ)
  1319. if not pairs:
  1320. raise RuntimeError("No paired PSG/Hypnogram files found in DATA_DIR.")
  1321. print(f" Found {len(pairs)} pair(s):")
  1322. for p, h in pairs[:5]:
  1323. print(f" {os.path.basename(p)} + {os.path.basename(h)}")
  1324. if len(pairs) > 5:
  1325. print(f" ... and {len(pairs) - 5} more")
  1326. print("\n[2] Computing Psi and baseline metrics on real EEG...")
  1327. real_rows = []
  1328. for subj_idx, (psg_path, hyp_path) in enumerate(pairs):
  1329. try:
  1330. raw = mne.io.read_raw_edf(psg_path, preload=True, verbose=False)
  1331. eeg_chs = [ch for ch in raw.ch_names
  1332. if "EEG" in ch.upper() or ch.startswith("Fp") or ch.startswith("Pz")]
  1333. if not eeg_chs:
  1334. eeg_chs = raw.ch_names[:2]
  1335. raw.pick_channels(eeg_chs[:2], verbose=False)
  1336. annot = mne.read_annotations(hyp_path)
  1337. raw.set_annotations(annot, emit_warning=False)
  1338. if raw.info["sfreq"] != REAL_FS:
  1339. print(f" Subject {subj_idx}: skipping (sfreq={raw.info['sfreq']})")
  1340. continue
  1341. n_ok = 0
  1342. for stage_label, stage_key in STAGES.items():
  1343. onsets = [ann["onset"] for ann in raw.annotations
  1344. if ann["description"] == stage_label]
  1345. if not onsets:
  1346. continue
  1347. n_use = min(len(onsets), N_EPOCHS)
  1348. # Stratified epoch sampling across the night
  1349. if len(onsets) > n_use:
  1350. sel = np.linspace(0, len(onsets) - 1, n_use).round().astype(int)
  1351. selected_onsets = [onsets[int(i)] for i in np.unique(sel)]
  1352. else:
  1353. selected_onsets = list(onsets)
  1354. for onset in selected_onsets:
  1355. start = int(onset * REAL_FS)
  1356. end = start + EPOCH_DUR_SEC * REAL_FS
  1357. if end > raw.n_times:
  1358. continue
  1359. ep = raw.get_data()[:, start:end]
  1360. if np.any(np.abs(ep) > 500e-6):
  1361. continue
  1362. ep_z = (ep - ep.mean(axis=-1, keepdims=True)) / (
  1363. ep.std(axis=-1, keepdims=True) + 1e-12)
  1364. m = compute_all_metrics(ep_z, REAL_FS, gamma_high=GAMMA_CAP)
  1365. m["LZC"] = _lzc_broadband(ep_z, REAL_FS)
  1366. m["spec_slope"] = _spectral_slope(ep_z, REAL_FS)
  1367. m["alpha_power"]= _alpha_power(ep_z, REAL_FS)
  1368. _, _, _, pac_z = compute_pac_surrogate(ep_z, REAL_FS,
  1369. gamma_high=GAMMA_CAP)
  1370. m["PAC_z"] = pac_z
  1371. m["stage"] = stage_key
  1372. m["subject"] = subj_idx
  1373. real_rows.append(m)
  1374. n_ok += 1
  1375. print(f" Subject {subj_idx}: {n_ok} epochs")
  1376. except Exception as e:
  1377. print(f" Subject {subj_idx}: skip ({e})")
  1378. if not real_rows:
  1379. raise RuntimeError("No epochs extracted — check file contents.")
  1380. real_df = pd.DataFrame(real_rows)
  1381. print(f"\nTotal epochs: {len(real_df)}")
  1382. print(real_df.groupby("stage")[["H", "D", "M", "LZC", "PAC_z"]].mean().round(3))
  1383. # ── Per-subject H_eff calibration ───────────────────────────────────
  1384. print("\n[3] Per-subject H_eff calibration (KDE mode of Wake alpha_dfa)...")
  1385. real_df["H_eff"] = np.nan
  1386. H_opt_per_subj = {}
  1387. for subj in real_df["subject"].unique():
  1388. wake_a = real_df.loc[(real_df["subject"] == subj) &
  1389. (real_df["stage"] == "W"), "H"].values
  1390. H_opt_subj = (wake_mode_kde(wake_a)
  1391. if len(wake_a) >= 3 else
  1392. float(real_df.loc[real_df["stage"] == "W", "H"].median()))
  1393. H_opt_per_subj[subj] = H_opt_subj
  1394. mask = real_df["subject"] == subj
  1395. subj_range = float(real_df.loc[mask, "H"].max() -
  1396. real_df.loc[mask, "H"].min())
  1397. if subj_range < 0.02:
  1398. subj_range = float(real_df["H"].max() - real_df["H"].min())
  1399. real_df.loc[mask, "H_eff"] = real_df.loc[mask, "H"].apply(
  1400. lambda a: max(0.0, 1.0 - abs(a - H_opt_subj) / subj_range))
  1401. H_opt_real = float(np.median(list(H_opt_per_subj.values())))
  1402. print(f" Formula : range-normalised triangular")
  1403. print(f" Median per-subject H_opt (KDE mode of Wake) = {H_opt_real:.3f}")
  1404. print(f" Mean H_eff by stage: "
  1405. f"{real_df.groupby('stage')['H_eff'].mean().round(3).to_dict()}")
  1406. # Normalise with NORM_REF from the synthetic MC
  1407. for col, ncol in [("H_eff", "H_eff_n"), ("D", "D_n"), ("M", "M_n")]:
  1408. lo, hi = NORM_REF[col]
  1409. real_df[ncol] = np.clip((real_df[col].values - lo) /
  1410. (hi - lo + 1e-12), 0.0, 1.0)
  1411. real_df["PSI"] = w_H*real_df["H_eff_n"] + w_D*real_df["D_n"] + w_M*real_df["M_n"]
  1412. return real_df, H_opt_per_subj
  1413. # Run the pipeline — wrap in try/except so a bad DATA_DIR does not
  1414. # prevent earlier sections' figures from remaining visible in the notebook.
  1415. try:
  1416. real_df, H_opt_per_subj = run_sleep_edf_validation()
  1417. SLEEP_EDF_OK = True
  1418. except Exception as e:
  1419. print(f"\n[!] Sleep-EDF pipeline could not complete:\n{e}\n")
  1420. print(" Troubleshooting:")
  1421. print(" - If the error mentions 'mne': run the first cell of this")
  1422. print(" notebook to install dependencies (it includes mne).")
  1423. print(" - Otherwise check that DATA_DIR points to a folder containing")
  1424. print(" paired *-PSG.edf and *-Hypnogram.edf files.")
  1425. real_df = None
  1426. H_opt_per_subj = {}
  1427. SLEEP_EDF_OK = False
  1428. # %%
  1429. # ── Sleep-EDF stats + Fig (subject-level Psi with significance) ─────────
  1430. if SLEEP_EDF_OK and real_df is not None and len(real_df) > 0:
  1431. subj_means = (real_df.groupby(["subject", "stage"])["PSI"]
  1432. .mean().reset_index()
  1433. .rename(columns={"PSI": "PSI_subj"}))
  1434. piv = subj_means.pivot(index="subject", columns="stage", values="PSI_subj")
  1435. for s in ["W", "N2", "R"]:
  1436. if s not in piv.columns: piv[s] = np.nan
  1437. piv = piv[["W", "N2", "R"]]
  1438. print("Subject-level Psi per stage (rows = subjects):")
  1439. print(piv.round(3).to_string())
  1440. # Summary
  1441. subj_summary = {}
  1442. for s in ["W", "N2", "R"]:
  1443. vals = piv[s].dropna().values
  1444. subj_summary[s] = {
  1445. "mean": float(np.mean(vals)),
  1446. "sd": float(np.std(vals, ddof=1)) if len(vals) > 1 else 0.0,
  1447. "sem": (float(np.std(vals, ddof=1) / np.sqrt(len(vals)))
  1448. if len(vals) > 1 else 0.0),
  1449. "n": len(vals),
  1450. "vals": vals,
  1451. }
  1452. n_subj = len(piv)
  1453. print(f"\nSubject-level summary (n={n_subj}):")
  1454. for s, info in subj_summary.items():
  1455. print(f" {s}: mean = {info['mean']:.3f} SEM = {info['sem']:.3f} (SD={info['sd']:.3f})")
  1456. # Friedman + Wilcoxon (Bonferroni-corrected)
  1457. print("\nPaired-design statistics on subject means:")
  1458. stats_result = paired_stats_table(
  1459. piv=piv, stages=["W", "N2", "R"],
  1460. pairs=[("W", "N2"), ("W", "R"), ("N2", "R")],
  1461. label_fn=str, indent=" ",
  1462. )
  1463. pairwise_results = stats_result["pairs"]
  1464. # Ordering report
  1465. w_m = subj_summary["W"]["mean"]
  1466. n2_m = subj_summary["N2"]["mean"]
  1467. r_m = subj_summary["R"]["mean"]
  1468. print(f"\nOrdering: W({w_m:.3f}) vs N2({n2_m:.3f}) vs R({r_m:.3f})")
  1469. if r_m < n2_m:
  1470. print("NOTE: REM < N2 in Psi. Mechanistic explanation:")
  1471. print(" D ~ 0 for all real stages (no detectable θ-γ PAC in 2-channel data).")
  1472. print(" Psi is driven by H_eff and M.")
  1473. print(" REM alpha_dfa slightly higher than N2 → marginally lower H_eff.")
  1474. print(" Acknowledged as a limitation in the paper.")
  1475. # ── Fig : subject-level bar + paired-line scatter ──────────────
  1476. stage_labels = {"W": "Wake", "N2": "N2 sleep", "R": "REM sleep"}
  1477. stage_colors = {"W": "#4393C3", "N2": "#FDAE61", "R": "#66C2A5"}
  1478. stages_plot = ["W", "N2", "R"]
  1479. all_vals = [v for s in stages_plot for v in subj_summary[s]["vals"]]
  1480. ymin = max(0.0, min(all_vals) - 0.08) if all_vals else 0.0
  1481. ymax = min(1.0, max(all_vals) + 0.15) if all_vals else 1.0
  1482. fig7, (ax_bar, ax_box) = plt.subplots(1, 2, figsize=(11, 4.8),
  1483. constrained_layout=True)
  1484. # Left panel: mean + SEM
  1485. for xi, s in enumerate(stages_plot):
  1486. info = subj_summary[s]
  1487. ax_bar.bar(xi, info["mean"], color=stage_colors[s], alpha=0.88,
  1488. yerr=info["sem"], capsize=6,
  1489. error_kw=dict(elinewidth=1.2, capthick=1.2, ecolor="#222222"),
  1490. zorder=3)
  1491. ax_bar.text(xi, info["mean"] + info["sem"] + (ymax-ymin)*0.04,
  1492. f"{info['mean']:.3f}", ha="center", va="bottom", fontsize=9)
  1493. # Significance brackets (Bonferroni p<0.0167)
  1494. bon_thresh = 0.05 / 3
  1495. y_br_base = ymax - (ymax-ymin)*0.10
  1496. br_level = 0
  1497. for xi1, xi2, (s1, s2) in [(0, 1, ("W","N2")), (0, 2, ("W","R")), (1, 2, ("N2","R"))]:
  1498. res = pairwise_results.get((s1, s2), {})
  1499. if res.get("p", 1.0) < bon_thresh:
  1500. y = y_br_base + br_level * (ymax-ymin)*0.07
  1501. ax_bar.plot([xi1, xi1, xi2, xi2], [y-0.01, y, y, y-0.01],
  1502. color="#333333", lw=0.9)
  1503. sig = "***" if res["p"] < 0.001 else "**" if res["p"] < 0.01 else "*"
  1504. ax_bar.text((xi1+xi2)/2, y+0.005, sig, ha="center", va="bottom",
  1505. fontsize=10, fontweight="bold")
  1506. br_level += 1
  1507. ax_bar.set_xticks(range(len(stages_plot)))
  1508. ax_bar.set_xticklabels([stage_labels[s] for s in stages_plot], fontsize=10)
  1509. ax_bar.set_ylim(ymin, ymax)
  1510. ax_bar.set_ylabel(rf"$\Psi$ (subject-level mean ± SEM, N={n_subj})")
  1511. fr_chi2 = stats_result["friedman_chi2"]; fr_p = stats_result["friedman_p"]
  1512. if np.isfinite(fr_p):
  1513. fr_txt = f"Friedman chi2={fr_chi2:.2f}, p={fr_p:.3f}"
  1514. if fr_p >= 0.05: fr_txt += " (ns)"
  1515. else:
  1516. fr_txt = "Friedman: insufficient data"
  1517. ax_bar.set_title(r"$\Psi$ across vigilance states" + "\n" + fr_txt)
  1518. # Right panel: subject scatter + paired grey lines
  1519. rng_f = np.random.default_rng(77)
  1520. for xi, s in enumerate(stages_plot):
  1521. vals = subj_summary[s]["vals"]
  1522. jitter = rng_f.uniform(-0.12, 0.12, len(vals))
  1523. ax_box.scatter(np.full(len(vals), xi) + jitter, vals,
  1524. color=stage_colors[s], s=28, alpha=0.80, zorder=3)
  1525. ax_box.plot([xi - 0.22, xi + 0.22],
  1526. [subj_summary[s]["mean"]] * 2,
  1527. color=stage_colors[s], lw=3.0, zorder=4)
  1528. for subj in piv.index:
  1529. row = piv.loc[subj]
  1530. ys = [row.get(s, np.nan) for s in stages_plot]
  1531. if not any(np.isnan(ys)):
  1532. ax_box.plot(range(len(stages_plot)), ys,
  1533. color="#BBBBBB", lw=0.6, alpha=0.5, zorder=2)
  1534. ax_box.set_xticks(range(len(stages_plot)))
  1535. ax_box.set_xticklabels([stage_labels[s] for s in stages_plot], fontsize=10)
  1536. ax_box.set_ylim(ymin, ymax)
  1537. ax_box.set_ylabel(r"Subject mean $\Psi$")
  1538. ax_box.set_title(f"Individual subjects (N={n_subj}, dots = subject means)")
  1539. plt.show()
  1540. plt.close(fig7)
  1541. else:
  1542. print("Skipping Sleep-EDF stats & Fig 7 (data not available).")
  1543. pairwise_results = {}
  1544. # %%
  1545. # ── Fig : empirical benchmarking heatmap ─────────────────────────────────
  1546. if SLEEP_EDF_OK and real_df is not None and len(real_df) > 0:
  1547. print("\nBenchmarking Psi vs single-metric baselines on real EEG...")
  1548. bench_cols_emp = [
  1549. ("H_eff", "H_eff"),
  1550. ("D", "D"),
  1551. ("M", "M"),
  1552. ("Psi (composite)", "PSI"),
  1553. ("LZC", "LZC"),
  1554. ("Spectral slope", "spec_slope"),
  1555. ("Alpha power", "alpha_power"),
  1556. ]
  1557. auc_res = subject_auc_benchmark(
  1558. df=real_df, pairs=[("W","N2"), ("W","R"), ("N2","R")],
  1559. bench_cols=bench_cols_emp,
  1560. subject_col="subject", stage_col="stage",
  1561. n_boot=2000, seed=42, do_print=True, indent=" ",
  1562. )
  1563. # Component-contribution transparency (D and M tend to collapse with 2 ch)
  1564. print("\nComponent contribution to Psi per stage:")
  1565. agg = real_df.groupby("stage")[["H_eff_n", "D_n", "M_n", "PSI"]].mean()
  1566. for stage, row in agg.iterrows():
  1567. psi_val = row["PSI"]
  1568. if psi_val <= 0:
  1569. print(f" {stage}: Psi = {psi_val:.3f} (nonpositive)")
  1570. continue
  1571. parts = [("H_eff", w_H*row["H_eff_n"]/psi_val*100),
  1572. ("D", w_D*row["D_n"]/psi_val*100),
  1573. ("M", w_M*row["M_n"]/psi_val*100)]
  1574. flag = " [Psi ~ H_eff collapse]" if parts[0][1] > 90 else ""
  1575. print(f" {stage}: Psi={psi_val:.3f} " +
  1576. " ".join(f"{n}={p:4.0f}%" for n, p in parts) + flag)
  1577. # ── Fig heatmap ───────────────────────────────────────────────
  1578. pairs8 = [("W","N2"), ("W","R"), ("N2","R")]
  1579. cols8 = [("H_eff","H_eff"), ("D","D"), ("M","M"), ("PSI","PSI"),
  1580. ("LZC","LZC"), ("Slope","spec_slope"), ("Alpha","alpha_power")]
  1581. auc_mat = np.full((len(pairs8), len(cols8)), np.nan)
  1582. for pi, (s1, s2) in enumerate(pairs8):
  1583. for ci, (lbl, col) in enumerate(cols8):
  1584. entry = auc_res.get((s1, s2, col))
  1585. if entry is not None:
  1586. auc_mat[pi, ci] = entry[0]
  1587. row_lbl8 = ["Wake vs N2", "Wake vs REM", "N2 vs REM"]
  1588. col_lbl8 = [
  1589. r"$H_\mathrm{eff}$" + "\n(scale-free)",
  1590. r"$D$" + "\n(cross-freq.)",
  1591. r"$M$" + "\n(metastab.)",
  1592. r"$\Psi$", "LZC", "Sp. slope", r"$\alpha$ power",
  1593. ]
  1594. fig8, ax8 = plt.subplots(figsize=(9.0, 3.6), constrained_layout=True)
  1595. im8 = ax8.imshow(auc_mat, vmin=0.1, vmax=1.0, cmap="RdYlGn", aspect="auto")
  1596. for ri in range(len(pairs8)):
  1597. best_val = np.nanmax(auc_mat[ri])
  1598. for ci in range(len(cols8)):
  1599. v = auc_mat[ri, ci]
  1600. if np.isnan(v): continue
  1601. ax8.text(ci, ri, f"{v:.3f}", ha="center", va="center",
  1602. fontsize=9.5,
  1603. color="white" if (v < 0.28 or v > 0.88) else "black",
  1604. fontweight="bold" if v == best_val else "normal")
  1605. ax8.set_xticks(range(len(cols8)))
  1606. ax8.set_xticklabels(col_lbl8, fontsize=10)
  1607. ax8.set_yticks(range(len(pairs8)))
  1608. ax8.set_yticklabels(row_lbl8, fontsize=10)
  1609. cb8 = plt.colorbar(im8, ax=ax8, label="AUC", fraction=0.025, pad=0.03)
  1610. cb8.ax.tick_params(labelsize=9)
  1611. ax8.axvline(3.5, color="black", lw=1.5, ls="--", alpha=0.65)
  1612. ax8.annotate("Psi components", xy=(1.5, 3.4), fontsize=8.5,
  1613. color="#2166AC", ha="center", annotation_clip=False)
  1614. ax8.annotate("Single-metric baselines", xy=(5.0, 3.4), fontsize=8.5,
  1615. color="#555555", ha="center", annotation_clip=False)
  1616. n_subj_fig = real_df["subject"].nunique()
  1617. ax8.set_title(r"Empirical Benchmarking: $\Psi$ vs Single-metric Baselines" + "\n"
  1618. f"Real Sleep-EDF (N={n_subj_fig} subjects; bold = best per row)",
  1619. fontsize=10)
  1620. plt.show()
  1621. plt.close(fig8)
  1622. else:
  1623. print("Skipping Fig (Sleep-EDF data not available).")

Ugail-Howard-Consciousness-Index_Validation_updated.ipynb at commit 89ad301, no license · at the source

Overview

Authors: Hassan Ugail1, Newton Howard2
ORCID iDs: Hassan Ugail
  1. Centre for Visual Computing and Intelligent Systems, University of Bradford,Bradford, UK
  2. School of Individualized Study, Rochester Institute of Technology,Rochester, USA
Institutions: University of Bradford (United Kingdom); Rochester Institute of Technology (United States)
Journal: Biological cybernetics, volume 120, issue 3-4, article 20
Dates: received 23 November 2025; accepted 22 June 2026; published online 13 July 2026; in print 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1007/s00422-026-01049-1 · PMID 42439951 · PMCID PMC13364888 · OpenAlex W7168188724
Open access: hybrid, a free copy (OpenAlex)
Status: code verified
Categories: EEG (modality), human (organism), cognitive (subfield)
Methods: Spectral & time-frequency, Connectivity, Statistics, Complexity, Preprocessing, Evoked potentials
MeSH: Brain*, Consciousness*, Models, Neurological*, Electroencephalography, Humans, Sleep, Wakefulness (* major topic)
Topic: EEG and Brain-Computer Interfaces (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Citations: not cited yet (Europe PMC); 52 references in the paper

Abstract

Quantifying consciousness from brain activity remains a major challenge in neuroscience and clinical practice. Many existing EEG measures focus on a single feature of neural activity, such as complexity, synchrony, or spectral structure, but no single feature appears sufficient across different brain states. We introduce a composite dynamical framework that combines three complementary properties of brain activity, i.e., scale-free temporal organisation, cross-frequency organisation, and metastable flexibility in large-scale synchronisation. These components are normalised and combined into a single index designed to capture organised dynamical complexity rather than raw signal complexity alone. We test the framework in both synthetic and empirical settings. In a generative model of nine EEG-like brain states, including wakefulness, dreaming, anaesthesia, non-conscious states, and seizure states, the index separates the synthetic conscious and non-conscious classes without overlap and remains stable across ablation, sensitivity, and Monte Carlo analyses. We then apply the framework to two-channel Sleep-EDF recordings from 30 healthy adults, where it provides a proof-of-principle subject-level separation of wakefulness from N2 and REM sleep. The framework is dynamical-systems-inspired and is not committed to any single theory of consciousness, making it compatible with a range of theoretical perspectives. With further validation, the framework may be applicable across multichannel brain recordings, including anaesthesia, disorders of consciousness, and basic consciousness-research settings.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 25 matches between paragraphs and lines of code.

ugail/Index-for-Consciousness-Dynamics

License: none: the authors keep all their rights
State: the link answers, verified on 27 September 2026
Evidence: files inventoried
Commit: 89ad301bf5b457920b457ff4d58f96c41512299c, 13 July 2026
Languages: Jupyter (3)
Size: 4 files, 3 scripts
Software Heritage: not archived
Found in: “Data Availability”
Holds: README, 3 notebooks
Not found: license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: Matplotlib (3 files), NumPy (3 files), pandas (3 files), SciPy (3 files), NetworkX (2 files), MNE-Python (1 file), scikit-learn (1 file)
Availability: 1 check, the latest on 27 September 2026: the link answers
  • 27 September 2026: the link answers
4 files

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 3 scripts, each with its path and the digest of its content;
  • 25 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data Availability

The code to compute the consciousness index Ψ Ψ, including the feature pipeline, the synthetic EEG generator, and validation scripts, is publicly available in a dedicated GitHub repository: https://github.com/ugail/Index-for-Consciousness-Dynamics. The real EEG recordings used for validation were obtained from the publicly available Sleep-EDF “Expanded” dataset, hosted on PhysioNet (https://physionet.org/content/sleep-edfx/). This dataset is distributed under the Open PhysioNet License. For full reproducibility and independent verification, the code for all experiments, including the ablation, sensitivity, Monte Carlo, and subject-level Sleep-EDF validation analyses, is available in the GitHub repository. The released code includes the full state-wise simulator configuration used for all synthetic analyses, including band weights, coupling constants, PAC strengths, noise parameters, slow-drift settings, and seizure-burst parameters.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 2, 28 September 2026

  • Publisher: n/a → Springer Science+Business Media

Version 1, 27 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 2 authors, 7 MeSH terms, 50 references.

Cite

This paper

Ugail, H., & Howard, N. (2026). A three-component dynamical index of consciousness-related neural organisation. Biological cybernetics, 120(3-4), 20. https://doi.org/10.1007/s00422-026-01049-1

BibTeX

@article{ugail2026three,
author = {Ugail, Hassan and Howard, Newton},
title = {{A three-component dynamical index of consciousness-related neural organisation}},
journal = {Biological cybernetics},
year = {2026},
month = jul,
volume = {120},
number = {3-4},
pages = {20},
publisher = {Springer Science+Business Media},
issn = {0340-1200},
doi = {10.1007/s00422-026-01049-1},
url = {https://doi.org/10.1007/s00422-026-01049-1},
pmid = {42439951},
pmcid = {PMC13364888}
}

RIS

TY - JOUR
AU - Ugail, Hassan
AU - Howard, Newton
TI - A three-component dynamical index of consciousness-related neural organisation
T2 - Biological cybernetics
J2 - Biol Cybern
PY - 2026
DA - 2026/07/13
VL - 120
IS - 3-4
SP - 20
SN - 0340-1200
PB - Springer Science+Business Media
DO - 10.1007/s00422-026-01049-1
UR - https://doi.org/10.1007/s00422-026-01049-1
LA - en
ER -

CSL-JSON

{
"id": "10.1007/s00422-026-01049-1",
"type": "article-journal",
"title": "A three-component dynamical index of consciousness-related neural organisation",
"container-title": "Biological cybernetics",
"author": [
{
"family": "Ugail",
"given": "Hassan"
},
{
"family": "Howard",
"given": "Newton"
}
],
"container-title-short": "Biol Cybern",
"volume": "120",
"issue": "3-4",
"page": "20",
"DOI": "10.1007/s00422-026-01049-1",
"PMID": "42439951",
"PMCID": "PMC13364888",
"ISSN": "0340-1200",
"publisher": "Springer Science+Business Media",
"URL": "https://doi.org/10.1007/s00422-026-01049-1",
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
13
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1016/j.celrep.2026.117782 [code]
Thermodynamics of consciousness: Non-equilibrium brain dynamics track conscious states.
Journal: Cell reports
In common: pandas, SciPy, Matplotlib, 1 other tool, EEG, cognitive, 6 references
[2] doi:10.1038/s41597-026-07350-9 [code]
An open multi-center MEG-EEG dataset for studying conscious visual perception.
Journal: Scientific data
In common: MNE-Python, scikit-learn, pandas, 3 other tools, EEG, 4 references
[3] doi:10.1093/nc/niag029 [code]
A data-driven approach to identifying and evaluating connectivity-based neural correlates of conscious visual perception.
Journal: Neuroscience of consciousness
In common: MNE-Python, NetworkX, scikit-learn, 4 other tools, cognitive, 3 references
[4] doi:10.3389/fncom.2026.1786996 [code]
Schumann-anchored golden ratio organization of human neural oscillations.
Journal: Frontiers in computational neuroscience
In common: MNE-Python, NetworkX, scikit-learn, 4 other tools, EEG, 2 references
[5] doi:10.1371/journal.pone.0348005 [code]
THOI: An efficient and accessible library for computing higher-order interactions enhanced by batch-processing.
Journal: PloS one
In common: NetworkX, scikit-learn, pandas, 3 other tools, 3 references
[6] doi:10.1038/s41593-026-02205-3 [code]
Competitive interactions shape mammalian brain network dynamics and computation.
Journal: Nature neuroscience
In common: NetworkX, scikit-learn, pandas, 3 other tools, 3 references
[7] doi:10.7554/elife.100605 [code]
Age-related changes in ‘cortical’ 1/f dynamics are linked to cardiac activity
Journal: n/a
In common: MNE-Python, NetworkX, scikit-learn, 4 other tools, 2 references
[8] doi:10.3390/s26103065 [code]
Subject-Wise Depression Screening from Eight-Channel Resting-State EEG Using Asymmetry-Aware Spectral Features and Connectivity Ablation.
Journal: Sensors (Basel, Switzerland)
In common: scikit-learn, pandas, SciPy, 2 other tools, EEG, author Hassan Ugail
[9] doi:10.1371/journal.pone.0351872 [code]
Decoding visual object recognition from EEG signals.
Journal: PloS one
In common: MNE-Python, scikit-learn, pandas, 3 other tools, EEG, 2 references
[10] doi:10.1038/s41593-026-02285-1 [code]
Fixation duration on natural scenes is explained by memory encoding not processing demand.
Journal: Nature neuroscience
In common: MNE-Python, scikit-learn, pandas, 3 other tools, cognitive, 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.