PyLossless: A non-destructive EEG processing pipeline.
The 18 matches · 1 of them tie a paragraph to a whole file, not to given lines: a weak match, whose lines are not tinted
- [1] § Materials and methods › Pipeline steps › Flag bridged sensors ↔ notebooks/pipeline_algorithms.ipynb, lines 446–527 · score 0.89 · flagging bridged sensors, high median correlation, inter quantile range, low IQR, bridging threshold, bridge indicator
- [2] § Materials and methods › Pipeline steps › Filtering ↔ examples/plot_0_implementation.py, lines 394–404 · score 0.84 · low frequency drifts, 1–100 Hz, notch filter, ICLabel, trained, classifier
- [3] § Materials and methods › Pipeline steps › Flag uncorrelated sensors ↔ notebooks/pipeline_algorithms.ipynb, lines 558–642 · score 0.77 · lower quantile range, low correlation, rejecting sensors, define epoch, sensors dimension, uncorrelated
- [4] § Materials and methods › Pipeline steps › Notation ↔ examples/plot_0_implementation.py, lines 1–53 · score 0.75 · lowercase letters, capital letters, subscripts, superscripts, scalars, denote
- [5] § Materials and methods › Pipeline steps › Flag noisy sensors ↔ pylossless/pipeline.py, lines 1034–1063 · score 0.71 · voltage variance, detect outlier, standard deviation, Flag noisy, outlying, threshold
- [6] § Materials and methods › Pipeline steps › Flag uncorrelated epochs ↔ pylossless/pipeline.py, lines 1418–1498 · score 0.65 · flag uncorrelated epochs, high impedance, neighboring, thresholds, EEG
- [7] § Materials and methods › Pipeline steps › Initial ICA and flagging of outlying independent component (IC) ↔ pylossless/pipeline.py, lines 1418–1498 · score 0.64 · flag_noisy_ics, flag noisy epochs, ICLabel, ICA, EEG, pipeline
- [8] § Materials and methods › Pipeline steps › Applying the PyLossless decisions ↔ pylossless/config/rejection.py, lines 13–56 · score 0.63 · RejectionPolicy, subtract, brain, confidence, EOG, class
- [9] § Materials and methods › Pipeline steps › Flag noisy sensors ↔ examples/plot_0_implementation.py, lines 210–225 · score 0.59 · right tail, spread, multiply, UQR, median, deviation
- [10] § Materials and methods › Pipeline steps › Saving the pipeline output ↔ pylossless/flagging.py, lines 252–330 · score 0.59 · MNE ICALabel, independent component, TSV, derivatives, root, preprocessing
- [11] § Materials and methods › Pipeline steps › Initial ICA and flagging of outlying independent component (IC) ↔ examples/plot_0_implementation.py, lines 394–404 · score 0.58 · ICLabel, ICA decompositions, flag noisy, classifier, epochs, pipeline
- [12] § Materials and methods › Pipeline steps › Flag bridged sensors ↔ pylossless/pipeline.py, lines 1136–1165 · score 0.56 · flagging bridged, bridged channel, IQR, median, correlation, pipeline
- [13] § Materials and methods › Licence and dependencies ↔ pylossless/datasets/datasets.py, the whole file · a weak match · score 0.56 · optional dependency, MNE BIDS, installation, PyLossless, EEG, pipeline
- [14] § Materials and methods › Pipeline steps › Saving the pipeline output ↔ pylossless/dash/qcgui.py, lines 48–87 · score 0.55 · project_root, PyLossless pipeline, derivatives, BIDS, MNE, Python
- [15] § Materials and methods › Pipeline steps › Final ICA and IC classification ↔ pylossless/flagging.py, lines 252–330 · score 0.55 · MNE ICALabel, independent component, classifier, ICLabel
- [16] § Materials and methods › Pipeline steps › Notation ↔ examples/plot_0_implementation.py, lines 1–53 · score 0.54 · single sensor, single epoch, denote, Notation, matrices, dimension
- [17] § Materials and methods › Pipeline steps › Flag the rank sensor ↔ notebooks/pipeline_algorithms.ipynb, lines 536–556 · score 0.54 · highest median, correlation coefficient, good sensors, rank, flags, channels
- [18] § Materials and methods › Pipeline steps › Flag noisy time periods ↔ examples/plot_0_implementation.py, lines 227–246 · score 0.53 · matrix indicates, standard deviation, vectors, outlier, thresholds, sensors
Paper
Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC
The paper is loaded when this pane is shown.
The authors' code
Python · 750 lines · 25 KB · MIT · 6 matches
- r"""
- PyLossless Algorithms
- =====================
- This tutorial explains the calculations that PyLossless performs at each step of the
- pipeline. We will use example EEG data to demonstrate the
- calculations.
- .. note::
- You can open this notebook in
- `Google Colab <https://colab.research.google.com/drive/1ecyNo10oFgpbVNuD7Ztgr2fs8XkpfOYo?usp=sharing>`_!
- .. _notation:
- Notation
- --------
- Before we begin, we define some notation that will be used throughout the text:
- - We start with a 3D matrix of EEG data,
- :math:`X \in \mathbb{R}^{S_\mathcal{G} \times E_\mathcal{G} \times T}`,
- where :math:`S_\mathcal{G}` and :math:`E_\mathcal{G}` are the sets of good sensors and
- epochs, respectively, and :math:`T`, is the number of samples(i.e. time-points).
- - ``s``, ``e``, and ``t`` are sensor, epochs, and samples, respectively.
- - We use superscripts to denote operations across a dimension, and we use subscripts to
- denote indexing a dimension.
- - We refer to a single sensor :math:`i` as
- :math:`X\big|_{s=i} \in \mathbb{R}^{E_\mathcal{G} \times T}`,
- with :math:`i \in S_\mathcal{G}`.
- - We refer to a single epoch :math:`j` as
- :math:`X\big|_{e=j} \in \mathbb{R}^{S_\mathcal{G} \times T}`,
- with :math:`j \in E_\mathcal{G}`.
- - We denote sensor-specific thresholds for rejecting epochs as
- :math:`\tau^e_i \in \mathbb{R}^{S_\mathcal{G}}`
- - We denote epoch-specific thresholds for rejecting sensors as
- :math:`\tau^s_j \in \mathbb{R}^{E_\mathcal{G}}`
- - We denote *quantiles* as :math:`Q\#^{dim}`: i.e. :math:`Q75^s` is the 75th *quantile*
- along the sensor dimension. The function :math:`Q75^s(X)` computes the 75th quantile
- along the :math:`s` dimension of matrix :math:`X`, resulting in a matrix noted
- :math:`X^{Q75^s} \in \mathbb{R}^{E \times T}`.
- Throughout the text, we use capital letters for matrices and lowercase letters for
- scalars. For example, the data point for sensor :math:`i`, epoch :math:`j`, and
- time :math:`k` is denoted as :math:`X\big|_{s=i; e=j; t=k} = x_{ijk} \in \mathbb{R}`,
- and :math:`X=\{x_{ijk}\}`.
- """
- # %%
- # Imports and data loading
- # ------------------------
- from pathlib import Path
- import numpy as np
- import mne
- from mne.datasets import sample
- import pylossless as ll
- # Load example mne data
- raw = ll.datasets.load_simulated_raw()
- # %%
- # Load a PyLossless configuration file
- # ------------------------------------
- # Let's load a PyLossless configuration file. This file contains the parameters that
- # will be used for each step of the pipeline. For example, the ``noisy_channels``
- # section contains the parameters for the :ref:`noisy_sensors` step. We can modify
- # these parameters to change the behavior of the pipeline. For example, we can change
- # the percent of epochs that a sensor must be noisy for it to be flagged via the
- # ``flag_crit`` parameter.
- config = ll.config.Config()
- config.load_default()
- config["noisy_channels"]["outliers_kwargs"]["lower"] = 0.25 # lower quantile
- config["noisy_channels"]["outliers_kwargs"]["upper"] = 0.75 # upper quantile
- config["noisy_channels"]["flag_crit"] = 0.30 # percent of epochs that a sensor must be noisy
- config.save("test_config.yaml")
- # %%
- # Create a pipeline instance
- # --------------------------
- pipeline = ll.LosslessPipeline("test_config.yaml")
- pipeline.raw = raw
- raw.plot()
- # %%
- # Input Data
- # ----------
- #
- # First, we epoch the data to be used for subsequent steps.
- # Let our 3D matrix below be defined as :math:`X \in \mathbb{R}^{S \times E \times T}`
- # where :math:`X` is a matrix of real numbers and of dimension :math:`S` sensors
- # :math:`\times$ E` epochs `\times T` times.
- #
- epochs = pipeline.get_epochs()
- # %%
- #
- # Let's convert our epochs object into a named :class:`xarray.DataArray` object.
- from pylossless.pipeline import epochs_to_xr
- #
- epochs_xr = epochs_to_xr(epochs, kind="ch")
- epochs_xr # 277 epochs, 50 sensors, 602 samples per epoch
- # %%
- # .. _robust_reference:
- #
- # Robust Average Reference
- # ------------------------
- #
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/robust_rereference.png
- # :align: center
- # :alt: Robust Average Reference graphic.
- #
- # Robust Average Reference. The figure shows the steps for robust average referencing.
- # See the text below for descriptions of mathematical notation.
- #
- # Before the pipeline can begin, we must average reference the data. This is because
- # the pipeline uses data distributions to identify noisy sensors, and For EEG data that
- # uses an online reference to a single electrode, sensors that are further from the
- # reference will have a higher voltage variance, and the pipeline will be biased to
- # flag these sensors as noisy. The average reference, which subtracts the average
- # signal across sensors from each individual sensor, will ensure an even playing field.
- # Howeer, we dont want to include noisy sensors in the average reference signal. So we
- # will identify noisy sensors and and leave them out of the average reference signal.
- # %%
- sample_std = epochs_xr.std("time")
- q25_ch = sample_std.quantile(0.25, dim="ch")
- q50_ch = sample_std.median(dim="ch")
- q75_ch = sample_std.quantile(0.75, dim="ch")
- ch_dist = sample_std - q50_ch # center the data
- ch_dist /= q75_ch - q25_ch # shape (chans, epoch)
- mean_ch_dist = ch_dist.mean(dim="epoch") # shape (chans)
- # find the median and 25 and 75 percentiles
- # of the mean of the channel distributions
- mdn = np.median(mean_ch_dist)
- deviation = np.diff(np.quantile(mean_ch_dist, [0.25, 0.75]))
- leave_out = mean_ch_dist.ch[mean_ch_dist > mdn + 6 * deviation].values.tolist()
- leave_out
- # %%
- ref_chans = [ch for ch in epochs.pick("eeg").ch_names if ch not in leave_out]
- pipeline.raw.set_eeg_reference(ref_channels=ref_chans)
- # %%
- #
- # .. _noisy_sensors:
- #
- # Flag Noisy Sensors
- # ------------------
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/Flag_noisy_sensors.png
- # :align: center
- # :alt: Flag Noisy Sensors graphic.
- #
- # Flag Noisy Sensors. The figure shows the steps for flagging noisy sensors. See the text below
- # for descriptions of mathematical notation.
- #
- # %%
- # Since we applied a robust average reference to the raw data, we will need to re-epoch
- # the data:
- epochs = pipeline.get_epochs()
- epochs_xr = epochs_to_xr(epochs, kind="ch")
- # First we take standard deviation of
- # :math:`X \in \mathbb{R}^{S \times E \times T}` across the samples dimension :math:`t`
- # resulting in a 2D matrix :math:`X^{\sigma_{t}} \in \mathbb{R}^{S \times E}`
- trim_ch_sd = epochs_xr.std("time")
- trim_ch_sd
- # %%
- # a) Take the 50th and 75th quantile across dimension sensor of :math:`X^{\sigma_{t}}`
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # This operation results in two 1D vectors of size :math:`E`:
- #
- # .. math::
- # X^{{\sigma}_t{Q50^s}} = Q50^s(X^{\sigma_{t}}) \in \mathbb{R}^{E}
- # .. math::
- # X^{{\sigma}_t{Q75^s}} = Q75^s(X^{\sigma_{t}}) \in \mathbb{R}^{E}
- # %%
- q50, q75 = trim_ch_sd.quantile([0.5, 0.75], dim="ch")
- q50 # a 1D array of median standard deviation values across channels for each epoch
- # %%
- # b) Define an Upper Quantile Range as :math:`Q75 - Q50`
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # .. math::
- # UQR^s = X^{{\sigma}_T{Q75}^s} - X^{{\sigma}_T{Q50}^s}
- #
- # This operation results in a 1D vector of size :math:`E`.
- uqr = q75 - q50
- uqr
- # %%
- # c) Identify outlier Indices :math:`(i, j)`
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # We multiply a constant :math:`k` by the :math:`UQR` to define a measure for the
- # spread of the right tail of the distribution of :math:`X^{\sigma_{t}}` values and
- # add it to the median of :math:`X^{\sigma_{t}}` to obtain epoch-specific standard
- # deviation threshold for outliers:
- #
- # .. math::
- # \tau^s_j = X^{{\sigma}_T{Q50}^S} + UQR^s\times k
- #
- # That is, :math:`\tau^s_j` is the epoch-specific threshold for the epoch :math:`j`
- k = 3
- upper_threshold = q50 + q75 * k
- upper_threshold # epoch specific thresholds
- # %%
- # Now, we compare our 2D standard deviation matrix to the threshold vector of
- # :math:`\tau^e_j`:
- #
- # .. math::
- # X^{\sigma_{t}} \big|_{e=j} > \tau^s_j
- #
- # resulting in the indicator matrix :math:`C \in \{0, 1\}^{S \times E}=\{c_{ij}\}`:
- #
- # .. math::
- # c_{ij} =
- # \begin{cases}
- # 0 & \text{if } x^{\sigma_{t}}_{ij} < \tau^s_j \\
- # 1 & \text{if } x^{\sigma_{t}}_{ij} \geq \tau^s_j
- # \end{cases}
- #
- # Each element of this matrix indicates whether sensor :math:`i` is an outlier at an epoch
- # :math:`j`.
- outlier_mask = trim_ch_sd > upper_threshold
- outlier_mask
- # %%
- # d) Identify noisy sensors part 1
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # To identify outlier sensors, we average across the epoch dimension of our indicator
- # matrix :math:`C` and obtain :math:`C^{\mu_e} \in \mathbb{R}^{S_\mathcal{G}}`, which
- # is a vector of fractional numbers :math:`c^{\mu_e}_i` representing the percentage of
- # epochs for which that sensor is an outlier.
- percent_outliers = outlier_mask.astype(float).mean("epoch")
- percent_outliers # percent of epochs that sensor is an outlier
- # %%
- # e) Identify noisy sensors part 2
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Next, we define a threshold :math:`\tau^{p}` (:math:`p` for percentile;
- # default, ``.20``) as a cutoff point for determining if a sensor should be marked
- # artifactual. The sensor :math:`i` is flagged as noisy if
- # :math:`c^{\mu_e}_i > \tau^{p}`. That is, if the sensor is an outlier for more than
- # :math:`\tau^{p}` percent of the epochs, it is flagged as noisy.
- p_threshold = config["noisy_channels"]["flag_crit"] # 0.3, or 30%
- noisy_chs = percent_outliers[percent_outliers > p_threshold].coords.to_index().values
- noisy_chs
- # %%
- # f) Add the noisy sensors to the pipeline flags
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Let's add the noisy sensors to the pipeline flags.
- pipeline.flags["ch"].add_flag_cat(kind="noisy", bad_ch_names=noisy_chs)
- pipeline.raw.info["bads"].extend(pipeline.flags["ch"]["noisy"].tolist())
- pipeline.flags["ch"]
- # %%
- #
- # .. _noisy_epochs:
- #
- # Flag Noisy Epochs
- # -----------------
- #
- # This step closely resembles the :ref:`noisy_sensors` step. For the sake of brevity
- # we will be more concise in the documentation.
- # %%
- #
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/Flag_noisy_epochs.png
- # :align: center
- #
- # Flag Noisy Epochs. The figure shows the steps for flagging noisy epochs. See the text below
- # for descriptions of mathematical notation.
- #
- # %%
- # a) Take standard deviation across the samples dimension :math:`t`
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Take a moment below to notice that the sensors flagged in the prior setp are not in
- # ``epochs_xr`` below:
- epochs = pipeline.get_epochs()
- # Let's make our epochs array into a named Array
- epochs_xr = epochs_to_xr(epochs, kind="ch")
- trim_ch_sd = epochs_xr.std("time")
- trim_ch_sd.coords["ch"]
- # %%
- # b) Compute 50th and 75th quantile across epochs and the UQR
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Like before, We Take the median and 70th quantile, but now we operate across epochs,
- # resulting in two 1D vector's of size ``n_good_sensors`` :math:`S_\mathcal{G}`
- #
- # .. math::
- # X^{{\sigma}_t{Q50^e}} = Q50^e(X^{\sigma_{t}}) \in \mathbb{R}^{S_\mathcal{G}}
- # .. math::
- # X^{{\sigma}_t{Q75^e}} = Q75^e(X^{\sigma_{t}}) \in \mathbb{R}^{S_\mathcal{G}}
- # .. math::
- # UQR^e = (X^{{\sigma}_T{Q75}^e} - X^{{\sigma}_T{Q50}^e})
- # %%
- # c) Define sensor-specific thresholds for rejecting epochs :math:`\tau^e_i`
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Our sensor-specifc threshold for rejecting epochs is defined by:
- #
- # .. math::
- # \tau^e_i = X^{{\sigma}_T{Q50}^e} + UQR^e\times k
- q50, q75 = trim_ch_sd.quantile([0.5, 0.75], dim="epoch")
- uqr_epoch = q75 - q50
- uqr_epoch
- # %%
- k = 8
- upper_threshold = q50 + uqr_epoch * k
- upper_threshold
- # %%
- # d) Identify Outlier indices
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # The indicator matrix is defined by:
- #
- # .. math::
- # c_{ij} =
- # \begin{cases}
- # 0 & \text{if } x^{\sigma_{t}}_{ij} < \tau^e_i \\
- # 1 & \text{if } x^{\sigma_{t}}_{ij} \geq \tau^e_i
- # \end{cases}
- #
- #
- # To identify outlier **epochs**, we average across the **sensor** dimension of our
- # indicator matrix :math:`C` and obtain
- # :math:`C^{\mu_s} \in \mathbb{R}^{E_\mathcal{G}}`, which is a vector of numbers
- # :math:`c^{\mu_s}_j` representing the percentage of **sensors** for which that epoch
- # is an outlier.
- outlier_mask = trim_ch_sd > upper_threshold
- outlier_mask
- # %%
- percent_outliers = outlier_mask.astype(float).mean("ch")
- percent_outliers
- # %%
- # e) Identify noisy epochs
- # ^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Next, we define a fractional threshold :math:`\tau^{p}` as a cutoff point for
- # determining if a epoch should be marked artifactual. The epoch :math:`j` is flagged
- # as noisy if :math:`c^{\mu_s}_j > \tau^{p}`.
- bad_epochs = percent_outliers[percent_outliers > p_threshold].coords.to_index().values
- bad_epochs
- # %%
- # f) Add the noisy epochs to the pipeline flags
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # Let's add the outlier epochs to our flags
- # These will be added directly as :class:`mne.Annotations` to the raw data.
- pipeline.flags["epoch"].add_flag_cat(
- kind="noisy", bad_epoch_inds=bad_epochs, epochs=epochs
- )
- pipeline.raw.annotations.description
- # %%
- pipeline.raw.plot()
- # %%
- # Filtering
- # ---------
- #
- # After flagging noisy sensors and epochs, we filter the data. By default,
- # The pipeline uses a 1-100Hz bandpass filter. This is because 1), ICA decompositions
- # are more stable when low frequency drifts are removed, and 2) the ICLabel classifier
- # is trained on data that has been filtered between 1-100Hz. A notch filter can also be
- # optionally specified.
- pipeline.config["filtering"]["notch_filter_args"]["freqs"] = [50]
- pipeline.filter()
- # %%
- # Find Nearest Neighbours & return Maximum Correlation
- # ----------------------------------------------------
- #
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/Nearest_neighbors.png
- # :align: center
- # :alt: Nearest Neighbors graphic.
- #
- # Nearest Neighbors. The figure shows the steps for finding nearest neighbors. See the text below
- # for descriptions of mathematical notation.
- #
- # %%
- # Whereas :ref:`noisy_sensors` and :ref:`noisy_epochs` operated on a 2D matrix of
- # standard deviation values, The next few steps will operate on correlation
- # coefficients. Here we describe the procedure for defining the 2D matrix of correlation
- # coefficients.
- # %%
- from pylossless.pipeline import chan_neighbour_r
- # %%
- #
- # Notice that our flagged epochs are dropped.
- epochs = pipeline.get_epochs()
- # %%
- # a) Calculate Correlation Coefficients between each Sensor and its neighboring eighbors
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # - For each good sensor $i$ in :math:`S_{\mathcal{G}}`, we select its :math:`N` nearest
- # neighbors. I.e. the :math:`N` sensors that are closest to it.
- #
- # - We call the sensor :math:`i` the *origin*, and its nearest neighbors :math`\hat{s_l}`
- # with :math:`l \in \{1, 2, \ldots, N\}`
- #
- # - Then, for each epoch :math:`j`, we calculate the correlation coefficient
- # :math:`\rho^t_{(i,\hat{s_l}),j}` between origin sensor :math:`i` and each neighbor
- # :math:`\hat{s_l}` across dimension :math:`t` (samples), returning a 3D matrice of
- # correlation coefficients:
- #
- # .. math::
- # \mathrm{P}^t = \{\rho^t_{(i, \hat{s_l}),j}\} \in \mathbb{R}^{S_G \times E_G \times n}
- #
- # Finally, we select the maximum correlation coefficient across the neighbor dimension
- # :math:`n`:
- #
- # .. math::
- # \mathrm{P}^{t,{\text{max}}^n}= \max\limits_{\hat{s_l}} \rho^t_{(i, \hat{s_l}),j}
- #
- # Returning a 2D matrix where each value at :math:`(i, j)` is the maximum correlation
- # coefficient between sensor :math:`i` and its :math:`N` nearest neighbors, at each epoch
- # :math:`j`
- # %%
- data_r_ch = chan_neighbour_r(epochs, nneigbr=3, method="max")
- # maximum correlation out of correlations between ch and its 3 neighbors
- data_r_ch
- # %%
- # This matrix :math:`\mathrm{P}^{t,{\text{max}}^n}` will be used in the steps below.
- # %%
- # Flag Bridged Sensors
- # --------------------
- #
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/Flag_bridged_sensors.png
- # :align: center
- # :alt: Flag Bridged Sensors graphic.
- #
- # Flag Bridged Sensors. The figure shows the steps for flagging bridged sensors.
- # See the text below for descriptions of mathematical notation.
- #
- # %%
- # a) Calculate the 50th, 75th quantile and IQR across epochs
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # .. math::
- # IQR^e = \mathrm{P}^{t,{\text{max}}^nQ75^e} - \mathrm{P}^{t,{\text{max}}^nQ25^e}
- #
- # For each sensor, divide the median across epochs by the IQR across epochs. Bridged
- # channels should have a high median correlation but a low IQR of the correlation.
- # We call this measure the bridge-indicator.
- #
- # .. math::
- # \mathcal{B}_s = \frac{\mathrm{P}^{t,{\text{max}}^nQ50^e}}{IQR^e}
- #
- # b) Define a bridging threshold
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- # Now, take the 25th, 50th, and 75th quantile of :math:`\mathcal{B}_s` across sensors,
- # And calculate the :math:`IQR^s`. A channel :math:`i` is bridged if
- #
- # .. math::
- # \mathcal{B}_i > B^{Q50^s} +k \times IQR^s
- #
- # %%
- import scipy
- from functools import partial
- # %%
- msr = data_r_ch.median("epoch") / data_r_ch.reduce(scipy.stats.iqr, dim="epoch")
- # msr is a 1D vector of size n_sensors
- config_trim = 40
- config_bridge_z = 6
- #
- trim = config_trim
- if trim >= 1:
- trim /= 100
- trim /= 2 # .20 and will be used as (.20, .20)
- #
- trim_mean = partial(scipy.stats.mstats.trimmed_mean, limits=(trim, trim))
- trim_std = partial(scipy.stats.mstats.trimmed_std, limits=(trim, trim))
- #
- z_val = config_bridge_z # 6
- mask = msr > msr.reduce(trim_mean, dim="ch") + z_val * msr.reduce(
- trim_std, dim="ch"
- ) # bridged chans
- #
- bridged_ch_names = data_r_ch.ch.values[mask]
- bridged_ch_names
- # %%
- # Let's add the outlier channels to our flags
- bad_chs = bridged_ch_names
- pipeline.flags["ch"].add_flag_cat(kind="bridged", bad_ch_names=bad_chs)
- pipeline.flags["ch"]
- # %%
- # Identify the Rank Channel
- # -------------------------
- #
- # Because the pipeline uses an average reference before the ICA decomposition, it is
- # necessary to account for rank deficiency (i.e., every sensor in the montage is
- # linearly dependent on the other channels due to the common average reference). To
- # account for this, the pipeline flags the sensor (out of the remaining good sensors)
- # with the highest median of the max correlation coefficient with its neighbors
- # (across epochs):
- #
- # .. math::
- # \begin{equation}
- # i = \text{arg}\max\limits_i \rho_{i}^{t,{\text{max}}^n,median^j}
- # \end{equation}
- #
- # This sensor has the least unique time-series out of the remaining set of good sensors
- # :math:`S_\mathcal{G}` and is flagged by the pipeline as ``”rank”``. Note that this
- # sensor is not flagged because it contains artifact, but only because one of the
- # remaining sensors needs to be removed to address rank deficiency before ICA
- # decomposition is performed. By choosing this sensor, we are likely to lose little
- # information because of its high correlation with its neighbors. This sensor can be
- # reintroduced after the ICA has been applied for artifact corrections.
- # %%
- good_chs = [
- ch for ch in data_r_ch.ch.values if ch not in pipeline.flags["ch"].get_flagged()
- ]
- data_r_ch_good = data_r_ch.sel(ch=good_chs)
- flag_ch = [str(data_r_ch_good.median("epoch").idxmax(dim="ch").to_numpy())]
- pipeline.flags["ch"].add_flag_cat(kind="rank", bad_ch_names=flag_ch)
- pipeline.flags["ch"]
- # %%
- # Flag low correlation Epochs
- # ---------------------------
- #
- # This step is designed to identify time periods in which many sensors are
- # uncorrelated with neighboring sensors. It is similar to the :ref:`noisy_sensors` step,
- #
- # Again we calculate the 25th and 50th quantile
- # of :math:`\mathrm{P}^{t,{\text{max}}^n}`, across the epochs dimension, and calculate
- # the lower quantile range :math:`LQR^s`. This results in vectors
- # :math:`\mathrm{P}^{t,{\text{max}}^nQ25^e}` and
- # :math:`\mathrm{P}^{t,{\text{max}}^nQ50^e}` of size :math:`S_\mathcal{G}`. As for previous
- # steps, we define sensor-specific thresholds for flagging epochs:
- #
- # .. math::
- # \begin{equation}
- # \tau^e = \mathrm{P}^{t,{\text{max}}^nQ50^e} - LQR^e\times k
- # \end{equation}
- #
- # And the corresponding indicator matrix:
- #
- # .. math::
- # \begin{equation}
- # c_{ij} =
- # \begin{cases}
- # 1 & \text{if } \rho^{t,{\text{max}}^n}_{ij} < \tau^e_i \\
- # 0 & \text{if } \rho^{t,{\text{max}}^n}_{ij} \geq \tau^e_i
- # \end{cases}
- # \end{equation}
- #
- # We average the indicator matrix across sensors and obtain a vector :math:`C^{\mu_s}`
- # that we use to flag uncorrelated epochs using the following criterion:
- #
- # .. math::
- # c^{\mu_e}_i > \tau^{p}.
- #
- # %%
- # Step a
- q25, q50 = data_r_ch.quantile([0.25, 0.5], dim="epoch")
- #
- # Define the LQR
- lqr = q50 - q25
- #
- # define a threshold
- k = 3
- lower_threshold = q50 - lqr * k
- #
- outlier_mask = data_r_ch < lower_threshold
- #
- percent_outliers = outlier_mask.astype(float).mean("ch")
- #
- p_threshold = 0.2
- bad_epochs = percent_outliers[percent_outliers > p_threshold].coords.to_index().values
- #
- # Add the outlier epochs to our flags
- pipeline.flags["epoch"].add_flag_cat(
- kind="uncorrelated", bad_epoch_inds=bad_epochs, epochs=epochs
- )
- pipeline.raw.annotations.description
- # %%
- # in this case, no epochs were flagged as uncorrelated.
- #
- # %%
- # Flag low correlation Sensors
- # -----------------------------
- #
- # .. figure:: https://raw.githubusercontent.com/scott-huberty/wip_pipeline-figures/main/Flag_uncorrelated_sensors.png
- # :align: center
- # :alt: Flag Uncorrelated Sensors graphic.
- #
- # Flag Uncorrelated Sensors. The figure shows the steps for flagging uncorrelated
- # sensors. See the text below for descriptions of mathematical notation.
- #
- # This step is designed to identify sensors that have an unusually low correlation with
- # neighboring sensors. The operations involved by this step are similar to those of the
- # :ref:`noisy_sensors` step, except we use maximal nearest neighbor correlations instead
- # of dispersion and the left instead of the right tail of the distribution to set
- # the threshold for outliers.
- # %%
- # a) Take lower quantile range and defined sensor-specific thresholds
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # We get the indicator matrix as described previously, using
- #
- # .. math::
- # \tau^e_i = \mathrm{P}^{t,{\text{max}}^nQ50^e} - LQR^e\times k
- #
- # and
- #
- # .. math::
- # c_{ij} =
- # \begin{cases}
- # 1 & \text{if } \rho^{t,{\text{max}}^n}_{ij} < \tau^e_i \\
- # 0 & \text{if } \rho^{t,{\text{max}}^n}_{ij} \geq \tau^e_i
- # \end{cases}
- # %%
- # b) Identify uncorrelated sensors
- # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- #
- # We define a threshold as we did in the previous step and flag uncorrelated epochs
- # :math:`j` if :math:`c^{\mu_s}_j > \tau^{p}`.
- # %%
- q25, q50 = data_r_ch.quantile([0.25, 0.5], dim="ch")
- #
- # Define LQR
- lqr = q50 - q25
- #
- # define a threshold
- k = 3
- lower_threshold = q50 - lqr * k
- #
- # Identify correlations less than the threshold
- outlier_mask = data_r_ch < lower_threshold
- percent_outliers = outlier_mask.astype(float).mean("epoch")
- #
- p_threshold = 0.2
- bad_chs = percent_outliers[percent_outliers > p_threshold].coords.to_index().values
- #
- # Add the outlier channels to our flags
- pipeline.flags["ch"].add_flag_cat(kind="uncorrelated", bad_ch_names=bad_chs)
- pipeline.flags["ch"]
- # %%
- # In this case, no sensors were flagged as uncorrelated.
- # %%
- # Run Initial ICA
- # ---------------
- #
- # The pipeline by runs ICA two times. The first ICA is only used to identify
- # noisy periods in its IC activation time-series. For this reason, the pipeline
- # uses the FastICA algorithm for speed.
- # %%
- pipeline.run_ica("run1")
- # %%
- # Flag Noisy IC Activation time-periods
- # -------------------------------------
- #
- # This step follows the same procedure as the :ref:`noisy_sensors` step, except that
- # the data is now the IC activation time-series. thus we start with a 3D matrix
- # :math:`X_{ica} \in \mathbb{R}^{I_\mathcal{G} \times E_\mathcal{G} \times T}` of
- # IC time-courses rather than scalp EEG data and where :math:`I` is the set of
- # independent components.
- #
- # %%
- pipeline.flag_noisy_ics()
- # %%
- pipeline.raw.annotations.description
- # %%
- # Run Final ICA
- # -------------
- #
- # Now The pipeline runs the final ICA decomposition, this time using the extended
- # Infomax algorithm. Note that any sensors or time-periods that have been flagged
- # up to this point will not be passed into the ICA decomposition. For the sake of
- # time, we will not run the second ICA here, as there are no more pipeline calculations.
- # %%
- # Run ICLabel Classifier
- # ----------------------
- #
- # The pipeline will run the ICLabel classifier on the final ICA, which will produce a
- # label for each IC, one of ``"brain"``, ``"muscle"``, ``"eog"`` (eye), ``"ecg"``
- # (heart), ``line_noise``, or ``"channel_noise"``.
- #
- #
- # Conclusion
- # ----------
- # And that's all! See the other pylossless tutorials for brief examples on running the
- # pipeline on your own data, and rejecting the flagged data.
- #
plot_0_implementation.py at commit 0f4bccf, under MIT · at the source
Overview
- Montreal Neurological Institute-Hospital, McGill University,Montreal, Canada
- Compute Ontario,St. Catharines, Canada
- Department of Computer Science and Engineering, University of South Carolina,Columbia, SC USA
- Artificial Intelligence Institute, University of South Carolina,Columbia, SC USA
- Carolina Autism and Neurodevelopment Research Center, Columbia, SC USA
- Institute for Mind and Brain, University of South Carolina,Columbia, SC USA
Abstract
EEG recordings are typically long and contain large amounts of data, making manual cleaning a time-consuming and error-prone task. Automated preprocessing pipelines can facilitate the efficient and objective extraction of artifacts, enabling standardized and reproducible analyses. However, automated preprocessing pipelines typically remove data considered artifacts and return a subset of irreversibly transformed signals. This approach obfuscates preprocessing decisions and often makes it impossible to recover the original data or modify the preprocessing steps. Further, it complicates collaboration among research teams working on a common dataset, as different analyses may require specific preprocessing steps. Given the large amount of resources devoted to collecting EEG, tools that can efficiently and transparently preprocess data are greatly needed. PyLossless addresses this need by creating a non-destructive, automated preprocessing pipeline that maintains the continuous EEG structure. It offers a user-friendly API, is well documented, tested through continuous integration, easily deployable, and integrates with the popular MNE-Python environment. The pipeline also provides a browser-based quality control review (QCR) dashboard that allows researchers to visualize and edit automated artifact flags for sensors, time periods, and independent components. The end product of PyLossless is a lossless annotated data state that can be shared and used with analysis-specific artifact rejection policies, allowing for an optimal balance between flexibility and standardization.
Supplementary Information: The online version contains supplementary material available at 10.3758/
Reproduced under the paper's license (CC BY), from the paper cited above.
Repository
Its files are read in the Code ↔ Paper reader above, with 18 matches between paragraphs and lines of code.
lina-usc/pylossless
0f4bccfe2657984a5363be80ef29f00bd3c1a07a, 3 September 2026Availability: 1 check, the latest on 27 September 2026: the link answers
- 27 September 2026: the link answers
45 files
- docs/
source/ , Python, 129 linesconf.py - examples/
plot_0_implementation.py , Python, 750 lines, 6 matches - examples/
plot_10_run_pipeline.py , Python, 147 lines - examples/
usage.py , Python, 22 lines - notebooks/
pipeline_algorithms.ipyn , Jupyter, 828 lines, 3 matchesb - notebooks/
qc_example.ipynb , Jupyter, 25 lines - pylossless/
__init__.py , Python, 4 lines - pylossless/
_logging.py , Python, 69 lines - pylossless/
bids.py , Python, 151 lines - pylossless/
config/ , Python, 2 lines__init__.py - pylossless/
config/ , Python, 66 linesconfig.py - pylossless/
config/ , Python, 173 lines, 1 matchrejection.py - pylossless/
conftest.py , Python, 87 lines - pylossless/
dash/ , Python, 23 lines__init__.py - pylossless/
dash/ , Python, 25 linesapp.py - pylossless/
dash/ , Python, 160 linescss_defaults.py - pylossless/
dash/ , Python, 645 linesmne_visualizer.py - pylossless/
dash/ , Python, 47 linespylossless_qc.py - pylossless/
dash/ , Python, 307 linesqcannotations.py - pylossless/
dash/ , Python, 482 lines, 1 matchqcgui.py - pylossless/
dash/ , Python, 1 linetests/ __init__.py - pylossless/
dash/ , Python, 18 linestests/ conftest.py - pylossless/
dash/ , Python, 25 linestests/ test_mne_visualizer.py - pylossless/
dash/ , Python, 24 linestests/ test_qcannotations.py - pylossless/
dash/ , Python, 84 linestests/ test_topo_viz.py - pylossless/
dash/ , Python, 845 linestopo_viz.py - pylossless/
dash/ , Python, 24 linesutils.py - pylossless/
datasets/ , Python, 2 lines__init__.py - pylossless/
datasets/ , Python, 85 lines, 1 matchdatasets.py - pylossless/
datasets/ , Python, 127 linessimulated.py - pylossless/
flagging.py , Python, 330 lines, 2 matches - pylossless/
pipeline.py , Python, 1,608 lines, 4 matches - pylossless/
tests/ , Python, 1 linetest_bids.py - pylossless/
tests/ , Python, 243 linestest_pipeline.py - pylossless/
tests/ , Python, 141 linestest_random_seed.py - pylossless/
tests/ , Python, 53 linestest_rejection.py - pylossless/
tests/ , Python, 63 linestest_simulated.py - pylossless/
tests/ , Python, 59 linestest_utils.py - pylossless/
utils/ , Python, 3 lines__init__.py - pylossless/
utils/ , Python, 40 lines_utils.py - pylossless/
utils/ , Python, 96 linescheck.py - pylossless/
utils/ , Python, 34 lineshtml.py - setup.py, Python, 48 lines
- LICENSE, License, 21 lines
- README.md, Text, 121 lines
Code Availability
Materials and analysis code are available at https://
Reproduced under the paper's license (CC BY), from the paper cited above.
Tracing map
Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.
What the map holds:
- 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
- 43 scripts, each with its path and the digest of its content;
- 18 matches between paragraphs of the paper and lines of the code (method lexical-v1);
- neither the text of the paper nor the code itself.
Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.
Data
Datasets cited
- bids-specification.readt
hedocs.io , at bids-specification.readthedocs.io; found in the references - github.com/
andesha/ , at github.com; found in “Availability of Data and Materials”face13
Availability of Data and Materials
The EEG data used for this analysis is available at https://
Reproduced under the paper's license (CC BY), from the paper cited above.
Versions
The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.
Version 2, 28 September 2026
- Publisher: n/a → Springer Science+Business Media
Version 1, 27 September 2026: the first record
Recorded: type, language, journal, volume, issue, pages, dates, 5 authors, 5 keywords, 5 MeSH terms, 1 funder, 19 references.
Cite
This paper
Huberty, S., Desjardins, J., Collins, T., Elsabbagh, M., & O’Reilly, C. (2026). PyLossless: A non-destructive EEG processing pipeline. Behavior research methods, 58(8), 220. https://
BibTeX
@article{huberty2026pylo
author = {Huberty, Scott and Desjardins, James and Collins, Tyler and Elsabbagh, Mayada and O’Reilly, Christian},
title = {{PyLossless: A non-destructive EEG processing pipeline}},
journal = {Behavior research methods},
year = {2026},
month = jul,
volume = {58},
number = {8},
pages = {220},
publisher = {Springer Science+Business Media},
issn = {1554-351X},
doi = {10.3758/
url = {https://
pmid = {42410267},
pmcid = {PMC13337736}
}
RIS
TY - JOUR
AU - Huberty, Scott
AU - Desjardins, James
AU - Collins, Tyler
AU - Elsabbagh, Mayada
AU - O’Reilly, Christian
TI - PyLossless: A non-destructive EEG processing pipeline
T2 - Behavior research methods
J2 - Behav Res Methods
PY - 2026
DA - 2026/
VL - 58
IS - 8
SP - 220
SN - 1554-351X
PB - Springer Science+Business Media
DO - 10.3758/
UR - https://
LA - en
ER -
CSL-JSON
{
"id": "10.3758/
"type": "article-journal",
"title": "PyLossless: A non-destructive EEG processing pipeline",
"container-title": "Behavior research methods",
"author": [
{
"family": "Huberty",
"given": "Scott"
},
{
"family": "Desjardins",
"given": "James"
},
{
"family": "Collins",
"given": "Tyler"
},
{
"family": "Elsabbagh",
"given": "Mayada"
},
{
"family": "O’Reilly",
"given": "Christian"
}
],
"container-title-short":
"volume": "58",
"issue": "8",
"page": "220",
"DOI": "10.3758/
"PMID": "42410267",
"PMCID": "PMC13337736",
"ISSN": "1554-351X",
"publisher": "Springer Science+Business Media",
"URL": "https://
"language": "en",
"issued": {
"date-parts": [
[
2026,
7,
6
]
]
}
}
The tracing map gets a citation of its own once an author has validated it and it has a DOI.
Similar papers
The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.
- [1] doi:10.1038/s41597-025-05174-7 [code]
- A large-scale MEG and EEG dataset for object recognition in naturalistic scenesJournal: n/aIn common: MNE-BIDS, ICLabel, MNE-Python, 4 other tools, methods / tools, EEG, 4 references
- [2] doi:10.1097/j.pain.0000000000004044 [code]
- No effect of rhythmic visual stimulation on experimental pain perception.Journal: PainIn common: MNE-BIDS, ICLabel, MNE-Python, 3 other tools, EEG, 5 references
- [3] doi:10.1162/imag.a.1269 [code]
- From early to contemporary normative modeling: Mapping individual differences in neurophysiological signals.Journal: Imaging neuroscience (Cambridge, Mass.)In common: xarray, ICLabel, MNE-Python, 4 other tools, EEG, 3 references
- [4] doi:10.1038/s41597-026-07350-9 [code]
- An open multi-center MEG-EEG dataset for studying conscious visual perception.Journal: Scientific dataIn common: MNE-BIDS, xarray, MNE-Python, 3 other tools, EEG, 2 references
- [5] doi:10.1093/cercor/bhag113 [code]
- Long-term reliability and stability of parameterized resting state EEG: evidence from a five-year follow-up.Journal: Cerebral cortex (New York, N.Y. : 1991)In common: MNE-BIDS, ICLabel, MNE-Python, 3 other tools, methods / tools, EEG, 2 references
- [6] doi:10.3390/s26134019 [code]
- NeuroStat: An Open-Source EEG Connectivity Platform for Randomised Controlled Trials.Journal: Sensors (Basel, Switzerland)In common: ICLabel, MNE-Python, pandas, 2 other tools, EEG, 4 references
- [7] doi:10.1093/nc/niag029 [code]
- A data-driven approach to identifying and evaluating connectivity-based neural correlates of conscious visual perception.Journal: Neuroscience of consciousnessIn common: MNE-BIDS, xarray, MNE-Python, 3 other tools, 2 references
- [8] doi:10.1038/s41597-026-07146-x [code]
- Intention-Action Conflict EEG-Hand Kinematics Dataset for Unimanual Control under Congruent and Incongruent Conditions.Journal: Scientific dataIn common: ICLabel, MNE-Python, pandas, 2 other tools, methods / tools, EEG, 2 references
- [9] doi:10.1016/j.dib.2026.113064 [code]
- A reproducible EEG hyperscanning dataset for triadic social decision-making during an iterated 3-player Prisoner's Dilemma.Journal: Data in briefIn common: ICLabel, MNE-Python, pandas, 2 other tools, methods / tools, EEG, 2 references
- [10] doi:10.1371/journal.pcbi.1014043 [code]
- EEG-Pype: An accessible MNE-Python pipeline with graphical user interface for preprocessing and analysis of resting-state electroencephalography data.Journal: PLoS computational biologyIn common: ICLabel, MNE-Python, pandas, 2 other tools, methods / tools, EEG, 2 references
Contribute
The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.
Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.
Claim this paper
Correct its record
Say what each link of this record is, remove the ones that are not the paper's, add the ones that are missing. The correction becomes a new version of the record, in its Versions section.
Validate its tracing map
You validate the map as this page shows it: 1 repository of the authors' code, each at its verified commit and with its license, 43 scripts, and 18 matches between paragraphs and code (see the Code and Map sections). It then receives a DOI on Zenodo, with you (your ORCID iD) and OSCR as its creators; the code itself is not deposited.
The map's fingerprint: sha256:2ececc24ff557b65…
Add the badge to its README
The badge links the code to this page. Copy one of these into the README of the paper's code: only you decide where it goes, and nothing is changed for you.
Markdown
[, paste the snippet at the top, then “Commit changes…” and, to review it first, “Create a new branch and start a pull request”. You open the pull request; OSCR asks for no permission.
Request its removal
To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).
Discussion, reproductions, activity
Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.
Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.
Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.
