OSCR

High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework.

Code ↔ Paper

2 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 2 matches
  1. [1] § Materials and methods › Linguistic feature adaptor training and ablation test ↔ codes/models.py, lines 14–100 · score 0.75 · positional encoding, decoder layers, encoder layers, heads, linear, token
  2. [2] § Materials and methods › Linguistic feature adaptor training and ablation test ↔ codes/codes/reconstruction.py, lines 138–179 · score 0.52 · weight decay, L2, Adam, optimizer, model, trained

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 133 lines · 5.2 KB · GPL-3.0 · 1 match

  1. class PositionalEncoding(nn.Module):
  2. def __init__(self, d_model, max_len=5000):
  3. super().__init__()
  4. pe = torch.zeros(max_len, d_model)
  5. position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
  6. div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
  7. pe[:, 0::2] = torch.sin(position * div_term)
  8. pe[:, 1::2] = torch.cos(position * div_term)
  9. self.register_buffer('pe', pe.unsqueeze(0))
  11. def forward(self, x):
  12. return x + self.pe[:, :x.size(1)]
  14. class Seq2SeqTransformer(nn.Module):
  15. def __init__(self, input_dim=42, output_dim=1024, d_model=256, nhead=8,
  16. num_encoder_layers=3, num_decoder_layers=3, dim_feedforward=1024):
  17. super().__init__()
  18. self.d_model = d_model
  19. self.output_dim = output_dim
  21. self.encoder_embed = nn.Linear(input_dim, d_model)
  22. self.decoder_embed = nn.Linear(output_dim, d_model)
  23. self.pos_encoder = PositionalEncoding(d_model)
  24. self.pos_decoder = PositionalEncoding(d_model)
  26. self.transformer = nn.Transformer(
  27. d_model=d_model,
  28. nhead=nhead,
  29. num_encoder_layers=num_encoder_layers,
  30. num_decoder_layers=num_decoder_layers,
  31. dim_feedforward=dim_feedforward,
  32. batch_first=True,
  33. )
  34. self.fc_out = nn.Linear(d_model, output_dim)
  36. # Enhanced length prediction head
  37. self.length_head = nn.Sequential(
  38. nn.Linear(d_model, d_model),
  39. nn.SiLU(),
  40. nn.Linear(d_model, d_model),
  41. nn.SiLU(),
  42. nn.Linear(d_model, 1)
  43. )
  45. def generate_square_subsequent_mask(self, sz):
  46. return torch.triu(torch.full((sz, sz), float('-inf')), diagonal=1)
  48. def forward(self, src, tgt=None, max_len=50, is_inference=False):
  49. #print('src.shape',src.shape)
  50. src = self.encoder_embed(src) * math.sqrt(self.d_model)
  51. src = self.pos_encoder(src)
  52. #print('src.shape',src.shape)
  53. if is_inference:
  54. return self._inference_forward(src, max_len)
  56. tgt = self.decoder_embed(tgt) * math.sqrt(self.d_model)
  57. tgt = self.pos_decoder(tgt)
  58. sz = tgt.size(1)
  59. tgt_mask = self.generate_square_subsequent_mask(sz).to(tgt.device)
  60. output = self.transformer(src=src, tgt=tgt, tgt_mask=tgt_mask)
  62. output_tokens = self.fc_out(output)
  63. # Predict length from mean of encoder output
  64. output_length = F.softplus(self.length_head(src.mean(dim=1)))
  66. return {
  67. 'tokens': output_tokens,
  68. 'length': output_length.squeeze(-1)
  69. }
  71. def _inference_forward(self, src, max_len):
  72. memory = self.transformer.encoder(src)
  73. batch_size = src.size(0)
  75. # Predict sequence length first
  76. #length_embed = self.length_head(memory.mean(dim=1))
  77. length_embed = self.length_head(src.mean(dim=1))
  78. pred_length = int(F.softplus(length_embed).round().item())
  79. pred_length = min(max(1, pred_length), max_len) # Clamp to valid range
  81. # Generate sequence based on predicted length
  82. tgt = torch.zeros(batch_size, 1, self.output_dim).to(src.device)
  83. output_tokens = []
  85. for _ in range(pred_length):
  86. tgt_embed = self.decoder_embed(tgt) * math.sqrt(self.d_model)
  87. tgt_embed = self.pos_decoder(tgt_embed)
  89. output = self.transformer.decoder(
  90. tgt_embed,
  91. memory,
  92. tgt_mask=self.generate_square_subsequent_mask(tgt.size(1)).to(src.device)
  93. )
  95. next_token = self.fc_out(output[:, -1:, :])
  96. output_tokens.append(next_token)
  97. tgt = torch.cat([tgt, next_token], dim=1)
  99. output_tokens = torch.cat(output_tokens, dim=1)
  100. return output_tokens, pred_length
  101. class DynamicSequenceLoss(nn.Module):
  102. def __init__(self, token_weight=1, length_weight=1):
  103. super().__init__()
  104. self.token_weight = token_weight
  105. self.length_weight = length_weight
  106. self.token_loss = nn.KLDivLoss(reduction='batchmean')
  107. self.length_loss = nn.HuberLoss()
  109. def forward(self, preds, targets):
  110. # 对预测值取log_softmax（KL散度要求）
  111. pred_log_probs = F.log_softmax(preds['tokens'], dim=-1)
  112. # 确保目标是有效的概率分布
  113. target_probs = F.softmax(targets['tokens'], dim=-1)
  114. # Token-level KL散度损失
  115. token_loss = self.token_loss(
  116. pred_log_probs, # 输入需要是log probabilities
  117. target_probs # 目标需要是probabilities
  118. )
  120. # Length prediction loss (保持不变)
  121. length_loss = self.length_loss(
  122. preds['length'],
  123. targets['length']
  124. )
  126. total_loss = (self.token_weight * token_loss +
  127. self.length_weight * length_loss)
  129. return {
  130. 'total': total_loss,
  131. 'token': token_loss,
  132. 'length': length_loss
  133. }

models.py at commit 0c19bbc, under GPL-3.0 · at the source

Overview

Authors: Jiawei Li1,2, Chunxu Guo1, Chao Zhang3,4, Edward F Chang5, Yuanning Li1,2,4,6,7
ORCID iDs: Edward F Chang, Yuanning Li
  1. School of Biomedical Engineering, ShanghaiTech University Shanghai China
  2. State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University Shanghai China
  3. Department of Electronic Engineering, Tsinghua University Beijing China
  4. Shanghai Artificial Intelligence Laboratory Shanghai China
  5. Department of Neurological Surgery, University of California, San Francisco San Francisco United States
  6. Shanghai Clinical Research and Trial Center Shanghai China
  7. Lin Gang Laboratory Shanghai China
Institutions: ShanghaiTech University (China); Tsinghua University (China); Shanghai Artificial Intelligence Laboratory (China); University of California, San Francisco (United States); Shanghai Clinical Research Center (China)
Journal: eLife, volume 14, article RP109400
Dates: published online 5 March 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.7554/elife.109400 · PMID 41784218 · PMCID PMC12962650 · OpenAlex W4417400061
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: intracranial EEG (iEEG / ECoG / SEEG) (modality), human (organism), cognitive (subfield)
Methods: Connectivity, Statistics, Machine learning, Spectral & time-frequency, Preprocessing
Keywords: electrocorticography, speech decoding, brain-computer interfaces, auditory cortex, brain-to-speech, deep neural networks, Human
MeSH: Brain-Computer Interfaces*, Linguistics*, Speech*, Electrocorticography, Generative Adversarial Networks, Humans, Long Short Term Memory, Speech Intelligibility (* major topic)
Journal subjects: Neuroscience
Topic: EEG and Brain-Computer Interfaces (Cognitive Neuroscience, Neuroscience), according to OpenAlex
Funding: National Natural Science Foundation of China (32371154); National Science and Technology Major Project (2025ZD0217000); Science and Technology Commission of Shanghai Municipality (24QA2705500); Lin Gang Laboratory (LG-GG-202402-06, LGL-1987-18)
Citations: cited by 1 paper (Europe PMC); 48 references in the paper

Abstract

Reconstructing speech from neural recordings is crucial for understanding human speech coding and developing brain-computer interfaces (BCIs). However, existing methods trade off acoustic richness (pitch, prosody) for linguistic intelligibility (words, phonemes). To overcome this limitation, we propose a dual-path framework to concurrently decode acoustic and linguistic representations. The acoustic pathway uses a long-short term memory (LSTM) decoder and a high-fidelity generative adversarial network (HiFi-GAN) to reconstruct spectrotemporal features. The linguistic pathway employs a transformer adaptor and text-to-speech (TTS) generator for word tokens. These two pathways merge via voice cloning to combine both acoustic and linguistic validity. Using only 20 min of electrocorticography (ECoG) data per human subject, our approach achieves highly intelligible synthesized speech (mean opinion score = 4.0/5.0, word error rate = 18.9%). Our dual-path framework reconstructs natural and intelligible speech from ECoG, resolving the acoustic-linguistic trade-off.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 2 matches between paragraphs and lines of code.

CCTN-BCI/Neural2Speech2

License: GPL-3.0
State: the link answers, verified on 30 September 2026
Evidence: files inventoried
Commit: 0c19bbc4e393ac76f565ac2059d66bdb6850a777, 16 May 2026
Languages: Python (15)
Size: 143 files, 15 scripts
Software Heritage: not archived
Found in: “Data availability”
Holds: README, license file
Not found: CITATION.cff, environment file, tests, continuous integration, documentation
Tools: PyTorch (6 files), NumPy (3 files), SciPy (3 files), Hugging Face Transformers (3 files), PyTorch Lightning (2 files), Matplotlib (1 file)
Availability: 1 check, the latest on 30 September 2026: the link answers
  • 30 September 2026: the link answers
17 files
At the source: github.com/CCTN-BCI/Neural2Speech2

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 15 scripts, each with its path and the digest of its content;
  • 2 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability

The data that support the findings of this study are available on request from the lead contact. The data are not publicly available because they could compromise research participant privacy and consent. All original code and preprocessed anonymized data to replicate the main findings of this study can be found at https://github.com/CCTN-BCI/Neural2Speech2, copy archived at CCTN-BCI, 2026. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 30 September 2026: the first record

Recorded: type, language, journal, volume, pages, dates, 5 authors, 7 keywords, 8 MeSH terms, 4 funders, 37 references.

Cite

This paper

Li, J., Guo, C., Zhang, C., Chang, E. F., & Li, Y. (2026). High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework. eLife, 14, RP109400. https://doi.org/10.7554/elife.109400

BibTeX

@article{li2026high,
author = {Li, Jiawei and Guo, Chunxu and Zhang, Chao and Chang, Edward F and Li, Yuanning},
title = {{High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework}},
journal = {eLife},
year = {2026},
month = mar,
volume = {14},
pages = {RP109400},
publisher = {eLife Sciences Publications, Ltd},
issn = {2050-084X},
doi = {10.7554/elife.109400},
url = {https://doi.org/10.7554/elife.109400},
pmid = {41784218},
pmcid = {PMC12962650}
}

RIS

TY - JOUR
AU - Li, Jiawei
AU - Guo, Chunxu
AU - Zhang, Chao
AU - Chang, Edward F
AU - Li, Yuanning
TI - High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework
T2 - eLife
J2 - Elife
PY - 2026
DA - 2026/03/05
VL - 14
SP - RP109400
SN - 2050-084X
PB - eLife Sciences Publications, Ltd
DO - 10.7554/elife.109400
UR - https://doi.org/10.7554/elife.109400
LA - en
ER -

CSL-JSON

{
"id": "10.7554/elife.109400",
"type": "article-journal",
"title": "High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework",
"container-title": "eLife",
"author": [
{
"family": "Li",
"given": "Jiawei"
},
{
"family": "Guo",
"given": "Chunxu"
},
{
"family": "Zhang",
"given": "Chao"
},
{
"family": "Chang",
"given": "Edward F"
},
{
"family": "Li",
"given": "Yuanning"
}
],
"container-title-short": "Elife",
"volume": "14",
"page": "RP109400",
"DOI": "10.7554/elife.109400",
"PMID": "41784218",
"PMCID": "PMC12962650",
"ISSN": "2050-084X",
"publisher": "eLife Sciences Publications, Ltd",
"URL": "https://doi.org/10.7554/elife.109400",
"language": "en",
"issued": {
"date-parts": [
[
2026,
3,
5
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1038/s41586-026-10691-5 [code]
Mapping the neuronal building blocks of human language with language models.
Journal: Nature
In common: SciPy, Matplotlib, NumPy, cognitive, 8 references
[2] doi:10.1038/s41467-026-75455-1 [code]
Shared latent representations of speech production for cross-patient speech decoding.
Journal: Nature communications
In common: PyTorch Lightning, PyTorch, SciPy, 2 other tools, intracranial EEG (iEEG / ECoG / SEEG), cognitive, 4 references
[3] doi:10.1162/imag.a.1227 [code]
Large language models reveal the neural tracking of linguistic context in attended and unattended multi-talker speech.
Journal: Imaging neuroscience (Cambridge, Mass.)
In common: Hugging Face Transformers, PyTorch, SciPy, 2 other tools, intracranial EEG (iEEG / ECoG / SEEG), cognitive, 4 references
[4] doi:10.1038/s41597-026-07248-6 [code]
A large-scale fMRI dataset for vision-language semantic association.
Journal: Scientific data
In common: PyTorch, SciPy, Matplotlib, 1 other tool, 5 references
[5] doi:10.1038/s41467-026-71284-4 [code]
Encoding of speech modes and loudness in ventral precentral gyrus.
Journal: Nature communications
In common: SciPy, Matplotlib, NumPy, cognitive, 5 references
[6] doi:10.1162/nol.a.244 [code]
A Novel Approach to Map the Causal Impact of Brain Stimulation on Semantic Processing With Language Models.
Journal: Neurobiology of language (Cambridge, Mass.)
In common: Hugging Face Transformers, PyTorch, SciPy, 2 other tools, 4 references
[7] doi:10.7554/elife.106543 [code]
Stimulus dependencies-rather than next-word prediction-can explain pre-onset brain encoding in naturalistic listening designs.
Journal: eLife
In common: Hugging Face Transformers, PyTorch, SciPy, 2 other tools, cognitive, 3 references
[8] doi:10.1038/s41593-026-02258-4 [code]
Laminar organization of cellular microcircuits modulating human interictal epileptiform discharges.
Journal: Nature neuroscience
In common: SciPy, Matplotlib, NumPy, 2 references, author Edward F Chang
[9] doi:10.1038/s41467-026-72253-7 [code]
Spurious alignment between large language models and brains can emerge from non-robust methods and overlooked confounds.
Journal: Nature communications
In common: Hugging Face Transformers, PyTorch, SciPy, 2 other tools, 3 references
[10] doi:10.1038/s41467-026-73947-8 [code]
Frontal cortex organization supporting audiovisual processing during naturalistic viewing.
Journal: Nature communications
In common: PyTorch, SciPy, Matplotlib, 1 other tool, intracranial EEG (iEEG / ECoG / SEEG), cognitive, 3 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.