OSCR

Comparative genomics reveals contrasting candidate adaptive signatures in two ecologically similar plateau fishes.

Code ↔ Paper

25 matches between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

18 blocks of code here have a permanent identifier (oscr:b:…), listed with the matches below: citable for good, with a page of its own. Select a match and press Cite to copy it. What a block identifier is.

The 25 matches
  1. [1] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/setup_mcmctree.py, lines 1 to 35 · score 0.94 · soft bound, Cyprinidae crown, 28–50, 49–150, 5–25, calibrated tree · block oscr:b:E1RSKYADK6VZ6-M · Report a mismatch
  2. [2] § Results › Phylogenomic framework and evolutionary placement of the two focal species ↔ scripts/render_revised_figures_v03.py, lines 451 to 511 · score 0.85 · Schizopygopsis younghusbandi, Oxygymnocypris stewartii, Danio rerio, Gymnocypris eckloni, root age, Diptychus maculatus · block oscr:b:60DVB9CGFY6B3-4 · Report a mismatch
  3. [3] § Results › Phylogenomic framework and evolutionary placement of the two focal species ↔ Notebooks/3-ultimate-Visualization4OrthoFinder.qmd, lines 960 to 1022 · score 0.83 · Schizopygopsis younghusbandi, Oxygymnocypris stewartii, Danio rerio, Gymnocypris eckloni, Diptychus maculatus, Aspiorhynchus laticeps · block oscr:b:DCDSBKDA29H3X-O · Report a mismatch
  4. [4] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/setup_mcmctree.py, lines 1 to 35 · score 0.82 · Branch length Hessians, MCMCTree, approximate likelihood, posterior, topology, PAML · block oscr:b:E1RSKYADK6VZ6-M · Report a mismatch
  5. [5] § Material and methods › Genome datasets and gene model preparation ↔ Notebooks/0-ultimate-preprocessing.qmd, lines 684 to 758 · score 0.78 · OrthoDB, protein database, masked genome, protein evidence, Gene prediction, GFF3 · block oscr:b:TGZPH76BRAE7Y-R · Report a mismatch
  6. [6] § Material and methods › Corrected fourfold degenerate transversion analysis ↔ scripts/run_4dtv_analysis.py, lines 1 to 48 · score 0.75 · HKY correction, degenerate sites, codon positions, transversions, Gene pairs, DTv · block oscr:b:V5XX2SPAK0YX3-N · Report a mismatch
  7. [7] § Results › Phylogenomic framework and evolutionary placement of the two focal species ↔ scripts/plot_calibrated_phylogeny.R, lines 1 to 31 · score 0.74 · ultrafast bootstrap, supermatrix ML, node age, root age, gene duplications, Phylogenomic · block oscr:b:R3RBEEQC3JV3P-9 · Report a mismatch
  8. [8] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/build_supermatrix_tree.py, lines 1 to 31 · score 0.72 · multiple sequence alignments, single copy, concatenated, supermatrix, gap, longest · block oscr:b:V61VZXA29DSY1-B · Report a mismatch
  9. [9] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/build_supermatrix_tree.py, lines 1 to 31 · score 0.66 · ultrafast bootstrap, IQ TREE, MFP, partition, phylogeny, species · block oscr:b:V61VZXA29DSY1-B · Report a mismatch
  10. [10] § Material and methods › Chromosome-scale synteny and self-synteny analysis ↔ scripts/run_mcscanx_pairwise.py, lines 1 to 49 · score 0.64 · BLASTP searches, Python scripts, MCScanX, scaffolds, synteny, chromosome · block oscr:b:JXFA19S917714-9 · Report a mismatch
  11. [11] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/plot_duplications_upset.py, lines 1 to 32 · score 0.62 · Gene duplication events, species tree, OrthoFinder, inferred, mapped, terminal · block oscr:b:V4HC91YEDQP1P-P · Report a mismatch
  12. [12] § Material and methods › Ka/Ks analysis of coding-sequence divergence ↔ scripts/run_4dtv_analysis.py, lines 392 to 434 · score 0.62 · PAL2NAL, codon alignments, protein sequences, translated · block oscr:b:C6M91ZNSXTWEJ-O · Report a mismatch
  13. [13] § Material and methods › Functional annotation and enrichment analysis ↔ Notebooks/4-ultimate-EnrichmentAnalysis-preparing.qmd, lines 84 to 191 · score 0.62 · eggNOG mapper, KEGG pathways, GO terms, enrichment, focal · block oscr:b:2VFRBXGF12DJ9-9 · Report a mismatch
  14. [14] § Material and methods › Functional annotation and enrichment analysis ↔ scripts/compute_focal_annotation_coverage.py, lines 50 to 83 · score 0.62 · EggNOG, KEGG pathways, GO terms, KO, focal species, gene · block oscr:b:J92DVESS156RR-8 · Report a mismatch
  15. [15] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ scripts/plot_calibrated_phylogeny.R, lines 1 to 31 · score 0.61 · Gene duplication events, species tree, OrthoFinder, internal, mapped, phylogenetic · block oscr:b:R3RBEEQC3JV3P-9 · Report a mismatch
  16. [16] § Material and methods › Chromosome-scale synteny and self-synteny analysis ↔ scripts/run_mcscanx_pairwise.py, lines 1 to 49 · score 0.61 · BLASTP searches, protein sequences, MCScanX, synteny, Chromosome, laticeps · block oscr:b:JXFA19S917714-9 · Report a mismatch
  17. [17] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ Notebooks/3-ultimate-Visualization4OrthoFinder.qmd, lines 960 to 1022 · score 0.60 · TimeTree, Ma, calibrations, schizothoracine, Carassius, Cyprinus · block oscr:b:DCDSBKDA29H3X-O · Report a mismatch
  18. [18] § Material and methods › Ka/Ks analysis of coding-sequence divergence ↔ Notebooks/2-ultimate-KaKs-analysis.qmd, lines 11 to 77 · score 0.59 · pal2nal, synonymous substitution rates, PAML, yn00, Ka, Ks · block oscr:b:T318X54FXVRSN-4 · Report a mismatch
  19. [19] § Results › Chromosome-scale organization and duplicated gene-pair divergence ↔ scripts/run_4dtv_analysis.py, lines 863 to 938 · score 0.59 · near zero peak, DTv distribution, gene pairs · block oscr:b:GXK0WKCF49QPX-K · Report a mismatch
  20. [20] § Material and methods › Chromosome-scale synteny and self-synteny analysis ↔ Notebooks/7-ultimate-StateOfTheArt-Visualization.qmd, lines 1396 to 1511 · score 0.58 · gene position, syntenic blocks, MCScanX, collinear, synteny, Chromosome · block oscr:b:YYCDS36GNHNE7-I · Report a mismatch
  21. [21] § Material and methods › Gene-family analysis ↔ Notebooks/2-ultimate-CAFE-analysis.qmd, lines 1002 to 1096 · score 0.57 · expanded families, contracted families, family change, lost, gene family, CAFE · block oscr:b:72HYZ79BR39AM-1 · Report a mismatch
  22. [22] § Material and methods › Phylogenomic reconstruction and divergence-time estimation ↔ Notebooks/2-ultimate-KaKs-analysis.qmd, lines 1911 to 1996 · score 0.56 · single copy orthogroups, filtered orthogroup, gap, alignments, sequence, species · block oscr:b:JF1YF7EDSW4J4-V · Report a mismatch
  23. [23] § Material and methods › Corrected fourfold degenerate transversion analysis ↔ scripts/run_4dtv_analysis.py, lines 1 to 48 · score 0.53 · degenerate sites, aligner, gene pair, Biopython, transversion, codon · block oscr:b:V5XX2SPAK0YX3-N · Report a mismatch
  24. [24] § Results › Gene-family turnover in the two focal species ↔ Notebooks/2-ultimate-CAFE-analysis.qmd, lines 1002 to 1096 · score 0.53 · expanded families, contracted families, family change, lost, gene family, expansion · block oscr:b:72HYZ79BR39AM-1 · Report a mismatch
  25. [25] § Results › Orthogroup composition and species-specific orthogroups ↔ scripts/plot_orthogroup_overview.py, lines 1 to 39 · score 0.52 · orthogroup repertoire, orthogroup presence, OrthoFinder, Jaccard, Pairwise, species · block oscr:b:28N2PZEQQXRM8-5 · Report a mismatch

Paper

Loaded from Europe PMC by your browser; OSCR does not store it: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 1,326 lines · 42 KB · CC-BY-4.0 · 4 matches

  1. #!/usr/bin/env python3
  2. """4DTv (Four-Fold Degenerate Transversion) Analysis Pipeline.
  3. Calculates transversion rates at four-fold degenerate third-codon positions
  4. for syntenic gene pairs identified by MCScanX. Used to estimate relative
  5. timing of whole-genome duplication (WGD) and speciation events.
  6. Pipeline steps:
  7. 1. Parse MCScanX collinearity file to extract syntenic gene pairs
  8. 2. Load CDS sequences and map MCScanX IDs to BRAKER3 transcript IDs
  9. 3. For each pair: translate → align proteins → back-translate → extract
  10. four-fold degenerate sites → count transversions → apply HKY correction
  11. 4. Output per-pair TSV results
  12. 5. Generate publication-quality histogram with KDE overlay
  13. Input data requirements:
  14. - .collinearity file from MCScanX (self- or inter-species)
  15. - braker.codingseq : CDS sequences from BRAKER3 (per species)
  16. Usage:
  17. # Self-comparisons for WGD detection:
  18. python scripts/run_4dtv_analysis.py \\
  19. --self Aspiorhynchus_laticeps --self Diptychus_maculatus
  20. # Inter-species comparison (requires collinearity file):
  21. python scripts/run_4dtv_analysis.py \\
  22. --pair Aspiorhynchus_laticeps Diptychus_maculatus
  23. # With options:
  24. python scripts/run_4dtv_analysis.py \\
  25. --self Aspiorhynchus_laticeps --self Diptychus_maculatus \\
  26. --aligner biopython --workers 8
  27. Author: Comparative genomics pipeline
  28. """
  29. import argparse
  30. import math
  31. import os
  32. import re
  33. import subprocess
  34. import sys
  35. import tempfile
  36. import time
  37. from collections import Counter
  38. from concurrent.futures import ProcessPoolExecutor, as_completed
  39. from pathlib import Path
  40. from typing import Dict, List, Optional, Tuple
  41. # ---------------------------------------------------------------------------
  42. # Constants
  43. # ---------------------------------------------------------------------------
  44. # Standard genetic code: codon -> amino acid (T used instead of U)
  45. CODON_TABLE: Dict[str, str] = {
  46. "TTT": "F", "TTC": "F", "TTA": "L", "TTG": "L",
  47. "CTT": "L", "CTC": "L", "CTA": "L", "CTG": "L",
  48. "ATT": "I", "ATC": "I", "ATA": "I", "ATG": "M",
  49. "GTT": "V", "GTC": "V", "GTA": "V", "GTG": "V",
  50. "TCT": "S", "TCC": "S", "TCA": "S", "TCG": "S",
  51. "CCT": "P", "CCC": "P", "CCA": "P", "CCG": "P",
  52. "ACT": "T", "ACC": "T", "ACA": "T", "ACG": "T",
  53. "GCT": "A", "GCC": "A", "GCA": "A", "GCG": "A",
  54. "TAT": "Y", "TAC": "Y", "TAA": "*", "TAG": "*",
  55. "CAT": "H", "CAC": "H", "CAA": "Q", "CAG": "Q",
  56. "AAT": "N", "AAC": "N", "AAA": "K", "AAG": "K",
  57. "GAT": "D", "GAC": "D", "GAA": "E", "GAG": "E",
  58. "TGT": "C", "TGC": "C", "TGA": "*", "TGG": "W",
  59. "CGT": "R", "CGC": "R", "CGA": "R", "CGG": "R",
  60. "AGT": "S", "AGC": "S", "AGA": "R", "AGG": "R",
  61. "GGT": "G", "GGC": "G", "GGA": "G", "GGG": "G",
  62. }
  63. # First two positions of codons where the third position is four-fold
  64. # degenerate (any nucleotide at position 3 yields the same amino acid).
  65. FOURFOLD_DEGENERATE_PREFIXES = frozenset({
  66. "GC", # Ala: GCT, GCC, GCA, GCG
  67. "GG", # Gly: GGT, GGC, GGA, GGG
  68. "GT", # Val: GTT, GTC, GTA, GTG
  69. "CT", # Leu: CTT, CTC, CTA, CTG
  70. "CC", # Pro: CCT, CCC, CCA, CCG
  71. "TC", # Ser: TCT, TCC, TCA, TCG
  72. "CG", # Arg: CGT, CGC, CGA, CGG
  73. "AC", # Thr: ACT, ACC, ACA, ACG
  74. })
  75. PURINES = frozenset({"A", "G"})
  76. PYRIMIDINES = frozenset({"C", "T"})
  77. # Gene pair line pattern in .collinearity files
  78. # e.g. " 0- 0:\tal_g14\tal_g2476\t 0"
  79. COLLINEARITY_PAIR_RE = re.compile(
  80. r"^\s*\d+-\s*\d+:\s+(\S+)\s+(\S+)\s+"
  81. )
  82. # ---------------------------------------------------------------------------
  83. # Utility helpers
  84. # ---------------------------------------------------------------------------
  85. def get_species_prefix(species_name: str) -> str:
  86. """Derive a 2-letter MCScanX prefix from a species name.
  87. Takes the first letter of genus + first letter of species epithet,
  88. lowercased.
  89. Args:
  90. species_name: e.g. 'Aspiorhynchus_laticeps'
  91. Returns:
  92. Prefix string, e.g. 'al'
  93. """
  94. parts = species_name.split("_")
  95. if len(parts) < 2:
  96. print(
  97. f"Error: Species name must be 'Genus_species' format: {species_name}",
  98. file=sys.stderr,
  99. )
  100. sys.exit(1)
  101. return (parts[0][0] + parts[1][0]).lower()
  102. def get_display_name(species_name: str, abbreviated: bool = False) -> str:
  103. """Convert species name to display format.
  104. Args:
  105. species_name: e.g. 'Aspiorhynchus_laticeps'
  106. abbreviated: If True, abbreviate genus (e.g. 'A. laticeps')
  107. Returns:
  108. Display name string
  109. """
  110. parts = species_name.split("_")
  111. genus, epithet = parts[0], parts[1]
  112. if abbreviated:
  113. return f"{genus[0]}. {epithet}"
  114. return f"{genus} {epithet}"
  115. def get_mathtext_name(species_name: str, abbreviated: bool = False) -> str:
  116. """Format species name as matplotlib mathtext italic.
  117. Args:
  118. species_name: e.g. 'Aspiorhynchus_laticeps'
  119. abbreviated: If True, abbreviate genus
  120. Returns:
  121. Mathtext string, e.g. '$\\it{A.\\ laticeps}$'
  122. """
  123. parts = species_name.split("_")
  124. genus, epithet = parts[0], parts[1]
  125. if abbreviated:
  126. return f"$\\it{{{genus[0]}.\\ {epithet}}}$"
  127. return f"$\\it{{{genus}\\ {epithet}}}$"
  128. def is_transversion(base1: str, base2: str) -> bool:
  129. """Check if a substitution is a transversion (purine <-> pyrimidine)."""
  130. if base1 == base2:
  131. return False
  132. return (base1 in PURINES) != (base2 in PURINES)
  133. # ---------------------------------------------------------------------------
  134. # I/O functions
  135. # ---------------------------------------------------------------------------
  136. def parse_collinearity(collinearity_path: Path) -> List[Tuple[str, str]]:
  137. """Parse MCScanX collinearity file to extract syntenic gene pairs.
  138. Args:
  139. collinearity_path: Path to .collinearity file
  140. Returns:
  141. List of (gene1_id, gene2_id) tuples with MCScanX-style IDs
  142. """
  143. pairs = []
  144. with open(collinearity_path) as fh:
  145. for line in fh:
  146. m = COLLINEARITY_PAIR_RE.match(line)
  147. if m:
  148. pairs.append((m.group(1), m.group(2)))
  149. return pairs
  150. def load_cds_sequences(cds_path: Path, prefix: str) -> Dict[str, str]:
  151. """Load CDS FASTA and build mapping from MCScanX-style IDs to sequences.
  152. BRAKER3 CDS headers are '>g14.t1'. MCScanX IDs are 'al_g14'.
  153. This function maps '{prefix}_g{N}' -> CDS sequence of 'g{N}.t1'.
  154. Only the primary transcript (.t1) is kept per gene. If .t1 is missing,
  155. falls back to the first available transcript for that gene.
  156. Args:
  157. cds_path: Path to braker.codingseq FASTA file
  158. prefix: Species prefix (e.g. 'al')
  159. Returns:
  160. Dict mapping MCScanX IDs to uppercase DNA sequences
  161. """
  162. sequences: Dict[str, str] = {}
  163. # Track all transcripts per gene for fallback
  164. gene_transcripts: Dict[str, Dict[str, str]] = {}
  165. current_id = None
  166. current_seq: List[str] = []
  167. with open(cds_path) as fh:
  168. for line in fh:
  169. line = line.rstrip()
  170. if line.startswith(">"):
  171. # Save previous sequence
  172. if current_id is not None:
  173. full_seq = "".join(current_seq).upper()
  174. # Parse gene and transcript: "g14.t1" -> gene="g14", tx="t1"
  175. parts = current_id.split(".")
  176. gene_id = parts[0]
  177. tx_id = parts[1] if len(parts) > 1 else "t1"
  178. if gene_id not in gene_transcripts:
  179. gene_transcripts[gene_id] = {}
  180. gene_transcripts[gene_id][tx_id] = full_seq
  181. current_id = line[1:].split()[0] # strip > and take first token
  182. current_seq = []
  183. else:
  184. current_seq.append(line)
  185. # Don't forget the last entry
  186. if current_id is not None:
  187. full_seq = "".join(current_seq).upper()
  188. parts = current_id.split(".")
  189. gene_id = parts[0]
  190. tx_id = parts[1] if len(parts) > 1 else "t1"
  191. if gene_id not in gene_transcripts:
  192. gene_transcripts[gene_id] = {}
  193. gene_transcripts[gene_id][tx_id] = full_seq
  194. # Build MCScanX-keyed dict, preferring .t1
  195. for gene_id, transcripts in gene_transcripts.items():
  196. mcscanx_id = f"{prefix}_{gene_id}"
  197. if "t1" in transcripts:
  198. sequences[mcscanx_id] = transcripts["t1"]
  199. else:
  200. # Fallback: take the first transcript
  201. sequences[mcscanx_id] = next(iter(transcripts.values()))
  202. return sequences
  203. # ---------------------------------------------------------------------------
  204. # Core algorithm
  205. # ---------------------------------------------------------------------------
  206. def translate_sequence(dna_seq: str) -> str:
  207. """Translate DNA sequence to protein using standard genetic code.
  208. Handles sequences whose length is not a multiple of 3 by ignoring the
  209. trailing partial codon. Stops translation at the first stop codon.
  210. Args:
  211. dna_seq: Uppercase DNA string
  212. Returns:
  213. Protein string (single-letter amino acids)
  214. """
  215. protein = []
  216. for i in range(0, len(dna_seq) - 2, 3):
  217. codon = dna_seq[i:i + 3]
  218. if "N" in codon:
  219. protein.append("X") # ambiguous
  220. continue
  221. aa = CODON_TABLE.get(codon, "X")
  222. if aa == "*":
  223. break
  224. protein.append(aa)
  225. return "".join(protein)
  226. def align_proteins_biopython(seq1: str, seq2: str) -> Tuple[str, str]:
  227. """Align two protein sequences using BioPython PairwiseAligner.
  228. Uses BLOSUM62 substitution matrix for scoring. This is a fast fallback
  229. when MAFFT is not available.
  230. Args:
  231. seq1: First protein sequence
  232. seq2: Second protein sequence
  233. Returns:
  234. Tuple of (aligned_seq1, aligned_seq2)
  235. """
  236. from Bio.Align import PairwiseAligner, substitution_matrices
  237. aligner = PairwiseAligner()
  238. aligner.substitution_matrix = substitution_matrices.load("BLOSUM62")
  239. aligner.open_gap_score = -11
  240. aligner.extend_gap_score = -1
  241. aligner.mode = "global"
  242. alignments = aligner.align(seq1, seq2)
  243. best = alignments[0]
  244. # Extract aligned sequences from the Alignment object
  245. aln_seqs = best.format("fasta").strip().split("\n")
  246. # Parse FASTA-formatted alignment
  247. aligned1_parts: List[str] = []
  248. aligned2_parts: List[str] = []
  249. current_parts: Optional[List[str]] = None
  250. for line in aln_seqs:
  251. if line.startswith(">"):
  252. if current_parts is None:
  253. current_parts = aligned1_parts
  254. else:
  255. current_parts = aligned2_parts
  256. else:
  257. if current_parts is not None:
  258. current_parts.append(line)
  259. return "".join(aligned1_parts), "".join(aligned2_parts)
  260. def align_proteins_mafft(
  261. seq1: str,
  262. seq2: str,
  263. seq1_name: str = "seq1",
  264. seq2_name: str = "seq2",
  265. use_docker: bool = True,
  266. container_name: str = "gpu-jupyter",
  267. ) -> Tuple[str, str]:
  268. """Align two protein sequences using MAFFT.
  269. Args:
  270. seq1: First protein sequence
  271. seq2: Second protein sequence
  272. seq1_name: Name for first sequence
  273. seq2_name: Name for second sequence
  274. use_docker: Run via Docker container
  275. container_name: Docker container name
  276. Returns:
  277. Tuple of (aligned_seq1, aligned_seq2)
  278. """
  279. with tempfile.NamedTemporaryFile(
  280. mode="w", suffix=".fa", delete=False, dir="/tmp"
  281. ) as tmp:
  282. tmp.write(f">{seq1_name}\n{seq1}\n>{seq2_name}\n{seq2}\n")
  283. tmp_path = tmp.name
  284. try:
  285. if use_docker:
  286. # Map host /tmp to container /tmp (assumed accessible)
  287. cmd = [
  288. "docker", "exec", container_name,
  289. "mafft", "--auto", "--quiet", tmp_path,
  290. ]
  291. else:
  292. cmd = ["mafft", "--auto", "--quiet", tmp_path]
  293. result = subprocess.run(
  294. cmd, capture_output=True, text=True, timeout=60,
  295. )
  296. if result.returncode != 0:
  297. raise RuntimeError(f"MAFFT failed: {result.stderr[:200]}")
  298. # Parse FASTA output
  299. aligned_seqs: List[str] = []
  300. current_seq: List[str] = []
  301. for line in result.stdout.strip().split("\n"):
  302. if line.startswith(">"):
  303. if current_seq:
  304. aligned_seqs.append("".join(current_seq))
  305. current_seq = []
  306. else:
  307. current_seq.append(line.strip())
  308. if current_seq:
  309. aligned_seqs.append("".join(current_seq))
  310. if len(aligned_seqs) != 2:
  311. raise RuntimeError(
  312. f"Expected 2 aligned sequences, got {len(aligned_seqs)}"
  313. )
  314. return aligned_seqs[0], aligned_seqs[1]
  315. finally:
  316. os.unlink(tmp_path)
  317. def back_translate_alignment(
  318. prot_aln1: str,
  319. prot_aln2: str,
  320. cds1: str,
  321. cds2: str,
  322. ) -> Tuple[str, str]:
  323. """Back-translate protein alignment to codon alignment (PAL2NAL logic).
  324. Walks through aligned protein sequences position by position:
  325. - Amino acid in both: emit corresponding codon from each CDS
  326. - Gap in one: emit '---' for that sequence, codon for the other
  327. Args:
  328. prot_aln1: Aligned protein sequence 1 (may contain '-')
  329. prot_aln2: Aligned protein sequence 2 (may contain '-')
  330. cds1: Original CDS for sequence 1 (unaligned, uppercase)
  331. cds2: Original CDS for sequence 2 (unaligned, uppercase)
  332. Returns:
  333. Tuple of (codon_aln1, codon_aln2) — equal length, multiple of 3
  334. Raises:
  335. ValueError: If codon/protein mismatch detected
  336. """
  337. codon_aln1: List[str] = []
  338. codon_aln2: List[str] = []
  339. pos1 = 0 # codon cursor for CDS1
  340. pos2 = 0 # codon cursor for CDS2
  341. for aa1, aa2 in zip(prot_aln1, prot_aln2):
  342. if aa1 == "-":
  343. codon_aln1.append("---")
  344. else:
  345. codon_aln1.append(cds1[pos1 * 3:(pos1 + 1) * 3])
  346. pos1 += 1
  347. if aa2 == "-":
  348. codon_aln2.append("---")
  349. else:
  350. codon_aln2.append(cds2[pos2 * 3:(pos2 + 1) * 3])
  351. pos2 += 1
  352. return "".join(codon_aln1), "".join(codon_aln2)
  353. def extract_4d_sites(
  354. codon_aln1: str, codon_aln2: str
  355. ) -> Tuple[str, str]:
  356. """Extract third-position bases at four-fold degenerate sites.
  357. A site qualifies if BOTH codons in the alignment column have their
  358. first two positions in FOURFOLD_DEGENERATE_PREFIXES, and neither
  359. codon contains gaps or ambiguous bases.
  360. Args:
  361. codon_aln1: Codon-aligned sequence 1 (length multiple of 3)
  362. codon_aln2: Codon-aligned sequence 2 (length multiple of 3)
  363. Returns:
  364. Tuple of (sites1, sites2) — strings of 3rd-position bases at 4D sites
  365. """
  366. sites1: List[str] = []
  367. sites2: List[str] = []
  368. valid_bases = frozenset("ATCG")
  369. for i in range(0, min(len(codon_aln1), len(codon_aln2)) - 2, 3):
  370. c1 = codon_aln1[i:i + 3]
  371. c2 = codon_aln2[i:i + 3]
  372. # Skip if either codon has gaps or ambiguous bases
  373. if not all(b in valid_bases for b in c1):
  374. continue
  375. if not all(b in valid_bases for b in c2):
  376. continue
  377. # Check if both codons are four-fold degenerate at position 3
  378. prefix1 = c1[:2]
  379. prefix2 = c2[:2]
  380. if prefix1 in FOURFOLD_DEGENERATE_PREFIXES and \
  381. prefix2 in FOURFOLD_DEGENERATE_PREFIXES:
  382. sites1.append(c1[2])
  383. sites2.append(c2[2])
  384. return "".join(sites1), "".join(sites2)
  385. def calculate_4dtv(
  386. sites1: str, sites2: str
  387. ) -> Tuple[float, float, int, int]:
  388. """Calculate raw and HKY-corrected 4DTv.
  389. HKY correction formula:
  390. piR = piA + piG (purine frequency)
  391. piY = piC + piT (pyrimidine frequency)
  392. Q = transversion proportion
  393. corrected = -2 * piR * piY * ln(1 - Q / (2 * piR * piY))
  394. Args:
  395. sites1: Third-position bases from sequence 1 at 4D sites
  396. sites2: Third-position bases from sequence 2 at 4D sites
  397. Returns:
  398. Tuple of (raw_4dtv, corrected_4dtv, n_4d_sites, n_transversions)
  399. corrected_4dtv is NaN if saturated.
  400. """
  401. n_sites = len(sites1)
  402. if n_sites == 0:
  403. return (float("nan"), float("nan"), 0, 0)
  404. # Count transversions
  405. n_tv = sum(
  406. 1 for b1, b2 in zip(sites1, sites2) if is_transversion(b1, b2)
  407. )
  408. raw_4dtv = n_tv / n_sites
  409. # Compute base frequencies from both sequences combined
  410. all_bases = sites1 + sites2
  411. counts = Counter(all_bases)
  412. total = len(all_bases)
  413. pi_a = counts.get("A", 0) / total
  414. pi_t = counts.get("T", 0) / total
  415. pi_g = counts.get("G", 0) / total
  416. pi_c = counts.get("C", 0) / total
  417. pi_r = pi_a + pi_g # purine frequency
  418. pi_y = pi_t + pi_c # pyrimidine frequency
  419. # HKY correction
  420. denom = 2.0 * pi_r * pi_y
  421. if denom <= 0:
  422. return (raw_4dtv, float("nan"), n_sites, n_tv)
  423. arg = 1.0 - raw_4dtv / denom
  424. if arg <= 0:
  425. # Saturated — too many transversions for reliable correction
  426. return (raw_4dtv, float("nan"), n_sites, n_tv)
  427. corrected = -denom * math.log(arg)
  428. return (raw_4dtv, corrected, n_sites, n_tv)
  429. def process_gene_pair(
  430. gene1: str,
  431. gene2: str,
  432. cds_seqs: Dict[str, str],
  433. aligner: str = "biopython",
  434. use_docker: bool = True,
  435. container_name: str = "gpu-jupyter",
  436. min_4d_sites: int = 10,
  437. ) -> Optional[Dict]:
  438. """Process a single gene pair through the full 4DTv pipeline.
  439. Args:
  440. gene1: MCScanX gene ID (e.g. 'al_g14')
  441. gene2: MCScanX gene ID (e.g. 'al_g2476')
  442. cds_seqs: Dict mapping MCScanX IDs to CDS sequences
  443. aligner: 'mafft' or 'biopython'
  444. use_docker: Use Docker for MAFFT
  445. container_name: Docker container name
  446. min_4d_sites: Minimum 4D sites required
  447. Returns:
  448. Result dict or None if pair fails
  449. """
  450. # Step 1: Look up CDS
  451. cds1 = cds_seqs.get(gene1)
  452. cds2 = cds_seqs.get(gene2)
  453. if cds1 is None or cds2 is None:
  454. return None
  455. # Step 2: Validate CDS length
  456. if len(cds1) < 6 or len(cds2) < 6:
  457. return None
  458. # Step 3: Translate to protein
  459. prot1 = translate_sequence(cds1)
  460. prot2 = translate_sequence(cds2)
  461. if len(prot1) < 2 or len(prot2) < 2:
  462. return None
  463. # Step 4: Align proteins
  464. try:
  465. if aligner == "mafft":
  466. aln1, aln2 = align_proteins_mafft(
  467. prot1, prot2, gene1, gene2, use_docker, container_name
  468. )
  469. else:
  470. aln1, aln2 = align_proteins_biopython(prot1, prot2)
  471. except Exception:
  472. return None
  473. # Step 5: Back-translate to codon alignment
  474. # Truncate CDS to match translated protein length (codon-aligned)
  475. cds1_trimmed = cds1[:len(prot1) * 3]
  476. cds2_trimmed = cds2[:len(prot2) * 3]
  477. try:
  478. codon_aln1, codon_aln2 = back_translate_alignment(
  479. aln1, aln2, cds1_trimmed, cds2_trimmed
  480. )
  481. except (ValueError, IndexError):
  482. return None
  483. # Step 6: Extract 4D sites
  484. sites1, sites2 = extract_4d_sites(codon_aln1, codon_aln2)
  485. if len(sites1) < min_4d_sites:
  486. return None
  487. # Step 7: Calculate 4DTv
  488. raw, corrected, n_sites, n_tv = calculate_4dtv(sites1, sites2)
  489. return {
  490. "gene1": gene1,
  491. "gene2": gene2,
  492. "n_4d_sites": n_sites,
  493. "n_transversions": n_tv,
  494. "raw_4dtv": raw,
  495. "corrected_4dtv": corrected,
  496. }
  497. # ---------------------------------------------------------------------------
  498. # Worker function for multiprocessing (must be at module level)
  499. # ---------------------------------------------------------------------------
  500. # Global references set before spawning workers
  501. _WORKER_CDS_SEQS: Dict[str, str] = {}
  502. _WORKER_ALIGNER: str = "biopython"
  503. _WORKER_USE_DOCKER: bool = True
  504. _WORKER_CONTAINER: str = "gpu-jupyter"
  505. _WORKER_MIN_4D: int = 10
  506. def _init_worker(
  507. cds_seqs: Dict[str, str],
  508. aligner: str,
  509. use_docker: bool,
  510. container_name: str,
  511. min_4d_sites: int,
  512. ) -> None:
  513. """Initialize worker process globals."""
  514. global _WORKER_CDS_SEQS, _WORKER_ALIGNER, _WORKER_USE_DOCKER
  515. global _WORKER_CONTAINER, _WORKER_MIN_4D
  516. _WORKER_CDS_SEQS = cds_seqs
  517. _WORKER_ALIGNER = aligner
  518. _WORKER_USE_DOCKER = use_docker
  519. _WORKER_CONTAINER = container_name
  520. _WORKER_MIN_4D = min_4d_sites
  521. def _worker_process_pair(pair: Tuple[str, str]) -> Optional[Dict]:
  522. """Worker function to process a single gene pair."""
  523. return process_gene_pair(
  524. pair[0], pair[1],
  525. _WORKER_CDS_SEQS,
  526. _WORKER_ALIGNER,
  527. _WORKER_USE_DOCKER,
  528. _WORKER_CONTAINER,
  529. _WORKER_MIN_4D,
  530. )
  531. # ---------------------------------------------------------------------------
  532. # Parallelization
  533. # ---------------------------------------------------------------------------
  534. def process_pairs_parallel(
  535. pairs: List[Tuple[str, str]],
  536. cds_seqs: Dict[str, str],
  537. n_workers: int = 4,
  538. aligner: str = "biopython",
  539. use_docker: bool = True,
  540. container_name: str = "gpu-jupyter",
  541. min_4d_sites: int = 10,
  542. ) -> List[Dict]:
  543. """Process all gene pairs with parallel workers.
  544. Args:
  545. pairs: List of (gene1, gene2) tuples
  546. cds_seqs: Dict mapping MCScanX IDs to CDS sequences
  547. n_workers: Number of parallel workers
  548. aligner: Alignment method
  549. use_docker: Use Docker for MAFFT
  550. container_name: Docker container name
  551. min_4d_sites: Minimum 4D sites per pair
  552. Returns:
  553. List of result dicts (only successful pairs)
  554. """
  555. results: List[Dict] = []
  556. total = len(pairs)
  557. t0 = time.time()
  558. if n_workers <= 1 or aligner == "mafft":
  559. # Sequential processing (MAFFT is already I/O-bound with subprocess)
  560. for idx, (g1, g2) in enumerate(pairs, 1):
  561. result = process_gene_pair(
  562. g1, g2, cds_seqs, aligner, use_docker, container_name,
  563. min_4d_sites,
  564. )
  565. if result is not None:
  566. results.append(result)
  567. if idx % 1000 == 0:
  568. elapsed = time.time() - t0
  569. rate = idx / elapsed
  570. print(
  571. f" Processed {idx}/{total} pairs "
  572. f"({len(results)} valid, {rate:.0f} pairs/sec)",
  573. flush=True,
  574. )
  575. else:
  576. # Parallel with ProcessPoolExecutor for BioPython aligner
  577. from multiprocessing import Pool
  578. with Pool(
  579. processes=n_workers,
  580. initializer=_init_worker,
  581. initargs=(cds_seqs, aligner, use_docker, container_name, min_4d_sites),
  582. ) as pool:
  583. done = 0
  584. for result in pool.imap_unordered(_worker_process_pair, pairs, chunksize=100):
  585. done += 1
  586. if result is not None:
  587. results.append(result)
  588. if done % 1000 == 0:
  589. elapsed = time.time() - t0
  590. rate = done / elapsed
  591. print(
  592. f" Processed {done}/{total} pairs "
  593. f"({len(results)} valid, {rate:.0f} pairs/sec)",
  594. flush=True,
  595. )
  596. elapsed = time.time() - t0
  597. print(
  598. f" Completed {total} pairs in {elapsed:.1f}s — "
  599. f"{len(results)} valid results ({len(results)/total*100:.1f}%)",
  600. flush=True,
  601. )
  602. return results
  603. # ---------------------------------------------------------------------------
  604. # Output
  605. # ---------------------------------------------------------------------------
  606. def write_results_tsv(results: List[Dict], output_path: Path) -> None:
  607. """Write per-pair 4DTv results to TSV file.
  608. Args:
  609. results: List of result dicts from process_gene_pair
  610. output_path: Output TSV file path
  611. """
  612. output_path.parent.mkdir(parents=True, exist_ok=True)
  613. with open(output_path, "w") as fh:
  614. fh.write(
  615. "gene1\tgene2\tn_4d_sites\tn_transversions\t"
  616. "raw_4dtv\tcorrected_4dtv\n"
  617. )
  618. for r in results:
  619. fh.write(
  620. f"{r['gene1']}\t{r['gene2']}\t{r['n_4d_sites']}\t"
  621. f"{r['n_transversions']}\t{r['raw_4dtv']:.6f}\t"
  622. f"{r['corrected_4dtv']:.6f}\n"
  623. )
  624. def write_summary(
  625. all_results: Dict[str, List[Dict]],
  626. comparison_labels: Dict[str, Tuple[str, str]],
  627. output_path: Path,
  628. ) -> None:
  629. """Write summary statistics to text file.
  630. Args:
  631. all_results: Maps comparison key to results list
  632. comparison_labels: Maps comparison key to (species1, species2) tuple
  633. output_path: Output file path
  634. """
  635. with open(output_path, "w") as fh:
  636. fh.write("4DTv Analysis Summary\n")
  637. fh.write("=" * 60 + "\n\n")
  638. for key, results in all_results.items():
  639. sp1, sp2 = comparison_labels[key]
  640. label = (
  641. f"{get_display_name(sp1)} (self)"
  642. if sp1 == sp2
  643. else f"{get_display_name(sp1)} vs {get_display_name(sp2)}"
  644. )
  645. fh.write(f"Comparison: {label}\n")
  646. fh.write(f" Total valid pairs: {len(results)}\n")
  647. if results:
  648. import numpy as np
  649. corrected = [
  650. r["corrected_4dtv"] for r in results
  651. if not math.isnan(r["corrected_4dtv"])
  652. ]
  653. if corrected:
  654. arr = np.array(corrected)
  655. fh.write(f" Corrected 4DTv statistics:\n")
  656. fh.write(f" Mean: {np.mean(arr):.4f}\n")
  657. fh.write(f" Median: {np.median(arr):.4f}\n")
  658. fh.write(f" Std: {np.std(arr):.4f}\n")
  659. fh.write(f" Min: {np.min(arr):.4f}\n")
  660. fh.write(f" Max: {np.max(arr):.4f}\n")
  661. raw_vals = [r["raw_4dtv"] for r in results]
  662. fh.write(f" Raw 4DTv mean: {np.mean(raw_vals):.4f}\n")
  663. sites = [r["n_4d_sites"] for r in results]
  664. fh.write(
  665. f" Mean 4D sites per pair: {np.mean(sites):.1f}\n"
  666. )
  667. saturated = len(results) - len(corrected)
  668. if saturated > 0:
  669. fh.write(
  670. f" Saturated pairs (NaN): {saturated}\n"
  671. )
  672. fh.write("\n")
  673. # ---------------------------------------------------------------------------
  674. # Visualization
  675. # ---------------------------------------------------------------------------
  676. def plot_4dtv_distribution(
  677. all_results: Dict[str, List[Dict]],
  678. comparison_labels: Dict[str, Tuple[str, str]],
  679. output_dir: Path,
  680. bin_width: float = 0.02,
  681. x_max: float = 0.8,
  682. ) -> None:
  683. """Generate publication-quality 4DTv distribution histogram with KDE.
  684. Args:
  685. all_results: Maps comparison key to results list
  686. comparison_labels: Maps comparison key to (species1, species2) tuple
  687. output_dir: Output directory for figures
  688. bin_width: Histogram bin width
  689. x_max: Maximum x-axis value
  690. """
  691. import matplotlib
  692. matplotlib.use("Agg")
  693. import matplotlib.pyplot as plt
  694. import numpy as np
  695. from scipy.stats import gaussian_kde
  696. # Publication rcParams (project convention)
  697. plt.rcParams.update({
  698. "figure.dpi": 300,
  699. "savefig.dpi": 300,
  700. "font.family": "serif",
  701. "font.size": 14,
  702. "axes.titlesize": 16,
  703. "axes.titleweight": "bold",
  704. "axes.labelsize": 14,
  705. "axes.labelweight": "bold",
  706. "xtick.labelsize": 12,
  707. "ytick.labelsize": 12,
  708. "legend.fontsize": 12,
  709. "legend.title_fontsize": 13,
  710. })
  711. bins = np.arange(0, x_max + bin_width, bin_width)
  712. # Color palette for comparisons
  713. color_list = ["#1f77b4", "#ff7f0e", "#2ca02c", "#d62728",
  714. "#9467bd", "#8c564b", "#e377c2", "#7f7f7f"]
  715. # Collect data for all comparisons
  716. plot_data: list = [] # (arr, color, label) tuples
  717. for idx, (key, results) in enumerate(all_results.items()):
  718. sp1, sp2 = comparison_labels[key]
  719. # Filter out NaN corrected values and negative zeros
  720. values = [
  721. max(0.0, r["corrected_4dtv"]) for r in results
  722. if not math.isnan(r["corrected_4dtv"]) and r["corrected_4dtv"] < x_max
  723. ]
  724. if not values:
  725. continue
  726. arr = np.array(values)
  727. color = color_list[idx % len(color_list)]
  728. # Build label with italicized species names
  729. if sp1 == sp2:
  730. label = f"{get_mathtext_name(sp1, abbreviated=True)} (self)"
  731. else:
  732. label = (
  733. f"{get_mathtext_name(sp1, abbreviated=True)} vs "
  734. f"{get_mathtext_name(sp2, abbreviated=True)}"
  735. )
  736. plot_data.append((arr, color, label))
  737. if not plot_data:
  738. print(" WARNING: No valid data to plot.")
  739. return
  740. # --- Figure 1: Frequency histogram (count-based, y-axis capped) ---
  741. fig, ax = plt.subplots(figsize=(10, 7))
  742. for arr, color, label in plot_data:
  743. counts, _, patches = ax.hist(
  744. arr, bins=bins, alpha=0.4,
  745. color=color, edgecolor=color, linewidth=0.5,
  746. label=label,
  747. )
  748. ax.set_xlabel("4DTv (corrected)")
  749. ax.set_ylabel("Number of gene pairs")
  750. ax.set_title("Distribution of 4DTv Values")
  751. ax.set_xlim(0, x_max)
  752. # Cap y-axis to make the tail visible (exclude the first bin from max)
  753. all_counts = []
  754. for arr, color, label in plot_data:
  755. h, _ = np.histogram(arr, bins=bins)
  756. all_counts.extend(h[1:]) # exclude first bin (near-zero peak)
  757. if all_counts:
  758. y_cap = max(all_counts) * 1.5
  759. # Only cap if the first bin is much larger than the rest
  760. for arr, _, _ in plot_data:
  761. h, _ = np.histogram(arr, bins=bins)
  762. if h[0] > y_cap * 2:
  763. ax.set_ylim(0, y_cap)
  764. ax.annotate(
  765. f"Peak at 0 truncated",
  766. xy=(0.02, 0.97), xycoords="axes fraction",
  767. fontsize=10, fontstyle="italic",
  768. ha="left", va="top",
  769. )
  770. break
  771. ax.legend(loc="upper right")
  772. ax.spines["top"].set_visible(False)
  773. ax.spines["right"].set_visible(False)
  774. fig.tight_layout()
  775. output_dir.mkdir(parents=True, exist_ok=True)
  776. fig.savefig(output_dir / "4dtv_distribution.pdf")
  777. fig.savefig(output_dir / "4dtv_distribution.png")
  778. plt.close(fig)
  779. # --- Figure 2: Frequency histogram with log y-axis ---
  780. # Log scale reveals secondary peaks masked by the dominant near-zero peak
  781. fig2, ax2 = plt.subplots(figsize=(10, 7))
  782. for arr, color, label in plot_data:
  783. ax2.hist(
  784. arr, bins=bins, alpha=0.4,
  785. color=color, edgecolor=color, linewidth=0.5,
  786. label=label,
  787. )
  788. ax2.set_xlabel("4DTv (corrected)")
  789. ax2.set_ylabel("Number of gene pairs (log scale)")
  790. ax2.set_title("Distribution of 4DTv Values")
  791. ax2.set_xlim(0, x_max)
  792. ax2.set_yscale("log")
  793. ax2.legend(loc="upper right")
  794. ax2.spines["top"].set_visible(False)
  795. ax2.spines["right"].set_visible(False)
  796. fig2.tight_layout()
  797. fig2.savefig(output_dir / "4dtv_distribution_log.pdf")
  798. fig2.savefig(output_dir / "4dtv_distribution_log.png")
  799. plt.close(fig2)
  800. # --- Figure 3: KDE density plot with y-axis capped ---
  801. # Exclude values below a threshold from KDE to focus on divergence signal
  802. fig3, ax3 = plt.subplots(figsize=(10, 7))
  803. for arr, color, label in plot_data:
  804. try:
  805. kde = gaussian_kde(arr, bw_method=0.05)
  806. x_kde = np.linspace(0, x_max, 1000)
  807. y_kde = kde(x_kde)
  808. ax3.plot(x_kde, y_kde, color=color, linewidth=2.5, label=label)
  809. ax3.fill_between(x_kde, y_kde, alpha=0.15, color=color)
  810. except Exception:
  811. pass
  812. ax3.set_xlabel("4DTv (corrected)")
  813. ax3.set_ylabel("Density")
  814. ax3.set_title("4DTv Density Distribution")
  815. ax3.set_xlim(0, x_max)
  816. # Cap y-axis to make the tail visible
  817. all_kde_max = []
  818. for arr, color, label in plot_data:
  819. try:
  820. kde = gaussian_kde(arr, bw_method=0.05)
  821. # Find density at x=0.05 (beyond the near-zero peak shoulder)
  822. shoulder_density = kde(np.array([0.05]))[0]
  823. all_kde_max.append(shoulder_density)
  824. except Exception:
  825. pass
  826. if all_kde_max:
  827. y_cap_kde = max(all_kde_max) * 3.0
  828. ax3.set_ylim(0, y_cap_kde)
  829. ax3.annotate(
  830. "Near-zero peak truncated",
  831. xy=(0.02, 0.97), xycoords="axes fraction",
  832. fontsize=10, fontstyle="italic",
  833. ha="left", va="top",
  834. )
  835. ax3.legend(loc="upper right")
  836. ax3.spines["top"].set_visible(False)
  837. ax3.spines["right"].set_visible(False)
  838. fig3.tight_layout()
  839. fig3.savefig(output_dir / "4dtv_density.pdf")
  840. fig3.savefig(output_dir / "4dtv_density.png")
  841. plt.close(fig3)
  842. print(
  843. f" Saved figures to {output_dir}/"
  844. f"{{4dtv_distribution,4dtv_density}}.{{pdf,png}}"
  845. )
  846. # ---------------------------------------------------------------------------
  847. # Pipeline orchestration
  848. # ---------------------------------------------------------------------------
  849. def resolve_collinearity_path(
  850. species1: str,
  851. species2: str,
  852. mcscanx_base: Path,
  853. ) -> Optional[Path]:
  854. """Find the collinearity file for a species pair.
  855. Tries multiple naming conventions:
  856. {p1}_{p2}/{p1}_{p2}.collinearity
  857. {p2}_{p1}/{p2}_{p1}.collinearity
  858. Args:
  859. species1: First species name
  860. species2: Second species name
  861. mcscanx_base: Base directory for MCScanX results
  862. Returns:
  863. Path to collinearity file, or None if not found
  864. """
  865. p1 = get_species_prefix(species1)
  866. p2 = get_species_prefix(species2)
  867. for a, b in [(p1, p2), (p2, p1)]:
  868. name = f"{a}_{b}"
  869. path = mcscanx_base / name / f"{name}.collinearity"
  870. if path.exists():
  871. return path
  872. return None
  873. def run_4dtv_pipeline(
  874. comparisons: List[Tuple[str, str]],
  875. braker_base: Path,
  876. mcscanx_base: Path,
  877. output_dir: Path,
  878. n_workers: int = 4,
  879. aligner: str = "biopython",
  880. use_docker: bool = True,
  881. container_name: str = "gpu-jupyter",
  882. min_4d_sites: int = 10,
  883. bin_width: float = 0.02,
  884. ) -> None:
  885. """Run the complete 4DTv analysis pipeline.
  886. Args:
  887. comparisons: List of (species1, species2) tuples
  888. braker_base: Base dir containing {species}/braker.codingseq
  889. mcscanx_base: Base dir with MCScanX results
  890. output_dir: Output directory
  891. n_workers: Parallel workers
  892. aligner: Alignment method ('mafft' or 'biopython')
  893. use_docker: Use Docker for MAFFT
  894. container_name: Docker container name
  895. min_4d_sites: Minimum 4D sites per pair
  896. bin_width: Histogram bin width
  897. """
  898. all_results: Dict[str, List[Dict]] = {}
  899. comparison_labels: Dict[str, Tuple[str, str]] = {}
  900. # Cache loaded CDS sequences per species
  901. cds_cache: Dict[str, Dict[str, str]] = {}
  902. output_dir.mkdir(parents=True, exist_ok=True)
  903. for sp1, sp2 in comparisons:
  904. p1 = get_species_prefix(sp1)
  905. p2 = get_species_prefix(sp2)
  906. comp_key = f"{p1}_{p2}"
  907. if sp1 == sp2:
  908. print(f"\n{'='*60}")
  909. print(f"Self-comparison: {get_display_name(sp1)}")
  910. print(f"{'='*60}")
  911. else:
  912. print(f"\n{'='*60}")
  913. print(
  914. f"Inter-species: {get_display_name(sp1)} vs "
  915. f"{get_display_name(sp2)}"
  916. )
  917. print(f"{'='*60}")
  918. # 1. Find collinearity file
  919. coll_path = resolve_collinearity_path(sp1, sp2, mcscanx_base)
  920. if coll_path is None:
  921. print(
  922. f" WARNING: No collinearity file found for {comp_key}. "
  923. f"Run MCScanX first. Skipping.",
  924. file=sys.stderr,
  925. )
  926. continue
  927. print(f" Collinearity: {coll_path}")
  928. # 2. Parse gene pairs
  929. pairs = parse_collinearity(coll_path)
  930. print(f" Syntenic gene pairs: {len(pairs)}")
  931. # 3. Load CDS sequences (cached per species)
  932. species_to_load = {sp1} if sp1 == sp2 else {sp1, sp2}
  933. merged_cds: Dict[str, str] = {}
  934. for sp in species_to_load:
  935. if sp not in cds_cache:
  936. prefix = get_species_prefix(sp)
  937. cds_path = braker_base / sp / "braker.codingseq"
  938. if not cds_path.exists():
  939. print(
  940. f" ERROR: CDS file not found: {cds_path}",
  941. file=sys.stderr,
  942. )
  943. continue
  944. print(f" Loading CDS: {cds_path}")
  945. cds_cache[sp] = load_cds_sequences(cds_path, prefix)
  946. print(f" Loaded {len(cds_cache[sp])} genes")
  947. merged_cds.update(cds_cache[sp])
  948. if not merged_cds:
  949. print(" ERROR: No CDS sequences loaded. Skipping.")
  950. continue
  951. # 4. Process all pairs
  952. print(f" Processing pairs (aligner={aligner}, workers={n_workers})...")
  953. results = process_pairs_parallel(
  954. pairs, merged_cds, n_workers, aligner, use_docker,
  955. container_name, min_4d_sites,
  956. )
  957. # 5. Write TSV
  958. tsv_path = output_dir / f"{comp_key}_4dtv.tsv"
  959. write_results_tsv(results, tsv_path)
  960. print(f" Results: {tsv_path}")
  961. all_results[comp_key] = results
  962. comparison_labels[comp_key] = (sp1, sp2)
  963. if not all_results:
  964. print("\nNo valid results. Exiting.")
  965. return
  966. # 6. Write summary
  967. summary_path = output_dir / "summary.txt"
  968. write_summary(all_results, comparison_labels, summary_path)
  969. print(f"\nSummary: {summary_path}")
  970. # 7. Generate visualization
  971. print("\nGenerating 4DTv distribution plot...")
  972. plot_4dtv_distribution(
  973. all_results, comparison_labels, output_dir, bin_width,
  974. )
  975. # ---------------------------------------------------------------------------
  976. # CLI
  977. # ---------------------------------------------------------------------------
  978. def parse_args() -> argparse.Namespace:
  979. """Parse command-line arguments."""
  980. parser = argparse.ArgumentParser(
  981. description=(
  982. "4DTv analysis: calculate transversion rates at four-fold "
  983. "degenerate sites for syntenic gene pairs from MCScanX."
  984. ),
  985. formatter_class=argparse.RawDescriptionHelpFormatter,
  986. epilog="""
  987. Examples:
  988. # Self-comparisons for WGD detection:
  989. python scripts/run_4dtv_analysis.py \\
  990. --self Aspiorhynchus_laticeps --self Diptychus_maculatus
  991. # Inter-species comparison:
  992. python scripts/run_4dtv_analysis.py \\
  993. --pair Aspiorhynchus_laticeps Diptychus_maculatus
  994. # Combined with BioPython aligner:
  995. python scripts/run_4dtv_analysis.py \\
  996. --self Aspiorhynchus_laticeps --self Diptychus_maculatus \\
  997. --aligner biopython --workers 8
  998. """,
  999. )
  1000. parser.add_argument(
  1001. "--self",
  1002. dest="self_species",
  1003. action="append",
  1004. default=[],
  1005. metavar="SPECIES",
  1006. help=(
  1007. "Self-comparison for WGD detection "
  1008. "(e.g. Aspiorhynchus_laticeps). Repeatable."
  1009. ),
  1010. )
  1011. parser.add_argument(
  1012. "--pair",
  1013. dest="pair_species",
  1014. action="append",
  1015. nargs=2,
  1016. default=[],
  1017. metavar=("SP1", "SP2"),
  1018. help=(
  1019. "Inter-species comparison "
  1020. "(e.g. Aspiorhynchus_laticeps Diptychus_maculatus). Repeatable."
  1021. ),
  1022. )
  1023. parser.add_argument(
  1024. "--braker-base",
  1025. type=Path,
  1026. default=Path("/home/jovyan/Outputs/Preprocessing/BRAKER_MASKED"),
  1027. help="Base directory containing {species}/braker.codingseq",
  1028. )
  1029. parser.add_argument(
  1030. "--mcscanx-base",
  1031. type=Path,
  1032. default=Path("/home/jovyan/Outputs_set3/MCScanX_Results"),
  1033. help="Base directory containing MCScanX results",
  1034. )
  1035. parser.add_argument(
  1036. "--output-dir",
  1037. type=Path,
  1038. default=Path("/home/jovyan/Outputs_set3/4DTv"),
  1039. help="Output directory (default: /home/jovyan/Outputs_set3/4DTv)",
  1040. )
  1041. parser.add_argument(
  1042. "--aligner",
  1043. choices=["mafft", "biopython"],
  1044. default="mafft",
  1045. help="Protein alignment method (default: mafft)",
  1046. )
  1047. parser.add_argument(
  1048. "--workers",
  1049. type=int,
  1050. default=4,
  1051. help="Number of parallel workers (default: 4)",
  1052. )
  1053. parser.add_argument(
  1054. "--no-docker",
  1055. action="store_true",
  1056. help="Run MAFFT directly without Docker",
  1057. )
  1058. parser.add_argument(
  1059. "--container",
  1060. default="gpu-jupyter",
  1061. help="Docker container name (default: gpu-jupyter)",
  1062. )
  1063. parser.add_argument(
  1064. "--min-4d-sites",
  1065. type=int,
  1066. default=10,
  1067. help="Minimum four-fold degenerate sites per pair (default: 10)",
  1068. )
  1069. parser.add_argument(
  1070. "--bin-width",
  1071. type=float,
  1072. default=0.02,
  1073. help="Histogram bin width (default: 0.02)",
  1074. )
  1075. return parser.parse_args()
  1076. def main() -> None:
  1077. """Entry point."""
  1078. args = parse_args()
  1079. if not args.self_species and not args.pair_species:
  1080. print(
  1081. "Error: Specify at least one --self or --pair comparison.",
  1082. file=sys.stderr,
  1083. )
  1084. sys.exit(1)
  1085. # Build comparison list
  1086. comparisons: List[Tuple[str, str]] = []
  1087. for sp in args.self_species:
  1088. comparisons.append((sp, sp))
  1089. for sp1, sp2 in args.pair_species:
  1090. comparisons.append((sp1, sp2))
  1091. use_docker = not args.no_docker
  1092. print("4DTv Analysis Pipeline")
  1093. print("=" * 60)
  1094. print(f"Comparisons: {len(comparisons)}")
  1095. print(f"Aligner: {args.aligner}")
  1096. print(f"Workers: {args.workers}")
  1097. print(f"Docker: {'yes' if use_docker else 'no'}")
  1098. print(f"Min 4D sites: {args.min_4d_sites}")
  1099. print(f"BRAKER base: {args.braker_base}")
  1100. print(f"MCScanX base: {args.mcscanx_base}")
  1101. print(f"Output dir: {args.output_dir}")
  1102. run_4dtv_pipeline(
  1103. comparisons=comparisons,
  1104. braker_base=args.braker_base,
  1105. mcscanx_base=args.mcscanx_base,
  1106. output_dir=args.output_dir,
  1107. n_workers=args.workers,
  1108. aligner=args.aligner,
  1109. use_docker=use_docker,
  1110. container_name=args.container,
  1111. min_4d_sites=args.min_4d_sites,
  1112. bin_width=args.bin_width,
  1113. )
  1114. if __name__ == "__main__":
  1115. main()

run_4dtv_analysis.py at commit 937b6c6, under CC-BY-4.0 · at the source

Overview

Authors: Yibang Wang1, Yuxiang Huang2, Wenyu Sun1,3, Hui Zhang1,3, Motomu Matsui2,4
  1. Laboratory of Marine Ecology and Environmental Sciences, Institute of Oceanology, Chinese Academy of Sciences, Qingdao 266071, People's Republic of China
  2. Department of Biological Sciences, Graduate School of Science, University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan
  3. University of Chinese Academy of Sciences, Beijing 100000, People's Republic of China
  4. Institute for Chemical Research, Kyoto University, Uji, Kyoto 611-0011, Japan
Journal: Royal Society open science, volume 13, issue 11, article 261481
Dates: received 7 July 2026; accepted 16 September 2026; published online 18 November 2026
Type: Research article · Language: English
License: CC BY
Identifiers: DOI 10.1098/rsos.261481 · PMID 43048745 · PMCID PMC13777439 · OpenAlex W7235059628
Open access: gold, a free copy (OpenAlex)
Status: code verified (how the code is found and verified)
Categories: other (organism), cellular / molecular (subfield)
Methods: Statistics, Connectivity
Keywords: Aspiorhynchus laticeps, Diptychus maculatus, plateau fish, comparative genomics, highland adaptation, candidate adaptive signatures, gene-family turnover
Topic: Genomics and Phylogenetic Studies (Molecular Biology, Biochemistry, Genetics and Molecular Biology), according to OpenAlex
Funding: Dalian Institute of Chemical Physics; National Key Research and Development Program of China
Citations: not cited yet (Europe PMC); 57 references in the paper

Abstract

Plateau aquatic environments expose fishes to chronic cold, hypoxia, ultraviolet radiation and low productivity, providing natural systems for studying genomic features associated with environmental adaptation. Whether related fishes inhabiting similar cold-water plateau environments exhibit comparable or divergent genomic responses remains unclear. We compared two plateau cypriniform fishes, Aspiorhynchus laticeps and Diptychus maculatus, within a 12-species phylogenomic framework, integrating orthogroup inference, gene-family turnover, coding-sequence divergence, chromosome-scale synteny and duplicated gene-pair divergence profiling. The strongest contrasts occurred in species-specific orthogroups and gene-family turnover. Aspiorhynchus laticeps had a larger set of apparent species-specific orthogroups, enriched mainly in oocyte and reproductive processes, and its expanded families and elevated-ω orthogroups were associated with immune, receptor-mediated and membrane-associated functions. By contrast, D. maculatus had fewer species-specific orthogroups enriched for neural and synaptic processes and a contraction-dominated gene-family profile involving host defence, mucosal immunity and chromatin-related processes. The focal contrast did not exceed empirical background variation, and expansion-related computational analysis of gene family evolution (CAFE) estimates were associated with assembly continuity and input-gene properties. These findings indicate that ecologically similar plateau fishes may exhibit different combinations of candidate genomic features, highlighting the joint influence of ecological adaptation and genome history.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repositories

Its files are read in the Code ↔ Paper reader above, with 25 matches between paragraphs and lines of code.

enantiomer-h/aspiorhynchus-laticeps-genomics-public

License: CC-BY-4.0
State: the link answers, verified on 20 November 2026
Evidence: files inventoried
Commit: 937b6c67df4d9281fee029598a1b741129c1c40e, 15 September 2026
Languages: Python (28), Quarto (18), Shell (10), R (7)
Size: 82 files, 63 scripts
Software Heritage: not archived
Found in: “Data accessibility”
Holds: README, license file, CITATION.cff, environment (docker-compose.yml, .build/Dockerfile), documentation, 18 notebooks
Not found: tests, continuous integration
Tools: pandas (26 files), Matplotlib (25 files), NumPy (15 files), Biopython (14 files), ggplot2 (10 files), tidyverse (10 files), seaborn (9 files), clusterProfiler (7 files), SciPy (5 files), ggpubr (2 files), data.table (1 file), patchwork (1 file), Pillow (1 file), scikit-learn (1 file)
Availability: 1 check, the latest on 20 November 2026: the link answers
  • 20 November 2026: the link answers
65 files

Zenodo 22767265

License: CC-BY-4.0
State: the link answers, verified on 20 November 2026
Evidence: files inventoried
Size: 1 file
Software Heritage: not checked
Found in: the references
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Tools: pandas (26 files), Matplotlib (25 files), NumPy (15 files), Biopython (14 files), ggplot2 (10 files), tidyverse (10 files), seaborn (9 files), clusterProfiler (7 files), SciPy (5 files), ggpubr (2 files), data.table (1 file), patchwork (1 file), Pillow (1 file), scikit-learn (1 file)
Availability: 1 check, the latest on 20 November 2026: the link answers (HTTP 200)
  • 20 November 2026: the link answers (HTTP 200)
65 files
At the source:

doi:10.6084/m9.figshare.c.8773029

License: none: the authors keep all their rights
State: unreachable at the last attempt, verified on 20 November 2026
Evidence: found in the paper
Software Heritage: not checked
Found in: the references
Not found: README, license file, CITATION.cff, environment file, tests, continuous integration, documentation
Availability: 1 check, the latest on 20 November 2026: unreachable at the last attempt (HTTP 202)
  • 20 November 2026: unreachable at the last attempt (HTTP 202)

The paper's code and data availability statement is in the Data section.

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 3 repositories of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 126 scripts, each with its path and the digest of its content;
  • 25 matches between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

Datasets cited

Data accessibility

Genome assemblies and associated sequencing data for A. laticeps and D. maculatus are available from NCBI under assembly accessions number GCA_023376895.1 and GCA_041504395.2. respectively. The assembly records can be accessed through NCBI Assembly at https://www.ncbi.nlm.nih.gov/assembly/GCA_023376895.1 and https://www.ncbi.nlm.nih.gov/assembly/GCA_041504395.2. Data and relevant code for this research work are stored in GitHub: https://github.com/enantiomer-h/aspiorhynchus-laticeps-genomics-public and have been archived within the Zenodo repository [56].

Supplementary material is available online [57].

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 20 November 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 5 authors, 7 keywords, 2 funders, 57 references.

Cite

This paper

Wang, Y., Huang, Y., Sun, W., Zhang, H., & Matsui, M. (2026). Comparative genomics reveals contrasting candidate adaptive signatures in two ecologically similar plateau fishes. Royal Society open science, 13(11), 261481. https://doi.org/10.1098/rsos.261481

BibTeX

@article{wang2026comparative,
author = {Wang, Yibang and Huang, Yuxiang and Sun, Wenyu and Zhang, Hui and Matsui, Motomu},
title = {{Comparative genomics reveals contrasting candidate adaptive signatures in two ecologically similar plateau fishes}},
journal = {Royal Society open science},
year = {2026},
month = nov,
volume = {13},
number = {11},
pages = {261481},
publisher = {The Royal Society},
issn = {2054-5703},
doi = {10.1098/rsos.261481},
url = {https://doi.org/10.1098/rsos.261481},
pmid = {43048745},
pmcid = {PMC13777439}
}

RIS

TY - JOUR
AU - Wang, Yibang
AU - Huang, Yuxiang
AU - Sun, Wenyu
AU - Zhang, Hui
AU - Matsui, Motomu
TI - Comparative genomics reveals contrasting candidate adaptive signatures in two ecologically similar plateau fishes
T2 - Royal Society open science
J2 - R Soc Open Sci
PY - 2026
DA - 2026/11/18
VL - 13
IS - 11
SP - 261481
SN - 2054-5703
PB - The Royal Society
DO - 10.1098/rsos.261481
UR - https://doi.org/10.1098/rsos.261481
LA - en
ER -

CSL-JSON

{
"id": "10.1098/rsos.261481",
"type": "article-journal",
"title": "Comparative genomics reveals contrasting candidate adaptive signatures in two ecologically similar plateau fishes",
"container-title": "Royal Society open science",
"author": [
{
"family": "Wang",
"given": "Yibang"
},
{
"family": "Huang",
"given": "Yuxiang"
},
{
"family": "Sun",
"given": "Wenyu"
},
{
"family": "Zhang",
"given": "Hui"
},
{
"family": "Matsui",
"given": "Motomu"
}
],
"container-title-short": "R Soc Open Sci",
"volume": "13",
"issue": "11",
"page": "261481",
"DOI": "10.1098/rsos.261481",
"PMID": "43048745",
"PMCID": "PMC13777439",
"ISSN": "2054-5703",
"publisher": "The Royal Society",
"URL": "https://doi.org/10.1098/rsos.261481",
"language": "en",
"issued": {
"date-parts": [
[
2026,
11,
18
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.7554/elife.107393 [code]
Chromosome-scale genome assembly of the European common cuttlefish &lt;i&gt;Sepia officinalis&lt;/i&gt;.
Journal: eLife
In common: clusterProfiler, data.table, ggplot2, 5 other tools, other, cellular / molecular, 7 references
[2] doi:10.1038/s41586-026-10629-x [code]
Whole-genome duplication shaped cell-type evolution in the vertebrate brain.
Journal: Nature
In common: clusterProfiler, patchwork, ggpubr, 8 other tools, other, cellular / molecular, 5 references
[3] doi:10.1016/j.xgen.2026.101194 [code]
Genetic diversity and regulatory features of human-specific NOTCH2NL duplications.
Journal: Cell genomics
In common: Biopython, patchwork, ggpubr, 9 other tools, cellular / molecular, 3 references
[4] doi:10.1016/j.cell.2023.08.027 [code]
Stepwise emergence of the neuronal gene expression program in early animal evolution.
Journal: Cell
In common: Biopython, data.table, ggplot2, 6 other tools, cellular / molecular, 4 references
[5] doi:10.1038/s41593-025-02050-w [code]
TDP-43 loss induces cryptic polyadenylation in ALS/FTD.
Journal: Nature neuroscience
In common: Biopython, patchwork, ggpubr, 9 other tools, cellular / molecular, 1 reference
[6] doi:10.1038/s41586-024-07473-2 [code]
The complete sequence and comparative analysis of ape sex chromosomes.
Journal: Nature
In common: Biopython, ggplot2, seaborn, 4 other tools, cellular / molecular, 5 references
[7] doi:10.7554/elife.90100 [code]
Robust optogenetic inhibition with red-light-sensitive anion-conducting channelrhodopsins.
Journal: eLife
In common: Biopython, patchwork, ggpubr, 7 other tools, 2 references
[8] doi:10.1186/s12915-026-02514-w [code]
Butterfly brains change in morphology and in gene splicing patterns after brief pheromone exposure.
Journal: BMC biology
In common: clusterProfiler, ggpubr, ggplot2, 1 other tool, other, cellular / molecular, 5 references
[9] doi:10.1186/s12915-023-01661-8 [code]
Expanded olfactory system in ray-finned fishes capable of terrestrial exploration.
Journal: BMC biology
In common: Biopython, ggpubr, data.table, 5 other tools, other, cellular / molecular, 2 references
[10] doi:10.1111/eva.13653 [code]
Investigating structural variant, indel and single nucleotide polymorphism differentiation between locally adapted Atlantic salmon populations.
Journal: Evolutionary applications
In common: patchwork, ggpubr, data.table, 6 other tools, other, 2 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Add the badge to the code's README

OSCR: the badge of this paper

If you are an author or a maintainer of this paper's code, you can add this badge. It says what OSCR knows of the paper (its code, its validated tracing map) and links the code to this page. Paste one of these snippets under the README's title. Keep the two marks around it, so that the badge can be replaced in place and is never added twice. OSCR changes nothing in your repository.

Markdown
<!-- oscr-badge:start -->
[![OSCR](https://openscicode.org/badge/doi/10.1098/rsos.261481.svg)](https://openscicode.org/badge/doi/10.1098/rsos.261481)
<!-- oscr-badge:end -->
reStructuredText
.. oscr-badge:start

.. image:: https://openscicode.org/badge/doi/10.1098/rsos.261481.svg
   :target: https://openscicode.org/badge/doi/10.1098/rsos.261481
   :alt: OSCR

.. oscr-badge:end
HTML
<!-- oscr-badge:start -->
<a href="https://openscicode.org/badge/doi/10.1098/rsos.261481"><img src="https://openscicode.org/badge/doi/10.1098/rsos.261481.svg" alt="OSCR"></a>
<!-- oscr-badge:end -->

On GitHub, open the README, choose Edit (the pencil), paste the snippet under the title, then "Commit changes".

If you choose to, you can have the badge kept up to date for you, with oscr badge --write from the command line or with the GitHub Action in your repository's workflows. Every other form and option (a card, AsciiDoc, LaTeX, shields.io) is on the badge page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: code errors and mismatches between the code and the paper about this paper and its code, from readers and its authors, beside the paper and its tracing map.

Report a mismatch between the code and the paper · Report a code error (signed in)

Reproductions: reports from readers who ran the authors' code and did not get the paper's result, what they ran, with which environment, commit and data.

Report a reproduction failure (signed in)

Versions of its code: the releases of its code tied to a version of this paper (the preprint, the accepted manuscript, the version of record…), each with the tracing map versioned with it.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.