OSCR

Microbiome functional gene pathways are indicative of cognitive performance in older adults at risk for Alzheimer's disease.

Code ↔ Paper

1 match between paragraphs of the paper and lines of its authors' code, computed by the harvester (lexical-v1). Click a colored paragraph or line to see its counterpart.

The 1 match
  1. [1] § Materials and methods › Sample handling and DNA sequencing ↔ kneaddata/knead_data.py, lines 3–40 · score 0.67 · Quality control, KneadData, tools, Metagenomic, contamination, host

Paper

Loaded from Europe PMC by your browser, not stored by OSCR: doi.org · Europe PMC

The paper is loaded when this pane is shown.

The authors' code

Python · 613 lines · 25 KB · MIT · 1 match

  1. #!/usr/bin/env python
  3. """
  4. KneadData
  6. KneadData is a tool designed to perform quality control on metagenomic
  7. sequencing data, especially data from microbiome experiments. In these
  8. experiments, samples are typically taken from a host in hopes of learning
  9. something about the microbial community on the host. However, metagenomic
  10. sequencing data from such experiments will often contain a high ratio of host
  11. to bacterial reads. This tool aims to perform principled in silico separation
  12. of bacterial reads from these "contaminant" reads, be they from the host,
  13. from bacterial 16S sequences, or other user-defined sources.
  15. Dependencies: Trimmomatic, Bowtie2 or BMTagger, and TRF (optional)
  17. To Run: kneaddata -i <input.fastq> -o <output_dir>
  19. Copyright (c) 2015 Harvard School of Public Health
  21. Permission is hereby granted, free of charge, to any person obtaining a copy
  22. of this software and associated documentation files (the "Software"), to deal
  23. in the Software without restriction, including without limitation the rights
  24. to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  25. copies of the Software, and to permit persons to whom the Software is
  26. furnished to do so, subject to the following conditions:
  28. The above copyright notice and this permission notice shall be included in
  29. all copies or substantial portions of the Software.
  31. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  32. IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  33. FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  34. AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  35. LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  36. OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  37. THE SOFTWARE.
  38. """
  40. import sys
  42. # check for the required python version
  43. required_python_version_major = 2
  44. required_python_version_minor = 7
  46. try:
  47. if (sys.version_info[0] < required_python_version_major or
  48. (sys.version_info[0] == required_python_version_major and
  49. sys.version_info[1] < required_python_version_minor)):
  50. sys.exit("CRITICAL ERROR: The python version found (version "+
  51. str(sys.version_info[0])+"."+str(sys.version_info[1])+") "+
  52. "does not match the version required (version "+
  53. str(required_python_version_major)+"."+
  54. str(required_python_version_minor)+"+)")
  55. except (AttributeError,IndexError):
  56. sys.exit("CRITICAL ERROR: The python version found (version 1) " +
  57. "does not match the version required (version "+
  58. str(required_python_version_major)+"."+
  59. str(required_python_version_minor)+"+)")
  61. import os
  62. import logging
  63. import argparse
  64. import re
  65. import itertools
  67. # Try to load one of the kneaddata modules to check the installation
  68. try:
  69. from kneaddata import utilities
  70. except ImportError:
  71. sys.exit("ERROR: Unable to find the kneaddata python package." +
  72. " Please check your install.")
  74. from kneaddata import run
  75. from kneaddata import config
  77. VERSION="0.12.4"
  79. # name global logging instance
  80. logger=logging.getLogger(__name__)
  81. # Global input files path list for FASTQC
  82. original_input_files=[]
  84. def parse_arguments(args):
  85. """
  86. Parse the arguments from the user
  87. """
  89. parser = argparse.ArgumentParser(
  90. description= "KneadData\n",
  91. formatter_class=argparse.RawTextHelpFormatter,
  92. prog="kneaddata")
  93. group1 = parser.add_argument_group("global options")
  94. group1.add_argument(
  95. "--version",
  96. action="version",
  97. version="%(prog)s v"+VERSION)
  98. parser.add_argument(
  99. "-v","--verbose",
  100. action="store_true",
  101. help="additional output is printed\n")
  102. group1.add_argument(
  103. "-i1", "--input1",
  104. help="Pair 1 input FASTQ file",
  105. dest='input1')
  106. group1.add_argument(
  107. "-i2", "--input2",
  108. help="Pair 2 input FASTQ file",
  109. dest='input2')
  110. group1.add_argument(
  111. "-un","--unpaired",
  112. help="unparied input FASTQ file",
  113. dest='unpaired')
  114. group1.add_argument(
  115. "-o", "--output",
  116. dest='output_dir',
  117. help="directory to write output files",
  118. required=True)
  119. group1.add_argument(
  120. "-s", "--scratch",
  121. dest='scratch_dir',
  122. help="directory to write temp files",
  123. default="")
  124. group1.add_argument(
  125. "-db", "--reference-db",
  126. default=[], action="append",
  127. help="location of reference database (additional arguments add databases)")
  128. group1.add_argument(
  129. "--bypass-trim",
  130. action="store_true",
  131. help="bypass the trim step")
  132. group1.add_argument(
  133. "--output-prefix",
  134. help="prefix for all output files\n[ DEFAULT : $SAMPLE_kneaddata ]")
  135. group1.add_argument(
  136. "-t","--threads",
  137. type=int,
  138. default=config.threads,
  139. metavar="<" + str(config.threads) + ">",
  140. help="number of threads\n[ Default : "+str(config.threads)+" ]")
  141. group1.add_argument(
  142. "-p","--processes",
  143. type=int,
  144. default=config.processes,
  145. metavar="<" + str(config.processes) + ">",
  146. help="number of processes\n[ Default : "+str(config.processes)+" ]")
  147. group1.add_argument(
  148. "-q","--quality-scores",
  149. default=config.quality_scores,
  150. choices=config.quality_scores_options,
  151. dest='trimmomatic_quality_scores',
  152. help="quality scores\n[ DEFAULT : "+config.quality_scores+" ]")
  153. group1.add_argument(
  154. "--run-bmtagger",
  155. default=False,
  156. action="store_true",
  157. dest='bmtagger',
  158. help="run BMTagger instead of Bowtie2 to identify contaminant reads")
  159. group1.add_argument(
  160. "--bypass-trf",
  161. action="store_true",
  162. help="option to bypass the removal of tandem repeats")
  163. group1.add_argument(
  164. "--run-trf",
  165. action="store_true",
  166. help="legacy option to run the removal of tandem repeats (now run by default)")
  167. group1.add_argument(
  168. "--run-fastqc-start",
  169. default=False,
  170. dest='fastqc_start',
  171. action="store_true",
  172. help="run fastqc at the beginning of the workflow")
  173. group1.add_argument(
  174. "--run-fastqc-end",
  175. default=False,
  176. dest='fastqc_end',
  177. action="store_true",
  178. help="run fastqc at the end of the workflow")
  179. group1.add_argument(
  180. "--store-temp-output",
  181. action="store_true",
  182. help="store temp output files\n[ DEFAULT : temp output files are removed ]")
  183. group1.add_argument(
  184. "--remove-intermediate-output",
  185. action="store_true",
  186. help="remove intermediate output files\n[ DEFAULT : intermediate output files are stored ]")
  187. group1.add_argument(
  188. "--cat-final-output",
  189. action="store_true",
  190. help="concatenate all final output files\n[ DEFAULT : final output is not concatenated ]")
  191. group1.add_argument(
  192. "--log-level",
  193. default=config.log_level,
  194. choices=config.log_level_choices,
  195. help="level of log messages\n[ DEFAULT : "+config.log_level+" ]")
  196. group1.add_argument(
  197. "--log",
  198. help="log file\n[ DEFAULT : $OUTPUT_DIR/$SAMPLE_kneaddata.log ]")
  200. group2 = parser.add_argument_group("trimmomatic arguments")
  201. group2.add_argument(
  202. "--trimmomatic",
  203. dest='trimmomatic_path',
  204. help="path to trimmomatic\n[ DEFAULT : $PATH ]")
  205. group2.add_argument(
  206. "--run-trim-repetitive",
  207. default=False,
  208. dest='run_trim_repetitive',
  209. action="store_true",
  210. help="Trim fastqc generated overrepresented sequences\n")
  211. group2.add_argument(
  212. "--max-memory",
  213. default=config.trimmomatic_memory,
  214. help="max amount of memory\n[ DEFAULT : "+config.trimmomatic_memory+" ]")
  215. group2.add_argument(
  216. "--trimmomatic-options",
  217. action="append",
  218. help="options for trimmomatic\n[ DEFAULT : "+" ".join(utilities.get_default_trimmomatic_options())+" ]\n"+\
  219. "MINLEN is set to "+str(config.trimmomatic_min_len_percent)+" percent of total input read length. The user can alternatively specify a length (in bases) for MINLEN.")
  220. group2.add_argument(
  221. "--sequencer-source",
  222. dest='sequencer_source',
  223. default=config.trimmomatic_provided_sequencer_default,
  224. choices=config.trimmomatic_provided_sequencer_source,
  225. help="options for sequencer-source\n[ DEFAULT : "+config.trimmomatic_provided_sequencer_default+"]")
  227. group3 = parser.add_argument_group("bowtie2 arguments")
  228. group3.add_argument(
  229. "--bowtie2",
  230. dest='bowtie2_path',
  231. help="path to bowtie2\n[ DEFAULT : $PATH ]")
  232. group3.add_argument(
  233. "--bowtie2-options",
  234. action="append",
  235. help="options for bowtie2\n[ DEFAULT : "+ " ".join(config.bowtie2_options)+" ]")
  236. group3.add_argument(
  237. "--decontaminate-pairs",
  238. choices=["strict","lenient","unpaired"],
  239. default="strict",
  240. help="options for filtering of paired end reads (strict='remove both R1+R2 if either align', lenient='remove only if both R1+R2 align', unpaired='ignore pairing and remove as single end')\n"+\
  241. "[ DEFAULT : %(default)s ]")
  242. group3.add_argument(
  243. "--reorder",
  244. action="store_true",
  245. help="order the sequences in the same order as the input\n[ DEFAULT : Sequences are not ordered ]")
  246. group3.add_argument(
  247. "--serial",
  248. action="store_true",
  249. help="filter the input in serial for multiple databases so a subset of reads are processed in each database search (the default when running with a single process)")
  251. group4 = parser.add_argument_group("bmtagger arguments")
  252. group4.add_argument(
  253. "--bmtagger",
  254. dest='bmtagger_path',
  255. help="path to BMTagger\n[ DEFAULT : $PATH ]")
  257. group5 = parser.add_argument_group("trf arguments")
  258. group5.add_argument(
  259. "--trf",
  260. dest='trf_path',
  261. help="path to TRF\n[ DEFAULT : $PATH ]")
  262. group5.add_argument(
  263. "--match",
  264. type=int,
  265. default=config.trf_match,
  266. help="matching weight\n[ DEFAULT : "+str(config.trf_match)+" ]")
  267. group5.add_argument(
  268. "--mismatch",
  269. type=int,
  270. default=config.trf_mismatch,
  271. help="mismatching penalty\n[ DEFAULT : "+str(config.trf_mismatch)+" ]")
  272. group5.add_argument(
  273. "--delta",
  274. type=int,
  275. default=config.trf_delta,
  276. help="indel penalty\n[ DEFAULT : "+str(config.trf_delta)+" ]")
  277. group5.add_argument(
  278. "--pm",
  279. type=int,
  280. default=config.trf_match_probability,
  281. help="match probability\n[ DEFAULT : "+str(config.trf_match_probability)+" ]")
  282. group5.add_argument(
  283. "--pi",
  284. type=int,
  285. default=config.trf_pi,
  286. help="indel probability\n[ DEFAULT : "+str(config.trf_pi)+" ]")
  287. group5.add_argument(
  288. "--minscore",
  289. type=int,
  290. default=config.trf_minscore,
  291. help="minimum alignment score to report\n[ DEFAULT : "+str(config.trf_minscore)+" ]")
  292. group5.add_argument(
  293. "--maxperiod",
  294. type=int,
  295. default=config.trf_maxperiod,
  296. help="maximum period size to report\n[ DEFAULT : "+str(config.trf_maxperiod)+" ]")
  297. group6 = parser.add_argument_group("fastqc arguments")
  298. group6.add_argument(
  299. "--fastqc",
  300. dest='fastqc_path',
  301. help="path to fastqc\n[ DEFAULT : $PATH ]")
  303. return parser.parse_args()
  305. def update_configuration(args):
  306. """ Update the run settings based on the arguments provided """
  308. # if only a single processor is to be used, default to serial mode for efficiency
  309. if args.processes == 1:
  310. args.serial=True
  312. # get the full path for the output directory
  313. args.output_dir = os.path.abspath(args.output_dir)
  314. if args.scratch_dir:
  315. args.scratch_dir = os.path.abspath(args.scratch_dir)
  317. # set if temp output should be removed
  318. args.remove_temp_output = not args.store_temp_output
  320. # if intermediate output should be removed, then also remove temp output
  321. if args.remove_intermediate_output:
  322. args.remove_temp_output = True
  324. # check the input files are non-empty and readable
  325. args.input=[]
  326. if (args.input1 and args.input2):
  327. args.input.append(os.path.abspath(args.input1))
  328. args.input.append(os.path.abspath(args.input2))
  329. if (args.unpaired):
  330. args.input.append(os.path.abspath(args.unpaired))
  331. utilities.is_file_readable(args.input[0],exit_on_error=True)
  332. if len(args.input) == 2:
  333. utilities.is_file_readable(args.input[1],exit_on_error=True)
  334. elif len(args.input) > 2:
  335. sys.exit("ERROR: Please provide at most 2 input files.")
  336. elif len(args.input) == 0:
  337. sys.exit("ERROR: Please provide --input1/--input2 or --unpaired (input) files.")
  339. #Store original file paths for FASTQC
  340. for input in args.input:
  341. original_input_files.append(input)
  343. # create the output directory and scratch if needed
  344. utilities.create_directory(args.output_dir)
  345. if args.scratch_dir:
  346. utilities.create_directory(args.scratch_dir)
  348. # set bowtie2 options
  349. if args.bowtie2_options:
  350. # parse the options from the user into any array of options
  351. args.bowtie2_options=utilities.format_options_to_list(args.bowtie2_options)
  352. else:
  353. # if not set by user, then set to default options
  354. args.bowtie2_options = config.bowtie2_options
  356. # add the quality scores to the bowtie2 options
  357. args.bowtie2_options+=[config.bowtie2_flag_start+args.trimmomatic_quality_scores]
  359. # set the mode for single end input file
  360. if len(args.input) == 1:
  361. args.decontaminate_pairs = "unpaired"
  363. # set the bowtie2 mode based on the pairs input
  364. args.discordant = False
  365. if args.decontaminate_pairs != "lenient" :
  366. args.discordant = True
  368. # update the quality score option into a flag for trimmomatic
  369. args.trimmomatic_quality_scores=config.trimmomatic_flag_start+args.trimmomatic_quality_scores
  371. # find the location of trimmomatic, trimmomatic does not need to be executable
  372. if not args.bypass_trim:
  373. args.trimmomatic_path=utilities.find_dependency(args.trimmomatic_path,config.trimmomatic_jar,"trimmomatic",
  374. "--trimmomatic", bypass_permissions_check=True)
  376. # find the location of bmtagger, if set to run
  377. if args.reference_db:
  378. if args.bmtagger:
  379. args.bmtagger_path=utilities.find_dependency(args.bmtagger_path,config.bmtagger_exe,"bmtagger",
  380. "--bmtagger", bypass_permissions_check=False)
  381. # add this folder to path, so as to be able to find other dependencies like bmfilter
  382. utilities.add_exe_to_path(os.path.dirname(args.bmtagger_path))
  383. else:
  384. # find the location of bowtie2, if not running with bmtagger
  385. args.bowtie2_path=utilities.find_dependency(args.bowtie2_path, config.bowtie2_exe, "bowtie2",
  386. "--bowtie2", bypass_permissions_check=False)
  388. # find the location of trf, if set to run
  389. if not args.bypass_trf:
  390. args.trf_path=utilities.find_dependency(args.trf_path,config.trf_exe,"trf",
  391. "--trf", bypass_permissions_check=False)
  393. # if fastqc is set to be run, check if the executable can be found
  394. if args.fastqc_start or args.fastqc_end or args.run_trim_repetitive:
  395. args.fastqc_path=utilities.find_dependency(args.fastqc_path,config.fastqc_exe,"fastqc",
  396. "--fastqc",bypass_permissions_check=False)
  398. # set the default output prefix
  399. if args.output_prefix == None:
  400. if args.input[0].endswith(".gz") or args.input[0].endswith(".bz2"):
  401. # remove compression extension if present
  402. infile_base = os.path.splitext(os.path.splitext(os.path.basename(args.input[0]))[0])[0]
  403. else:
  404. infile_base = os.path.splitext(os.path.basename(args.input[0]))[0]
  405. args.output_prefix = infile_base + "_kneaddata"
  407. # find the bowtie2 indexes for each of the reference databases
  408. # reference database inputs can be directories, indexes, or index files
  409. if args.reference_db:
  410. reference_indexes=[]
  411. database_type="bowtie2"
  412. if args.bmtagger:
  413. database_type="bmtagger"
  414. for directory in args.reference_db:
  415. reference_indexes.append(utilities.find_database_index(os.path.abspath(directory),database_type))
  417. args.reference_db=reference_indexes
  419. return args
  421. def setup_logging(args):
  422. """ Set up the log file """
  424. if not args.log:
  425. args.log = os.path.join(args.output_dir,args.output_prefix+".log")
  427. # configure the logger
  428. logging.basicConfig(filename=args.log,format='%(asctime)s - %(name)s - %(levelname)s: %(message)s',
  429. level=getattr(logging,args.log_level), filemode='w', datefmt='%m/%d/%Y %I:%M:%S %p')
  431. # write the version of the software to the log
  432. logger.info("Running kneaddata v"+VERSION)
  434. # write the location of the output files to the log
  435. message="Output files will be written to: " + args.output_dir
  436. logger.info(message)
  438. # write out all of the argument settings
  439. message="Running with the following arguments: \n"
  440. for key,value in vars(args).items():
  441. if isinstance(value,list) or isinstance(value,tuple):
  442. value_string=" ".join([str(i) for i in value])
  443. else:
  444. value_string=str(value)
  446. message+=key+" = "+value_string+"\n"
  447. logger.debug(message)
  449. def main():
  450. # Parse the arguments from the user
  451. args = parse_arguments(sys.argv)
  453. # Update the configuration
  454. args = update_configuration(args)
  456. # Start logging
  457. setup_logging(args)
  459. # set the prefix for the output files
  460. final_output_dir = args.output_dir
  461. if args.scratch_dir:
  462. full_path_output_prefix = os.path.join(args.scratch_dir, args.output_prefix)
  463. args.output_dir = args.scratch_dir
  464. else:
  465. full_path_output_prefix = os.path.join(args.output_dir, args.output_prefix)
  467. temp_output_files=[]
  468. # Check for compressed files, bam files, or sam files
  469. for index in range(len(args.input)):
  471. # check for gzipped/bz2 files
  472. if args.input[index].endswith(".gz") or args.input[index].endswith(".bz2"):
  473. args.input[index]=utilities.get_decompressed_file(args.input[index], args.output_dir, temp_output_files, args.input)
  474. elif args.input[index].endswith(".bam"):
  475. input_files_set=utilities.get_fastq_from_bam_file(args.input[index], args.output_dir, temp_output_files, args.input)
  476. if isinstance(input_files_set,list):
  477. args.input=input_files_set
  478. else:
  479. args.input[index]=input_files_set
  480. elif args.input[index].endswith(".sam"):
  481. args.input[index]=utilities.get_fastq_from_sam_file(args.input[index], args.output_dir, temp_output_files, args.input)
  483. # Get the format of the first input file
  484. file_format=utilities.get_file_format(args.input[0])
  486. if file_format != "fastq":
  487. message="Your input file is of type: "+file_format+". Please provide an input file of fastq format."
  488. logger.critical(message)
  489. sys.exit(message)
  491. # if this is the new illumina identifier format, create temp files after reformatting the headers
  492. for index in range(len(args.input)):
  493. args.input[index]=utilities.get_reformatted_identifiers(args.input[index],index,args.output_dir, temp_output_files, args.input)
  495. # check for reads that are not ordered and order if needed (if trimmomatic is run)
  496. if not args.bypass_trim and len(args.input)==2:
  497. args.input=utilities.check_and_reorder_reads(args.input, args.output_dir, temp_output_files)
  499. # remove any temp files from decompress/reformat that are no longer needed
  500. utilities.update_temp_output_files(temp_output_files, [], args.input)
  502. # set trimmomatic options
  503. # this is done after the decompression and conversions from sam/bam
  504. # as the default requires the read length from the input sequences
  505. if args.trimmomatic_options:
  506. # parse the options from the user into an array of options
  507. args.trimmomatic_options = utilities.format_options_to_list(args.trimmomatic_options)
  508. else:
  509. # if trimmomatic options not set by user, then set to default options
  510. # use read length of input file for minlen
  511. args.trimmomatic_options = utilities.get_default_trimmomatic_options(utilities.get_read_length_fastq(args.input[0]),
  512. path=config.trimmomatic_adapter_folder,type="PE" if len(args.input) == 2 else "SE", sequencer_source=args.sequencer_source)
  515. # Get the number of reads initially
  516. utilities.log_read_count_for_files(args.input,"raw","Initial number of reads",args.verbose)
  518. # Run fastqc if set to run at start of workflow
  519. if args.fastqc_start or args.run_trim_repetitive:
  520. run.fastqc(args.fastqc_path, args.output_dir, original_input_files, args.threads, args.verbose)
  521. #Setting fastqc output zip and txt file path
  522. output_txt_files=[]
  523. for input_file_name in original_input_files:
  524. temp_file = os.path.splitext(input_file_name)[0]
  525. if (temp_file.count('fastq')>0 or temp_file.count('fq')>0 ):
  526. temp_file = os.path.splitext(temp_file)[0]
  527. output_txt_files.append(args.output_dir+"/fastqc/"+temp_file.split('/')[-1]+"_fastqc/fastqc_data.txt")
  529. if not args.bypass_trim:
  530. if args.run_trim_repetitive:
  531. # Get the Min Overrepresented Seq Length
  532. args.trimmomatic_options = utilities.get_updated_trimmomatic_parameters(output_txt_files, args.output_dir, args.trimmomatic_options)
  534. trimmomatic_output_files = run.trim(
  535. args.input, full_path_output_prefix, args.trimmomatic_path,
  536. args.trimmomatic_quality_scores, args.max_memory, args.trimmomatic_options,
  537. args.threads, args.verbose)
  539. else:
  540. message="Bypass trimming"
  541. logger.info(message)
  542. print(message)
  543. trimmomatic_output_files=[args.input]
  545. # Get the number of reads after trimming
  546. utilities.log_read_count_for_files(trimmomatic_output_files,"trimmed","Total reads after trimming",args.verbose)
  548. # run TRF, if set
  549. if not args.bypass_trf:
  550. # run trf on all output files
  551. trf_output_files=run.tandem(trimmomatic_output_files, full_path_output_prefix, args.match,
  552. args.mismatch,args.delta,args.pm,args.pi,
  553. args.minscore,args.maxperiod,args.trf_path,
  554. args.processes,args.verbose,args.remove_temp_output,args.threads)
  555. # remove the aligment files, if intermediate output files should be removed
  556. if args.reference_db and args.remove_intermediate_output:
  557. temp_output_files+=utilities.resolve_sublists(trimmomatic_output_files)
  558. else:
  559. trf_output_files = trimmomatic_output_files
  560. # If a reference database is not provided, then bypass decontamination step
  561. if not args.reference_db:
  562. message="Bypass decontamination"
  563. logger.info(message)
  564. print(message)
  565. # resolve sub-lists if present
  566. final_output_files=trf_output_files
  567. else:
  568. final_output_files=run.decontaminate(args, full_path_output_prefix, trf_output_files)
  569. # remove trimmed output files, if set to remove intermediate outputx
  570. if not args.bypass_trim and args.remove_intermediate_output:
  571. temp_output_files+=utilities.resolve_sublists(trf_output_files)
  573. # If set, concat the final output files if there is more than one
  574. final_output_files = utilities.resolve_sublists(final_output_files)
  575. if args.cat_final_output and len(final_output_files) > 1:
  576. cat_output_file=full_path_output_prefix+config.fastq_file_extension
  577. utilities.cat_files(final_output_files,cat_output_file)
  579. # if removing intermediate output, then remove the files that were merged
  580. if args.remove_intermediate_output:
  581. temp_output_files+=final_output_files
  582. final_output_files=[cat_output_file]
  583. else:
  584. final_output_files.append(cat_output_file)
  586. # Remove any temp output files, if set
  587. if not args.store_temp_output:
  588. for file in temp_output_files:
  589. utilities.remove_file(file)
  591. # Run fastqc if set to run at end of workflow
  592. if args.fastqc_end:
  593. run.fastqc(args.fastqc_path, args.output_dir, final_output_files, args.threads, args.verbose)
  595. # If using scratch, then move final output files to output folder
  596. if args.scratch_dir:
  597. scratch_output_files=final_output_files
  598. final_output_files=[]
  599. for outfile in scratch_output_files:
  600. utilities.move_file(os.path.basename(outfile),args.output_dir,final_output_dir)
  601. final_output_files.append(os.path.join(final_output_dir,os.path.basename(outfile)))
  603. if len(final_output_files) > 1:
  604. message="\nFinal output files created: \n"
  605. else:
  606. message="\nFinal output file created: \n"
  608. message=message+ "\n".join(final_output_files) + "\n"
  609. logger.info(message)
  610. print(message)
  612. if __name__ == '__main__':
  613. main()

knead_data.py at commit af84ded, under MIT · at the source

Overview

Authors: Abigail L. Zeamer1,2, YuShuan Lai1,2, Ethan Loew1, Victoria Sanborn3, Matthew Tracy1, Cynthia Jo4, Danielle Ferdinand4, Doyle V. Ward1,2, Shakti K. Bhattarai1,2, Johnathan Drake3, Beth A. McCormick1,2, Vanni Bucci1,2, John P. Haran1,2,4
ORCID iDs: Ethan Loew, Danielle Ferdinand, Beth A. McCormick, John P. Haran
  1. Department of Microbiology, University of Massachusetts Chan Medical School, Worcester, MA, USA
  2. Program in Microbiome Dynamics, University of Massachusetts Chan Medical School, Worcester, MA, USA
  3. Rhode Island Hospital, Providence, Rhode Island, USA
  4. Department of Emergency Medicine, University of Massachusetts Chan Medical School, Worcester, MA, USA
Institutions: University of Massachusetts Chan Medical School (United States); Rhode Island Hospital (United States)
Journal: Gut microbes, volume 18, issue 1, article 2676162
Dates: published online 24 May 2026; in print December 2026
Type: Data paper · Language: English
License: CC BY
Identifiers: DOI 10.1080/19490976.2026.2676162 · PMID 42178714 · PMCID PMC13203045 · OpenAlex W7162288588
Open access: gold, a free copy (OpenAlex)
Status: code verified
Categories: human (organism), Alzheimer's / dementia (population)
Methods: Statistics, Machine learning, Preprocessing, Connectivity
Keywords: Alzheimer's disease, mild cognitive impairment, cognition, ADAS-Cog, clinical dementia rating (CDR) scale, NIH toolbox, memory, executive function, microbiome, methionine, putrescine, polyamines, cysteine, urea cycle, folate, vitamin B12
MeSH: Alzheimer Disease*, Bacteria*, Cognition*, Cognitive Dysfunction*, Gastrointestinal Microbiome*, Aged, Aged, 80 and over, Female, Humans, Male, Metabolic Networks and Pathways, Middle Aged (* major topic)
Journal subjects: Data Note
Topic: Gut microbiota and health (Molecular Biology, Biochemistry, Genetics and Molecular Biology), according to OpenAlex
Funding: Gates Family Foundation; National Institute of Allergy and Infectious Diseases (U01 AI172987); National Institute on Aging (2019-AARG-NTF-641955 and R01AG067483-01); Alzheimer’s Association (2019-AARG-NTF-641955)
Citations: not cited yet (Europe PMC); 114 references in the paper

Abstract

Disturbances in the gut microbiome are increasingly correlated with neurodegenerative disorders, including Alzheimer's disease. Multiple lines of emerging evidence are consistent with the microbiome's involvement in disease pathology in AD by triggering or potentiating systemic and neuroinflammation, thereby influencing disease pathology through the “microbiota–gut–brain axis.” Currently, the copathologies contributing to cognitive decline and symptomatic progression in AD remain unknown and understudied. Changes in the gut microbiome composition may offer clues to potential systemic physiologic and neuropathologic changes that contribute to cognitive decline. Here, we recruited a cohort of 260 older adults (aged 60 y or older) living in the community and followed them over time, tracking objective measures of cognition, clinical information, and gut microbiome samples. Subjects were classified as healthy controls, exhibiting mild cognitive impairment, or having dementia based on clinical assessments. Using metagenomic sequencing and gene pathway analyses, we found that certain microbial-encoded metabolic pathways correlated with worse cognitive performance. Specifically, genes involved in the urea cycle, polyamine synthesis, or the metabolism of methionine and cysteine predicted worse cognitive performance. Our study suggests that the gut microbiome composition may be linked to cognitive impairment along the AD continuum and points to microbial metabolic pathways that may potentiate disease.

Reproduced under the paper's license (CC BY), from the paper cited above.

Repository

Its files are read in the Code ↔ Paper reader above, with 1 match between paragraphs and lines of code.

biobakery/kneaddata

License: MIT
State: the link answers, verified on 28 September 2026
Evidence: files inventoried
Commit: af84ded250ba2f046c57e76bb66d59d542f541ab, 10 July 2026
Languages: Python (28)
Size: 62 files, 28 scripts
Software Heritage: archived
Found in: the text, “Sample handling and DNA sequencing”
Holds: README, license file, environment (setup.py), tests, continuous integration
Not found: CITATION.cff, documentation
Tools: Biopython (1 file)
Availability: 1 check, the latest on 28 September 2026: the link answers
  • 28 September 2026: the link answers
30 files
At the source: github.com/biobakery/kneaddata

Tracing map

Proposed by the machine: these links were found in the paper and verified at the source, without human review. The map will receive a Zenodo DOI once one of the paper's authors has validated it with their ORCID.

What the map holds:

  • 1 repository of the authors' code, each at its verified commit, with its license and how the link was found in the paper;
  • 28 scripts, each with its path and the digest of its content;
  • 1 match between paragraphs of the paper and lines of the code (method lexical-v1);
  • neither the text of the paper nor the code itself.

Its JSON (tracing-map.json) is deposited on Zenodo with its DOI once the map is validated.

Data

No dataset and no data link were found in the paper.

Data availability statement

Microbiome sequencing and related metadata are available on NCBI (BioProject ID PRJNA1446836).

Reproduced under the paper's license (CC BY), from the paper cited above.

Versions

The history of this record: each version stored by the harvester or made by a correction of its authors or of the maintainers of its code, and what changed in its facts. The texts of the paper (its abstract, its availability statements) are not part of it; versions that changed only those are not listed.

Version 1, 28 September 2026: the first record

Recorded: type, language, journal, volume, issue, pages, dates, 13 authors, 16 keywords, 12 MeSH terms, 4 funders, 109 references.

Cite

This paper

Zeamer, A. L., Lai, Y., Loew, E., Sanborn, V., Tracy, M., Jo, C., Ferdinand, D., Ward, D. V., Bhattarai, S. K., Drake, J., McCormick, B. A., Bucci, V., & Haran, J. P. (2026). Microbiome functional gene pathways are indicative of cognitive performance in older adults at risk for Alzheimer's disease. Gut microbes, 18(1), 2676162. https://doi.org/10.1080/19490976.2026.2676162

BibTeX

@article{zeamer2026microbiome,
author = {Zeamer, Abigail L. and Lai, YuShuan and Loew, Ethan and Sanborn, Victoria and Tracy, Matthew and Jo, Cynthia and Ferdinand, Danielle and Ward, Doyle V. and Bhattarai, Shakti K. and Drake, Johnathan and McCormick, Beth A. and Bucci, Vanni and Haran, John P.},
title = {{Microbiome functional gene pathways are indicative of cognitive performance in older adults at risk for Alzheimer's disease}},
journal = {Gut microbes},
year = {2026},
month = may,
volume = {18},
number = {1},
pages = {2676162},
publisher = {Taylor \& Francis},
issn = {1949-0976},
doi = {10.1080/19490976.2026.2676162},
url = {https://doi.org/10.1080/19490976.2026.2676162},
pmid = {42178714},
pmcid = {PMC13203045}
}

RIS

TY - JOUR
AU - Zeamer, Abigail L.
AU - Lai, YuShuan
AU - Loew, Ethan
AU - Sanborn, Victoria
AU - Tracy, Matthew
AU - Jo, Cynthia
AU - Ferdinand, Danielle
AU - Ward, Doyle V.
AU - Bhattarai, Shakti K.
AU - Drake, Johnathan
AU - McCormick, Beth A.
AU - Bucci, Vanni
AU - Haran, John P.
TI - Microbiome functional gene pathways are indicative of cognitive performance in older adults at risk for Alzheimer's disease
T2 - Gut microbes
J2 - Gut Microbes
PY - 2026
DA - 2026/05/24
VL - 18
IS - 1
SP - 2676162
SN - 1949-0976
PB - Taylor & Francis
DO - 10.1080/19490976.2026.2676162
UR - https://doi.org/10.1080/19490976.2026.2676162
LA - en
ER -

CSL-JSON

{
"id": "10.1080/19490976.2026.2676162",
"type": "article-journal",
"title": "Microbiome functional gene pathways are indicative of cognitive performance in older adults at risk for Alzheimer's disease",
"container-title": "Gut microbes",
"author": [
{
"family": "Zeamer",
"given": "Abigail L."
},
{
"family": "Lai",
"given": "YuShuan"
},
{
"family": "Loew",
"given": "Ethan"
},
{
"family": "Sanborn",
"given": "Victoria"
},
{
"family": "Tracy",
"given": "Matthew"
},
{
"family": "Jo",
"given": "Cynthia"
},
{
"family": "Ferdinand",
"given": "Danielle"
},
{
"family": "Ward",
"given": "Doyle V."
},
{
"family": "Bhattarai",
"given": "Shakti K."
},
{
"family": "Drake",
"given": "Johnathan"
},
{
"family": "McCormick",
"given": "Beth A."
},
{
"family": "Bucci",
"given": "Vanni"
},
{
"family": "Haran",
"given": "John P."
}
],
"container-title-short": "Gut Microbes",
"volume": "18",
"issue": "1",
"page": "2676162",
"DOI": "10.1080/19490976.2026.2676162",
"PMID": "42178714",
"PMCID": "PMC13203045",
"ISSN": "1949-0976",
"publisher": "Taylor & Francis",
"URL": "https://doi.org/10.1080/19490976.2026.2676162",
"language": "en",
"issued": {
"date-parts": [
[
2026,
5,
24
]
]
}
}

The tracing map gets a citation of its own once an author has validated it and it has a DOI.

Similar papers

The papers with a page that share the most with this one: the tools found in their code, their categories, datasets, cited references and authors, the rarest counting most.

[1] doi:10.1186/s40168-026-02342-8 [code]
Impacts of host genetics on gut microbiome composition in Alzheimer's disease.
Journal: Microbiome
In common: Alzheimer's / dementia, 5 references
[2] doi:10.1038/s41467-026-74775-6
Select microbial metabolites promote tau aggregation in a murine tauopathy model.
Journal: Nature communications
In common: Alzheimer's / dementia, 3 references
[3] doi:10.64898/2026.03.09.710596 [code]
Infant gut microbiomes contribute to metabolic states that impact brain function
Journal: bioRxiv (preprint)
In common: Biopython, 2 references
[4] doi:10.1016/j.tjpad.2026.100601
Identification of a CD44-dependent control of astrocytic autophagic activity in Alzheimer's disease.
Journal: The journal of prevention of Alzheimer's disease
In common: Alzheimer's / dementia, 4 references
[5] doi:10.1016/j.dcn.2026.101744 [code]
The longitudinal structure of cognition in the ABCD study and associations with neural function and psychopathology: A Bayesian probabilistic principal components analysis.
Journal: Developmental cognitive neuroscience
In common: 3 references
[6] doi:10.1038/s41746-026-02732-0 [code]
A dynamic risk prediction framework for Alzheimer's disease and related dementias with interpretability.
Journal: NPJ digital medicine
In common: Alzheimer's / dementia, 3 references
[7] doi:10.1016/j.bbih.2026.101275 [code]
A protocol for the Teen Bugs study: An integrative, multi-omics approach to understanding the role of the gut microbiome and mesocorticolimbic system in adolescent mental health following early adverse caregiving.
Journal: Brain, behavior, & immunity - health
In common: 3 references
[8] doi:10.1038/s41540-026-00677-4
Neuro-Dynamic Quantitative Systems Pharmacology (QSP) model describing Alzheimer's disease pathophysiology and treatment effects.
Journal: NPJ systems biology and applications
In common: Alzheimer's / dementia, 3 references
[9] doi:10.64898/2026.05.06.26352540 [code]
Generating synthetic tau-PET scans in Alzheimer’s disease from MRI, blood biomarkers and demographics with deep learning
Journal: medRxiv (preprint)
In common: Alzheimer's / dementia, 3 references
[10] doi:10.1007/s12035-026-06014-4
Hypothalamus Amyloid Levels Are Associated with Early Sex-Dependent Alterations in Peripheral Energy Homeostasis in TgF344-AD Rats.
Journal: Molecular neurobiology
In common: Alzheimer's / dementia, 3 references

Contribute

The authors of this paper can claim it, correct its record and validate its tracing map, and the maintainers of its code (its owner, or a public member of its organization) correct what it says of their repository; anyone signed in can ask for its removal. Every request goes to OSCR's own machine, which answers it; your account page follows them.

Sign in with ORCID to claim this paper as one of its authors, correct its record or validate its tracing map: when the paper's metadata lists your ORCID iD, you are recognized at once. Maintainers of its code: sign in with GitHub, then claim the repository on your account page.

Request its removal

To ask OSCR to remove this record, the copies of its authors' scripts or its tracing map, use the removal request page: signed in, you say who you are, what to remove and why, then review and confirm the request. Published rules decide every request (how).

Discussion, reproductions, activity

Discussion: questions and error reports about this paper and its code, from signed-in readers and its authors. It opens with sign-in.

Reproductions: reports from readers who ran the authors' code: what they reproduced, with which environment, commit and data. It opens with sign-in.

Activity: what happens around this paper: new versions of its record, its map's validation, discussions and reproductions. It opens with sign-in.