A hybrid pipeline for reconstruction & analysis
of viral and host genomes at multi-organ level.
TRACESPipe is a next-generation sequencing pipeline for identification, assembly, and analysis of viral and human-host genomes at multi-organ level. The identification and assembly of viral genomes rely on cooperation between three modalities:
- compression-based predictors;
- sequence alignments;
- de-novo assembly.
For the human-host variant call identification, the same procedure is followed although directly starting within the second point, given the use of the same reference (revised Cambridge Reference) to all the cases.
The previous image shows the architecture of TRACESPipe, where the green line stands for the mitochondrial human line. This pipeline has been tested in Illumina HiSeq and NovaSeq platforms. The operating system required to run it is Linux. In windows use cygwin (https://www.cygwin.com/) and make sure that it is included in the installation: cmake, make, zcat, unzip, wget, tr, grep (and any dependencies). If you install the complete cygwin packet then all these will be installed. After, all steps will be the same as in Linux.
The TRACESPipe includes methods for ancient DNA authentication, namely using the quantification of damage (in the tips of the reads) relative to a reference. Other feature is the quantification of y-chromosome presence through compression-based predictors.
Additionally, the TRACESPipe includes read trimming and filtering, PhiX removal, and redundancy controls (at the Database level and for each candidate reference genomes) to improve the consistency and quality of the data.
is needed for installation.
To install Conda use the following steps:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
Additional instructions can be found here:
https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html
To install TRACESPipe, run the following commands in a Linux OS:
git clone https://github.com/viromelab/tracespipe.git
cd tracespipe/src/
chmod +x TRACES*.sh
./TRACESPipe.sh --install
./TRACESPipe.sh --get-all-aux
In the tracespipe/ folder the following structure exists:
tracespipe/
│
├── meta_data/ # information about the filenames in input_data/ and organ names
│ └── meta_info.txt # see Configuration section for this file.
│
├── input_data/ # where the NGS reads must be placed (and compressed with gzip)
│
├── output_data/ # where the results will appear using the following subfolders:
│ │
│ ├── TRACES_results/ # where the files regarding the metagenomic
│ │ # analysis, redundancy (complexity) and control will appear
│ ├── TRACES_results/profiles/ # where the redundancy (complexity) profiles appear
│ │
│ ├── TRACES_viral_alignments/ # where viral alignments and index will appear
│ ├── TRACES_viral_consensus/ # where viral consensus (FASTA) will appear
│ ├── TRACES_viral_bed/ # where viral BED files will appear (SNPs and Coverage)
│ ├── TRACES_viral_statistics/ # where viral statistics appear (depth/wide coverage)
│ │
│ ├── TRACES_mtdna_alignments/ # where mtdna alignments and index will appear
│ ├── TRACES_mtdna_consensus/ # where mtdna consensus (FASTA) will appear
│ ├── TRACES_mtdna_bed/ # where mtdna BED files will appear (SNPs and Coverage)
│ ├── TRACES_mtdna_statistics/ # where mtdna statistics appear (depth/wide coverage)
│ ├── TRACES_mtdna_authentication/ # where mtdna species and population authentication appears
│ │
│ ├── TRACES_cy_alignments/ # where cy alignments and index will appear
│ ├── TRACES_cy_consensus/ # where cy consensus (FASTA) will appear
│ ├── TRACES_cy_bed/ # where cy BED files will appear (SNPs and Coverage)
│ ├── TRACES_cy_statistics/ # where cy statistics appear (depth/wide coverage)
│ │
│ ├── TRACES_specific_alignments/ # where specific alignments and index will appear
│ ├── TRACES_specific_consensus/ # where specific consensus (FASTA) will appear
│ ├── TRACES_specific_bed/ # where specific BED files will appear
│ ├── TRACES_specific_statistics/ # where specific statistics appear (depth/wide coverage)
│ │
│ ├── TRACES_mtdna_damage_<ORGAN>/ # where the mtdna damage estimation files will appear
│ │
│ ├── TRACES_denovo_<ORGAN>/ # where the output of de-novo assembly appears
│ │
│ ├── TRACES_hybrid_alignments/ # where the hybrid data appears
│ ├── TRACES_hybrid_consensus/ # where the hybrid data appears
│ ├── TRACES_hybrid_bed/ # where the hybrid data appears
│ │
│ ├── TRACES_hybrid_R2_alignments/ # where the second round hybrid data appears
│ ├── TRACES_hybrid_R2_consensus/ # where the second round hybrid data appears
│ ├── TRACES_hybrid_R2_bed/ # where the second round hybrid data appears
│ │
│ ├── TRACES_hybrid_R3_alignments/ # where the third round hybrid data appears
│ ├── TRACES_hybrid_R3_consensus/ # where the third round hybrid data appears
│ ├── TRACES_hybrid_R3_bed/ # where the third round hybrid data appears
│ │
│ ├── TRACES_hybrid_R4_alignments/ # where the fourth round hybrid data appears
│ ├── TRACES_hybrid_R4_consensus/ # where the fourth round hybrid data appears
│ ├── TRACES_hybrid_R4_bed/ # where the fourth round hybrid data appears
│ │
│ ├── TRACES_hybrid_R5_consensus/ # where the automatic choosen hybrid consensus
│ │ # appears (diff will be made using this data)
│ │
│ ├── TRACES_multiorgan_alignments/ # where the multi-organ alignments data appears
│ ├── TRACES_multiorgan_consensus/ # where the multi-organ consensus data appears
│ │
│ ├── TRACES_diff/ # where the dnadiff results appear (identity & SNPs)
│ ├── TRACES_specific_diff/ # where the dnadiff results appear for specific
│ │
│ └── TRACES_blasts/ # where the specific blasted results appears
│
├── to_encrypt_data/ # where the NGS files to encrypt must be before encryption
├── encrypted_data/ # where the encrypted data will appear
├── decrypted_data/ # where the decrypted data will appear
│
├── logs/ # where the logs (stdout, stderr, and system) will appear
│
├── src/ # where the bash code is and where the commands must be call
│
└── imgs/ # images related with the pipelineTo configure TRACESPipe add your FASTQ files gziped at the folder
input_data/
Then, add a file exclusively with name meta_info.txt at the folder
meta_data/
This file needs to specify the organ type (with a single word name) and the filenames for the paired end reads. An example of the content of meta_info.txt is the following:
skin:V1_S44_R1_001.fastq.gz:V1_S44_R2_001.fastq.gz
brain:V2_S29_R1_001.fastq.gz:V2_S29_R2_001.fastq.gz
colon:V3_S45_R1_001.fastq.gz:V3_S45_R2_001.fastq.gz
Then, at the src/ folder run:
./TRACESPipe.sh --get-all-aux
To run TRACES Pipeline, use the following command:
./TRACESPipe.sh <parameters>
There are many parameters and configurations that can be used.
See the next section for more information about the usage.
./TRACESPipe.sh -h
████████╗ ██████╗ █████╗ ██████╗ ███████╗ ███████╗
╚══██╔══╝ ██╔══██╗ ██╔══██╗ ██╔════╝ ██╔════╝ ██╔════╝
██║ ██████╔╝ ███████║ ██║ █████╗ ███████╗
██║ ██╔══██╗ ██╔══██║ ██║ ██╔══╝ ╚════██║
██║ ██║ ██║ ██║ ██║ ╚██████╗ ███████╗ ███████║
╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚══════╝ ╚══════╝
P I P E L I N E
| A hybrid pipeline for reconstruction & analysis |
| of viral and host genomes at multi-organ level. |
Usage: ./TRACESPipe.sh [options]
-h, --help Show this help message and exit,
-v, --version Show the version and some information,
-f, --force Force running and overwrite of files,
-flog, --flush-logs Flush logs (delete logs),
-fout, --flush-output Flush output data (delete all output_data),
-i, --install Installation of all the tools,
-up, --update Update all the tools in TRACESPipe,
-spv, --show-prog-ver Show included programs versions,
-st, --sample Creates human ref. VDB and sample organ,
-gmt, --get-max-threads Get the number of maximum machine threads,
-t <THREADS>, --threads <THREADS>
Number of threads to use,
-dec, --decrypt Decrypt (all files in ../encrypted_data),
-enc, --encrypt Encrypt (all files in ../to_encrypt_data),
-vdb, --build-viral Build viral database (all) [Recommended],
-vdbr, --build-viral-r Build viral database (references only),
-udb, --build-unviral Build non viral database (control),
-afs <FASTA>, --add-fasta <FASTA>
Add a FASTA sequence to the VDB.fa,
-aes <ID>, --add-extra-seq <ID>
Add extra sequence to the VDB.fa,
-gx, --get-extra-vir Downloads/appends (VDB) extra viral seq,
-gad, --gen-adapters Generate FASTA file with adapters,
-gp, --get-phix Extracts PhiX genomes (Needs viral DB),
-gm, --get-mito Downloads human Mitochondrial genome,
-dwms, --download-mito-species
Downloads the complete NCBI mitogenomes
database containing the existing species,
-dwmp, --download-mito-population
Downloads two complete mitogenome databases
with healthy and pathogenic sequences,
-aums, --auth-mito-species
Autheticate the mitogenome species,
-aump, --auth-mito-population
Authenticate closest population,
-cmt <ID>, --change-mito <ID>
Set any Mitochondrial genome by ID,
-gy, --get-y-chromo Downloads human Y-chromosome,
-gax, --get-all-aux Runs -gad -gp -gm -gy,
-cbn, --create-blast-db It creates a nucleotide blast database,
-ubn, --update-blast-db It updates a nucleotide blast database,
-sfs <FASTA>, --search-blast-db <FASTA>
It blasts the nucleotide (nt) blast DB,
-sfrs <FASTA>, --search-blast-remote-db <FASTA>
It blasts remotly thenucleotide (nt) blast
database (it requires internet connection),
-rdup, --remove-dup Remove duplications (e.g. PCR dup),
-vhs, --very-sensitive Aligns with very high sensitivity (slower),
-gbb, --best-of-bests Identifies the best of bests references
between multiple organs [similar reference],
-iss <SIZE>, --inter-sim-size <SIZE>
Inter-genome similarity top size (control),
-rpro, --run-profiles Run complexity and relative profiles (control),
-rpgi <ID>, --run-gid-complexity-profile <ID>
Run complexity profiles by GID,
-cpwi <VALUE>, --complexity-profile-window <VALUE>
Complexity profile window size,
-cple <VALUE>, --complexity-profile-level <VALUE>
Complexity profile compression level [1;10],
-rm, --run-meta Run viral metagenomic identification,
-ro, --run-meta-nv Run NON-viral metagenomic identification,
-mis <VALUE>, --min-similarity <VALUE>
Minimum similarity value to consider the
sequence for alignment-consensus (filter),
-top <VALUE>, --view-top <VALUE>
Display the top <VALUE> with the highest
similarity (by descending order),
-rava, --run-all-v-alig Run all viral align/sort/consensus seqs
from a specific list,
-rsd <ID>, --run-de-novo-specific <ID/PATTERN>
Run specific alignments of the de-novo
to the reference genome,
-rsr <ID>, --run-specific <ID/PATTERN>
Run specific reference align/consensus,
-rsx <ID>, --run-extreme <ID/PATTERN>
Run specific reference align/consensus
using extreme sensitivity,
-rmt, --run-mito Run Mito align and consensus seq,
-rmtd, --run-mito-dam Run Mito damage only,
-rgid <ID>, --run-gid-damage <ID>
Run damage pattern analysis by GID,
-rya, --run-cy-align Run CY align and consensus seq,
-ryq, --run-cy-quant Estimate the quantity of CY DNA,
-rda, --run-de-novo Run de-novo assembly,
-rhyb, --run-hybrid Run hybrid assembly (align/de-novo),
-rmhc, --run-multiorgan-consensus
Run alignments/consensus between all the
reconstructed organ sequences,
-vis, --visual-align Run Visualization tool for alignments,
-covl, --coverage-latex Run coverage table in Latex format,
-covc, --coverage-csv Run coverage table in CSV format,
-covp <NAME>, --coverage-profile <BED_NAME_FILE>
Run coverage profile for specific BED file,
-cmax <MAX>, --max-coverage <MAX_COVERAGE>
Maximum depth coverage (depth normalization),
-clog <VALUE>, --coverage-log-scale <VALUE>
Coverage profile logarithmic scale VALUE=Base,
-cwis <VALUE>, --coverage-window-size <VALUE>
Coverage window size for low-pass filter,
-cdro <VALUE>, --coverage-drop <VALUE>
Coverage drop size (sampling),
-diff, --run-diff Run diff -> reference and hybrid (ident/SNPs),
-sdiff <V_NAME> <ID/PATTERN>, --run-specific-diff <V_NAME> <ID/PATTERN>
Run specific diff of reconstructed to a virus
pattern of ID. Example: -sdiff B19 AY386330.1,
-brec, --blast-reconstructed
Run local blast over reconstructed genomes,
-ra, --run-analysis Run data analysis (core),
-all, --run-all Run all the options (excluding the specific).
Ex: ./TRACESPipe.sh --flush-output --flush-logs --run-mito --run-meta
--remove-dup --run-de-novo --run-hybrid --min-similarity 1 --run-diff
--very-sensitive --best-of-bests --run-multiorgan-consensus
Add the file meta_info.txt at ../meta_data/ folder. Example:
meta_info.txt -> 'organ:reads_forward.fa.gz:reads_reverse.fa.gz'
The reads must be GZIPed in the ../input_data/ folder.
The output results are at ../output_data/ folder.
Contact: tracespipe@gmail.com
The common use of TRACESPipe as command is:
./TRACESPipe.sh \
--flush-logs \
--run-meta \
--inter-sim-size 2 \
--run-all-v-alig \
--run-mito \
--remove-dup \
--run-de-novo \
--run-hybrid \
--min-similarity 1.5 \
--view-top 5 \
--best-of-bests \
--very-sensitive \
--run-multiorgan-consensus \
--run-diff
From the run all the output is provided at folder output_data and it can be human inspected using IGV.
Nevertheless, for specific runs, below some examples are described.
5.1 Building viral consensus sequences with fixed reference sequence in all organs (if exists in the FASTQ samples):
./TRACESPipe.sh --run-meta --run-all-v-alig --remove-dup --min-similarity 3 --best-of-bests
The output consensus sequence is included at
output_data/TRACES_viral_consensus
while the alignments at
output_data/TRACES_viral_alignments
and the BED files at
output_data/TRACES_viral_bed
./TRACESPipe.sh --run-mito --remove-dup
The output consensus sequence is included at
output_data/TRACES_mtdna_consensus
while the alignments at
output_data/TRACES_mtdna_alignments
and the BED files at
output_data/TRACES_mtdna_bed
TRACESPipe supports secure encryption of genomic data. This allows outsourcing of the sequencing service while maintaining secure transmission and storage of the files.
Place the files from sequencing (e.g. FASTQ gziped files) in the folder to_encrypt_data and, then, run:
./TRACESPipe.sh --encrypt
Insert a strong password.
The encrypted files are in the encrypted_data folder.
Place the encrypted files in the folder encrypted_data and, then, run:
./TRACESPipe.sh --decrypt
Insert the password that has been used in encryption.
The decrypted files are in the decrypted_data folder.
./TRACESPipe.sh --run-meta --run-all-v-alig
The output consensus sequence is included at
output_data/TRACES_viral_consensus
while the alignments at
output_data/TRACES_viral_alignments
and the BED files at
output_data/TRACES_viral_bed
./TRACESPipe.sh --run-cy-quant
The output quantify is included at
output_data/TRACES_results/REP_CY_<organ_name>.txt
./TRACESPipe.sh --run-meta
The output is included at
../output_data/TRACES_results/REPORT_META_VIRAL_ALL.txt
./TRACESPipe.sh --run-meta-nv
The output is included at
../output_data/TRACES_results/REPORT_META_NON_VIRAL_<organ_name>.txt
./TRACESPipe.sh --run-de-novo
The outputs are included at
../output_data/TRACES_denovo_alignments
../output_data/TRACES_denovo_consensus
../output_data/TRACES_denovo_bed
5.9 Run specific viral alignment (AF037218.1) for all organs using extreme sensitivity without duplications:
./TRACESPipe.sh --remove-dup --run-extreme AF037218.1
The output is included at
../output_data/TRACES_specific_alignments
and the depth and breadth coverage values at
cat ../output_data/TRACES_specific_statistics
./TRACESPipe.sh --run-mito-dam
The output is included at
../output_data/TRACES_mtdna_damage_<organ_name>
./TRACESPipe.sh --search-blast-remote-db AF037218.1
The output is included at
../output_data/TRACES_blastn
This approach assumes that the reconstruction has already been processed:
./TRACES_normalized_depth.sh ../output_data/TRACES_viral_bed/B19-coverage-blood.bed 200
The output is provided to the stdout.
TRACES Pipeline uses a combination of the following tools:
| Tool | URL | Article |
|---|---|---|
| 💚 Cryfa | [https://github.com/cobilab/cryfa] |  |
| 💚 Entrez | [https://www.ncbi.nlm.nih.gov/genome] | |
| 💚 GTO | [https://github.com/cobilab/gto] | |
| 💚 Trimmomatic | [http://www.usadellab.org/cms/?page=trimmomatic] | |
| 💚 MAGNET | [https://github.com/cobilab/magnet] | |
| 💚 FALCON-meta | [https://github.com/cobilab/falcon] | |
| 💚 Bowtie2 | [http://bowtie-bio.sourceforge.net/bowtie2] | |
| 💚 Bwa | [http://bio-bwa.sourceforge.net/] | |
| 💚 metaSPAdes | [http://cab.spbu.ru/software/meta-spades/] | |
| 💚 Samtools | [http://samtools.sourceforge.net/] | |
| 💚 Bcftools | [http://www.htslib.org/doc/bcftools.html] | |
| 💚 Tabix | [http://htslib.org/doc/tabix.html] | |
| 💚 BEDtools | [https://bedtools.readthedocs.io/en/latest/] | |
| 💚 IGV | [https://software.broadinstitute.org/software/igv/] | |
| 💚 mapDamage2 | [https://ginolhac.github.io/mapDamage/] | |
| 💚 Blastn | [https://blast.ncbi.nlm.nih.gov/] | |
| 💚 mummer4 | [https://mummer4.github.io/] | |
| 💚 ivar | [https://andersen-lab.github.io/ivar] | |
If you use this pipeline, please cite:
Pratas, D., Toppinen, M., Pyöriä, L., Hedman, K., Sajantila, A. and Perdomo, M.F., 2020.
A hybrid pipeline for reconstruction and analysis of viral genomes at multi-organ level.
GigaScience, 9(8), p.giaa086.
For any issue let us know at issues link.
GPL v3.
For more information see LICENSE file or visit
http://www.gnu.org/licenses/gpl-3.0.html