CWL based workflow to assemble haploid/diploid eukaryote genomes of non-model organisms

The workflow is designed to use both PacBio long-reads and Illumina short-reads. The workflow first extracts, corrects, trims and decontaminates the long reads. Decontaminated trimmed reads are then used to assemble the genome and raw reads are used to polish it. Next, Illumina reads are cleaned and used to further polish the resultant assembly. Finally, the polished assembly is masked using inferred repeats and haplotypes are eliminated. The workflow uses BioConda and DockerHub to install required software and is therefore fully automated. In addition to final assembly, the workflow produces intermediate assemblies before and after polishing steps. The workflow follows the syntax for CWL v1.0.

Dependencies

Programs

The pipeline can be run either using Cromwell or cwltool reference implementation and docker containers can be run either using Singularity or udocker.

Cromwell implementation

• cromwell v44
• java-jdk v8.0.112

Reference implementation

• cwltool v1.0.20181012180214
• nodejs v10.4.1 required by cwltool
• Python library galaxy-lib v18.5.7

Singularity software packages have to be installed server-wide by administrator

• Singularity v3.2.1
• squashfs-tools v4.3.0

Udocker software package can be installed locally

• udocker v1.1.2

Data

• Illumina adapters converted to FASTA format
• NCBI nucleotide non-redundant sequences for decontamination with Centrifuge
• RepBase v17.02 file RMRBSeqs.embl

Installation

Install miniconda using installation script installConda.sh. To install CWL, use either installation script installCromwell.sh or installCwltool.sh. To install udocker, use installation script installUdocker.sh. To install singularity, ask your system administrator.

# First confirm that you have the program 'git' installed in your system
> cd
> git clone -b 'v0.1.3-beta' --single-branch --depth 1 https://github.com/vetscience/Assemblosis
> cd Assemblosis
> bash installConda.sh
> bash installCromwell.sh # or bash installCwltool.sh
> bash installUdocker.sh # if singularity cannot be installed or does not run

For data dependencies: download and extract RepBase database, download Centrifuge version of NCBI nt database and create Illumina adapter FASTA file to your preferred locations. If your reads are clean from adapters, the adapter FASTA file can be empty. Give the location of these data in the configuration (.yml) file (see Usage).

Usage

You have to create a YAML (.yml) file for each assembly. This file defines the required parameters and the location for both PacBio and Illumina raw-reads.

> cd
> export PATH=~/miniconda3/bin:$PATH
> cd Assemblosis/Run
> cp ../Examples/assemblyCele.yml .

"Edit assemblyCele.yml to fit your computing environment and to define the location for the read files, databases and Illumina adapters"

"Running docker images using Cromwell and singularity:"
> java -Dconfig.file=cromwell.udocker.conf -jar cromwell-44.jar run -t CWL -v v1.0 assembly.cwl -i assemblyCele.yml

"Running docker images using Cromwell and udocker:"
> java -Dconfig.file=cromwell.singularity.conf -jar cromwell-44.jar run -t CWL -v v1.0 assembly.cwl -i assemblyCele.yml

"Running docker images using Cwltool and singularity:"
> cwltool --tmpdir-prefix /home//Tmp --beta-conda-dependencies --cachedir /home//Cache --singularity --leave-tmpdir assembly.cwl assemblyCele.yml

"Running docker images using Cwltool and udocker:"
> cwltool --tmpdir-prefix /home//Tmp --beta-conda-dependencies --cachedir /home//Cache --user-space-docker-cmd udocker --leave-tmpdir assembly.cwl assemblyCele.yml

An annotated example of the YAML file for Caenorhabditis elegans assembly.

## Directory, which contains the PacBio raw data
# NOTE! The software looks for all .h5 file (or bam files if bacBioInBam below is defined true) in given directory
pacBioDataDir:
  class: Directory
  location: /home//Dna

## PacBio files are in bam format as returned from Sequel platform
pacBioInBam: true

## Prefix for the resultant assembly files
prefix: cele

## Maximum number of threads used in the pipeline
threads: 24

## Minimum number of threads per job used in canu assembler
minThreads: 4

## Number of concurrent jobs in canu assembler (recommended to use threads / minThreads)
canuConcurrency: 6

### Parameters for the program Canu are described in https://canu.readthedocs.io/en/latest/parameter-reference.html
## Expected genome size. This parameter is forwarded to Canu assembler.
genomeSize: 100m

## Minimum length for the PacBio reads used for the assembly. This parameter is forwarded to Canu assembler.
# The maximum resolvable repeat regions becomes 2 x minReadLength
minReadLen: 6000

## Parameter for Canu assembler to adjust to GC-content. Should be 0.15 for high or low GC content.
corMaxEvidenceErate: 0.20

### Parameters for the program Trimmomatic are described in http://www.usadellab.org/cms/?page=trimmomatic
## Paired-end (PE) reads of Illumina raw data. These files are given to the program Trimmomatic.
# NOTE! Data for two paired libraries is given below.
readsPe1:
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598966_1.fastq.gz
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598967_1.fastq.gz
readsPe2:
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598966_2.fastq.gz
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598967_2.fastq.gz

## Phred coding of Illumina data. This parameter is forwarded to Trimmomatic.
# NOTE! Each read-pair needs one phred value.
phredsPe: ['33','33']

## Sliding window and illuminaClip parameters for Trimmomatic
slidingWindow:
    windowSize: 4
    requiredQuality: 25
illuminaClip:
    adapters:
        class: File
        path: 
    seedMismatches: 2
    palindromeClipThreshold: 30
    simpleClipThreshold: 10
    minAdapterLength: 20
    keepBothReads: true
## Further parameters for Trimmomatic
# Required phred-quality for leading 5 nucleotides
leading: 25
# Required phred-quality for trailing 5 nucleotides
trailing: 25
# Minimum accepted read-length to keep the read after trimming
minlen: 40

### Parameters for the program bowtie2 are described in http://bowtie-bio.sourceforge.net/bowtie2/manual.shtml
## Illumina PE fragment length. Program bowtie2 parameter -X.
# NOTE! Each read-pair needs one phred value.
maxFragmentLens: [500, 600]
# Orientation of pair-end reads e.g. 'fr', 'rf', 'ff': Program bowtie2 parameters --fr, --rf or --ff
orientation: 'fr'

### Parameters for the program Pilon are described in https://github.com/broadinstitute/pilon/wiki/Requirements-&-Usage
# Prefix for the resultant pilon polished assembly. Pilon parameter --output
polishedAssembly: celePilon
# This is set 'true' for an organism with diploid genome: Pilon parameter --diploid
diploidOrganism: true
# Value 'bases' fixes snps and indels: Pilon parameter --fix
fix: bases

### Parameters for the program centrifuge are described in http://www.ccb.jhu.edu/software/centrifuge/manual.shtml
# Path to the directory, that contains NCBI nt database in nt.?.cf files. Centrifuge parameter -x
database:
  class: Directory
  path:  /home//ntDatabase
# Lenght of the identical match in nucleotides required to infer a read as contaminant. Centrifuge parameter --min-hitlen
partialMatch: 100
# NCBI taxon root identifers for the species considered contaminants: e.g. bacteria (=2), viruses (=10239), fungi (=4751), mammals (=40674), artificial seqs (=81077). Pipeline specific parameter.
taxons: [2,10239,4751,40674,81077]

## Parameters for the RepeatModeler and RepeatMasker are described in http://www.repeatmasker.org
repBaseLibrary:
  class: File
  # This is the RepBase file from https://www.girinst.org/repbase. RepeatMasker parameter -lib
  path: /home//RepBaseLibrary/RMRBSeqs.embl
# Constant true and false values for repeat masker
trueValue: true
falseValue: false

Runtimes and hardware requirements

The workflow was tested in Linux environment (CentOS Linux release 7.2.1511) in a server with 24 physical CPUs (48 hyperthreaded CPUs) and 512 GB RAM.

Maximum memory usage of 134.1 GB was claimed by the program Centrifuge for each assembly.

Software tools used in this pipeline

• Dextractor v1.0
• Trimmomatic v0.36
• Centrifuge v1.0.3
• Canu v1.8
• Arrow in SmrtLink v7.0.1
• Bowtie 2 v2.2.8
• SAMtools v1.6
• Pilon v1.22
• RepeatMasker v4.0.6
• RepeatModeler v1.0.11
• RepBase v17.02
• HaploMerger2 build_20160512

Cite

If you use the pipeline, please cite: Korhonen, Pasi K., Ross S. Hall, Neil D. Young, and Robin B. Gasser. "Common Workflow Language (CWL)-based software pipeline for de novo genome assembly from long-and short-read data." GigaScience 8, no. 4 (2019): giz014.