Assemblosis
Version 1

Workflow Type: Common Workflow Language

CWL based workflow to assemble haploid/diploid eukaryote genomes of non-model organisms

The workflow is designed to use both PacBio long-reads and Illumina short-reads. The workflow first extracts, corrects, trims and decontaminates the long reads. Decontaminated trimmed reads are then used to assemble the genome and raw reads are used to polish it. Next, Illumina reads are cleaned and used to further polish the resultant assembly. Finally, the polished assembly is masked using inferred repeats and haplotypes are eliminated. The workflow uses BioConda and DockerHub to install required software and is therefore fully automated. In addition to final assembly, the workflow produces intermediate assemblies before and after polishing steps. The workflow follows the syntax for CWL v1.0.

Dependencies

Programs

The pipeline can be run either using Cromwell or cwltool reference implementation and docker containers can be run either using Singularity or udocker.

Cromwell implementation

Reference implementation

Singularity software packages have to be installed server-wide by administrator

Udocker software package can be installed locally

Data

Installation

Install miniconda using installation script installConda.sh. To install CWL, use either installation script installCromwell.sh or installCwltool.sh. To install udocker, use installation script installUdocker.sh. To install singularity, ask your system administrator.

# First confirm that you have the program 'git' installed in your system
> cd
> git clone -b 'v0.1.3-beta' --single-branch --depth 1 https://github.com/vetscience/Assemblosis
> cd Assemblosis
> bash installConda.sh
> bash installCromwell.sh # or bash installCwltool.sh
> bash installUdocker.sh # if singularity cannot be installed or does not run

For data dependencies: download and extract RepBase database, download Centrifuge version of NCBI nt database and create Illumina adapter FASTA file to your preferred locations. If your reads are clean from adapters, the adapter FASTA file can be empty. Give the location of these data in the configuration (.yml) file (see Usage).

Usage

You have to create a YAML (.yml) file for each assembly. This file defines the required parameters and the location for both PacBio and Illumina raw-reads.

> cd
> export PATH=~/miniconda3/bin:$PATH
> cd Assemblosis/Run
> cp ../Examples/assemblyCele.yml .

"Edit assemblyCele.yml to fit your computing environment and to define the location for the read files, databases and Illumina adapters"

"Running docker images using Cromwell and singularity:"
> java -Dconfig.file=cromwell.udocker.conf -jar cromwell-44.jar run -t CWL -v v1.0 assembly.cwl -i assemblyCele.yml

"Running docker images using Cromwell and udocker:"
> java -Dconfig.file=cromwell.singularity.conf -jar cromwell-44.jar run -t CWL -v v1.0 assembly.cwl -i assemblyCele.yml

"Running docker images using Cwltool and singularity:"
> cwltool --tmpdir-prefix /home//Tmp --beta-conda-dependencies --cachedir /home//Cache --singularity --leave-tmpdir assembly.cwl assemblyCele.yml

"Running docker images using Cwltool and udocker:"
> cwltool --tmpdir-prefix /home//Tmp --beta-conda-dependencies --cachedir /home//Cache --user-space-docker-cmd udocker --leave-tmpdir assembly.cwl assemblyCele.yml

An annotated example of the YAML file for Caenorhabditis elegans assembly.

## Directory, which contains the PacBio raw data
# NOTE! The software looks for all .h5 file (or bam files if bacBioInBam below is defined true) in given directory
pacBioDataDir:
  class: Directory
  location: /home//Dna

## PacBio files are in bam format as returned from Sequel platform
pacBioInBam: true

## Prefix for the resultant assembly files
prefix: cele

## Maximum number of threads used in the pipeline
threads: 24

## Minimum number of threads per job used in canu assembler
minThreads: 4

## Number of concurrent jobs in canu assembler (recommended to use threads / minThreads)
canuConcurrency: 6

### Parameters for the program Canu are described in https://canu.readthedocs.io/en/latest/parameter-reference.html
## Expected genome size. This parameter is forwarded to Canu assembler.
genomeSize: 100m

## Minimum length for the PacBio reads used for the assembly. This parameter is forwarded to Canu assembler.
# The maximum resolvable repeat regions becomes 2 x minReadLength
minReadLen: 6000

## Parameter for Canu assembler to adjust to GC-content. Should be 0.15 for high or low GC content.
corMaxEvidenceErate: 0.20

### Parameters for the program Trimmomatic are described in http://www.usadellab.org/cms/?page=trimmomatic
## Paired-end (PE) reads of Illumina raw data. These files are given to the program Trimmomatic.
# NOTE! Data for two paired libraries is given below.
readsPe1:
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598966_1.fastq.gz
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598967_1.fastq.gz
readsPe2:
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598966_2.fastq.gz
  - class: File
    format: edam:format_1930  # fastq
    path: /home//Dna/SRR2598967_2.fastq.gz

## Phred coding of Illumina data. This parameter is forwarded to Trimmomatic.
# NOTE! Each read-pair needs one phred value.
phredsPe: ['33','33']

## Sliding window and illuminaClip parameters for Trimmomatic
slidingWindow:
    windowSize: 4
    requiredQuality: 25
illuminaClip:
    adapters:
        class: File
        path: 
    seedMismatches: 2
    palindromeClipThreshold: 30
    simpleClipThreshold: 10
    minAdapterLength: 20
    keepBothReads: true
## Further parameters for Trimmomatic
# Required phred-quality for leading 5 nucleotides
leading: 25
# Required phred-quality for trailing 5 nucleotides
trailing: 25
# Minimum accepted read-length to keep the read after trimming
minlen: 40

### Parameters for the program bowtie2 are described in http://bowtie-bio.sourceforge.net/bowtie2/manual.shtml
## Illumina PE fragment length. Program bowtie2 parameter -X.
# NOTE! Each read-pair needs one phred value.
maxFragmentLens: [500, 600]
# Orientation of pair-end reads e.g. 'fr', 'rf', 'ff': Program bowtie2 parameters --fr, --rf or --ff
orientation: 'fr'

### Parameters for the program Pilon are described in https://github.com/broadinstitute/pilon/wiki/Requirements-&-Usage
# Prefix for the resultant pilon polished assembly. Pilon parameter --output
polishedAssembly: celePilon
# This is set 'true' for an organism with diploid genome: Pilon parameter --diploid
diploidOrganism: true
# Value 'bases' fixes snps and indels: Pilon parameter --fix
fix: bases

### Parameters for the program centrifuge are described in http://www.ccb.jhu.edu/software/centrifuge/manual.shtml
# Path to the directory, that contains NCBI nt database in nt.?.cf files. Centrifuge parameter -x
database:
  class: Directory
  path:  /home//ntDatabase
# Lenght of the identical match in nucleotides required to infer a read as contaminant. Centrifuge parameter --min-hitlen
partialMatch: 100
# NCBI taxon root identifers for the species considered contaminants: e.g. bacteria (=2), viruses (=10239), fungi (=4751), mammals (=40674), artificial seqs (=81077). Pipeline specific parameter.
taxons: [2,10239,4751,40674,81077]

## Parameters for the RepeatModeler and RepeatMasker are described in http://www.repeatmasker.org
repBaseLibrary:
  class: File
  # This is the RepBase file from https://www.girinst.org/repbase. RepeatMasker parameter -lib
  path: /home//RepBaseLibrary/RMRBSeqs.embl
# Constant true and false values for repeat masker
trueValue: true
falseValue: false

Runtimes and hardware requirements

The workflow was tested in Linux environment (CentOS Linux release 7.2.1511) in a server with 24 physical CPUs (48 hyperthreaded CPUs) and 512 GB RAM.

Assembly Runtime in CPU hours RAM usage (GB)
Caenorhabditis elegans 1537 134.1
Drosophila melanogaster 6501 134.1
Plasmodium falciparum 424 134.1

Maximum memory usage of 134.1 GB was claimed by the program Centrifuge for each assembly.

Software tools used in this pipeline

Cite

If you use the pipeline, please cite: Korhonen, Pasi K., Ross S. Hall, Neil D. Young, and Robin B. Gasser. "Common Workflow Language (CWL)-based software pipeline for de novo genome assembly from long-and short-read data." GigaScience 8, no. 4 (2019): giz014.

Click and drag the diagram to pan, double click or use the controls to zoom.

Inputs

ID Name Description Type
pacBioDataDir n/a n/a
  • Directory
pacBioTmpDir n/a n/a
  • string
pacBioInBam n/a n/a
  • boolean
prefix n/a n/a
  • string
genomeSize n/a n/a
  • string
minReadLen n/a n/a
  • int
corMaxEvidenceErate n/a n/a
  • float
readsPe1 n/a n/a
  • File[]
readsPe2 n/a n/a
  • File[]
phredsPe n/a n/a
  • string[]
slidingWindow n/a n/a
  • assembly-typedef.yml#slidingWindow
illuminaClip n/a n/a
  • assembly-typedef.yml#illuminaClipping?
leading n/a n/a
  • int
trailing n/a n/a
  • int
minlen n/a n/a
  • int
threads n/a n/a
  • int
minThreads n/a n/a
  • int
canuConcurrency n/a n/a
  • int
orientation n/a n/a
  • string
maxFragmentLens n/a n/a
  • int[]
polishedAssembly n/a n/a
  • string
diploidOrganism n/a n/a
  • boolean
fix n/a n/a
  • string
database n/a n/a
  • Directory
taxons n/a n/a
  • int[]
partialMatch n/a n/a
  • int
repBaseLibrary n/a n/a
  • File
trueValue n/a n/a
  • boolean
falseValue n/a n/a
  • boolean

Steps

ID Name Description
cleanIlluminaReads n/a n/a
hdf5check n/a n/a
correct n/a n/a
renameReads n/a n/a
classifyReads n/a n/a
decontaminate n/a n/a
assemble n/a n/a
arrow n/a n/a
indexReference n/a n/a
expressionToolBowtie n/a n/a
mapIlluminaReads n/a n/a
sortMappedReads n/a n/a
indexBamFile n/a n/a
expressionToolBam n/a n/a
pilon n/a n/a
indexAssembly n/a n/a
expressionToolRepeatModeler n/a n/a
inferRepeats n/a n/a
maskCustomRepeats n/a n/a
maskTranspRepeats n/a n/a
maskSimpleRepeats n/a n/a
combineCatFiles n/a n/a
haploMerge n/a n/a

Outputs

ID Name Description Type
correctedReads n/a n/a
  • File
trimmedReads n/a n/a
  • File
canuAssembly n/a n/a
  • File
arrowAssembly n/a n/a
  • File
pilonAssembly n/a n/a
  • File
trimmedReadFiles1 n/a n/a
  • File[]
trimmedReadFiles2 n/a n/a
  • File[]
sortedBamIndexFileOut n/a n/a
  • File[]
deconReport n/a n/a
  • File
deconClassification n/a n/a
  • File
decontaminatedReads n/a n/a
  • File
contaminatedReads n/a n/a
  • File
assemblyMasked n/a n/a
  • File
assemblyMerged n/a n/a
  • File

Version History

v0.1.3-beta (earliest) Created 20th Apr 2022 at 00:10 by Pasi Korhonen

Preparing support for Cromwell


Frozen v0.1.3-beta c498607
help Creators and Submitter
Creator
Submitter
Activity

Views: 1056

Created: 20th Apr 2022 at 00:10

Last updated: 20th Apr 2022 at 00:23

help Tags

This item has not yet been tagged.

help Attributions

None

Total size: 1.46 MB
Powered by
(v.1.14.1)
Copyright © 2008 - 2023 The University of Manchester and HITS gGmbH