Quick start guide?

[MikeEHMatson]

Hello,
Would it be possible to provide a quickstart guide or some kind of basic tutorial that explains how to setup the file and directory structure? It is currently not very clear to myself, at least.

For example, is the directory my illumina files (.fastq only, or .fastq.gz?) reside required to be named "Reads"? Do my files need to have the prefix "Reads"?

Apologies for the low level technical question.

[atifrahman]

Hi,
I've expanded the README file.

The version provided assumes that illumina files from each sample is in separate directory and the directories have the prefix. The read files can have any names as long as those are the only fastq (or fastq.gz) files in the directory.

Please let me know if you have other queries.

[MikeEHMatson]

Thanks for the README expansion, but I still seem to be having an issue getting it to work.

In the countKmers script, I have my directory defined as:

dir=/path/to/dir/hawkReads
I'm fairly confident the jellyfish, hawk, and sort directories are correct since what I have filled in came straight from our sysadmin's recommendations.

There are two directories in hawkReads:

/path/to/dir/hawkReads/Reads_nonPath
/path/to/dir/hawkReads/Reads_Path

And one file in each directory:

/path/to/dir/hawkReads/Reads_nonPath/AL1.fastq.gz
/path/to/dir/hawkReads/Reads_Path/11-12.fastq.gz

However, I still get an error:

$ ./countKmers     
                                                                                                                                                                                                       
./countKmers: line 13: 78637 Killed                  ${jellyfishDir}/jellyfish count -C -o ${OUTPREFIX}_kmers/tmp -m ${KMERSIZE} -t ${CORES} -s 20G <( zcat *.fastq.gz )
ls: cannot access Reads_Path_kmers/tmp*: No such file or directory
Error: Requires at least 2 arguments.
Usage: jellyfish merge [options] input:string+
Use --help for more information
ls: cannot access Reads_Path_kmers_jellyfish: No such file or directory
./countKmers: line 13: 78719 Killed                  ${jellyfishDir}/jellyfish count -C -o ${OUTPREFIX}_kmers/tmp -m ${KMERSIZE} -t ${CORES} -s 20G <( zcat *.fastq.gz )
ls: cannot access Reads_nonPath_kmers/tmp*: No such file or directory
Error: Requires at least 2 arguments.
Usage: jellyfish merge [options] input:string+
Use --help for more information
ls: cannot access Reads_nonPath_kmers_jellyfish: No such file or directory

I feel like I am missing something simple, though I can't quite isolate it.

Thanks for any help you can provide.

[atifrahman]

Would it be possible to send me the whole script?

[MikeEHMatson]

Sure. I will add, since it is not apparent here, that the scripts and their parent directory are not in the same directory as specified in line 6 at "dir=/bigdata/..."

CORES=4 #number of cores to use for blast searches
KMERSIZE=31 # RD:61

#modified from NIKS script

dir=/bigdata/judelsonlab/mmatson/usdaTemp/fastqFiles/mps/untrimmed/hawkReads            #directory for read files 
hawkDir=/opt/linux/centos/7.x/x86_64/pkgs/HAWK/0.8.3/                   #directory where hawk is installed
jellyfishDir=/opt/linux/centos/7.x/x86_64/pkgs/jellyfish-hawk/bin/              #directory where jellyfish is installed
sortDir=/usr/bin                #directory where parallel sort is installed

cd ${dir}

for file in `ls -d Reads*`
do
        OUTPREFIX=$file

        cd ${file}

        mkdir ${OUTPREFIX}_kmers

        ${jellyfishDir}/jellyfish count -C -o ${OUTPREFIX}_kmers/tmp -m ${KMERSIZE} -t ${CORES} -s 20G <( zcat *.fastq.gz ) #change if not gzipped

        COUNT=$(ls ${OUTPREFIX}_kmers/tmp* |wc -l)

        if [ $COUNT -eq 1 ]
        then
                mv ${OUTPREFIX}_kmers/tmp_0 ${OUTPREFIX}_kmers_jellyfish
        else
                ${jellyfishDir}/jellyfish merge -o ${OUTPREFIX}_kmers_jellyfish ${OUTPREFIX}_kmers/tmp*
        fi
        rm -rf ${OUTPREFIX}_kmers

        COUNT=$(ls ${OUTPREFIX}_kmers_jellyfish |wc -l)

        if [ $COUNT -eq 1 ]
        then

                ${jellyfishDir}/jellyfish histo -f -o ${OUTPREFIX}.kmers.hist.csv -t ${CORES} ${OUTPREFIX}_kmers_jellyfish
                awk '{print $2"\t"$1}' ${OUTPREFIX}.kmers.hist.csv > ${OUTPREFIX}_tmp
                mv ${OUTPREFIX}_tmp ${OUTPREFIX}.kmers.hist.csv

                awk -f ${hawkDir}/countTotalKmer.awk ${OUTPREFIX}.kmers.hist.csv >> ${dir}/total_kmer_counts.txt

                CUTOFF=1
                echo $CUTOFF > ${OUTPREFIX}_cutoff.csv


                ${jellyfishDir}/jellyfish dump -c -L `expr $CUTOFF + 1` ${OUTPREFIX}_kmers_jellyfish > ${OUTPREFIX}_kmers.txt
                ${sortDir}/sort --parallel=${CORES} -n -k 1 ${OUTPREFIX}_kmers.txt > ${OUTPREFIX}_kmers_sorted.txt

                rm ${OUTPREFIX}_kmers_jellyfish
                rm ${OUTPREFIX}_kmers.txt

                echo "${dir}/Reads_${OUTPREFIX}/${OUTPREFIX}_kmers_sorted.txt" >> ${dir}/sorted_files.txt

        fi


        cd ..

done

[MikeEHMatson]

Hello,
Apologies for sounding impatient, but I was wondering if you had found out what the issue was yet, or if this was the script you were referring to?

Thanks,
Mike

[atifrahman]

Yeah... this the script I wanted but couldn't figure out what the issue is yet.

[MikeEHMatson]

Hi,
I was able to move past the error by giving the script a considerable amount of memory (>200gb). I think Jellyfish needed the memory boost.

Mike

[atifrahman]

Ah great!

[robertwhbaldwin]

Had the same problem working on AWS EC2 service. The problem was how I mounted the instance store device. I used the ext4 file system. So i remounted with sudo mkfs -t xfs /dev/nvme1n1 and countKmers_jf2 and the error from countKmer_jf2 described above disappeared and it worked. So it may have something to do with the file system, not sure.