Error using fastq-dl on HPC Cluster with Sturm
Sven-Winter opened this issue · comments
Hi,
I am trying to use fastq-dl on a HPC cluster with slurm.
However, the download fails due to the wget error Exit Code 4.
If I try it on the head node without job submission the tool and wget works fine.
I've read that HPC cluster nodes often require a proxy for wget.
Is there a wait to use fastq-dl on a cluster node?
Hello @Sven1990 !
Can you confirm whether or not the compute nodes have internet access or not?
Cheers,
Robert
Hi Robert,
I talked to the cluster service unit and they suggest to just change ftp:// to https:// in the wget command that is run by fastq-dl.
I changed that in the script and it downloaded 11 Gb for the first file but now the processes are „sleeping“.
Is there anything in the script that can be adjusted to make this work better?
Best
Sven
… Am 11.04.2022 um 15:00 schrieb Robert A. Petit III ***@***.***>:
Hello @Sven1990 <https://github.com/sven1990> !
Can you confirm whether or not the compute nodes have internet access or not?
Cheers,
Robert
—
Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIVJVI46LQCVFNMDEWH33EDVEQPAVANCNFSM5TDDLTTQ>.
You are receiving this because you were mentioned.
Hmm I had to change to ftp for ENA (https://github.com/rpetit3/fastq-dl/releases/tag/v1.0.6).
You can always try using SRA for the downloads