Giters

ENCODE-DCC / atac-seq-pipeline

ENCODE ATAC-seq pipeline

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Job atac.align:0:2 exited with return code 3

icen234 opened this issue · comments

commented

Describe the bug

I have followed the instructions for installation of the pipeline using conda, the installation was successfully completed but when I run it I get the following error:

Started troubleshooting workflow: id=d4caa269-70bb-4cd7-981a-ccfa4cf0090e, status=Failed
Found failures JSON object.
[
{
"causedBy": [
{
"message": "Job atac.align:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
},
{
"message": "Job atac.align_mito:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"message": "Workflow failed"
}
]

  • Recursively finding failures in calls (tasks)...

==== NAME=atac.align_mito, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=3, JOB_ID=3741
START=2023-02-16T19:31:48.580Z, END=2023-02-16T19:32:01.499Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stderr
STDERR_CONTENTS=

  • Error: pipeline environment (docker, singularity or conda) not found.

STDERR_BACKGROUND_CONTENTS=

  • Error: pipeline environment (docker, singularity or conda) not found.
    ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/non_mito/.samstats.qc’: No such file or directory
    ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/    .read_length.txt’: No such file or directory
    ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/.bam’: No such file or directory
    ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/    .bai’: No such file or directory
    ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/*.samstats.qc’: No such file or directory

OS/Platform

  • OS/Platform: linux supercomputing cluster
  • Conda version: conda 22.11.1
  • Pipeline version: v2.2.2.
  • Caper version: v2.2.3

Caper configuration file

Paste contents of ~/.caper/default.conf.

backend=local

# Local directory for localized files and Cromwell's intermediate files.
# If not defined then Caper will make .caper_tmp/ on CWD or `local-out-dir`.
# /tmp is not recommended since Caper store localized data files here.
local-loc-dir=

cromwell=/user/leuven/345/vsc34531/.caper/cromwell_jar/cromwell-82.jar
womtool=/user/leuven/345/vsc34531/.caper/womtool_jar/womtool-82.jar

Input JSON file

Paste contents of your input JSON file.

{
"atac.title" : "mESCs in serum",
    "atac.description" : "Pluripotency factors functionally premark cell-type-restricted enhancers in ES cells",
    "atac.paired_end" : false,
    "atac.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v3/mm10.tsv",
    "atac.fastqs_rep1_R1" : [
        "/staging/leuven/stg_00072/Ionas/Old_Files/published_data/atacseq_analysis/Kim_Rosenfeld/50k_mESC_serum.fastq.gz"],
    "atac.auto_detect_adapter" : true,
    "atac.enable_gc_bias" : true,
    "atac.enable_tss_enrich" : true,
    "atac.enable_annot_enrich" : true,
    "atac.enable_jsd" : true,

    "atac.align_cpu" : 10,
    "atac.filter_cpu" : 10,
    "atac.bam2ta_cpu" : 10,
    "atac.call_peak_cpu" : 10
}

Troubleshooting result

If you ran caper run without Caper server then Caper automatically runs a troubleshooter for failed workflows. Find troubleshooting result in the bottom of Caper's screen log.

If you ran caper submit with a running Caper server then first find your workflow ID (1st column) with caper list and run caper debug [WORKFLOW_ID].

Paste troubleshooting result.

2023-02-16 20:32:13,623|caper.cromwell_workflow_monitor|INFO| Workflow: id=d4caa269-70bb-4cd7-981a-ccfa4cf0090e, status=Failed
2023-02-16 20:32:27,460|caper.cromwell_metadata|INFO| Wrote metadata file. /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/metadata.json
2023-02-16 20:32:27,460|caper.cromwell|INFO| Workflow failed. Auto-troubleshooting...
* Started troubleshooting workflow: id=d4caa269-70bb-4cd7-981a-ccfa4cf0090e, status=Failed
* Found failures JSON object.
[
    {
        "causedBy": [
            {
                "message": "Job atac.align:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
                "causedBy": []
            },
            {
                "message": "Job atac.align_mito:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
                "causedBy": []
            }
        ],
        "message": "Workflow failed"
    }
]
* Recursively finding failures in calls (tasks)...

==== NAME=atac.align_mito, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=3, JOB_ID=3741
START=2023-02-16T19:31:48.580Z, END=2023-02-16T19:32:01.499Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stderr
STDERR_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.

STDERR_BACKGROUND_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/non_mito/*.samstats.qc’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/*.read_length.txt’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/*.bam’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/*.bai’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/*.samstats.qc’: No such file or directory



==== NAME=atac.align_mito, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=3, JOB_ID=5928
START=2023-02-16T19:32:02.578Z, END=2023-02-16T19:32:13.482Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.

STDERR_BACKGROUND_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/non_mito/*.samstats.qc’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/*.read_length.txt’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/*.bam’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/*.bai’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/attempt-2/execution/*.samstats.qc’: No such file or directory



==== NAME=atac.align, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=3, JOB_ID=3024
START=2023-02-16T19:31:44.606Z, END=2023-02-16T19:31:56.505Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/stderr
STDERR_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.

STDERR_BACKGROUND_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/non_mito/*.samstats.qc’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/*.read_length.txt’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/*.bam’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/*.bai’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/execution/*.samstats.qc’: No such file or directory



==== NAME=atac.align, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=3, JOB_ID=5162
START=2023-02-16T19:31:58.581Z, END=2023-02-16T19:32:09.686Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.

STDERR_BACKGROUND_CONTENTS=

* Error: pipeline environment (docker, singularity or conda) not found.
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/non_mito/*.samstats.qc’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/*.read_length.txt’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/*.bam’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/*.bai’: No such file or directory
ln: failed to access ‘/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align/shard-0/attempt-2/execution/*.samstats.qc’: No such file or directory
commented

Please specify an environment to be used for the pipeline (--singularity, --docker or --conda). They are flags. We recommend to use singularity on HPCs and docker on PCs.

# PCs
$ caper run WDL -i input.json --docker

# HPCs: make sure to configure Caper's config for HPCs first. See README
$ caper hpc submit WDL -i input.json --singularity
commented

Thanks for the fast reply! I specify --conda. This is the command I use:
$ caper run atac.wdl -i "/staging/leuven/stg_00072/Ionas/test.json" --conda

commented

I am using PBS/Torque. When I set pbs as the caper backend, I get a different error:

2023-02-20 13:49:59,212|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2023-02-20 13:50:02,505|caper.nb_subproc_thread|INFO| Subprocess finished successfully.
2023-02-20 13:50:02,506|caper.cromwell|INFO| Passed Womtool validation.
2023-02-20 13:50:02,507|caper.caper_runner|INFO| launching run: wdl=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac.wdl, inputs=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/.caper_tmp/staging/leuven/stg_00072/Ionas/test.local.json, backend_conf=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/.caper_tmp/atac/20230220_134931_302733/backend.conf
2023-02-20 13:50:07,545|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Submitted
2023-02-20 13:50:07,580|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Running
2023-02-20 13:51:46,761|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Failed
2023-02-20 13:52:02,925|caper.cromwell_metadata|INFO| Wrote metadata file. /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/metadata.json
2023-02-20 13:52:02,925|caper.cromwell|INFO| Workflow failed. Auto-troubleshooting...
* Started troubleshooting workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Failed
* Found failures JSON object.
[
    {
        "causedBy": [
            {
                "causedBy": [
                    {
                        "message": "Could not find job ID from stdout file.Check the stderr file for possible errors: /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr.submit",
                        "causedBy": []
                    }
                ],
                "message": "java.lang.RuntimeException: Could not find job ID from stdout file.Check the stderr file for possible errors: /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr.submit"
            }
        ],
        "message": "Workflow failed"
    }
]
* Recursively finding failures in calls (tasks)...

==== NAME=atac.read_genome_tsv, STATUS=Failed, PARENT=
SHARD_IDX=-1, RC=None, JOB_ID=None
START=2023-02-20T12:50:15.666Z, END=2023-02-20T12:51:46.245Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr
2023-02-20 13:52:02,927|caper.nb_subproc_thread|ERROR| Cromwell failed. returncode=1
2023-02-20 13:52:02,928|caper.cli|ERROR| Check stdout in /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/cromwell.out.5

When I ran the pipeline using this command:
caper hpc submit test.wdl -i input.json --conda
I get an error asking me to define --leader-job-name

When I run it as
caper hpc submit test.wdl -i input.json --conda --leader-job-name test1
I get this error:

2023-02-20 13:55:54,260|caper.hpc|INFO| Running shell command: qsub -l walltime=48:00:00,mem=4gb -V -N CAPER_test1 /user/leuven/345/vsc34531/ncbkzi5h.sh
pbs_connect received error code 15008 ('trqauthd unable to authenticate. User or PID of calling process not found or does not match') from trqauthd

Unable to communicate with tier2-p-moab-2.tier2.hpc.kuleuven.be(10.118.224.19)
Unable to authorize user.
qsub: cannot connect to server tier2-p-moab-2.tier2.hpc.kuleuven.be (errno=15008) trqauthd unable to authenticate. User or PID of calling process not found or does not match

Any help? Thanks!

commented

First of all, you need to run caper init YOUR_CLUSTER_TYPE to initialize Caper's configuration file for your cluster.
backend=local doesn't work with caper hpc subcommands.

# for example of PBS
$ caper init pbs # supported cluster type: slurm, sge, pbs, lsf

Please read Caper README carefully to configure it correctly for your cluster type.
Also, we don't recommend Conda. Does your cluster support Singularity?