NFCORE_HIC:HIC:CUSTOM_DUMPSOFTWAREVERSIONS (1) terminated with an error exit status (1)
aghr opened this issue · comments
Description of the bug
The job CUSTOM_DUMPSOFTWAREVERSIONS should process software versions from file collated_versions.yml. I believe this file collated_versions.yml is created in some previous job of this pipeline. The creation of the file collated_versions.yml seems to have a bug as the resulting file collated_versions.yml does not agree to the YAML format, here is the excerpt that causes the error:
...
"NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:COMBINE_MATES":
python: 3.9.12
"NFCORE_HIC:HIC:TADS:COOLTOOLS_INSULATION":
cooltools: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-wano91yj because the default path (/home/XXXX/.cache/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
0.5.1
"NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN_TRIMMED":
bowtie2: 2.4.4
samtools: 1.16.1
pigz: 2.6
...
To avoid that error I've tried to set the env var MPLCONFIGDIR but without success. I believe the reason is that env vars I define before running nf-core hic will not be defined inside the singularity container where the pipeline jobs are executed. Therein in these containers the env var MPLCONFIGDIR is still undefined.
I think the creation of the file collated_versions.yml needs to be improved to avoid copying warnings and other stuff from the output of 'software x --version' into the file collated_versions.yml .
Command used and terminal output
export MPLCONFIGDIR=/scratch/XXXX/nf_core_hic/matplotlib_config_dir
export NXF_SINGULARITY_CACHEDIR=/scratch/singularity_cache
nextflow run nf-core/hic -profile test,singularity --outdir /scratch/XXXX/nf_core_hic/output
===================
-[nf-core/hic] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_HIC:HIC:CUSTOM_DUMPSOFTWAREVERSIONS (1)'
Caused by:
Process `NFCORE_HIC:HIC:CUSTOM_DUMPSOFTWAREVERSIONS (1)` terminated with an error exit status (1)
Command executed [/home/XXXX/.nextflow/agohrssets/nf-core/hic/./workflows/../modules/nf-core/custom/dumpsoftwareversions/templates/dumpsoftwa
reversions.py]:
#!/usr/bin/env python
"""Provide functions to merge multiple versions.yml files."""
import yaml
import platform
from textwrap import dedent
def _make_versions_html(versions):
"""Generate a tabular HTML output of all versions for MultiQC."""
html = [
dedent(
"""\
<style>
#nf-core-versions tbody:nth-child(even) {
background-color: #f2f2f2;
}
</style>
<table class="table" style="width:100%" id="nf-core-versions">
<thead>
<tr>
<th> Process Name </th>
<th> Software </th>
<th> Version </th>
</tr>
</thead>
"""
)
]
for process, tmp_versions in sorted(versions.items()):
html.append("<tbody>")
for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
html.append(
dedent(
f"""\
<tr>
<td><samp>{process if (i == 0) else ''}</samp></td>
<td><samp>{tool}</samp></td>
<td><samp>{version}</samp></td>
</tr>
"""
)
)
html.append("</tbody>")
html.append("</table>")
return "\n".join(html)
def main():
"""Load all version files and generate merged output."""
versions_this_module = {}
versions_this_module["NFCORE_HIC:HIC:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
"python": platform.python_version(),
"yaml": yaml.__version__,
}
with open("collated_versions.yml") as f:
versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module
# aggregate versions by the module name (derived from fully-qualified process name)
versions_by_module = {}
for process, process_versions in versions_by_process.items():
module = process.split(":")[-1]
try:
if versions_by_module[module] != process_versions:
raise AssertionError(
"We assume that software versions are the same between all modules. "
"If you see this error-message it means you discovered an edge-case "
"and should open an issue in nf-core/tools. "
)
except KeyError:
versions_by_module[module] = process_versions
versions_by_module["Workflow"] = {
"Nextflow": "23.04.3",
"nf-core/hic": "2.1.0",
}
versions_mqc = {
"id": "software_versions",
"section_name": "nf-core/hic Software Versions",
"section_href": "https://github.com/nf-core/hic",
"plot_type": "html",
"description": "are collected at run time from the software output.",
"data": _make_versions_html(versions_by_module),
}
with open("software_versions.yml", "w") as f:
yaml.dump(versions_by_module, f, default_flow_style=False)
with open("software_versions_mqc.yml", "w") as f:
yaml.dump(versions_mqc, f, default_flow_style=False)
with open("versions.yml", "w") as f:
yaml.dump(versions_this_module, f, default_flow_style=False)
if __name__ == "__main__":
main()
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File ".command.sh", line 101, in <module>
main()
File ".command.sh", line 61, in main
versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
return loader.get_single_data()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
while not self.check_event(MappingEndEvent):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
if self.check_token(KeyToken):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 115, in check_token
while self.need_more_tokens():
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 152, in need_more_tokens
self.stale_possible_simple_keys()
File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 291, in stale_possible_simple_keys
raise ScannerError("while scanning a simple key", key.mark,
yaml.scanner.ScannerError: while scanning a simple key
in "collated_versions.yml", line 15, column 1
could not find expected ':'
in "collated_versions.yml", line 16, column 1
Work dir:
/scratch/XXXX/nf_core_hic/work/e0/cfebcb3608555b7094944e8079bc42
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
-- Check '.nextflow.log' file for detailsRelevant files
System information
- N E X T F L O W, version 23.04.3 build 5875
- compute server with Linux
- run on local hard disk of compute server
- run without slurm etc; local executor
- run with singularity version 3.8.6
- openjdk 17.0.3-internal 2022-04-19
- OpenJDK Runtime Environment (build 17.0.3-internal+0-adhoc..src)
- OpenJDK 64-Bit Server VM (build 17.0.3-internal+0-adhoc..src, mixed mode, sharing)
- NAME="CentOS Linux"
- VERSION="7 (Core)"
- ID="centos"
- ID_LIKE="rhel fedora"
- VERSION_ID="7"
- PRETTY_NAME="CentOS Linux 7 (Core)"
- ANSI_COLOR="0;31"
- CPE_NAME="cpe:/o:centos:centos:7"
Not sure why it's happening, probably just needs to be bumped to the latest versions.
Hi, I have exactly the same error on the Bianca server from UPPMAX. anyway to skip this step?
Just in a config file and use nextflow -c bonus.config
process {
withName: "CUSTOM_DUMPSOFTWAREVERSIONS" {
ext.when = false
}
}
Thanks. it works after updated the config file