pack command fails on complex workflow with error 'MissingTypeName'

Question

pack command fails on complex workflow with error 'MissingTypeName'

alexiswl opened this issue a year ago · comments

Steps to reproduce

Download complex workflow

wget "https://github.com/umccr/cwl-ica/releases/download/bclconvert-with-qc-pipeline%2F4.0.3__1682577366/bclconvert-with-qc-pipeline__4.0.3__1682577366.zip"
unzip bclconvert-with-qc-pipeline__4.0.3__1682577366.zip
cd bclconvert-with-qc-pipeline__4.0.3__1682577366/

Validate cwltool --pack works fine

cwltool --version

/home/alexiswl/miniforge3/envs/cwl-ica/bin/cwltool 3.1.20230302145532

cwltool --pack "workflow.cwl" | jq 'keys'

[
  "$graph",
  "$namespaces",
  "$schemas",
  "cwlVersion"
]

Try cwltuils pack

import cwl_utils
import cwl_utils.__meta__
print(cwl_utils.__meta__.__version__)
0.26

from cwl_utils.pack import pack
pack('workflow.cwl')

Gives

Packing workflow.cwl
Parsing 1 types from /home/alexiswl/UMCCR/Projects/202305/create-bclconvert-test/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/bclconvert-run-configuration/2.0.0--4.0.3/bclconvert-run-configuration__2.0.0--4.0.3.yaml
Parsing 1 types from /home/alexiswl/UMCCR/Projects/202305/create-bclconvert-test/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/fastq-list-row/1.0.0/fastq-list-row__1.0.0.yaml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 285, in pack
    cwl = pack_process(cwl, full_url, cwl["cwlVersion"])
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 73, in pack_process
    cwl = resolve_schemadefs(cwl, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 177, in resolve_schemadefs
    cwl = schemadef.inline_types(cwl, "inputs", base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 131, in inline_types
    cwl[port] = [_inline_type(v, base_url, user_defined_types) for v in defs]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 131, in <listcomp>
    cwl[port] = [_inline_type(v, base_url, user_defined_types) for v in defs]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 230, in _inline_type
    v["type"] = _inline_type(_type, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 152, in _inline_type
    "items": _inline_type(v[:-2], base_url, user_defined_types),
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 188, in _inline_type
    return _inline_type(resolve_type, path_prefix, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 221, in _inline_type
    v["fields"] = [
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 222, in <listcomp>
    _inline_type(_f, base_url, user_defined_types) for _f in fields
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 230, in _inline_type
    v["type"] = _inline_type(_type, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 191, in _inline_type
    return [_inline_type(_v, base_url, user_defined_types) for _v in v]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 191, in <listcomp>
    return [_inline_type(_v, base_url, user_defined_types) for _v in v]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 196, in _inline_type
    raise errors.MissingTypeName(
cwl_utils.errors.MissingTypeName: In file /home/alexiswl/UMCCR/Projects/202305/create-bclconvert-test/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/bclconvert-run-configuration/2.0.0--4.0.3/bclconvert-run-configuration__2.0.0--4.0.3.yaml, type None is missing type name

Michael R. Crusoe · Answer 1 · Mon May 15 2023 15:21:17 GMT+0800 (China Standard Time)

Thanks for the report! That code came from sbpack. I wonder if rabix/sbpack#67 is the same issue and if that fix can be ported here.

Michael R. Crusoe · Answer 2 · Tue May 16 2023 22:22:20 GMT+0800 (China Standard Time)

@alexiswl I merged #222 ; did that help?

Alexis Lucattini · Answer 3 · Wed May 17 2023 06:38:43 GMT+0800 (China Standard Time)

I'm getting this now:

Packing workflow.cwl
Parsing 1 types from /tmp/tmp.JdR8Nozs1u/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/bclconvert-run-configuration/2.0.0--4.0.3/bclconvert-run-configuration__2.0.0--4.0.3.yaml
Parsing 1 types from /tmp/tmp.JdR8Nozs1u/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/fastq-list-row/1.0.0/fastq-list-row__1.0.0.yaml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 285, in pack
    cwl = pack_process(cwl, full_url, cwl["cwlVersion"])
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 73, in pack_process
    cwl = resolve_schemadefs(cwl, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/pack.py", line 177, in resolve_schemadefs
    cwl = schemadef.inline_types(cwl, "inputs", base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 132, in inline_types
    cwl[port] = [_inline_type(v, base_url, user_defined_types) for v in defs]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 132, in <listcomp>
    cwl[port] = [_inline_type(v, base_url, user_defined_types) for v in defs]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 237, in _inline_type
    v["type"] = _inline_type(_type, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 153, in _inline_type
    "items": _inline_type(v[:-2], base_url, user_defined_types),
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 189, in _inline_type
    return _inline_type(resolve_type, path_prefix, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 228, in _inline_type
    v["fields"] = [
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 229, in <listcomp>
    _inline_type(_f, base_url, user_defined_types) for _f in fields
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 237, in _inline_type
    v["type"] = _inline_type(_type, base_url, user_defined_types)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 192, in _inline_type
    return [_inline_type(_v, base_url, user_defined_types) for _v in v]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 192, in <listcomp>
    return [_inline_type(_v, base_url, user_defined_types) for _v in v]
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/schemadef.py", line 196, in _inline_type
    imported_type, import_base_url = utils.load_linked_file(
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/site-packages/cwl_utils/utils.py", line 92, in load_linked_file
    contents = pathlib.Path(new_url.path).open().read()
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp.JdR8Nozs1u/bclconvert-with-qc-pipeline__4.0.3__1682577366/schemas/samplesheet/2.0.0--4.0.3/samplesheet__2.0.0--4.0.3.yaml#samplesheet'

Alexis Lucattini · Answer 4 · Wed May 17 2023 06:41:09 GMT+0800 (China Standard Time)

Ah but #223 solves it!

Alexis Lucattini · Answer 5 · Wed May 17 2023 06:56:18 GMT+0800 (China Standard Time)

Sorry, will leave this open and let it be closed automatically by merge of #223