@params.txt parsing is various types of broken

Question

@params.txt parsing is various types of broken

gdevenyi opened this issue 5 years ago · comments

Gabriel A. Devenyi commented 5 years ago

I'm trying to move my variables defined at the top of pipeline into a config file so I can offer differently configured versions.

At first glance, I expected to be able to just move the variables into a file and add "-p" to the front, and everything would go.

So, a variable defined like this:

-p BEASTMODEL_DIR="${System.getenv().QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2"

Fails:

Could not understand command ${System.getenv().QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2 or find it as a file

Changing to the bash style:

-p BEASTMODEL_DIR="${QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2"

Still fails:

Could not understand command ${QUARANTINE_PATH}/resources/mni_icbm152_nlin_sym_09c_minc2 or find it as a file

Changing it to a real path:

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"

Still fails with java errors:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
	at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.io.FileNotFoundException: /opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2 (Is a directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:71)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1572)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600)
	at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source)
	at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
	at bpipe.Runner.loadPipelineSrc(Runner.groovy:662)
	at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
	at bpipe.Runner.main(Runner.groovy:377)
	... 6 more

Simon Sadedin · Answer 1 · Fri Apr 19 2019 05:32:20 GMT+0800 (China Standard Time)

Can you show the whole command line you are using? Those failures look a bit like what I would expect if the parameters are inserted after the script file rather than before. The general form has to be:

bpipe run <params here> pipeline.groovy <inputs here>

Can you confirm that your form follows the above?

Gabriel A. Devenyi · Answer 2 · Fri Apr 19 2019 08:53:48 GMT+0800 (China Standard Time)

I can confirm I'm running it that way:

bpipe run @/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ../subject*

Gabriel A. Devenyi · Answer 3 · Fri Apr 19 2019 09:03:32 GMT+0800 (China Standard Time)

Moving the -n300 had the same result

Simon Sadedin · Answer 4 · Sun Apr 21 2019 09:23:10 GMT+0800 (China Standard Time)

Very strange, I can't pick what's wrong.

The reading of the parameters from the file is being done under the hood by the same library that parses the command arguments (not Bpipe doing that explicitly). The behavior of that library is that it literally takes whatever is in the file and injects it as command line arguments.

So it ought to work if the contents of your file is like:

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"

Note the -p is necessary at the start.

Obviously, it's critical that there are no spaces or anything that would count as a space in there. Otherwise, I'm not sure what's wrong!

Simon Sadedin · Answer 5 · Sun Apr 21 2019 09:28:18 GMT+0800 (China Standard Time)

NB: it would be interesting to know if this test passes for you:

https://github.com/ssadedin/bpipe/tree/master/tests/params_from_file

It is not demonstrating ideal usage (the -p should be inside the file!) but it will establish if there's something else really strange interfering (your shell? environment? etc).

Gabriel A. Devenyi · Answer 6 · Mon Apr 22 2019 08:50:13 GMT+0800 (China Standard Time)

20:47:32 [/scratch/bpipe/tests/params_from_file] master ± bash run.sh

ERROR: Failed to find expected parameter text in output

20:49:40 [:/scratch/bpipe/tests/params_from_file] master(+5/-1) ± cat test.out
java.lang.ClassNotFoundException: bpipe.Runner
        at org.codehaus.groovy.tools.RootLoader.findClass(RootLoader.java:179)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.codehaus.groovy.tools.RootLoader.loadClass(RootLoader.java:151)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:99)
        at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)

Simon Sadedin · Answer 7 · Mon Apr 22 2019 09:55:54 GMT+0800 (China Standard Time)

Hmm, that's a problem, but not the one I expected! Just checking the obvious, did you build the code yet?

As in, in the top level directory:

./gradlew dist

Gabriel A. Devenyi · Answer 8 · Tue Apr 23 2019 02:17:24 GMT+0800 (China Standard Time)

I used my bpipe install 0.9.9.6 which has been functioning for me and the 20+ folks who use my pipelines written in bpipe otherwsie :)

Gabriel A. Devenyi · Answer 9 · Tue Apr 23 2019 21:11:52 GMT+0800 (China Standard Time)

0.9.9.7 release has the same bug.

In case it may be relevant:

± java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

Gabriel A. Devenyi · Answer 10 · Thu May 02 2019 21:40:41 GMT+0800 (China Standard Time)

Following up here. Cloning master and building bpipe from source results in a the test passing, so I'm learning towards this bug being an issue with the bpipe distribution version (+ maybe an interaction with my java)

Gabriel A. Devenyi · Answer 11 · Wed May 29 2019 03:32:42 GMT+0800 (China Standard Time)

Hi, just following up again.

The downloadable bpipe version is broken, but if I build locally these tests work. Any ideas why?

Gabriel A. Devenyi · Answer 12 · Fri Aug 02 2019 04:07:41 GMT+0800 (China Standard Time)

Followup here.

The test passes for 0.9.9.7.

If I use the "recommended format"

-p BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"
-p BEASTMODEL_NAME="mni_icbm152_t1_tal_nlin_sym_09c"
-p BEASTLIBRARY_DIR="/opt/quarantine/resources/BEaST_libraries/combined"
-p DEFACEMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c"
-p REGISTRATIONMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc"
-p REGISTRATIONBRAINMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_mask.mnc"
-p REGISTRATIONHEADMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_headmask.mnc"
-p REGISTRATIONOUTLINE="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_outline.mnc"
-p REGISTRATIONANTIMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_antimask.mnc"

bpipe run @/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ~/projects/personal/raw_data/minc/*mnc

Fails with error:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
	at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.io.FileNotFoundException: /opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2 (Is a directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:71)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1572)
	at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600)
	at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source)
	at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
	at bpipe.Runner.loadPipelineSrc(Runner.groovy:672)
	at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
	at bpipe.Runner.main(Runner.groovy:381)
	at bpipe.Runner9.main(Runner9.java:47)
	... 6 more

If instead I setup like the test:

BEASTMODEL_DIR="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2"
BEASTMODEL_NAME="mni_icbm152_t1_tal_nlin_sym_09c"
BEASTLIBRARY_DIR="/opt/quarantine/resources/BEaST_libraries/combined"
DEFACEMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c"
REGISTRATIONMODEL="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc"
REGISTRATIONBRAINMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_mask.mnc"
REGISTRATIONHEADMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_headmask.mnc"
REGISTRATIONOUTLINE="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_outline.mnc"
REGISTRATIONANTIMASK="/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c_antimask.mnc"

And run like the test:

bpipe run -p@/data/chamal/projects/gabriel/src/minc-bpipe-library/default2.conf -n300 ~/projects/src/minc-bpipe-library/pipeline.bpipe ~/projects/personal/raw_data/minc/*mnc

We get success:

====================================================================================================
|                              Starting Pipeline at 2019-08-01 16:04                               |
====================================================================================================

=============================== Stage convert (2015_02_02_DLG_t1w_2) ===============================

=============================== Stage convert (2018_07_16_GAD_t1w_5) ===============================

Gabriel A. Devenyi · Answer 13 · Tue Oct 01 2019 01:30:01 GMT+0800 (China Standard Time)

Hi,

I'm following up again here because I would like to change my pipeline to use configuration file options but I am not comfortable doing so until the difference between the docs and the functional implementation is resolved.

The working version (and the in-repo test code) uses -p@params.txt while the documentation specifies @params.txt directly. Which is considered correct? If the later, it is broken during use.

Gabriel A. Devenyi · Answer 14 · Thu Dec 03 2020 08:58:53 GMT+0800 (China Standard Time)

Followup here, documentation is still unclear.