buildcache create fails with Spack develop

[Rocco Meli]

Building a stack with Spack develop fails with the following error:

spack -e ./bootstrap buildcache create --rebuild-index --allow-root --only=package -m alpscache $(spack -e ./bootstrap find --format '/${hash}')
==> Warning: Using flags to specify mirrors is deprecated and will be removed in Spack 0.21, use positional arguments instead.
==> Error: unexpected tokens in the spec string
/$qopnyywo6pbzxnuwmhigd4atak36224e /$o4njaxpd2vbtg3xj6pm7riastfp4gdno /$zh6jx2cafu2oik2thrnfvzenymuwkqbf /$vtsmei3mpeijbilx6nrvnlhdhb6xlara /$qgixj7zzmq7zwtexq2nvhwwzgp5qptwn /$r5fc5voqdghbdr2dwrctqx7l2dxrkz2r /$nckz4pa6jigjgieo4s4yillr5a5pj56m /$6cbkl5kgjgoua7lkwm2573iar7zgmnvb /$x2f7nteaybed5tahr7da3sqnf4lmfy65 /$43334xnuikvddwtq4hwjyy4s2jhi2fxc /$j66ksktzvekpmnyetvwbu2xdovwntdmq /$dflv7d5jedfoiyry6k6i2idav6nqpbas /$lysz7xccwfwiuyh74aw4q43q5bgiv5vn /$y4yleakj6ts7eftfvyxknivvs4ycgris /$eoqugy3pfg53qncmi45fhm4lmcisds7y /$ed674flkfaa37bfvi2kygwthjdlxo7kh /$ed2as5owju3xj5ywvutc4oqskohqndzo /$hmssjulvwz3cnex4ig3xxqioz5yqn63s
^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^   ^^^^ ^                                   ^^
make[1]: *** [Makefile:16: bootstrap/generated/build_cache] Error 3
make[1]: *** Waiting for unfinished jobs....

I tracked the issue to spack/spack#37425.

---

During the bisection, the following error also creeped up:

/dev/shm/rmeli/test/bwrap-mutable-root.sh --tmpfs ~ --bind /dev/shm/rmeli/test/tmp /tmp --bind /dev/shm/rmeli/test/store /user-environment spack -C /dev/shm/rmeli/test/modules module tcl refresh --upstream-modules --delete-tree --yes-to-all
==> Error: Cannot use invalid module set default.    Valid module set names are []
make: *** [Makefile:50: modules] Error 1

[Rocco Meli]

Part of the problem comes from colorisation of spack find. A possible workaround is the following (see spack/spack#2441 (comment)):

diff --git a/stackinator/templates/Makefile.compilers b/stackinator/templates/Makefile.compilers
index 0aa3f6d..9be8a1e 100644
--- a/stackinator/templates/Makefile.compilers
+++ b/stackinator/templates/Makefile.compilers
@@ -18,7 +18,7 @@ all:{% for compiler in compilers %} {{ compiler }}/generated/build_cache{% endfo
 {% for compiler, config in compilers.items() %}
 {{ compiler }}/generated/build_cache: {{ compiler }}/generated/env
 {% if push_to_cache %}
-       $(SPACK) -e ./{{ compiler }} buildcache create --rebuild-index --allow-root --only=package -m alpscache $$($(SPACK) -e ./{{ compiler }} find --format '/$${hash}')
+       $(SPACK) -e ./{{ compiler }} buildcache create --rebuild-index --allow-root --only=package -m alpscache $$($(SPACK) -e ./{{ compiler }} find --format '{/hash}' | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" )
 {% endif %}
        touch $@

diff --git a/stackinator/templates/Makefile.environments b/stackinator/templates/Makefile.environments
index 63eb5e8..6173a54 100644
--- a/stackinator/templates/Makefile.environments
+++ b/stackinator/templates/Makefile.environments
@@ -17,7 +17,7 @@ all:{% for env in environments %} {{ env }}/generated/build_cache{% endfor %}
 {% for env, config in environments.items() %}
 {{ env }}/generated/build_cache: {{ env }}/generated/env
 {% if push_to_cache %}
-       $(SPACK) -e ./{{ env }} buildcache create --rebuild-index --allow-root --only=package -m alpscache $$($(SPACK) -e ./{{ env }} find --format '/$${hash}')
+       $(SPACK) -e ./{{ env }} buildcache create --rebuild-index --allow-root --only=package -m alpscache $$($(SPACK) -e ./{{ env }} find --format '{/hash}' | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g")
 {% endif %}
        touch $@

[Ben Cumming]

Thanks @RMeli.

About the module error that you found during bisection: ==> Error: Cannot use invalid module set default. Valid module set names are [].

Was it only present in some of the commits, or is it now something we have to address?

[Rocco Meli]

It is not present in the latest develop, it was only present in one or two commits during the bisection (unfortunately I forgot to note them down), but I think it was a transitory problem that has already been fixed.