Calling `cache publish` raises TypeError
nstarman opened this issue · comments
I'm trying to publish my cache for https://github.com/nstarman/Temperature-Diffusion-Spectral-Distortion-Paper. When I run showyourwork cache publish I get the error
Traceback (most recent call last):
File "/Users/nmrs/miniconda3/envs/spectral_distortions/bin/showyourwork", line 8, in <module>
sys.exit(entry_point())
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/cli/main.py", line 427, in publish
commands.zenodo_publish(branch)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/cli/commands/zenodo.py", line 42, in zenodo_publish
zenodo_doi = sandbox.copy_draft(zenodo_doi, branch=branch)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/zenodo.py", line 37, in wrapper
return method(self, *args, **kwargs)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/zenodo.py", line 994, in copy_draft
cache_folder = self._download_latest_draft()
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/zenodo.py", line 37, in wrapper
return method(self, *args, **kwargs)
File "/Users/nmrs/miniconda3/envs/spectral_distortions/lib/python3.10/site-packages/showyourwork/zenodo.py", line 963, in _download_latest_draft
url = entry["links"]["download"]
TypeError: string indices must be integers
I didn't find a related issue when I searched, but if there is one, sorry for the noise!
Unfortunately since Zenodo updated their API all the cache stuff is broken and I don't have capacity to fix any of it (I'm especially unmotivated since I don't think it's a good idea :D). So I'd treat the Zenodo caching as unmaintained!
@dfm in the absence of Zenodo, do you have any recommendations for how to handle rules that are too intensive to ever run on Github, e.g. training a neural network? (it's okay if not -- I might just set up a manual cache)
Yeah - I'd recommend doing it manually! That way you can be really confident that your data products will remain available and won't be surprised when syw randomly starts running an expensive computation :D
@jfcrenshaw I was wondering if you had any luck setting up a manual cache that's compatible with SYW? I'm running into problems with zenodo sandbox as well (see #434 ) and would love to hear if you have found a way to not rerun intensive jobs every build if its not too much work for you to share here. No worries if it would take a lot of work to explain though, I'm sure I can do some googling and figure it out too
@afarah18 I have started using github-release to cache outputs in Github release drafts. It's pretty easy and gives you lots of control over the cache. However I've only used this for some simpler scripts/pipelines, and haven't used this with any SYW workflows.
Thank you for the tip! I'll try setting it up with SYW :)