`resup put` fails when a directory content changes
meowcat opened this issue · comments
meowcat commented
Steps to reproduce:
Create a dir, put a file there, upload it with resup put
(resup) stravsmi@uch-mssvr:~$ mkdir reprex-keys
(resup) stravsmi@uch-mssvr:~$ cd reprex-keys/
(resup) stravsmi@uch-mssvr:~/reprex-keys$ head -c 200M </dev/urandom >myfile
(resup) stravsmi@uch-mssvr:~/reprex-keys$ resup put expozol-sediment-analysis-dia-positive-mode
Calculating checksum for myfile ...
time: 0.891764879227 seconds
sha256: c42fc6cb09e880c12754e774a1a4b46232828e6a5c140bc455ae2e444a08bb00
uploading myfile (209715200)
{'restricted_level': 'public', 'hashtype': 'sha256', 'package_id': 'expozol-sediment-analysis-dia-positive-mode', 'hash': 'c42fc6cb09e880c12754e774a1a4b46232828e6a5c140bc455ae2e444a08bb00', 'name': 'myfile', 'url': 'dummy', 'citation': '', 'resource_type': 'Dataset', 'size': 209715200}
[================================] 209715200/209716289 - 00:00:04
waiting for server-side processing ...
total time: 7.31933903694 s
Create a second file and upload again (to the same or any other repo. Note: anyway, this will duplicate and not replace myfile
if uploading to the same repo)
(resup) stravsmi@uch-mssvr:~/reprex-keys$ head -c 200M </dev/urandom >myfile2
(resup) stravsmi@uch-mssvr:~/reprex-keys$ resup put expozol-sediment-analysis-dia-positive-mode
loading existing checksums
Traceback (most recent call last):
File "/home/stravsmi/miniconda3/envs/resup/bin/resup", line 11, in <module>
load_entry_point('resup', 'console_scripts', 'resup')()
File "/home/stravsmi/resup/resup/resup.py", line 578, in main
put.upload()
File "/home/stravsmi/resup/resup/resup.py", line 375, in upload
self._chksum()
File "/home/stravsmi/resup/resup/resup.py", line 294, in _chksum
self.metadata[filename]['hash'] = oldmeta[filename]['hash']
KeyError: 'myfile2'
(resup) stravsmi@uch-mssvr:~/reprex-keys$
meowcat commented
Workaround: rm -r _checksums
meowcat commented
However then the parts-file get rewritten.
Harald von Waldow commented
That is intended behaviour. The source directory should not be modified manually after resup
has been run on it for the first time. Updated the docu to reflect that in a06a72d.