eawag-rdm / resup

Batch upload and download of resources to and from data package in CKAN.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`resup put` fails when a directory content changes

meowcat opened this issue · comments

Steps to reproduce:

Create a dir, put a file there, upload it with resup put

(resup) stravsmi@uch-mssvr:~$ mkdir reprex-keys
(resup) stravsmi@uch-mssvr:~$ cd reprex-keys/
(resup) stravsmi@uch-mssvr:~/reprex-keys$ head -c 200M </dev/urandom >myfile
(resup) stravsmi@uch-mssvr:~/reprex-keys$ resup put expozol-sediment-analysis-dia-positive-mode
Calculating checksum for myfile ...
    time: 0.891764879227 seconds
    sha256: c42fc6cb09e880c12754e774a1a4b46232828e6a5c140bc455ae2e444a08bb00
uploading myfile (209715200)
{'restricted_level': 'public', 'hashtype': 'sha256', 'package_id': 'expozol-sediment-analysis-dia-positive-mode', 'hash': 'c42fc6cb09e880c12754e774a1a4b46232828e6a5c140bc455ae2e444a08bb00', 'name': 'myfile', 'url': 'dummy', 'citation': '', 'resource_type': 'Dataset', 'size': 209715200}
[================================] 209715200/209716289 - 00:00:04

waiting for server-side processing ...
total time: 7.31933903694 s

Create a second file and upload again (to the same or any other repo. Note: anyway, this will duplicate and not replace myfile if uploading to the same repo)

(resup) stravsmi@uch-mssvr:~/reprex-keys$ head -c 200M </dev/urandom >myfile2   
(resup) stravsmi@uch-mssvr:~/reprex-keys$ resup put expozol-sediment-analysis-dia-positive-mode
loading existing checksums
Traceback (most recent call last):
  File "/home/stravsmi/miniconda3/envs/resup/bin/resup", line 11, in <module>
    load_entry_point('resup', 'console_scripts', 'resup')()
  File "/home/stravsmi/resup/resup/resup.py", line 578, in main
    put.upload()
  File "/home/stravsmi/resup/resup/resup.py", line 375, in upload
    self._chksum()
  File "/home/stravsmi/resup/resup/resup.py", line 294, in _chksum
    self.metadata[filename]['hash'] = oldmeta[filename]['hash']
KeyError: 'myfile2'
(resup) stravsmi@uch-mssvr:~/reprex-keys$

Workaround: rm -r _checksums

However then the parts-file get rewritten.

That is intended behaviour. The source directory should not be modified manually after resup has been run on it for the first time. Updated the docu to reflect that in a06a72d.