s3-to-dc faults on encountering proc-info.yaml
pindge opened this issue · comments
CMD: s3-to-dc --no-sign-request s3://dea-public-data-dev/derivative/ga_ls_fc_pi21_test_3/**/*.yaml ga_ls_fc_pi21_test_3
-11-19 09:20:28,329] {pod_launcher.py:128} WARNING - Pod not yet started: datacube-index.6d513ac719ad43bcac915ef9309ac677
[2021-11-19 09:20:29,337] {pod_launcher.py:198} INFO - Event: datacube-index.6d513ac719ad43bcac915ef9309ac677 had an event of type Pending
[2021-11-19 09:20:29,338] {pod_launcher.py:128} WARNING - Pod not yet started: datacube-index.6d513ac719ad43bcac915ef9309ac677
[2021-11-19 09:20:30,347] {pod_launcher.py:198} INFO - Event: datacube-index.6d513ac719ad43bcac915ef9309ac677 had an event of type Running
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - Traceback (most recent call last):
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - File "/usr/local/bin/s3-to-dc", line 8, in <module>
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - sys.exit(cli())
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - return self.main(*args, **kwargs)
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - rv = self.invoke(ctx)
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - return ctx.invoke(self.callback, **ctx.params)
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
[2021-11-19 09:20:32,376] {pod_launcher.py:149} INFO - return __callback(*args, **kwargs)
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 129, in cli
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - added, failed = dump_to_odc(
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 58, in dump_to_odc
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - index_update_dataset(metadata, uri, dc, doc2ds, update, update_if_exists, allow_unsafe)
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 146, in index_update_dataset
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - ds, err = doc2ds(metadata, uri)
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/datacube/index/hl.py", line 281, in __call__
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - dataset, err = self._ds_resolve(doc, uri)
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/datacube/index/hl.py", line 158, in resolve
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - db_dss = {str(ds.id): ds for ds in index.datasets.bulk_get(all_uuid)}
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/datacube/index/_datasets.py", line 108, in bulk_get
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - ids = [to_uuid(i) for i in ids]
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/datacube/index/_datasets.py", line 108, in <listcomp>
[2021-11-19 09:20:32,377] {pod_launcher.py:149} INFO - ids = [to_uuid(i) for i in ids]
[2021-11-19 09:20:32,378] {pod_launcher.py:149} INFO - File "/usr/local/lib/python3.8/dist-packages/datacube/index/_datasets.py", line 106, in to_uuid
[2021-11-19 09:20:32,378] {pod_launcher.py:149} INFO - return x if isinstance(x, UUID) else UUID(x)
[2021-11-19 09:20:32,378] {pod_launcher.py:149} INFO - File "/usr/lib/python3.8/uuid.py", line 165, in __init__
[2021-11-19 09:20:32,378] {pod_launcher.py:149} INFO - raise TypeError('one of the hex, bytes, bytes_le, fields, '
[2021-11-19 09:20:32,378] {pod_launcher.py:149} INFO - TypeError: one of the hex, bytes, bytes_le, fields, or int arguments must be given
[2021-11-19 09:23:25,957] {local_task_job.py:208} WARNING - State of this instance has been externally set to failed. Terminating instance.
fix applied in datacube-core opendatacube/datacube-core#1287