BadChunk: Array '1543661287/correlator_data

[Athanaseus Javas Ramaila]

I'm trying to download data from the archive using a link with token but it fails with bad chunk array data.

By running:
KATSDPTELSTATE_ALLOW_PICKLE=1 mvftoms.py --flags cam -f https://archive-gw-1.kat.ac.za/1543661287/1543661287_sdp_l0.full.rdb?token=********************

Gives the following output:

No calibration products will be applied                                                                  
Per user request the following antennas will be selected: 'm000', 'm001', 'm002', 'm003', 'm004', 'm005', 'm006', 'm007', 'm009', 'm010', 'm011', 'm012', 'm013', 'm014', 'm015', 'm016', 'm017', 'm018', 'm019', '
m020', 'm021', 'm022', 'm023', 'm024', 'm025', 'm026', 'm027', 'm028', 'm029', 'm030', 'm031', 'm032', 'm033', 'm034', 'm035', 'm036', 'm037', 'm038', 'm039', 'm040', 'm041', 'm042', 'm043', 'm044', 'm045', 'm04
7', 'm048', 'm049', 'm050', 'm051', 'm053', 'm054', 'm055', 'm056', 'm057', 'm058', 'm059', 'm060', 'm061', 'm062'       
Per user request the following target fields will be selected: 'J1939-6342', '1613-586', 'T16R02C02'
Per user request the following scans will be dumped: 1, 3, 5
Extract MS for spw 0: centre frequency 1284000000 Hz
Will create MS output in 1543661287_sdp_l0.ms

#### Producing a full polarisation MS (HH,HV,VH,VV) ####


Using array as the reference antenna. All targets and scans will be based on this antenna.


Iterating through scans in dataset(s)...

Writing static meta data...
scan   1 (  37 samples) loaded. Target: 'J1939-6342'. Writing to disk...
Traceback (most recent call last):
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/chunkstore.py", line 453, in chunk_metadata
    shape = tuple(s.stop - s.start for s in slices)
TypeError: 'int' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aramaila/.virtualenvs/katdal/bin/mvftoms.py", line 916, in <module>
    main()
  File "/home/aramaila/.virtualenvs/katdal/bin/mvftoms.py", line 666, in main
    scan_vis_data, scan_weight_data, scan_flag_data)
  File "/home/aramaila/.virtualenvs/katdal/bin/mvftoms.py", line 116, in load
    out=[vis, weights, flags])
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/lazy_indexer.py", line 585, in get
    kept = [dask_getitem(array.dataset, keep) for array in arrays]
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/lazy_indexer.py", line 585, in <listcomp>
    kept = [dask_getitem(array.dataset, keep) for array in arrays]
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/lazy_indexer.py", line 139, in dask_getitem
    dsk = dask.optimization.cull(out.dask, out.__dask_keys__())[0]
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/dask/optimization.py", line 51, in cull
    dependencies_k = get_dependencies(dsk, k, as_list=True)  # fuse needs lists
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/dask/core.py", line 258, in get_dependencies
    return keys_in_tasks(dsk, [arg], as_list=as_list)
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/dask/core.py", line 186, in keys_in_tasks
    if w in keys:
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/_collections_abc.py", line 666, in __contains__
    self[key]
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/dask/highlevelgraph.py", line 643, in __getitem__
    return self.layers[key[0]][key]
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/chunkstore.py", line 145, in __getitem__
    return self.getter(self.array_name, slices, self.dtype, **self.kwargs)
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/chunkstore.py", line 323, in get_chunk_or_placeholder
    return self.get_chunk(array_name, slices, dtype)
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/chunkstore_s3.py", line 631, in get_chunk
    chunk_name, shape = self.chunk_metadata(array_name, slices, dtype=dtype)
  File "/home/aramaila/.virtualenvs/katdal/lib/python3.6/site-packages/katdal/chunkstore.py", line 455, in chunk_metadata
    raise BadChunk(f'Array {array_name!r}: chunk ID should be '
katdal.chunkstore.BadChunk: Array '1543661287_sdp_l0/correlator_data': chunk ID should be a sequence of slice objects, not 0

System info

• Ubuntu 18.04
• Python3
• Katdal 0.20

[Ludwig Schwardt]

Thanks @Athanaseus, I've noticed something similar... Bumping this up the queue!