By default xds_to_table will create columns with standard (instead of tiled) storage managers

[Landman Bester]

• dask-ms version: master branch
• Python version: 3.9
• Operating System: ubunu 2004

Description

When writing a column that does not yet exist to a measurement set using xds_to_table it ends up having a standard storage manager associated with it. This can result in suboptimal performance when writing columns with channel and/or correlation axes.
dask-ms should detect if this is the case and use a tiled storage manager instead.

[Benjamin Hugo]

It may actually be advantageous to use an incremental storage manager for cases where multiple fields may be written one at a time - EG default initialization of MODEL_DATA which is only filled out by software later.

…

On Mon, 11 Apr 2022, 12:46 Landman Bester, ***@***.***> wrote: - dask-ms version: master branch - Python version: 3.9 - Operating System: ubunu 2004 Description When writing a column that does not yet exist to a measurement set using xds_to_table it ends up having a standard storage manager associated with it. This can result in suboptimal performance when writing columns with channel and/or correlation axes. dask-ms should detect if this is the case and use a tiled storage manager instead. — Reply to this email directly, view it on GitHub <#195>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6X5IHPDX2WMO553TVDVEP7KBANCNFSM5TCYQPAA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[JSKenyon]

Closed via #196. Will leave it in master for you to test out @landmanbester. Reopen if you have trouble.

[Landman Bester]

I just tried to test this locally with the following code snippet

import numpy as np
import dask
import dask.array as da
from daskms import xds_from_ms, xds_to_table

ms_name = "/home/landman/testing/pfb/MS/point_gauss_nb.MS_p0/"
xds = xds_from_ms(ms_name,
                  chunks={'row':10000})

test_data = da.zeros(xds[0].DATA.shape,
                     chunks=xds[0].DATA.chunks,
                     dtype=np.complex64)

xdso = []
for ds in xds:
    dso = ds.assign(**{'TESTCOL':(('row', 'chan', 'corr'), test_data)})
    xdso.append(dso)

writes = xds_to_table(xdso, ms_name, columns='TESTCOL')

dask.compute(writes)

and it fals over with

Traceback (most recent call last):
  File "/home/landman/software/scratch/test_xds_to_table.py", line 19, in <module>
    writes = xds_to_table(xdso, ms_name, columns='TESTCOL')
  File "/home/landman/software/dask-ms/daskms/dask_ms.py", line 88, in xds_to_table
    out_ds = write_datasets(table_name, xds, columns,
  File "/home/landman/software/dask-ms/daskms/writes.py", line 718, in write_datasets
    tp = _updated_table(table, datasets, columns, descriptor)
  File "/home/landman/software/dask-ms/daskms/writes.py", line 324, in _updated_table
    _dminfo = {} if _dminfo['*1']['NAME'] in odminfo else _dminfo
KeyError: '*1'

Not sure if I am doing something wrong?

[JSKenyon]

Ah ok - I know what I did wrong. Reopening for now.

[JSKenyon]

Weirdly, I cannot reproduce @landmanbester. If you change TESTCOL to DIFFTESTCOL, do you get the same error?

[Landman Bester]

Same error. As discussed I also tried on an actual (as opposed to simulated) MeerKAT MS and I didn't get the same issue. The MS I was using originally was produced by makems so maybe something is going wrong there

[JSKenyon]

The issue is the suffix on your MS. dask-ms where a column lives when building descriptors. Using name.MS_p0 confuses the descriptor builder. If you are happy with the explanation (and the fix i.e. just don't change the suffix), go ahead and close again.

[Landman Bester]

Ah ok, thanks @JSKenyon. The suffix inherited from makems never made sense to me, not sure if it has some kind of significance but I'm happy to just change it