Giters

ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS

Home Page:https://dask-ms.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

MS descriptors don't support optional columns

sjperkins opened this issue · comments

commented 
    I'm getting this error when running the command @sjperkins :

2022-11-02 10:34:29,585 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21032324 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,592 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21037373 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,597 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21037373 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,602 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21042422 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,607 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21042422 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,613 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21047471 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,617 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21047471 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,624 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21088424 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,629 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21088424 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,634 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21092912 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,640 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21092912 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,646 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21097400 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,651 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21097400 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,657 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21101888 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,661 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21101888 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,666 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21160232 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,672 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21160232 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,678 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21164720 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,683 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21164720 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,689 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21169208 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,694 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21169208 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,699 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21173696 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,704 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21173696 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,711 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21232040 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,716 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21232040 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,721 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21236528 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,726 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21236528 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,732 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21241016 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,736 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21241016 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,743 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21245504 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,748 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21245504 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,753 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21303848 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,758 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21303848 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,764 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21308336 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,769 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21308336 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,775 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21312824 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,780 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21312824 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,786 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21317312 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,792 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21317312 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:31,247 - dask-ms - INFO - Input: 'measurementset' file:///home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg
2022-11-02 10:34:31,247 - dask-ms - INFO - Output: 'measurementset' file:///home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg_time
2022-11-02 10:35:21,005 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,011 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,015 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,021 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,025 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,031 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,036 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,042 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,521 - dask-ms - WARNING - Ignoring SOURCE
2022-11-02 10:35:21,525 - dask-ms - WARNING - Ignoring 'TARGET': Unable to infer shape of column 'TARGET' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,526 - dask-ms - WARNING - Ignoring 'ENCODER': Unable to infer shape of column 'ENCODER' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,527 - dask-ms - WARNING - Ignoring 'POINTING_OFFSET': Unable to infer shape of column 'POINTING_OFFSET' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,527 - dask-ms - WARNING - Ignoring 'DIRECTION': Unable to infer shape of column 'DIRECTION' due to:
'TableProxy::getCell: no such row'
Traceback (most recent call last):
  File "/home/vicente/anaconda3/envs/pyralysis2/bin/dask-ms", line 8, in <module>
    sys.exit(main())
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/entrypoint.py", line 9, in main
    return EntryPoint(sys.argv[1:]).execute()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/entrypoint.py", line 33, in execute
    cmd.execute()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/convert.py", line 415, in execute
    writes = self.convert_table(self.args)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/convert.py", line 500, in convert_table
    writes.append(writer(datasets, out_store))
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/dask_ms.py", line 102, in xds_to_table
    out_ds = write_datasets(
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/writes.py", line 760, in write_datasets
    tp = _updated_table(table, datasets, columns, descriptor)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/writes.py", line 338, in _updated_table
    table_proxy.addcols(_table_desc, dminfo=_dminfo).result()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/table_proxy.py", line 114, in _impl
    return getattr(table, method)(*args, **kwargs)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/casacore/tables/table.py", line 1226, in addcols
    self._addcols(tdesc, dminfo, addtoparent)
RuntimeError: Invalid Table operation: Data manager name StandardStMan is already used in table /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg_time/POINTING

Originally posted by @miguelcarcamov in #159 (comment)

commented

@sjperkins

Maybe if I add the link to the ms here you can traceback the error?

The command line that I'm currently using is:

dask-ms convert HLTau_B6cont.calavg.tav300s -g "FIELD_ID,DATA_DESC_ID,SCAN_NUMBER" -i "ANTENNA1,ANTENNA2,TIME,FEED1,FEED2" -o output.ms --format ms --force

I'm not creating any folder before that.

Originally posted by @miguelcarcamov in #159 (comment)

commented

@miguelcarcamov I've moved the discussion from #159 into this issue as there are a number of separate concerns that must be addressed.