read tdms file should have option to convert group/channel names to valid python names by replacing invalid characters with "_"

[jj-github-jj]

Autocomplete features such as using namedtuples rely on valid python names for groups and channels. But TDMS files may have been saved using invalid characters such as blanks, brackets etc .
Read file should have option to convert group/channel names to valid strings without the invalid characters by replacing invalid characters with underscores for example.

I used codes such as following to rename group/channel names to be able to use python tools such as namedtuples with tdms files

def change_to_valid_names(list_of_names):
    s=[(re.sub("[/' ()-+]","_",x)) for x in list_of_names] #replace invalid char with _
    s=[(re.sub("-+","_",x)) for x in s] #replace 1 or more _ with single _ to compact
    s=[(re.sub("_+","_",x)) for x in s] #replace 1 or more _ with single _ to compact
    s=[x.replace("_","n",1) if (x[0] == "_") else x for x in s] #cant start with underscore so change first underscore
    return(s)

[Adam Reeve]

Hi, thanks for the suggestion but I don't think this should be the responsibility of npTDMS. Group and channel names are only handled as strings by npTDMS so names that are not valid python identifiers are not a problem for normal use. If you want to use these as names of members in a namedtuple then it's up to you to handle this conversion.

[jj-github-jj]

Thanks for the response. The reason I suggested the conversion to valid names option is to be able to use tab auto completion or selection features in jupyter notebook . There is another package nitdms (https://l-johnston.github.io/nitdms/) that is more pythonic in that it has auto completion features for group names and channel names as strings using dot notation so no need to change group/channel names to python valid names .Its great for smaller , simpler TDMS files. Unfortunately it has a bug in handling labview timestamp datatype and also can't handle large files which is why I use npTDMS. Is it possible to add nitdms like tab completion features to npTDMS ?

[Adam Reeve]

I don't want to change the API to be able to access groups and channels as properties of the file and group objects, but it looks like I can simply add an _ipython_key_completions_ method to support tab completion when indexing with a string, so eg. tdms_file["<tab> and tdms_group["<tab> will show auto-complete options.

[Adam Reeve]

The IPython completion behaviour seems a bit awkward, by default it includes a whole bunch of extra stuff like file names and magics (ipython/ipython#11359), but you can configure this to only show dictionary completions with:

import IPython.core.completer
IPython.core.completer.IPCompleter.dict_keys_only = True

Unfortunately this is only a global option and there doesn't seem to be a way to control this behaviour from the npTDMS side.

One other limitation is it won't evaluate intermediate results, so eg. tdms_file["group_name"]["<tab> won't autocomplete with channel names. This is supposedly supposed to work if setting %config IPCompleter.greedy = True, but that didn't work for me, and caused completion to stop working.

Those limitations aside, this definitely seems like it's worth adding.

[jj-github-jj]

Thank you for supporting this feature.

[Adam Reeve]

Closing this as fixed, the IPython completion has just been released in version 1.5.0.