Group Properties Not Reading
aequitz opened this issue · comments
The attached (example_file.zip) file has a group property within it. More specifically, it has a "name" property. However, the nptdms library does not pick up on any group properties.
When viewing the file in a TDMS Viewer like SuperViewer, the property looks like it exists.
SuperViewer is created with LabVIEW, so I also created some simple LabVIEW code that demonstrates the group property exists.
For additional context, this what the file looks like when viewing it in the TDM Excel Add-In for Microsoft Excel.
Here is code that can be used to view the problem:
from nptdms import RootObject, TdmsChannel, TdmsFile, TdmsGroup
orig_file = 'example_file.tdms'
orig = TdmsFile(orig_file)
root = RootObject(orig.properties)
groups: list[TdmsGroup] = orig.groups()
channels: list[TdmsChannel] = [c for g in groups for c in g.channels()]
print('Properties in group:')
print(groups[0].properties)This produces the following output:
Properties in group:
OrderedDict()
Hi @aequitz. I think what is happening here is that the name property doesn't actually exist in the file as an explicit property, but is being added as a property by LabVIEW and SuperViewer using the group name from its path.
Is this really a problem for you? You can access the name of a group with its name attribute. Any real properties written to the group will be included in the properties dictionary. And if you have a use case where you want to get all properties including the name, you can add the name yourself.
This becomes a problem when using nptdms to read a TDMS file and then write additional data to it. When that's done, the name property gets deleted from the file and is no longer seen by LabVIEW/SuperViewer. I would expect nptdms to retain those group properties when doing a read/write.
Additionally, MATLAB requires this name group property to exist in order to use its tdmsread function with TDMS files so I would think this is required of the TDMS file format (or maybe MATLAB is making a bad assumption that the name group property should always be there)
Hmm okay that's interesting. There definitely isn't a name property stored in the example file you provide (you can confirm this by running tdmsinfo -d example_file.tdms which logs all properties read), but there is a metadata entry for the group object in the file with no properties. Maybe missing this group object from the metadata when rewriting the file results in the name property not being displayed.
Are you able to try including the group object explicitly when writing the file to see if that fixes your issues with LabVIEW and SuperViewer not seeing the group name and MATLAB's tdmsread not working? If so, it would make sense for npTDMS to always insert an empty group object when one isn't provided explicitly.
Eg:
tdms_writer.write_segment([GroupObject("SandboxTest")])I tried out what you suggested with the following code:
from nptdms import (GroupObject, RootObject, TdmsChannel, TdmsFile, TdmsGroup,
TdmsWriter)
# Read TDMS file
ORIG_FILE = 'example_file.tdms'
NEW_FILE = 'new_file.tdms'
orig = TdmsFile(ORIG_FILE)
root = RootObject(orig.properties)
groups: list[TdmsGroup] = orig.groups()
channels: list[TdmsChannel] = [c for g in groups for c in g.channels()]
new_groups = [GroupObject('SandboxTest')]
with TdmsWriter(NEW_FILE) as copy:
copy.write_segment([root] + new_groups + channels)
new = TdmsFile(NEW_FILE)
new_group: TdmsGroup = new.groups()[0]
print(new_group.properties)This results in an empty OrderedDict() being printed:
However, if I manually force the group properties to the file with the following code:
from nptdms import RootObject, TdmsChannel, TdmsFile, TdmsGroup, TdmsWriter
# Read TDMS file
ORIG_FILE = 'example_file.tdms'
NEW_FILE = 'new_file.tdms'
orig = TdmsFile(ORIG_FILE)
root = RootObject(orig.properties)
groups: list[TdmsGroup] = orig.groups()
channels: list[TdmsChannel] = [c for g in groups for c in g.channels()]
for group in groups:
group.properties['name'] = group.name
with TdmsWriter(NEW_FILE) as copy:
copy.write_segment([root] + groups + channels)
new = TdmsFile(NEW_FILE)
new_group: TdmsGroup = new.groups()[0]
print(new_group.properties)I get the following print output:
Hi @aequitz, did you try opening the file from the first code with MATLAB's tdmsread though, or viewing it in SuperViewer? The behaviour you show from npTDMS is what I would expect.
With new_file.tdms, here is what SuperViewer sees:
So it is still saying that the name group property exists.
Great okay thanks so it looks like making sure a group object is included in the metadata should be enough to fix reading files written by npTDMS.
But I don't think there's any need to change npTDMS to return the name as a property and that could be a breaking change for some users.
Hey @adamreeve - I apologize, the SuperViewer image I took above was from the second snippet of code in this comment.
Here the new_file.tdms that results in the first snippet of code which does delete the group name property:
@adamreeve Any feedback on this? Could we add something like this when a GroupObject gets initialized to make sure that the name property gets retained?
if 'name' not in group_obj_properties:
group_obj_properties['name'] = group_obj.nameHi @aequitz, sorry I haven't spent any time properly looking into this yet. That really shouldn't be necessary given that other TMDS writers don't explicitly write the group name to the file, and your example file that works fine doesn't have a name property, so there must be something else about the written files that's causing a problem with these readers.
Hi @aequitz. I did some testing and found that the files produced by rewriting the root, group and channel objects from npTDMS without adding the group name property were nearly binary identical to your example_file.tdms, except for some small differences in wf_start_time timestamp values due to precision loss going via Python datetime, and the file format version written by npTDMS by default is 4712 but example_file.tdms has 4713. If I changed the version to 4713 then the group name does show up in SuperView and LabView's TDMS file viewer VI, so I guess there's a difference in how LabView treats these versions.
I probably originally set the default format to 4712 assuming this would be best for compatibility, but I've never found any documentation describing the difference between these versions there is some documentation at https://www.ni.com/docs/en-US/bundle/labview/page/lvconcepts/fileio_tdms_versions.html on the differences between the versions but it doesn't explain why the properties would be interpreted differently like this, and the 2.0 features are not things used or supported by npTDMS. I do plan to change the default to 4713 at some point as this seems to be the most common version used (see #264).
I also tested writing files without the root and group object and these caused major issues for SuperViewer where not all channels were correctly shown. LabView's TDMS file viewer did show some errors but could show the data and properties for all channels correctly. So it does look like that needs to be fixed, and we should automatically include a root and group object when required. The NI docs on the TDMS format actually mention this, which I hadn't noticed before:
In order for all TDMS client applications to work properly, every TDMS file must contain a file object. A file object must contain a group object for each group name used in a channel path.
I don't have access to Matlab so can't test the issue you mentioned with it not being able to read TDMS files. Can you test whether Matlab can read files as long as the root and group objects are present? And whether it also needs the version to be 4713?
You can set the version like this for example, since npTDMS version 1.5.0:
with TdmsWriter(NEW_FILE, version=4713) as copy:
copy.write_segment([root] + new_groups + channels)@adamreeve - It looks like this works just fine in MATLAB as well! Good find!
I see that you merged this into main. Will this be released in a newer version of npTDMS soon?
Yep I've just published this as version 1.7.0
Awesome, appreciate the quick turnaround!