Inconsistent Behaviour With Truncated Files

[nkraemer2]

I have a file where a chunk has been truncated. Reading all the data points returns a warning and a series of points (with the missing points zero-filled). Trying to read the last data point throws an error. It seems to me that either both situations should throw a warning, or both situations should throw an exception. See code snippet below.

with nptdms.TdmsFile.open(f_path) as tdms:
	group = tdms.groups()[-1]
	chan = group.channels()[-1]
	data = chan[:] #this issues a nptdms.base_segment cropping warning
	data2 = chan[-1] #this throws an IndexError

On a related topic, guidance in the documentation for how to programmatically detect the presence of partial/truncated file would be welcome.

[Adam Reeve]

Hi @nkraemer2, thanks for reporting this. I can't reproduce this with the example files I have with truncated data though, when I test this I correctly get the last readable value back when indexing with -1. Are you able to provide your file or some more information on the file structure to help reproduce this?

And I agree it would be helpful to have a way to detect this programmatically. This information isn't currently exposed in a way that would be easy to access though. You could probably capture the warning log, but it might make sense to expose some metadata through the API that includes the expected and actual lengths, or just whether data was truncated.

[nkraemer2]

Hi @adamreeve , thanks for the reply. I'm working on getting permission to share the file with you. The channel data is interleaved. There is only one group and 3 channels within that group. What other information about the file structure would be useful in the mean time?

[Adam Reeve]

Hmm I'm not sure what else might be important, I've also tested with interleaved data but haven't seen this issue. One thing that might help is sharing the output from the tdmsinfo command with debugging enabled. This is installed with nptdms so you should be able to run something like:

tdmsinfo --debug filename.tdms

This will include names of groups and channels in the output though so might still contain some sensitive information

[nkraemer2]

Thank you for your patience. I finally received permission to share the file with you, but I cannot post it publicly for the time being. What is the best way for me to send the file to you?

[Adam Reeve]

Okay no problem, you can email the file or a link to it. I try not to post my email address too publicly, but it's in the package metadata: https://github.com/adamreeve/npTDMS/blob/master/setup.cfg#L6C1-L6C13