struct.error: unpack requires a buffer of 4 bytes

[Huynh Ngoc Phien]

Hi @adamreeve ,

I am currently using nptdms to extract data from a 50MB TDMS file. However, I am encountering an issue with the extraction process, and would appreciate any assistance in resolving the matter. It is working properly with others.

[ISSUE]
Traceback (most recent call last):
File "C:\Users\hnphien\Documents\converter\program.py", line 64, in
run(log=proc_logger, config=proc_config)
File "C:\Users\hnphien\Documents\converter\program.py", line 40, in run
try_convert(
File "C:\Users\hnphien\Documents\converter\src\ft_converter\convert.py", line 60, in try_convert
dataframes = _convert(
File "C:\Users\hnphien\Documents\converter\src\ft_converter\convert.py", line 42, in _convert
df_hys = process_tdms(log, file_path, cache_file_path)
File "C:\Users\hnphien\Documents\converter\src\ft_converter\tdms_processor\tdms_process.py", line 27, in process_tdms
tdms: TdmsFile = TdmsFile(file_path)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms.py", line 131, in init
self._read_file(self._reader, read_metadata_only, keep_open)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms.py", line 289, in _read_file
self._read_data(tdms_reader)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms.py", line 301, in _read_data
for chunk in tdms_reader.read_raw_data():
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\reader.py", line 119, in read_raw_data
for chunk in segment.read_raw_data(self._file):
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms_segment.py", line 244, in read_raw_data
for chunk in self._read_data_chunks(f, data_objects, self.num_chunks):
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms_segment.py", line 359, in _read_data_chunks
for chunk in reader.read_data_chunks(file, data_objects, num_chunks):
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\base_segment.py", line 52, in read_data_chunks
yield self._read_data_chunk(file, data_objects, chunk)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms_segment.py", line 482, in _read_data_chunk
object_data[obj.path] = obj.read_values(file, number_values, self.endianness)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\tdms_segment.py", line 562, in read_values
return self.data_type.read_values(file, number_values, endianness)
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\types.py", line 219, in read_values
offsets.append(Uint32.read(file, endianness))
File "c:\Users\hnphien.conda\envs\wlt\lib\site-packages\nptdms\types.py", line 98, in read
return _struct_unpack(endianness + cls.struct_declaration, read_bytes)[0]
struct.error: unpack requires a buffer of 4 bytes

[Adam Reeve]

Hi, this usually means that npTDMS tried to read 4 bytes but has reached the end of the file so couldn't read as much data as expected. This could be a bug in npTDMS or could mean the file is truncated and there's less data than expected. If there are many segments in the file it might be possible to catch this error and skip reading the last segment.

[Huynh Ngoc Phien]

When opening the file via TDMS Excel extension, I got some messages via a log file:
***************** New logfile section *******************
ToChannelData.cpp(198): TDMS: ERROR: TDS Exception in GetValues: Tds Error: TdsErrEof(4):
ToChannelData.cpp(198): TDMS: ERROR: TDS Exception in GetValues: Tds Error: TdsErrEof(4):
ToChannelData.cpp(198): TDMS: ERROR: TDS Exception in GetValues: Tds Error: TdsErrEof(4):
ToChannelData.cpp(198): TDMS: ERROR: TDS Exception in GetValues: Tds Error: TdsErrEof(4):
ToChannelData.cpp(198): TDMS: ERROR: TDS Exception in GetValues: Tds Error: TdsErrEof(4):

How can I continue with this? Thank you so much, @adamreeve

[Adam Reeve]

Can you still read some data with the Excel extension even though you get those messages?

I can take a look at fixing this so npTDMS will log an error but return what data it can read, but I can't give a timeframe for when that will be done by. If it's possible for you to provide a link to the file that you're having the problem with that would help.

If you install npTDMS from source code you could also try catching this exception and returning some placeholder data for the chunk that has the problem, eg:

--- a/nptdms/types.py
+++ b/nptdms/types.py
@@ -218,14 +218,17 @@ class String(TdmsType):
             This is stored as an array of offsets
             followed by the contiguous string data.
         """
-        offsets = [0]
-        for i in range(number_values):
-            offsets.append(Uint32.read(file, endianness))
-        strings = []
-        for i in range(number_values):
-            s = file.read(offsets[i + 1] - offsets[i])
-            strings.append(String._decode(s))
-        return strings
+        try:
+            offsets = [0]
+            for i in range(number_values):
+                offsets.append(Uint32.read(file, endianness))
+            strings = []
+            for i in range(number_values):
+                s = file.read(offsets[i + 1] - offsets[i])
+                strings.append(String._decode(s))
+            return strings
+        except struct.error:
+            return [""] * number_values
 
     @staticmethod
     def _decode(string_bytes):