Problem with reading SpooledTemporaryFile
dulacskamty opened this issue · comments
Versions:
-python: 3.9.15
-nptdms: 1.6.1
I have the following python modules:
- client: open a .tdms file (with built-in open function) and uploads/sends as http POST request to server (with requests module)
- server: receives the file (with fastapi module) and tries to read it with nptdms
I attach these two files in zip as "simple_client.py" and "simple_server.py" (I think this error can be reproduced with any arbitrary .tdms file).
client_server.zip
The method of uploading files with FastAPI comes from here, FastAPI server is running through uvicorn according to this.
However, the main point is that in server I have a SpooledTemporaryFile, which is a file-like object (like an already opened file) and I want to read it with TdmsFile.read(), but it fails with the following error message:
Traceback (most recent call last): File "C:\Users\user\.conda\envs\py39\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi result = await app( # type: ignore[func-returns-value] File "C:\Users\user\.conda\envs\py39\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\fastapi\applications.py", line 270, in __call__ await super().__call__(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\applications.py", line 124, in __call__ await self.middleware_stack(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__ raise exc File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__ raise exc File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "C:\Users\user\.conda\envs\py39\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__ raise e File "C:\Users\user\.conda\envs\py39\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__ await self.app(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\routing.py", line 706, in __call__ await route.handle(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\routing.py", line 276, in handle await self.app(scope, receive, send) File "C:\Users\user\.conda\envs\py39\lib\site-packages\starlette\routing.py", line 66, in app response = await func(request) File "C:\Users\user\.conda\envs\py39\lib\site-packages\fastapi\routing.py", line 237, in app raw_response = await run_endpoint_function( File "C:\Users\user\.conda\envs\py39\lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function return await dependant.call(**values) File "C:\Users\user\Documents\lab\.\simple_server.py", line 10, in create_upload_file tdms = TdmsFile.read(file.file) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms.py", line 69, in read return TdmsFile(file, raw_timestamps=raw_timestamps, memmap_dir=memmap_dir) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms.py", line 131, in __init__ self._read_file( File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms.py", line 293, in _read_file self._read_data(tdms_reader) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms.py", line 305, in _read_data for chunk in tdms_reader.read_raw_data(): File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\reader.py", line 145, in read_raw_data for chunk in segment.read_raw_data(self._file): File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms_segment.py", line 244, in read_raw_data for chunk in self._read_data_chunks(f, data_objects, self.num_chunks): File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms_segment.py", line 359, in _read_data_chunks for chunk in reader.read_data_chunks(file, data_objects, num_chunks): File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\base_segment.py", line 52, in read_data_chunks yield self._read_data_chunk(file, data_objects, chunk) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms_segment.py", line 482, in _read_data_chunk object_data[obj.path] = obj.read_values(file, number_values, self.endianness) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\tdms_segment.py", line 557, in read_values return fromfile(file, dtype=dtype, count=number_values) File "C:\Users\user\.conda\envs\py39\lib\site-packages\nptdms\base_segment.py", line 147, in fromfile bytes_read = file.readinto(buffer[offset:]) AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto'
So it looks like to me that SpooledTemporaryFile doesnt have this method "readinto", what nptdms wants to call, so the program crashes.
My question is that is this a bug, or a normal behaviour (with other words, nptdms should be able to read temporary files or I should not expect that)? If it is a bug, then could this problem be fixed? If it is not a bug, then is there any plan for the future to handle temporary files in TdmsFile.read() input, or I should not hope for that?
Hi, my intention is that this should support any file-like object, but this relies on having a common interface for what should be considered a file. The readinto method is defined on io.RawIOBase or io.BufferedIOBase so any class inheriting from either of those or implementing the same interface should work.
According to the tempfile docs, SpooledTemporaryFile was changed in Python version 3.11 to fully implement the io.BufferedIOBase interface. Is upgrading to Python 3.11 an option for you?
Thank you for the quick answer and for looking into the problem. You are right, reading SpooledTemporaryFile works well with Python version 3.11, so I close this issue.