pyulog PX4 log file read error when using ulog2csv
njlawson1969 opened this issue · comments
Dear pyulog user group.
On running ulog2csv in Windows 10, I receive the current error log.
C:\Flights 23-06-21\Pixhawk>ulog2csv log001.ulg
Traceback (most recent call last):
File "C:\Users\eg3023\AppData\Local\Programs\Python\Python38\Scripts\ulog2csv-script.py", line 11, in
load_entry_point('pyulog==0.7.0', 'console_scripts', 'ulog2csv')()
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\ulog2csv.py", line 42, in main
convert_ulog2csv(args.filename, args.messages, args.output, args.delimiter, args.ignore)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\ulog2csv.py", line 59, in convert_ulog2csv
ulog = ULog(ulog_file_name, msg_filter, disable_str_exceptions)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 132, in init
self._load_file(log_file, message_name_filter_list)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 493, in _load_file
self._read_file_data(message_name_filter_list)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 664, in _read_file_data
msg_add_logged = self._MessageAddLogged(data, header,
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 387, in init
self.message_name = ULog.parse_string(data[3:])
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 91, in parse_string
ret = _parse_string(cstr)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 18, in _parse_string
return str(cstr, 'utf-8', errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 0: invalid start byte
Could you please advise if the data can be read into csv or recovered with the pyulog code. The px4 log file is found here: https://1drv.ms/u/s!AqvNv7Mai6RqhbUm0EkdBxk5MYOQCg?e=AtUwU3
The online px4 flight reviewer is flagging that the file is too large to read.
Many thanks, Nick (Cranfield Uni)
Hi
It means the file is corrupt. How did you retrieve it?
Thanks Beat. The file was just copied from the Pixhawk SD card after the flight. The byte count seems consistent with the SD card and transfer to my laptop (link in last message). It is the first time I have had this issue with the Pixhawk, so I was wondering whether it is possible to retrieve the data. Many thanks, Nick
Certainly not something to be expected. Did you at any point plug the power while it was still logging?
In any case you could try with ulog_info -i <file>, and you might have to change the ulog parser to ignore more errors if there are any more.
Ok thanks Beat. The unit was simply plugged into the battery pack at the start of the flight and disconnected when the flight finished. This has been what has always worked. So yes, unusual to see this error. I will try your suggestion. Thanks again, Nick
This is the new message on running with ilog_info -i filename.log. Possible looks like a corrupt file, or am I entering the correct format in the execution command?
C:\Users\eg3023\OneDrive - Cranfield University\eg3023\My Documents\Teaching\Pixhawk>ulog_info -i log001.ulg
Traceback (most recent call last):
File "C:\Users\eg3023\AppData\Local\Programs\Python\Python38\Scripts\ulog_info-script.py", line 11, in
load_entry_point('pyulog==0.7.0', 'console_scripts', 'ulog_info')()
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\info.py", line 85, in main
ulog = ULog(ulog_file_name, None, disable_str_exceptions)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 132, in init
self._load_file(log_file, message_name_filter_list)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 493, in _load_file
self._read_file_data(message_name_filter_list)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 664, in _read_file_data
msg_add_logged = self._MessageAddLogged(data, header,
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 390, in init
self._parse_format(message_formats)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 409, in _parse_format
self._parse_nested_type('', self.message_name, message_formats)
File "c:\users\eg3023\appdata\local\programs\python\python38\lib\site-packages\pyulog\core.py", line 418, in _parse_nested_type
message_format = message_formats[type_name]
KeyError: '2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00&\x00D\x04\x00kf\x00\x00\x00\x00\x04-;.?;\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00.\x00D\x05\x00kf\x00\x00\x00\x00=<h<\x04\x0b\x10\x0f\x00\x00\x00\x00\x00\x00\x01~\x14?
then pages and pages of what looks like similar HEX decode
<\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00.\x00D\x05\x00Pnf\x00\x00\x00\x00fۥ\x1e\u07bbs9\x0f\x00\x00\x00\x00\x00\x00S\x14?o>G \x0f\x00\x007\x00D\x13\x00Pnf\x00\x00\x00\x00\x1d\x0b6;%\x1b?ӂ$02<LK?\x7f?e6\x12:\x02<\x00D\x14\x00Qnf\x00\x00\x00\x00\x00'
C:\Users\eg3023\OneDrive - Cranfield University\eg3023\My Documents\Teaching\
Yes this is what I meant with you might have to ignore more errors by changing the parser. Looks like there's a corruption in the header already which unfortunately reduces chances of an easy recovery.
Dear pyulog user group.
A further query I'm afraid. On running ulog2csv in Windows 10, I receive all the csv files as expected, but the timestamp starting points on the csv's seem clipped and on the inertial data, I am losing most of the inertial flight data. When you view the log file on the online Pixhawk viewer https://logs.px4.io/, the data is there for all the parameters, for all of the flight. So the file is not corrupted, but ulog2csv does not seem to output all the data from the start to end of each data stream. The output data and log file is here:
https://1drv.ms/u/s!AqvNv7Mai6RqhbV7do9vmvjgXUPU8g?e=QXmlJW
It would be really appreciated, if anyone can suggest how I output the full data streams.
Thanks as always, Nick
Folks, I got to the bottom of this problem. The timestamps on the inertial output and the GPS output are running off different clocks. Sorry to bother you
Glad to see you got to the bottom of it.