Name mapping from sdlog2 format
jgoppert opened this issue · comments
I've coded px4tools to look for specific field names when using sdlog2. Now that the fields are renamed in ulog, it is a pain. It would be nice if there was an sdlog2 compatibility csv export.
Also, Is there a way to dump the csv into one file without listing all of the messages? The sdlog2 csv converter would dump it all into one file which I could then read directly into pandas for analysis in python.
Also, Is there a way to dump the csv into one file without listing all of the messages?
No, not easily. Different messages are typically recorded at different rates with different timestamps. Merging them would generally require a resampling, which is why multiple files are created in the first place.
The sdlog2 was just less accurate in terms of timestamps, which is why it could create a single file. We could do the same here, but I think we should do better than that.
I think we should try to use pandas. It has built in resampling which I've already used here with sdlog2 to resample the data at 1 ms: https://github.com/dronecrew/px4tools/blob/master/px4tools/logsysid.py#L20 Granted with multiple independent data series, it will be trickier. I think if you want to compare data though, you have to eventually resample at some fixed interval.
Do you think it makes sense to incorporate some plotting/analysis routines directly with pyulog? I would be ok with merging my px4tools work into pyulog if it meant less code maintenance for me.
I think if you want to compare data though, you have to eventually resample at some fixed interval.
Agreed. I just want to keep it transparent and make data manipulations explicit. Meaning we should not resample by default, but add convenience method to do so if the user wants it.
I haven't used pandas so far, but it looks like a good choice. I'd just like to avoid having it inside pyulog as the innermost dependency, but for analysis scripts we can definitely add it. From what I see, it's easy to use numpy arrays in pandas.
Do you think it makes sense to incorporate some plotting/analysis routines directly with pyulog?
I'd like to keep pyulog lean, just being a generic parsing library for ULog. There should only be minor changes once it's marked as stable. But I think we should create a separate repo for all the log analysis & plotting scripts (and some generic library code), probably based on jupyter notebook.
Also, this is relevant for @priseborough as he does plotting scripts based on ULog (#7) as well.
Maybe we can also share code with https://github.com/PX4/flight_review (the new web plotting tool). flight_review uses the bokeh plotting library, which can also be used within a jupyter notebook, with the advantage of having interactive plots (I added an example in https://github.com/PX4/flight_review/blob/master/testing_notebook.ipynb).
What do you think, should we swtich entirely to bokeh with jupyter notebook? It would allow us to easily generate the same plots on the server.
@Tumbili @Stifael your feedback would be welcome here as well.
I agree that we should create a separate repository for the plotting and data analysis tools. There is an up-front cost associated with getting developers to change tools (eg jupyter and bokeh) which for me is measured in weeks as I have no familiarity with either and will have to re-write what has been done already. The impact on schedule is something I will need to discuss with my customers.
I created a document here to try and capture the use cases for the EKF data analysis tool, but so far only Wingtra has contributed:
I will link to it in the document Dennis created for the vehicle level log analysis.
@bkueng I think we should use pandas as the underlying data analysis tool and then use bokeh for plotting as you suggest. Bokeh is actually made to work with pandas according to their docs. I already have a lot of flight analysis scripts written in pandas here, https://github.com/dronecrew/px4tools, and I already have it distributed on conda-forge with ci testing on all platforms (linux/windows/osx). Maybe we just move all analysis code to px4tools and move the repository to PX4 from dronecrew? We can then add the bokeh plotting to px4tools.
@priseborough I think you should run with what you have short term than then we can all help you convert it over to pandas/bokeh.
This is a good 10 min intro of what pandas can do: https://vimeo.com/59324550
@bkueng @priseborough I have added a ulog -> pandas data frame converter that keeps all data sparse.
https://github.com/dronecrew/px4tools/blob/master/examples/pyulog.ipynb
Any reason you convert to CSV first? Can we do it directly?
@bkueng You could write it natively in pyulog using the python class: http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe. I was just lazy.
Whats the state now. I saw in the setup.py from the pyulog package that is is considered as a control and statistical analysis tool. Does that mean that analysis modules such as px4tools are going to be merged into pyulog? Also, what about plotting in general. Do we have now a common plotting tool (bokeeh?).
No, I think it's a good separation as it is now, makes it easier to maintain and to keep a clear API.
If we want to display the plots interactively on the web as well, then yes, bokeh is the way to go. But I (or rather we) haven't really decided on that.
I agree, it keeps things cleaner when keeping the tools seperated. But now my real question: I expect that a lot of similar log analysis and plot crations are required for real flight test analysis (where usually the plots are first examined using flight_review), for ekf analysys using @priseborough script, and for general control analysis done with the px4tools from @jgoppert. Currently im thinking about improving the current px4 integration tests and want to include analysis from the log files as well. So.im wondering if it is possible to have a common way of doing analysis and plotting. Maybe even having the visual stuff independent from the analysis. IMHO that would make it easier to contribute to the existing tools.
Closing as stale.