Add notebooks to unit test
icouckuy opened this issue · comments
It would be useful to add the documentation notebooks to the testing framework somehow. It will easily detect API changes without manually having to run them all.
Of course, it will slow down the tests a lot (especially with Travis). Anyone has a solution for that?
Perhaps it's worth looking at the GPflow solution for testing notebooks. Really slow notebook are excluded from the tests. If you come up with something do please let us know.
I added the notebooks to the tests in #49 . I experience no timeouts, only a crash because latex is missing on Travis (I could install it but I think removing latex labels is likely to be faster). Main problem is that about a week ago our Travis builds suddenly doubled in runtime without us adding slow tests etc. I am in contact with their support to find out what is wrong. If this is fixed I think notebook tests are feasible.
@javdrher a bit of a stab in the dark as I don't have all context but the GPflow tests got slow at one point in the early days because we were using the tf default graph without resetting it. This meant that it got very large as we ran the tests. Discard if not relevant.
Got solved and turned out to be a mixture of many things, thanks for the input!