Collect and output timing information
krlmlr opened this issue · comments
If you want faster tests, it's nice to know how long each step takes
this is a great idea -- just to brainstorm, where do you think we should stick the timing data?
Into some well-known file (=hard-coded file name), each invocation of travis-tool.sh appends one row to that file. Output the file in the after_scripts hook, with a new command in travis-tool.sh.
But what do you mean by timing data for tests? There's not a consistent API across unit testing packages
I think the timing should be part of the testing framework; a quick scan of the RUnit manual suggests that timing can be part of the output of a run of a suite. Travis just needs to start your framework, and then do as little as possible to interfere with its working.
I should also say that having had timing tests run on travis; their timing is rather unreliable. Different runs of the same test give significantly different timing results.
The issue is about timing each action of travis-tool.sh -- bootstrap, installation, testing -- not timing the actual unit tests.
Actually, I think they're not necessarily so different -- I'd love to have an easy way to push data from a running travis test to somewhere we could pick it up later; while it works, I think "spit it out in the logs and grep for it" is a sub-par solution.
- For travis runs itself, it'd be nice to see how long the steps take.
- For tests, it'd be great to see pass/fail of individual tests
- For tests/benchmarks, it'd be nice to gather data -- though as @kasterma points out, that's likely to be noisy.
+1 for spit && grep. For how long are the Travis build logs available?
See mislav/hub#423 for a way to find the CI build for a specific Git ref.
I'm working on a way to analyze travis's log data in this package