Add continuous integration
travisdowns opened this issue · comments
Some type of CI would be nice.
It's clear that a lot of the non-functional behavior of uarch-bench is hardware specific: it wants to run on bare metal, and currently expects x86. The libpfc mode also wants to install a kernel driver, and set certain values that only root can set in /sys and a few other "root-needed" things - but you can definitely run as non root (perhaps the experience could be better though).
Just building and doing a "default" run of ./uarch-bench would be an awesome start.
I admit to being ignorant of how well cloud CI systems handle binary dependencies like nasm...
There is a patch in the master branch of nemequ/uarch-bench which adds Travis CI, but it fails due to an illegal instruction. I think it may need something like https://github.com/nemequ/uarch-bench/blob/0cce27819f45f790d4e4328be94fcc5e1b7dc5d4/psnip-bench-register.cpp#L20 to work.
@nemequ - thanks!
Right, one big issue with uarch-bench at the moment is that I've completely written it without really any regard to non-AVX2 targets (roughly, Haswell or newer and Ryzen) without any consideration for platforms that don't support that instruction set. It's been also on my TODO to formalize support for instruction sets: for example, if an instruction set isn't supported on a given platform, tests that use that should be excluded on that platform.
I think the benchmark that is failing is misc/add-32 implemented as misc_add_loop32 in x86_methods.asm which is the first test in the misc category, since it uses blsi which is an BMI instruction (what's up with that, it makes no sense: probably it's a copy/paste error and this test was intended to test blsi performance).
I can't tell what CPU arch the TravisCI servers are running, but it's not a recent intel based on Independent add chain 0.33 (only 3 integer ALUs), so maybe it's AMD (which doesn't have BMI until Ryzen, so almost certainly not what TravisCI has deployed).
I can create a bunch of issues to help here, e.g., outputting the instruction set extensions supported, the CPU line, etc, etc, but I think the best workaround to unblock you is to include --test-name=default/* to only run the default test for now.
Thoughts?
Only running some basic tests would at least provide a temporary workaround. I went ahead and added --test-name=default/*, so the build now passes.
It looks like Xeon E5-2680 v2, but I doubt all their machines are the same, and even if they are today I assume they'll buy new servers eventually. I think the only stable solution would be run-time CPU feature detection (i.e., conditionally adding tests based on __builtin_cpu_supports).
So if you want to add Travis CI, all you need to do is sign up for it and enable it for this repo on Travis CI. Then it should start building every commit as soon as you pull in the patch from my repo.
@nemequ - oh right, thats IvyBridge and I forgot they only had 3 integer ALUs (although they could still do 4 uops/cycle that could only happen if at least one used an EU other than the integer ALUs), so the 0.33 result makes perfect sense.
Methinks this can be closed, unless there is something missing that I'm unaware of…