Performance regression with newer KLEE versions

[Dennis Yurichev]

Hi all.
The following example: https://sat-smt.codes/current_tree/CRC/KLEE/klee_SILVER.c
It is intended to reconstruct a text string for a specified CRC32 hash.
It works lightning fast for KLEE 1.4.0 (version from docker), just checked today.

clang -emit-llvm -c -g klee_SILVER.c
klee klee_SILVER.bc

But version 2.0 and 3.0 (latest) also from docker - same example suck for a hour.
I probably run it incorrectly?
Something changes since version 1.4.0?

[Cristian Cadar]

Hi @DennisYurichev, it is hard to say what causes this performance regression for your example, as many things have changed between the 1.4.0 and 2.0 Docker images: KLEE itself, the underlying solvers, the LLVM compiler and the code it generates, etc.

I also note that KLEE 1.4.0 is really old: more than 6 years old!

It would be interesting to debug this: I would advise you to first try to use the same LLVM version with 1.4 and 2.0 as well as the same constraint solver version, and see if you still observe the performance difference. If so, we would know it's KLEE and we can try to debug the issue.

[Dennis Yurichev]

OK, thanks for comment.
But at least, do I run clang and klee correctly?

[Cristian Cadar]

I didn't look in detail at the code, but I don't see any obvious issues with how you are using KLEE here.

[Dennis Yurichev]

I used the same example file I mentioned in the first github post.

I tried the '--use-query-log=solver:smt2' option.

• Version 3.0

clang -emit-llvm -c -g klee_SILVER.c
time klee --use-query-log=solver:smt2 klee_SILVER.bc

... after just couple of minutes I breaked it...

cat /home/klee/klee-out-0/solver-queries.smt2 | grep Query | wc -l
12723

So much queries! Also, that file grows quickly during klee execution.

• Version 1.4.0

clang -emit-llvm -c -g klee_SILVER.c
time klee --use-query-log=solver:smt2 klee_SILVER.bc

Finished quickly.

cat /home/klee/klee-out-0/solver-queries.smt2 | grep Query | wc -l
15

Does it all sounds familiar to you?

What other debugging switches can be useful here?

[Cristian Cadar]

@DennisYurichev I looked a bit at this, and it's an LLVM code generation/optimisation issue. While LLVM 3.4 (used inside the KLEE 1.4 Docker image) generates a select instruction when compiling the for loop body in crc32, LLVM 6.0 (used inside the KLEE 2.0 Docker image) generates an explicit branch. With the explicit branch, KLEE has to fork a huge number of paths, which is not the case when a select is used.

To avoid relying on the LLVM behaviour, merge all the paths inside the loop:

for (k = 0; k < 8; k++) {
     klee_open_merge();
     crc = crc & 1 ? (crc >> 1) ^ 0xedb88320 : crc >> 1;
     klee_close_merge();
 }

and then run klee --use-merge klee_SILVER.bc

You should see a similar behaviour as with the KLEE 1.4 Docker image.

[Dennis Yurichev]

Thank you very much for your comment. Going to investigate...

[Frank Busse]

Can you try running klee with -optimize (the SimplifyCFG pass should fix this imho)?

[Dennis Yurichev]

Necro-bump (as youngsters say).

@251 - simply adding '-optimize' didn't solve the problem.

@ccadar Adding "klee_open_merge();" and "klee_close_merge();" and running with "--use-merge" solved it, thanks! But how can I know I should add these functions? When? Where?
Am I right it should be used if a loop body has a lot of branches?

[Frank Busse]

@DennisYurichev

You probably forgot to compile with -O0 -Xclang -disable-O0-optnone when -optimize didn't work.

As stated in the tutorials, we recommend above flags or CFLAGS="-g -O1 -Xclang -disable-llvm-passes -D__NO_STRING_INLINES -D_FORTIFY_SOURCE=0 -U__OPTIMIZE__" for larger projects.

[Dennis Yurichev]

Thank you very much for your replies. Going to fix my examples...