Giters

xxsds / DYNAMIC

Dynamic succinct/compressed data structures

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

abort: packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size()

smoe opened this issue · comments

commented

Hello,
ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1.
Anything you want me to do about it?

Cheers,
Steffen

commented

First invocation:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:252ms
rank:331ms
wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

and in gdb:

Core was generated by `./obj-x86_64-linux-gnu/wm_string'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f08b36ef55b in __GI_abort () at abort.c:79
#2  0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0", 
    file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
#3  0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, 
    function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
    at assert.c:101
#4  0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=<optimized out>, rank=0, c=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:1058
#5  0x000056266528c474 in speed_select (num=100000, num_of_alphabet=<optimized out>) at ./wm_string.cpp:74
#6  0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
#7  0x000056266528aa2e in main () at ./wm_string.cpp:394

This is very nicely reproducible.

commented
commented
commented

That is what I presume since I requested a pull from vgteam/DYNAMIC.

commented

Hello,
ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1.
Anything you want me to do about it?

Cheers,
Steffen

I thought I had solved this bug but apparently it's alive and well. I'll look into it next week.

commented

Hi, I solved the first part of the issue:

Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.

Actually, the code was correct and only the assert had a bug : -)

@ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.

commented
commented

There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?

commented
commented

All dependencies of DYNAMIC are in Debian/Ubuntu already - I guess this would be rather straight-forward to implement. Ping me if you need help.

commented

@nicolaprezza I'm not able to reproduce the problem with wm_string. It's at least not happened in the 20-30 runs that I've done (running it in a loop and watching...)

commented

Latest checkout:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:229ms
rank:293ms
wm_string: /home/moeller/git/med-team/dynamic/a/DYNAMIC/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

Maybe because of other compiler settings?

/usr/bin/cmake -E cmake_link_script CMakeFiles/wm_string.dir/link.txt --verbose=1
/usr/local/bin/c++  -g -O2 -fdebug-prefix-map=/home/moeller/git/med-team/dynamic/a/DYNAMIC=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --std=c++11  -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -rdynamic CMakeFiles/wm_string.dir/wm_string.cpp.o  -o wm_string

The benchmark seems stable now. Would you have other ideas what I should test over here?

./obj-x86_64-linux-gnu/cw-bwt /etc/passwd /tmp/bla.txt worked - but not on /proc/cpuinfo
./obj-x86_64-linux-gnu/h0_lz77 /etc/passwd /tmp/bla.txt worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txt worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txtworked

I can get a VM online somewhere for a common playground to investigate the wm_string issue.

commented

I can reproduce it. Have you enabled asserts? with cmake, it's simply cmake .. -DCMAKE_BUILD_TYPE=Debug

commented

Thanks :) That's my mistake. I can reproduce it by enabling the assert!

commented

It does look like a possible problem with the input. There is a quirky PRNG being used. I might switch to modern c++ random distributions.

commented

Thanks @ekg! now it works.