Unexpected InvalidDataSize
tthebst opened this issue · comments
Bug Report
KVDK version
System configuration
Reproduce steps
kvdk::Configs engine_configs;
{
engine_configs.pmem_file_size = 40 * 1024UL * 1024UL * 1024UL;
engine_configs.pmem_segment_blocks = (1ull << 8);
engine_configs.hash_bucket_num = (1ull << 10);
engine_configs.log_level = kvdk::LogLevel::Debug;
}
std::string engine_path{kvpmem_data_file_path};
// Purge old KVDK instance
system(std::string{"rm -rf " + engine_path + "\n"}.c_str());
status = kvdk::Engine::Open(engine_path, &engine, engine_configs, stdout);
assert(status == kvdk::Status::Ok);
DBUG_PRINT("KVDK",
("Successfully created KVDK engine %s", kvpmem_data_file_path));
status = engine->SSet("hi", "yo", "peeep");
ASSERT(status == kvdk::Status::Ok, (int)status);
}
Expect behavior
Should put value in the ordered set.
Current behavior
Assert fails with InvalidDataSize. If I print the size of the string view inside the engine I get following output [ERROR] time 40009 ms: SSET size collection: 1147430597 userkey: 1147430594 value: 1147430588.
I think there is an issue with the string view but I don't understand why since the usage is very similar to the usage guide. I also tried to pass a static string. Any ideas?
Thanks for your feedback.
I can't reproduce the issue with these codes, could you provide the complete code?
Thanks for the response. The compleete code is available here in the file storage/kvpmem/ha_kvpmem.cc.
A possibility to reproduce is to use my docker image.
docker push tthebst/mysql-test:bug
docker run --network host --cap-add=sys_nice -v /mnt/local2/tim/my-kvpmem.cnf:/etc/mysql/my.cnf -v /mnt/pmem2:/pmem -e MYSQL_ROOT_PASSWORD=secret -e MYSQL_USER=mysql -e MYSQL_PASSWORD=secret --name mysql8 -d tthebst/mysql-test:kvpmem mysqld --default-authentication-plugin=mysql_native_password --debug=d,KVDK:i
my-kvpmem.cnf
[mysqld]
#default_storage_engine=KVPMEM
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
secure-file-priv= NULL
disable_log_bin
kvpmem_data_file_path="/pmem/testkvpmem"
max_connections=5001
max_prepared_stmt_count=524224
To build the container you can use the following instructions
git clone -b 8.0-kvdk git@github.com:tthebst/mysql-server.git --depth 1
cd mysql-server
git submodule update --init --recursive
mkdir storage/kvpmem/lib
cd storage/kvpmem
bash update_kvdk.sh
cd ../..
# MySQL build instructions
mkdir build
# go into build container. This makes it easier to compile MySQL so you don't need any dependencies
docker run -it --entrypoint bash -v $PWD:/data tthebst/build-container:test
cd /data/
# compile kvdk
mkdir storage/kvpmem/lib
cd storage/kvpmem
bash update_kvdk.sh
cd ../..
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/out -DCPACK_MONOLITHIC_INSTALL=1 -DWITH_BOOST=/usr/local -DWITH_KVPMEM_STORAGE_ENGINE=1 -DWITH_DEBUG=1 -D CMAKE_C_COMPILER=gcc-8 -D CMAKE_CXX_COMPILER=g++-8
make -j11 && make -j11 package
exit
# build && run mysql container
docker build -t tthebst/mysql-test:kvdk-bug .
docker run --network host --cap-add=sys_nice -v /mnt/local2/tim/my-kvpmem.cnf:/etc/mysql/my.cnf -v /mnt/pmem2:/pmem -e MYSQL_ROOT_PASSWORD=secret -e MYSQL_USER=mysql -e MYSQL_PASSWORD=secret --name mysql8 -d tthebst/mysql-test:kvdk-bug mysqld --default-authentication-plugin=mysql_native_password --debug=d,KVDK:i
To test the issue connect with a mysql client and create a table. Look at docker logs.
docker run -it --network host --rm mysql bash -c 'mysql --user=root --password=secret --host=127.0.0.1' -p
CREATE database t;
use t;
CREATE TABLE shopp (
article INT UNSIGNED DEFAULT '0000' NOT NULL,
dealer CHAR(20) DEFAULT '' NOT NULL,
price DECIMAL(16,2) DEFAULT '0.00' NOT NULL,
PRIMARY KEY (article))
ENGINE = KVPMEM;
Let me know if that helps. Honestly, I'm not even sure if this is KVDK issue. Maybe I made a build mistake for KVDK or I'm accidentally using the wrong string view implementation.
Thank you Tim for providing such detailed steps to reproduce the problem.
I pulled your docker image and tried to create a table, saw the below error, is it the same as yours?
$ docker pull tthebst/mysql-test:bug
$ docker run --network host --cap-add=sys_nice -v /home/psi1/my-kvpmem.cnf:/etc/mysql/my.cnf -v /mnt/pmem0:/pmem -e MYSQL_ROOT_PASSWORD=secret -e MYSQL_USER=mysql -e MYSQL_PASSWORD=secret --name mysql8 -d tthebst/mysql-test:bug mysqld --default-authentication-plugin=mysql_native_password --debug=d,KVDK:I
$ docker exec -it mysql8 /bin/bash
$ mysql --user=root --password=secret --host=127.0.0.1
mysql> CREATE database t;
Query OK, 1 row affected (0.00 sec)
mysql> use t;
Database changed
mysql> CREATE TABLE shopp (
-> article INT UNSIGNED DEFAULT '0000' NOT NULL,
-> dealer CHAR(20) DEFAULT '' NOT NULL,
-> price DECIMAL(16,2) DEFAULT '0.00' NOT NULL,
-> PRIMARY KEY (article))
-> ENGINE = KVPMEM;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...If so, we may need more time to debug it.
Meanwhile, I have tried to run dbtest and bench on the container, it works. So I think the docker environment is ok.
root@ubuntu:/pmem/kvdk/build# pwd
/pmem/kvdk/build
root@ubuntu:/pmem/kvdk/build# ldd dbtest
linux-vdso.so.1 (0x00007ffea91f4000)
libengine.so => ./libengine.so (0x00007f5b67c00000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5b683d1000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5b679f4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b6828c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b67fe6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b6782f000)
libpmem.so.1 => /usr/lib/x86_64-linux-gnu/libpmem.so.1 (0x00007f5b67f92000)
libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007f5b675f2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5b683f8000)
libndctl.so.6 => /usr/lib/x86_64-linux-gnu/libndctl.so.6 (0x00007f5b67f6a000)
libdaxctl.so.1 => /usr/lib/x86_64-linux-gnu/libdaxctl.so.1 (0x00007f5b67f5d000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f5b67f4d000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f5b67f42000)
libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x00007f5b67f19000)
libuuid.so.1 => /usr/lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f5b67f10000)
libkmod.so.2 => /usr/lib/x86_64-linux-gnu/libkmod.so.2 (0x00007f5b67ef4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5b67eee000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5b67ec4000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f5b672fe000)
root@ubuntu:/pmem/kvdk/build# ./dbtest
[==========] Running 15 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 15 tests from EngineBasicTest
[ RUN ] EngineBasicTest.TestThreadManager
[INFO] time 96 ms: Map pmem space done
[INFO] time 97 ms: RestorePendingBatch done: iterated 0 records
[INFO] time 97 ms: RestoreData done: iterated 0 records
[INFO] time 97 ms: Rebuild skiplist done
[INFO] time 99 ms: Closing instance ...
[INFO] time 99 ms: Waiting bg threads exit ...
[INFO] time 197 ms: Instance closed
[ OK ] EngineBasicTest.TestThreadManager (232 ms)
[ RUN ] EngineBasicTest.TestBasicStringOperations
[INFO] time 96 ms: Map pmem space done
[INFO] time 96 ms: RestorePendingBatch done: iterated 0 records
[INFO] time 97 ms: RestoreData done: iterated 0 records
[INFO] time 97 ms: Rebuild skiplist done
[INFO] time 101 ms: Closing instance ...
[INFO] time 101 ms: Waiting bg threads exit ...
[INFO] time 197 ms: Instance closed
[ OK ] EngineBasicTest.TestBasicStringOperations (244 ms)
...
...
root@ubuntu:/pmem/kvdk/build# ./bench
[INFO] time 1440 ms: Map pmem space done
[INFO] time 2472 ms: RestorePendingBatch done: iterated 0 records
[INFO] time 4561 ms: RestoreData done: iterated 37034937 records
[INFO] time 4561 ms: Rebuild skiplist done
init 10 write threads
init 0 read threads
------- ops in seconds -----------
time (ms), read ops, not found, write ops, total read, total write
1000 0 0 12151000 0 12151000
2000 0 0 14326000 0 26477000
3000 0 0 12414000 0 38891000
4000 0 0 10645000 0 49536000
...
Yes this is the issue I'm facing. I updated the instruction to build MySQL. It now includes e build container such that you don't have to install any dependencies to compile. Let me know if it works.
I just also tried to use the same compile gcc 8.5 to compile kvdk and MySQL such that there is no ABI issue. But I still face the same issue.
This is the code section that causes the issue.
Can please you replace the stdout with a file to see if there is an error message of KVDK?
status = kvdk::Engine::Open(engine_path, &engine, engine_configs, stdout);
[INFO] time 79 ms: Map pmem space done
[INFO] time 80 ms: RestorePendingBatch done: iterated 0 records
[INFO] time 82 ms: RestoreData done: iterated 0 records
[INFO] time 82 ms: Rebuild skiplist done
[INFO] time 83 ms: Populating PMem space ...
[INFO] time 12040 ms: Populating done
This is the log. It seems fine. The issue happens when you access the string view inside KVKD since it points to a bogus memory location. Not sure why the argument passing does not work as intended. Could it be some ABI issue?
By default, KVDK is compiled with C++ 17
Line 11 in 76f276a
Since C++ 17 already introduced the std::string_view, so KVDK will use std::string_view instead of the pmem::obj::string_view
kvdk/extern/libpmemobj++/string_view.hpp
Line 28 in 76f276a
So, I guess you are right, it may be the ABI issue. Because in ha_kvpmem.cc, you use pmem::obj::string_view, but KVDK uses std::string_view. Could you please replace pmem::obj::string with std::string_view in your code and have a try again?
BTW, I will submit a PR to remove the pmem::obj::string_view in KVDK code to avoid confusion.
Thanks for the hints. I think the issue is the following. MySQL is compiled with C++14 (and I can't easily move to c++17...) and therefore std::string_view is not included and the arguments passed to KVDK are using the type of libpmemobj-cpp. KVDK is compiled with C++17 and uses std::string_view. This probably leads to some weird access.
So there is no issue with KVDK :). But I don't have much in-depth C++ knowledge and I can't think of a solution for this right now. Do you have any ideas? Thanks for the help!!!!
As a temporary workaround, can you please use the C API of KVDK? Here is the example of usage:
https://github.com/pmem/kvdk/blob/main/examples/tutorial/c_api_tutorial.c
Thanks for the suggestion. I got some things working with the C API. For example using SET and GET. But I still have issues with the iterator.
DBUG_PRINT("KVDK", ("rnd_init: a %s", active_table.c_str()));
assert(kvdk_engine != NULL);
KVDKIterator *iter =
KVDKCreateIterator(kvdk_engine, active_table.c_str(), SORTED);
if (iter) {
// seek to first element of table
KVDKIterSeekToFirst(iter);
DBUG_PRINT("KVDK", ("rnd_init:"));
} else {
// seek to first element of table
DBUG_PRINT("KVDK", ("rnd_init: hmmmmm"));
}This code fails when calling KVDKIterSeekToFirst. There is also no DBUG output but the stacktrace shows me that it failed there.
Thank you for your help.
Tim, We will debug it.
BTW, the latest commit make KVDK support C++11, 14 and 17, so you may compile it with C++14 to solve the ABI issue of MySQL.
change it to 14 for your case.
Line 11 in 6123387
Okay, thank you very much. The C++14 support will help me. The issue above is resolved it was an issue on my side.
I think the issue can be closed. Thanks again for the help.
You're welcome. If you have any more issues, please let us know.