segfault on debian buster calling ping after disconnect

Question

segfault on debian buster calling ping after disconnect

hashbangperl opened this issue 5 years ago · comments

Replicated with both MariaDB 10.3 and 10.4.

Will reliably get a segfault 1 in 2 to 1 in 4 times, when calling ping on a db handle after calling disconnect on it with latest DBD::MariaDB and mariadb 10.3 or 10.4 on debian buster, likely to be same on debian stretch. This was with auto_reconnect flag set.

my $dbh = DBI->connect("dbi:mysql:$dbname;host=localhost",$user, $password);
$dbh->disconnect();
ok( ! $dbh->ping(), 'dbh is disconnected and did not segv');

done_testing();

pali · Answer 1 · Fri Dec 06 2019 23:23:39 GMT+0800 (China Standard Time)

Hello! With specifying dbi:mysql in DSN string you are using DBD::mysql driver and not DBD::MariaDB. Please try again with DSN string "DBD::MariaDB:$dbname;host=localhost".

pali · Answer 2 · Fri Dec 06 2019 23:51:33 GMT+0800 (China Standard Time)

I compiled DBD::MariaDB 1.21 with MariaDB 10.3.8 client and run your example script under valgrind to check for any memory corruption and also under gdb to see any suspicious problems:

$ valgrind perl -Iblib/lib -Iblib/arch -MDBI -E 'my $dbh = DBI->connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket", "pali", ""); say "client version library: $dbh->{mariadb_clientversion}"; $dbh->disconnect(); say "ping after disconnect " . ($dbh->ping() ? "succeeded" : "failed"); say "no segfault"'
==26880== Memcheck, a memory error detector
==26880== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==26880== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==26880== Command: perl -Iblib/lib -Iblib/arch -MDBI -E my\ $dbh\ =\ DBI-\>connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket",\ "pali",\ "");\ say\ "client\ version\ library:\ $dbh-\>{mariadb_clientversion}";\ $dbh-\>disconnect();\ say\ "ping\ after\ disconnect\ "\ .\ ($dbh-\>ping()\ ?\ "succeeded"\ :\ "failed");\ say\ "no\ segfault"
==26880== 
client version library: 100308
ping after disconnect failed
no segfault
==26880== 
==26880== HEAP SUMMARY:
==26880==     in use at exit: 3,448,742 bytes in 16,299 blocks
==26880==   total heap usage: 37,089 allocs, 20,790 frees, 6,631,374 bytes allocated
==26880== 
==26880== LEAK SUMMARY:
==26880==    definitely lost: 20,635 bytes in 19 blocks
==26880==    indirectly lost: 3,319,127 bytes in 13,005 blocks
==26880==      possibly lost: 0 bytes in 0 blocks
==26880==    still reachable: 108,980 bytes in 3,275 blocks
==26880==         suppressed: 0 bytes in 0 blocks
==26880== Rerun with --leak-check=full to see details of leaked memory
==26880== 
==26880== For counts of detected and suppressed errors, rerun with: -v
==26880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

$ gdb -ex r -ex bt -ex quit --args perl -Iblib/lib -Iblib/arch -MDBI -E 'my $dbh = DBI->connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket", "pali", ""); say "client version library: $dbh->{mariadb_clientversion}"; $dbh->disconnect(); say "ping after disconnect " . ($dbh->ping() ? "succeeded" : "failed"); say "no segfault"'
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from perl...(no debugging symbols found)...done.
Starting program: /usr/bin/perl -Iblib/lib -Iblib/arch -MDBI -E my\ \$dbh\ =\ DBI-\>connect\(\"DBI:MariaDB:test\;host=localhost\;mariadb_socket=/tmp/socket\",\ \"pali\",\ \"\"\)\;\ say\ \"client\ version\ library:\ \$dbh-\>\{mariadb_clientversion\}\"\;\ \$dbh-\>disconnect\(\)\;\ say\ \"ping\ after\ disconnect\ \"\ .\ \(\$dbh-\>ping\(\)\ \?\ \"succeeded\"\ :\ \"failed\"\)\;\ say\ \"no\ segfault\"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
client version library: 100308
ping after disconnect failed
no segfault
[Inferior 1 (process 26868) exited normally]
No stack.

But I was not able to reproduce any crash.

Can you please specify exact version of MariaDB client which is causing segfault? Also can you provide valgrind and gdb output (for backtrace) of crash?

Aaron Trevena · Answer 3 · Sat Dec 07 2019 00:43:27 GMT+0800 (China Standard Time)

Hello! With specifying dbi:mysql in DSN string you are using DBD::mysql driver and not DBD::MariaDB. Please try again with DSN string "DBD::MariaDB:$dbname;host=localhost".

Yes, I just copied the same test code I used for mysql. It broke with DBD::MariaDB in the same way.

Aaron Trevena · Answer 4 · Sat Dec 07 2019 01:07:29 GMT+0800 (China Standard Time)

I compiled DBD::MariaDB 1.21 with MariaDB 10.3.8 client and run your example script under valgrind to check for any memory corruption and also under gdb to see any suspicious problems:

$ valgrind perl -Iblib/lib -Iblib/arch -MDBI -E 'my $dbh = DBI->connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket", "pali", ""); say "client version library: $dbh->{mariadb_clientversion}"; $dbh->disconnect(); say "ping after disconnect " . ($dbh->ping() ? "succeeded" : "failed"); say "no segfault"'
==26880== Memcheck, a memory error detector
==26880== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==26880== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==26880== Command: perl -Iblib/lib -Iblib/arch -MDBI -E my\ $dbh\ =\ DBI-\>connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket",\ "pali",\ "");\ say\ "client\ version\ library:\ $dbh-\>{mariadb_clientversion}";\ $dbh-\>disconnect();\ say\ "ping\ after\ disconnect\ "\ .\ ($dbh-\>ping()\ ?\ "succeeded"\ :\ "failed");\ say\ "no\ segfault"
==26880== 
client version library: 100308
ping after disconnect failed
no segfault
==26880== 
==26880== HEAP SUMMARY:
==26880==     in use at exit: 3,448,742 bytes in 16,299 blocks
==26880==   total heap usage: 37,089 allocs, 20,790 frees, 6,631,374 bytes allocated
==26880== 
==26880== LEAK SUMMARY:
==26880==    definitely lost: 20,635 bytes in 19 blocks
==26880==    indirectly lost: 3,319,127 bytes in 13,005 blocks
==26880==      possibly lost: 0 bytes in 0 blocks
==26880==    still reachable: 108,980 bytes in 3,275 blocks
==26880==         suppressed: 0 bytes in 0 blocks
==26880== Rerun with --leak-check=full to see details of leaked memory
==26880== 
==26880== For counts of detected and suppressed errors, rerun with: -v
==26880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

$ gdb -ex r -ex bt -ex quit --args perl -Iblib/lib -Iblib/arch -MDBI -E 'my $dbh = DBI->connect("DBI:MariaDB:test;host=localhost;mariadb_socket=/tmp/socket", "pali", ""); say "client version library: $dbh->{mariadb_clientversion}"; $dbh->disconnect(); say "ping after disconnect " . ($dbh->ping() ? "succeeded" : "failed"); say "no segfault"'
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from perl...(no debugging symbols found)...done.
Starting program: /usr/bin/perl -Iblib/lib -Iblib/arch -MDBI -E my\ \$dbh\ =\ DBI-\>connect\(\"DBI:MariaDB:test\;host=localhost\;mariadb_socket=/tmp/socket\",\ \"pali\",\ \"\"\)\;\ say\ \"client\ version\ library:\ \$dbh-\>\{mariadb_clientversion\}\"\;\ \$dbh-\>disconnect\(\)\;\ say\ \"ping\ after\ disconnect\ \"\ .\ \(\$dbh-\>ping\(\)\ \?\ \"succeeded\"\ :\ \"failed\"\)\;\ say\ \"no\ segfault\"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
client version library: 100308
ping after disconnect failed
no segfault
[Inferior 1 (process 26868) exited normally]
No stack.

But I was not able to reproduce any crash.

Can you please specify exact version of MariaDB client which is causing segfault? Also can you provide valgrind and gdb output (for backtrace) of crash?

I reproduced this several times using mariadb on debian buster using the mariadb debian packages provided for buster for mariadb both 10.3 and 10.4

I've rebooted the VM and now it's refusing to segfault on an empty database, I suspect that you won't see this problem on a freshly booted clean install, then. I'm going to continue trying to reproduce, once I get it to happen again, I'll throw strace and gdb at it

pali · Answer 5 · Thu Dec 12 2019 00:55:22 GMT+0800 (China Standard Time)

Hi! Have you able to reproduce this issue under gdb/valgrind with DBD::MariaDB?

Aaron Trevena · Answer 6 · Thu Dec 12 2019 05:46:05 GMT+0800 (China Standard Time)

I've managed to replicate it in GDB with DBD::mysql, but since reinstalling DBD::MariaDB I've not been able to get it to segfault, even when the mysql dbd client is, so it requires a specific combination of packages in buster that I had earlier and haven't replicated since reinstalling various mariadb and mysql servers and clients :(

Aaron Trevena · Answer 7 · Thu Dec 12 2019 05:47:19 GMT+0800 (China Standard Time)

Valgrind seems to prevent it happening, I was able to get it to fail with DBD::MariaDB initially with standard mariadb on buster without valgrind, it may be possible with GDB but I'd have to rebuild buster from scratch to replicate, and that's not worth my time at the moment

pali · Answer 8 · Mon Dec 16 2019 17:37:31 GMT+0800 (China Standard Time)

Valgrind may prevent segfaults, but it prints stacktrace of any memory corruptions which could lead to segfaults. So it is still useful.

I was trying to reproduce your reported issue on Debian 9.11 but I was not able. Even valgrind does not report any suspicious memory access.

So please provide whole reproducer, exact versions of MariaDB client, MariaDB server, DBI module, DBD::MariaDB module and test script which is causing it, together with stacktrace of crash (either from valgrind or gdb).

Because currently it looks like a glitch in setup as DBD::MariaDB is already tested on Travis with about 200 configurations of MariaDB client and server versions and there is no crash or failed test... https://travis-ci.org/gooddata/DBD-MariaDB

pali · Answer 9 · Sun Feb 02 2020 21:34:36 GMT+0800 (China Standard Time)

@hashbangperl: I tried to reproduce this issue on Debian 10 Buster with MariaDB 10.3.18, but without success. There is no memory corruption, no crash.

isync · Answer 10 · Tue Jun 30 2020 01:13:44 GMT+0800 (China Standard Time)

I see regular segfaults in a script where database connection times out between database accesses. The script issues db requests fine, then sleeps for extended periods. In these sleep()s db connection times out (as one can expect). And once the script tries to reconnect to db or does a ping() I get a segfault on next INSERT. It's reproducible.

I use DBD::MariaDB through Dancer::Plugin:Database. (I've checked that it uses DBD::MariaDB and not the MySQL DBD.) D::P::Database has an auto-reconnect feature that triggers the segfault. I tried to remedy this by issuing a ping() more regularly and before any "next database operation" - the result is segfaults triggered by this ping() or shortly after.

First I usually get a "MySQL server has gone away" message, although it's only the connection and server is still live. Then immediately or on the next operation I get a segfault. Here's a snippet from my script logs:

DBD::MariaDB::db do failed: MySQL server has gone away at /usr/share/perl5/Dancer/Plugin/Database/Core.pm line 344.
[10497] debug @155590.397738> Database connection went away, reconnecting in /usr/share/perl5/Dancer/Plugin/Database.pm l. 28
DBI::db=HASH(0x55ad189c1b00)->disconnect invalidates 1 active statement handle (either destroy statement handles or call finish on them before disconnecting) at /usr/share/perl5/Dancer/Plugin/Database/Core.pm line 94.
...
Sleeping... (60min)
Sleeping... (60min)
Start next iteration:
Inserting data...
Segmentation fault

I'm aware of how annoying and unhelpful this report is, but I hope even a small "I've seen it as well" is of any help. I'm too noobish with stacktraces etc. to provide any dumps or so... sorry.

$ cat /etc/issue
Debian GNU/Linux 10
> SELECT VERSION( ) ;
10.3.22-MariaDB-0+deb10u1
$ perl -MDBD::MariaDB -e 'print "$DBD::MariaDB::VERSION\n"'
1.21
$ perl -MDBI -e 'print "$DBI::VERSION\n"'
1.642
$ perl -MDancer::Plugin::Database::Core -e 'print "$Dancer::Plugin::Database::Core::VERSION\n"'
0.20
$ perl -MDancer::Plugin::Database -e 'print "$Dancer::Plugin::Database::VERSION\n"'
2.13

isync · Answer 11 · Mon Aug 24 2020 19:43:48 GMT+0800 (China Standard Time)

I'm seeing an increase of segfaults on long-standing db-connections since I've upgraded MariaDB (via apt upgrade) from
10.3.22-MariaDB-0+deb10u1 to
10.3.23-MariaDB-0+deb10u1

I've reinstalled/recompiled DBD::MariaDB against changed libs of the same apt upgrade, so that should be in sync.

Also MariaDB is quickly filling errors.log with notes of: MariaDB aborted connection. Got an error reading communication packets
I don't know if these 'aborted connections' are caused by the same issue or if that's due to other MariaDB changes between mentioned versions.

pali · Answer 12 · Sat Feb 11 2023 02:45:48 GMT+0800 (China Standard Time)

Could you please provide gdb backtrace of the crash? I tried different test scenarios on Debian Buster but I was not able to trigger segfault.

isync · Answer 13 · Mon Feb 13 2023 01:07:10 GMT+0800 (China Standard Time)

At least from my side: sorry, no. This feels like ages ago and I can't rebuild the same stack. But what I can say is I'm using MariaDB since then, and on a similar stack, and the error didn't come to my attention again.

pali · Answer 14 · Mon Feb 13 2023 01:35:49 GMT+0800 (China Standard Time)

@hashbangperl: Are you able to reproduce this problem?

pali · Answer 15 · Thu Aug 24 2023 02:49:19 GMT+0800 (China Standard Time)

I'm closing this issue as nobody was able to reproduce it for more years.