Segmentation fault in XS.so

Question

Segmentation fault in XS.so

rodolfojcj opened this issue 3 years ago · comments

Hello.

I'm facing an intermittent issue in a web site of mine, which runs in Ubuntu 20.04, mainly with Apache 2.4.41, Perl 5.30.0 and mod_perl 2.0.11.

After a lot of debugging I found a minimal way to reproduce it, according to the following details.

Setup the server environment

This assumes a fresh Ubuntu 20.04 installation for an x86_64 CPU.

Run these commands:
apt-get install apache2
apt-get install libapache2-mod-perl2
apt-get install libcrypt-jwt-perl # this one installs these dependencies: libcpanel-json-xs-perl{a} libcryptx-perl{a} and libjson-maybexs-perl{a}
mkdir /var/perl_code

Create the file /etc/apache2/sites-available/buggy-site.com.conf with this content:

<VirtualHost *:80>
  ServerName buggy-site.com

  DocumentRoot /var/perl_code

  ErrorLog ${APACHE_LOG_DIR}/buggy-site-error.log
  CustomLog ${APACHE_LOG_DIR}/buggy-site-access.log combined

  <Directory "/var/perl_code">
          Options ExecCGI
          DirectoryIndex index.pl
          <IfVersion >= 2.3>
                  Require all granted
          </IfVersion>
          AllowOverride All
          <Files *.pl>
                  SetHandler perl-script
                  Options +ExecCGI
                  PerlResponseHandler ModPerl::Registry
                  PerlSendHeader On
          </Files>
  </Directory>
</VirtualHost>

Create the file /var/perl_code/index.pl with this content:

#!/usr/bin/perl -w
use strict;
use warnings;
# this single line is enough to trigger the segmentation fault
use Crypt::JWT;
print 'Content-Type: ' . 'application/json' . "\r\n";
print 'Content-Disposition: attachment;filename="' . 'attached_file.json' . "\"\r\n";
print "Accept-Ranges: bytes\r\n";
my $response_content = '{"responses":[{"code":"200","message":"OK","key":""}],"data":{"found":1}}';
print 'Content-Length: ' . length( $response_content ) . "\r\n";
print "\r\n";
print $response_content;

Enable the virtual host site with these commands:
a2ensite buggy-site.com.conf
systemctl restart apache2.service

Setup the client environment

I assume the client has a terminal to run commands, with the wget installed to open URLs.

To emulate a public we site, edit its /etc/hosts file to add a line that associates the IP address of the server with the sample domain name of the configured web site. In my example, that line would be like this:

  192.168.1.100 buggy-site.com

An alternative to this is to have a DNS record that correctly associates the server IP address with the domain name of the sample web site.

Use the web site

From this moment on, I will mention two sides in turns, one for the server and other for the client.

In the client, I'm using this command to access the configured web site whenever I want:

wget --server-response --output-document=- http://buggy-site.com

No matter how many times I execute that wgetcommand, no errors will happen (or at least I did not observe hints about it). I presume that this is because the Apache web server is running multiple processes, given the MPM "prefork", "event" or "worker" modules that are generally used. Also, in my web site having such multiple processes increased the difficulty to debug and isolate the error.

In the server, open a terminal to monitor the Apache and operating system log files, for example with this command:

tail -f /var/log/apache2/*.log /var/log/syslog
In the server, stop the Apache web server with this command:

systemctl stop apache2.service
In the server, start the Apache web server in debug mode with these commands:

source /etc/apache2/envvars
apache2 -X

That is convenient to run the Apache web server with a single process.

In the client, run the wget command previously given to access the web site. That command finishes successfully, getting the expected content from the server. At this moment the server works normally, without showing symptoms of an error.

Trigger the error

In the server, go to the terminal that keeps running the apache2 -X command and stop it by pressing the "Ctrl + C" keys sequence.
After doing that, these things will happen:

The terminal where the apache2 -X was running will show this mesage:
```
Segmentation fault (core dumped)
```

The /var/log/syslog file will show a couple of lines like these:

May 20 15:18:44 localhost kernel: [ 1435.562727] apache2[6398]: segfault at 8 ip 00007f57c06b7f47 sp 00007ffd0bacf8e8 error 4 in XS.so[7f57c06b7000+f000]
May 20 15:18:44 localhost kernel: [ 1435.562743] Code: 1f 40 00 f3 0f 1e fa 48 63 15 dd 30 01 00 48 8b 87 78 0b 00 00 48 83 6f 78 04 48 8b 04 d0 48 8b 70 30 48 c7 40 30 00 00 00 00 <8b> 56 08 83 fa 01 76 11 83 ea 01 89 56 08 c3 66 2e 0f 1f 84 00 00

I presume that the mentioned XS.so file in the first line corresponds to the /usr/lib/x86_64-linux-gnu/perl5/5.30/auto/Cpanel/JSON/XS/XS.so file, which is probably opened as part of the following sequence of opened files:

  /usr/local/share/perl/5.30.0/Crypt/JWT.pm
  /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/Crypt/Misc.pm
  /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/CryptX.pm
  /usr/lib/x86_64-linux-gnu/perl5/5.30/Cpanel/JSON/XS.pm
  /usr/lib/x86_64-linux-gnu/perl5/5.30/auto/Cpanel/JSON/XS/XS.so

A core dump file will appear under the /var/crash/ directory.

My observations

The system has these packages installed:

libcrypt-jwt-perl (version 0.026-1)
libcryptx-perl (version 0.067-1)
libcpanel-json-xs-perl (version 4.19-1build1)
libjson-maybexs-perl (version 1.004000-1)

The Perl module Crypt::JWT depends on CryptX, which uses a JSON::XS implementation or JSON::PP, preferring Cpanel::JSON::XS when selecting one of them. This is evident from this code fragment taken from the /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/CryptX.pm file in the server:

...
our $VERSION = '0.069';
...
BEGIN {
  if (eval { require Cpanel::JSON::XS }) {
    Cpanel::JSON::XS->import(qw(encode_json decode_json));
    $has_json = 1;
  }
  elsif (eval { require JSON::XS }) {
    JSON::XS->import(qw(encode_json decode_json));
    $has_json = 2;
  }
  elsif (eval { require JSON::PP }) {
    JSON::PP->import(qw(encode_json decode_json));
    $has_json = 3;
  }
  else {
    $has_json = 0;
  }
}

Given that first preference that CryptX has, I think that the segmentation fault is caused by some bug in the Cpanel::JSON::XS implementation, however I could be wrong, in which case I will need to create an issue for that CryptX module.

A workaround

Installing JSON::XS and removing Cpanel::JSON::XS brings these two things:

Stops the segmentation fault from happening when executing the apache2 -X command.
Stops the intermittent issue I faced with the web site of mine.

To do that, these are the commands to execute in the server:

aptitude install libjson-xs-perl
aptitude remove libcpanel-json-xs-perl

What to do?

If possible I would like to have this issue confirmed, solved and with a fix published for Ubuntu 20.04.
Also, I would like to keep using the Cpanel::JSON::XS module because its documentation says that it "...has proper ithreads support..." and because it is frequently updated and enhanced.

In case there is something that I could help with, it would be my pleasure.
Thanks in advance for your time and attention.

rodolfojcj · Answer 1 · Fri May 21 2021 05:05:16 GMT+0800 (China Standard Time)

To verify if the problem is caused by outdated Perl modules in Ubuntu 20.04, I tried with uninstalling the relevant operating system packages and installing the recent ones available in CPAN, however I still get the "segmentation fault" error previously described.

These are the commands I executed this time:

  aptitude remove libcpanel-json-xs-perl
  aptitude remove libjson-xs-perl
  # I chose the option that removed the other packages that depended on them

  dpkg -l | grep -i perl | grep -i json
  # (returns an empty output)

  dpkg -l | grep -i perl | grep -i crypt
  # (returns an empty output)

  apt-get install cpanminus
  apt-get install make
  apt-get install gcc

  cpanm Cpanel::JSON::XS
  # (this successfully installs Cpanel::JSON::XS version 4.26)

  cpanm Crypt::JWT
  # (this successfully installs Crypt-JWT version 0.033 with its dependencies JSON version 4.03 and CryptX version 0.072)

  apache2 -X
  # (after the wget request, typed Ctrl+C to stop the web server)

rodolfojcj · Answer 2 · Fri May 21 2021 05:10:57 GMT+0800 (China Standard Time)

Continuing with my previous comment, I tried with manually removing the Cpanel::JSON::XS that I installed from CPAN, observing that again this stops the "segmentation fault" from happening.

This is the executed command, showing the files that were removed:

  cpanm --uninstall Cpanel::JSON::XS
  Cpanel::JSON::XS contains the following files:
  
    /usr/local/bin/cpanel_json_xs
    /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/Cpanel/JSON/XS.pm
    /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/Cpanel/JSON/XS/Boolean.pm
    /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/Cpanel/JSON/XS/Type.pm
    /usr/local/lib/x86_64-linux-gnu/perl/5.30.0/auto/Cpanel/JSON/XS/XS.so
    /usr/local/man/man1/cpanel_json_xs.1p
    /usr/local/man/man3/Cpanel::JSON::XS.3pm
    /usr/local/man/man3/Cpanel::JSON::XS::Boolean.3pm
    /usr/local/man/man3/Cpanel::JSON::XS::Type.3pm

rodolfojcj · Answer 3 · Fri May 21 2021 07:27:58 GMT+0800 (China Standard Time)

With Ubuntu 21.04 (Hirsute Hippo) the "segmentation fault" also happens, in a similar way to that described for Ubuntu 20.04.

The difference is that I needed to manually install the libcpanel-json-xs-perl package, because it is no longer a dependency for libcrypt-jwt-perl, which now depends on the libjson-perl package, which in turn ends installing the now preferred libjson-xs-perl package.

rodolfojcj · Answer 4 · Fri May 21 2021 07:54:22 GMT+0800 (China Standard Time)

With the older Ubuntu 18.04 (Bionic Beaver) the "segmentation fault" error does not happen.

I found these differences in this case:

The package libcrypt-jwt-perl does not exist in that distribution.

I manually installed the libcpanel-json-xs-perl and libjson-xs-perl with these commands:

apt-get install libcpanel-json-xs-perl # this installed version 3.0239-1
apt-get install libjson-xs-perl # this installed version 3.040-1

To use the cpanm command, I previously executed these commands:

apt-get install cpanminus
apt-get install make
apt-get install gcc

From CPAN, I installed Crypt::JWT with this command:

cpanm Crypt::JWT # this installed Crypt-JWT version 0.033 and its dependencies CryptX version 0.072 and JSON version 4.03

Reini Urban · Answer 5 · Fri May 21 2021 13:18:57 GMT+0800 (China Standard Time)

Thanks for the simple testcase, the rest is all irrelevant

Reini Urban · Answer 6 · Sun May 23 2021 19:32:21 GMT+0800 (China Standard Time)

Sorry, the testcase is still too baroque. I don't have a mod_perl with a threaded perl.

encode and decode with threads do work fine in this lib, but maybe mod_perl has problems with threaded DESTRUCT?
Cannot you come up with a simple testcase, similar to t/125_shared_boolean.t?

Crypt::JWT only uses JSON, not JSON::MaybeXS or this module.

rodolfojcj · Answer 7 · Mon May 24 2021 20:09:47 GMT+0800 (China Standard Time)

Hello, thanks for the feedback.

I would need to investigate about a possible problem with mod_perl and perl threads.

Were you able to reproduce the segmentation fault by following the steps I described?

Crypt:JWT uses CryptX, which in turn uses either Cpanel::JSON::XS as its preferred JSON::XS implementation, as mentioned under the "My observations" section in the initial comment for this issue.

I will review the approach in t/125_shared_boolean.t and will come with details if I find something.

Regards.

Reini Urban · Answer 8 · Tue May 25 2021 13:02:35 GMT+0800 (China Standard Time)

No, I couldn't repro it

Reini Urban · Answer 9 · Wed May 04 2022 15:40:23 GMT+0800 (China Standard Time)

I suspect it will be fixed in 4.28 with the new improvements in END and DESTROY.

Reini Urban · Answer 10 · Tue Jun 14 2022 19:04:16 GMT+0800 (China Standard Time)

Please re-open if 4.28 didn't fix it.