dtls_udp_echo server : DTLSv1_listen() still return 1 while client has been disconnected

Question

dtls_udp_echo server : DTLSv1_listen() still return 1 while client has been disconnected

fantansy opened this issue 5 years ago · comments

do anybody try the daemon?
I built and run dtls_udp_echo , on client side, socket can be set up and recv echo response successfully.
but on server side, DTLSv1_listen() still return 1 then new alloc thread to handle it.
my openssl version:
openssl-1.1.0j

Felix Weinrank · Answer 1 · Wed May 08 2019 15:19:01 GMT+0800 (China Standard Time)

Can you provide additional information?

OS
arguments
output

Thanks!

刘群 · Answer 2 · Wed Jun 05 2019 15:07:04 GMT+0800 (China Standard Time)

I got similar problems on

OS: Ubuntu 18.04 with openssl 1.1.0g via sudo apt install libssl-dev
arguments:

# Start echo server
./cert.sh
cd src/
./dtls_udp_echo -L 127.0.0.1 -v


# Start standard openssl dtls client to talk with the echo server
liuqun@liuqun-virtual-machine:~/DTLS-Examples/src $ 
openssl s_client -dtls1_2 -CAfile ../ca-cert.pem   -key certs/client-key.pem -cert certs/client-cert.pem -connect 127.0.0.1:23232

output:

liuqun@liuqun-virtual-machine:~/DTLS-Examples/src$ ./dtls_udp_echo -L 127.0.0.1 -v
socket: Too many open files
socket: Too many open files
socket: Too many open files
socket: Too many open files
socket: Too many open files
socket: Too many open files
socket: Too many open files
Thread 7fa9ad512700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
Thread 7fa9b4d21700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
socket: Too many open files
Thread 7fa9b1d1b700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)

刘群 · Answer 3 · Wed Jun 05 2019 18:36:36 GMT+0800 (China Standard Time)

On Ubuntu 19.04, the behavior of DTLSv1_listen() changes if 2 clients are connecting concurrently.

OS: Ubuntu 19.04 with openssl 1.1.1b (via sudo apt install libssl-dev)
dtls_udp_echo server arguments:

liuqun@vmware:~/DTLS-Examples/src$ ./dtls_udp_echo  -L 192.168.1.106 -V

start 2 openssl s_client

One from localhost, and the other one from 192.168.1.6, each host has the same pem files:

# the client from localhost
openssl s_client -dtls1_2 \
    -CAfile ../ca-cert.pem \
    -key certs/client-key.pem \
    -cert certs/client-cert.pem \
    -connect 192.168.1.106:23232

# the other client from another linux vmware (ubuntu 18.04 with default openssl 1.1.0g)
openssl s_client -dtls1_2 -CAfile ca-cert.pem   -key client-key.pem -cert client-cert.pem -connect 192.168.1.106:23232

output of tshark (tcpdump) on localhost:

liuqun@vmware:~/DTLS-Examples/src$ sudo tshark -i any "udp port 23232"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'any'
    1 0.000000000 192.168.1.106 192.168.1.106 DTLS 255 Client Hello
    2 0.000062566 192.168.1.106 192.168.1.106 DTLSv1.0 92 Hello Verify Request
    3 0.000145886 192.168.1.106 192.168.1.106 DTLSv1.0 275 Client Hello
    4 0.001316816 192.168.1.106 192.168.1.106 DTLSv1.2 1444 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
    5 0.002842575 192.168.1.106 192.168.1.106 DTLSv1.2 2317 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
    6 0.003229041 192.168.1.106 192.168.1.106 DTLSv1.2 1174 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
    7 4.264309086  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
    8 5.273251905  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
    9 7.289700319  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
   10 11.545936487  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
   11 19.738098951  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
   12 35.865849733  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
   13 40.982953252 192.168.1.106 192.168.1.106 DTLSv1.2 83 Encrypted Alert
   14 40.983072732 192.168.1.106 192.168.1.106 DTLSv1.2 83 Encrypted Alert
   15 69.912984522  192.168.1.6 192.168.1.106 DTLS 233 Client Hello
   16 69.913074957 192.168.1.106 192.168.1.6  DTLSv1.0 92 Hello Verify Request
   17 69.913297603  192.168.1.6 192.168.1.106 DTLSv1.0 253 Client Hello
   18 69.914061325 192.168.1.106 192.168.1.6  DTLSv1.2 1444 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
   19 69.914745176  192.168.1.6 192.168.1.106 DTLSv1.2 592 Certificate (Fragment)
   20 69.914761280  192.168.1.6 192.168.1.106 DTLSv1.2 592 Certificate (Fragment)
   21 69.914762760  192.168.1.6 192.168.1.106 DTLSv1.2 592 Certificate (Fragment)
   22 69.916146562  192.168.1.6 192.168.1.106 DTLSv1.2 592 Certificate (Reassembled), Client Key Exchange, Certificate Verify (Fragment)
   23 69.916161796  192.168.1.6 192.168.1.106 DTLSv1.2 225 Certificate Verify (Reassembled), Change Cipher Spec, Encrypted Handshake Message
   24 69.916659727 192.168.1.106 192.168.1.6  DTLSv1.2 1174 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
   25 95.510619568 192.168.1.106 192.168.1.6  DTLSv1.2 83 Encrypted Alert
   26 95.511064532  192.168.1.6 192.168.1.106 DTLSv1.2 83 Encrypted Alert

From the tshark/tcpdump record, we can see the handshake request packets of the second client were blocked by dtls_udp_echo.

The output from our echo server:

liuqun@vmware:~/DTLS-Examples/src$ ./dtls_udp_echo  -L 192.168.1.106 -V

Thread 7f9ceccc6700: accepted connection from 192.168.1.106:54965
------------------------------------------------------------
 countryName               = DE
 stateOrProvinceName       = NW
 localityName              = Steinfurt
 organizationName          = FH-Muenster
 organizationalUnitName    = NPLAB
 commonName                = dtls-client.nplab.de

 Cipher: ECDHE-RSA-AES256-GCM-SHA384
------------------------------------------------------------

Thread 7f9ceccc6700: done, connection closed.

Thread 7f9ceccc6700: accepted connection from 192.168.1.6:40594
------------------------------------------------------------
 countryName               = DE
 stateOrProvinceName       = NW
 localityName              = Steinfurt
 organizationName          = FH-Muenster
 organizationalUnitName    = NPLAB
 commonName                = dtls-client.nplab.de

 Cipher: ECDHE-RSA-AES256-GCM-SHA384
------------------------------------------------------------

Thread 7f9ceccc6700: done, connection closed.

I've repeated the same test for several time. It seems that the client would get blocked randomly, and sometime the handshake process would repeat again and again.

刘群 · Answer 4 · Wed Jun 05 2019 20:18:16 GMT+0800 (China Standard Time)

openssl s_client -dtls's default behavior:

retry sending "Client Hello" request again after 1, 2, 4, 8, 16, 32 seconds if the DTLS server did not respond anything.

Felix Weinrank · Answer 5 · Thu Jun 06 2019 16:07:02 GMT+0800 (China Standard Time)

Hey @liuqun,

I've tested the issue on three operating systems: OSX, FreeBSD 12 and Ubuntu 19.04
I just ran your dtls_udp_echo server with openssl client example several times and observed the output.

Operating Systems

Ubuntu 19.04 with OpenSSL_1_1_1-stable from GIT (commit d63d841fb510a920275c66d3e486089c5c718797)
OSX 10.14.5 with OpenSSL 1.1.1c 28 May 2019 from Homebrew
FreeBSD 12 with OpenSSL 1.1.1b-freebsd 26 Feb 2019 from default installation

Results

OSX seems to work fine.
Ubuntu sometimes works, sometimes it blocks in a way you already observed
FreeBSD never works: the server always exits with pthread_create: Too many processes

Felix Weinrank · Answer 6 · Thu Jun 06 2019 16:09:46 GMT+0800 (China Standard Time)

I've forgot to mention: I've seen the issue with a single client.
It wasn't necessary to connect with a second client.

Felix Weinrank · Answer 7 · Fri Jun 07 2019 05:58:34 GMT+0800 (China Standard Time)

We've fixed a bug and added a version check.

The pthread_create: Too many processes issue was caused by a mismatch between my OpenSSL version at compile time and the version that was dynamically linked.
I'm quite sure you've trapped into the same problem.
I've added a check which compares the OpenSSL version used at compile time and the version which has been linked.

The "blocking" issue was indeed a bug.
We've set an socket option (SO_REUSEPORT) which behaves differently on FreeBSD and Linux.
I've added an additional check to avoid this unwanted behaviour.

Can you try the fix and report back?
Felix

刘群 · Answer 8 · Fri Jun 14 2019 16:16:36 GMT+0800 (China Standard Time)

Edited...
(Note: I found the version of OpenSSL I used currently is not the latest security update version.
I should sudo apt-get update && sudo apt-get upgrade and then test against the latest official deb package. )

OS: Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-50-generic x86_64) with pre-built openssl 1.1.0g-2ubuntu4.3 deb package,
OpenSSL lib version: libssl1.1_1.1.0g-2ubuntu4.3_amd64.deb
OpenSSL header version: libssl-dev_1.1.0g-2ubuntu4.3_amd64.deb

sudo apt install libssl-dev=1.1.0g-2ubuntu4.3   libssl1.1=1.1.0g-2ubuntu4.3

OpenSSL source tarball is available from Ubuntu https://packages.ubuntu.com/source/bionic/openssl

wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.0g-2ubuntu4.3.dsc
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.0g.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.0g.orig.tar.gz.asc
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.0g-2ubuntu4.3.debian.tar.xz

dpkg-source -x openssl_1.1.0g-2ubuntu4.3.dsc

DTLS-Examples version
commit cdf60fa (HEAD -> master, origin/master, origin/HEAD)

Date:   Thu Jun 6 23:41:38 2019 +0200

    Check OpenSSL version (#14)

server arguments: ./dtls_udp_echo -L 127.0.0.1 -v
client arguments:
~/DTLS-Examples/src$ openssl s_client -dtls1_2 -CAfile ../ca-cert.pem -key certs/client-key.pem -cert certs/client-cert.pem -connect 127.0.0.1:23232
Bug Output:

liuqun@liuqun-virtual-machine:~/bbb/DTLS-Examples/src$ ./dtls_udp_echo -L 127.0.0.1 -v
Using OpenSSL 1.1.0g  2 Nov 2017
socket: Too many open files
Thread 7f928c440700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
socket: Too many open files
Thread 7f928bc3f700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)

Felix Weinrank · Answer 9 · Fri Jun 14 2019 17:53:12 GMT+0800 (China Standard Time)

You're right, I were able to reproduce the issue and we'll look into it... thanks!

刘群 · Answer 10 · Fri Jun 14 2019 19:36:42 GMT+0800 (China Standard Time)

OS: Ubuntu 18.04 after upgrade
OpenSSL lib and header deb package version: apt show libssl-dev

Package: libssl-dev
Version: 1.1.1-1ubuntu2.1~18.04.1

Output

./dtls_udp_echo -L 127.0.0.1 -v
Using OpenSSL 1.1.1  11 Sep 2018
socket: Too many open files
Thread 7fee25c3b700: done, connection closed.
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)

Felix Weinrank · Answer 11 · Mon Jun 17 2019 22:16:55 GMT+0800 (China Standard Time)

TL;DR: Use OpenSSL Version 1.1.1a or newer!

The issue has been reported earlier DTLSv1_listen() regression on 1.1.x to the OpenSSL project.

Since OpenSSL 1.1.0 only receives security fixes, the issue has only been fixed in 1.1.1a.

We've reproduced the issue with OpenSSL 1.1.0k and 1.1.1 on Linux and FreeBSD.
1.1.1a and higher are working as expected.