Data race in tls_process_new_session_ticket?
rschu1ze opened this issue · comments
Hi all,
we (ClickHouse, an open-source analytical database) recently migrated from boringssl to OpenSSL 3.2 (PR). Back then, session caching had to be disabled (commit as part of aforementioned PR) as our continuous tests using ThreadSanitizer builds found data races in OpenSSL and the effort had to be completed in time back then.
I re-visited session caching today (i.e. re-enabled it), and looked more closely into the the races:
I am not an OpenSSL expert but I think this is what happens:
This thread adds a SSL session to the session cache. This operation is performed by SSL_CTX_add_session under an exclusive lock:
E Previous write of size 8 at 0x7260000d0b30 by thread T672 (mutexes: write M1, write M2):
E #0 SSL_SESSION_list_add build_docker/./contrib/openssl/ssl/ssl_sess.c:1263:27 (clickhouse+0x20250cea) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #1 SSL_CTX_add_session build_docker/./contrib/openssl/ssl/ssl_sess.c:762:5 (clickhouse+0x20250cea)
E #2 ssl_update_cache build_docker/./contrib/openssl/ssl/ssl_lib.c:4527:13 (clickhouse+0x2024632c) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #3 tls_process_new_session_ticket build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2850:9 (clickhouse+0x202c3ba6) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
[...]
This thread wants to replace the existing cached session:
E WARNING: ThreadSanitizer: data race (pid=1)
E Read of size 8 at 0x7260000d0b30 by thread T674 (mutexes: write M0):
E #0 __tsan_memcpy <null> (clickhouse+0x7237feb) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #1 ssl_session_dup build_docker/./contrib/openssl/ssl/ssl_sess.c:146:5 (clickhouse+0x2024f2f9) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #2 tls_process_new_session_ticket build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2731:25 (clickhouse+0x202c354f) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #3 ossl_statem_client_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1126:16 (clickhouse+0x202bf906) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
[...]
tls_process_new_session_ticket has a comment Sessions must be immutable once they go into the session cache. Otherwise we can get multi-thread problems. Therefore we don't "update" sessions, we replace them with a duplicate. In TLSv1.3 we need to do this every time a NewSessionTicket arrives because those messages arrive post-handshake and the session may have already gone into the session cache..
"Duplicating" the session happens via memcpy in ssl_session_dup without lock and I did not find any other mechanism further up the stack that would protect the cached session against parallel modification or release. I guess thread sanitizer complains rightfully here unless I missed something. The copy needs to happen under a read lock to be kosher.
Please tell me I missed something 😄.
I'm not sure if this helps you, but since you appear to be using the aws python sdk, we recently closed this issue:
#24480
Which is in a completely different location, but also related to the aws python sdk, in which we found that parts of the sdk were creating SSL_CTX objects that it was sharing among SSL objects (which is fine) but mutating the shared objects while they were in use in other threads (which is not).
Not sure if it relates here, but it might be worth looking at.
Thanks for the detailed report (link to log helped). I think this is kind of false positive. I must admit I don't all details how thread sanitizer works internally. So my analysis here might be quite wrong.
The mutex M2 is rw-lock whcih belongs to SSL context (according to this T-SAN output here):
E Mutex M2 (0x721000275940) created at:
E #0 pthread_rwlock_init <null> (clickhouse+0x723e9bd) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #1 CRYPTO_THREAD_lock_new build_docker/./contrib/openssl/crypto/threads_pthread.c:54:9 (clickhouse+0x2044f271) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #2 SSL_CTX_new_ex build_docker/./contrib/openssl/ssl/ssl_lib.c:3883:17 (clickhouse+0x20244a35) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #3 SSL_CTX_new build_docker/./contrib/openssl/ssl/ssl_lib.c:4120:12 (clickhouse+0x2024533d) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #4 Poco::Net::Context::createSSLContext() build_docker/./base/poco/NetSSL_OpenSSL/src/Context.cpp (clickhouse+0x1d240abc) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
</snip>
Mutexes M0 and M1 don't belong to OpenSSL they seem to be created by DB application.
E Mutex M1 (0x722c0000a800) created at:
E #0 pthread_mutex_init <null> (clickhouse+0x723d7bf) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x2082e9a4) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x2082e9a4)
E #3 Poco::Net::SecureSocketImpl::SecureSocketImpl(Poco::AutoPtr<Poco::Net::SocketImpl>, Poco::AutoPtr<Poco::Net::Context>) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:64:19 (clickhouse+0x1d2591a3) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #4 Poco::Net::SecureStreamSocketImpl::SecureStreamSocketImpl(Poco::AutoPtr<Poco::Net::Context>) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:26:2 (clickhouse+0x1d25f8db) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #5 Poco::Net::SecureStreamSocket::SecureStreamSocket() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocket.cpp:30:19 (clickhouse+0x1d25c987) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #6 std::__1::__unique_if<Poco::Net::SecureStreamSocket>::__unique_single std::__1::make_unique[abi:v15000]<Poco::Net::SecureStreamSocket>() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0x195c81ee) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #7 DB::Connection::connect(DB::ConnectionTimeouts const&) build_docker/./src/Client/Connection.cpp:128:26 (clickhouse+0x195c81ee)
and stack here belongs to M0:
E Mutex M0 (0x722c0016c010) created at:
E #0 pthread_mutex_init <null> (clickhouse+0x723d7bf) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #1 std::__1::__libcpp_recursive_mutex_init[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:269:10 (clickhouse+0x2082e9a4) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #2 std::__1::recursive_mutex::recursive_mutex() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:61:14 (clickhouse+0x2082e9a4)
E #3 Poco::Net::SecureSocketImpl::SecureSocketImpl(Poco::AutoPtr<Poco::Net::SocketImpl>, Poco::AutoPtr<Poco::Net::Context>) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:64:19 (clickhouse+0x1d2591a3) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #4 Poco::Net::SecureStreamSocketImpl::SecureStreamSocketImpl(Poco::AutoPtr<Poco::Net::Context>) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:26:2 (clickhouse+0x1d25f8db) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #5 Poco::Net::SecureStreamSocket::SecureStreamSocket() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocket.cpp:30:19 (clickhouse+0x1d25c987) (BuildId: a38098db2d267457b06b8d4a8b29e4d4d8b98a0a)
E #6 std::__1::__unique_if<Poco::N
</snip>
So what really matters to OpenSSL is M2 (our mutex). The mutex is held as we insert session to cache created by thread T672. The matching code which does that can be found here:
712 int SSL_CTX_add_session(SSL_CTX *ctx, SSL_SESSION *c)
713 {
714 int ret = 0;
715 SSL_SESSION *s;
716
717 /*
718 * add just 1 reference count for the SSL_CTX's session cache even though
719 * it has two ways of access: each session is in a doubly linked list and
720 * an lhash
721 */
722 SSL_SESSION_up_ref(c);
723 /*
724 * if session c is in already in cache, we take back the increment later
725 */
726
727 if (!CRYPTO_THREAD_write_lock(ctx->lock)) {
728 SSL_SESSION_free(c);
729 return 0;
730 }
731 s = lh_SSL_SESSION_insert(ctx->sessions, c);
732
733 /*
734 * s != NULL iff we already had a session with the given PID. In this
735 * case, s == c should hold (then we did not really modify
736 * ctx->sessions), or we're in trouble.
737 */
So this where we grab the lock for SSL context (a.k.a. M2) and insert a chunk of memory (session) into global list (a.k.a. session cache). What's worth to note is that function lh_SSL_SESSION_insert() modifies link members (prev, next pointers) of session which is inserted to cache. The modification happens under M2 protection and thread sanitizer makes a note of it (I think).
Later there is a thread T674 which calls a dup on session created by T672. I'm sure the look up operation did happen after the protection M2. However as soon as T674 finds the session it grabs the reference and drops M2. Because the session is immutable. The memcpy() operation copies the whole session including prev,next pointers which were modified under protection of M2 when T672 was calling lh_SSL_SESSION_insert(). And that's the why the T-san assert is triggered.
So thread sanitizer is right we are reading memory without holding M2 but this is harmful here because thread sanitizer does not understand the object is read-only here. May be we can copy the object member by member avoiding to copy {prev, next} members instead of doing memcpy(), to make thread sanitizer happy. don't know if it would help. At least that's my understanding of the report.
I don't think this is a false positive. Thread sanitizer is correct that this code is a data race. It's a data race where you ignore the result of the race, but C says that data races are undefined behavior whether or not you ignore the result.
The issue isn't that TSan doesn't understand the object is read-only. It's not read-only. It is logically read-only, but the next and prev pointers are legitimately not read-only. TSan just doesn't understand that you then ignore the bad reads:
Lines 163 to 166 in 1977c00
But TSan is correct to not understand that because C's rules don't care.
We might try to provide thread sanitizer with a hint to ignore ssl_session_dup build(). this might help us with clang's sanitizer.
While that will silence the report, it won't fix the bug in OpenSSL. The data race is still undefined behavior in the language.
Could we move the next/prev members to the end of the structure and memcpy everything but those pointers?
Best to avoid the refcount too, since that's also synchronized.
FWIW, I'm not sure of the feasibility, but RCU might be a solution here. the lock/unlock functions for rcu are already annotated to convince tsan that there is no race condition here. The addition of a read side lock should be effectively no performance impact (ideally), but it sounds like a reasonably easy replacement.
I think RCU is way, way overkill here. There is no complex synchronization or anything going on here. Just an overbroad convenience memcpy that never needed to copy the offending fields in the first place.
If the memcpy can be constrained to only touch fields that aren't protected on the write side, then yes, I would agree, but I'm not currently looking at the code, so I'm not sure of the feasibility there.