[Bug] Crash while thread running after FileWatcherInotfiy destructed.

Question

[Bug] Crash while thread running after FileWatcherInotfiy destructed.

ashione opened this issue 2 years ago · comments

My main instruction likes

int main () {
FileWatcherInotify A = new FileWatcherInotify(...);
A.watch('...');
delete A;
return 0;
}

There are two threads cross running in process,

main thread
watcher thread

We found watcher thread crashed and this backtrace remained:

#4  0x00007f78d88248f3 in uw_update_context () from /usr/lib64/libgcc_s.so.1
#5  0x00007f78d88254ae in _Unwind_Backtrace () from /usr/lib64/libgcc_s.so.1
#6  0x00007f78d8759946 in backtrace () from /usr/lib64/libc.so.6
#7  0x00007f78da46f8cf in google::GetStackTrace(void**, int, int) () from /home/admin/.pyenv/versions/3.6.13/lib/python3.6/site-packages/ray/cpp/lib/libray_api.so
#8  0x00007f78da4707f9 in google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) ()
   from /home/admin/.pyenv/versions/3.6.13/lib/python3.6/site-packages/ray/cpp/lib/libray_api.so
#9  <signal handler called>
#10 0x00007f7876d34f2d in efsw::Private::ThreadMemberFunc<efsw::FileWatcherInotify>::run (this=<error reading variable: Cannot access memory at address 0x27f7773ffe798>)
    at bazel-out/k8-dbg/bin/external/com_github_efsw/_virtual_includes/efsw/efsw/Thread.hpp:78

main thread stack :

#0  0x00007f78d862cac7 in __pthread_timedjoin_ex () from /usr/lib64/libpthread.so.0
#1  0x00007f7876d43541 in efsw::Platform::ThreadImpl::wait (this=0x7f777cfa36a0) at external/com_github_efsw/src/efsw/platform/posix/ThreadImpl.cpp:25
#2  0x00007f7876d3de3b in efsw::Thread::wait (this=0x7f777c14c2a0) at external/com_github_efsw/src/efsw/Thread.cpp:22
#3  0x00007f7876d3dd48 in efsw::Thread::~Thread (this=0x7f777c14c2a0, __in_chrg=<optimized out>) at external/com_github_efsw/src/efsw/Thread.cpp:9
#4  0x00007f7876d3dd98 in efsw::Thread::~Thread (this=0x7f777c14c2a0, __in_chrg=<optimized out>) at external/com_github_efsw/src/efsw/Thread.cpp:12
#5  0x00007f7876d2cf0b in efsw::FileWatcherInotify::~FileWatcherInotify (this=0x7f7768002fe0, __in_chrg=<optimized out>)
    at external/com_github_efsw/src/efsw/FileWatcherInotify.cpp:50
#6  0x00007f7876d2d0d6 in efsw::FileWatcherInotify::~FileWatcherInotify (this=0x7f7768002fe0, __in_chrg=<optimized out>)
    at external/com_github_efsw/src/efsw/FileWatcherInotify.cpp:68
#7  0x00007f7876d2b351 in efsw::FileWatcher::~FileWatcher (this=0x7f777c847160, __in_chrg=<optimized out>) at external/com_github_efsw/src/efsw/FileWatcher.cpp:67
#8  0x00007f7876d2b378 in efsw::FileWatcher::~FileWatcher (this=0x7f777c847160, __in_chrg=<optimized out>) at external/com_github_efsw/src/efsw/FileWatcher.cpp:68

According to above stackes, it's watcher deletion happened before watcher entry_point running, which means m_objest had been deleted and memory undetected in that time

// Specialization using a member function
template <typename C> struct ThreadMemberFunc : ThreadFunc {
	ThreadMemberFunc( void ( C::*function )(), C* object ) :
		m_function( function ), m_object( object ) {}
	virtual void run() { ( m_object->*m_function )(); }
	void ( C::*m_function )();
	C* m_object;
};

Martín Lucas Golini · Answer 1 · Thu Mar 09 2023 21:28:15 GMT+0800 (China Standard Time)

Hi!
Why are you using FileWatcherInotify directly? That class is private. But as far as I can remember it should work if called directly.
I'm unable to reproduce the bug with a minimal example:

#include <efsw/efsw.hpp>

int main( int argc, char** argv ) {
	efsw::FileWatcher* fileWatcher = new efsw::FileWatcher();
	fileWatcher->watch();
	delete fileWatcher;
	return 0;
}

Neither with a watched dir:

#include <efsw/efsw.hpp>
#include <iostream>

class UpdateListener : public efsw::FileWatchListener {
  public:
	UpdateListener() {}

	void handleFileAction( efsw::WatchID, const std::string&, const std::string&, efsw::Action,
						   std::string ) {}
};

int main( int argc, char** argv ) {
	efsw::FileWatcher* fileWatcher = new efsw::FileWatcher();
	UpdateListener* ul = new UpdateListener();
	fileWatcher->addWatch( ".", ul );
	fileWatcher->watch();
	delete fileWatcher;
	delete ul;
	return 0;
}

I'm missing something?

Lingxuan Zuo · Answer 2 · Fri Mar 10 2023 01:47:33 GMT+0800 (China Standard Time)

Hi!

Why are you using FileWatcherInotify directly? That class is private. But as far as I can remember it should work if called directly.

I'm unable to reproduce the bug with a minimal example:

#include <efsw/efsw.hpp>



int main( int argc, char** argv ) {

	efsw::FileWatcher* fileWatcher = new efsw::FileWatcher();

	fileWatcher->watch();

	delete fileWatcher;

	return 0;

}

Neither with a watched dir:

#include <efsw/efsw.hpp>

#include <iostream>



class UpdateListener : public efsw::FileWatchListener {

  public:

	UpdateListener() {}



	void handleFileAction( efsw::WatchID, const std::string&, const std::string&, efsw::Action,

						   std::string ) {}

};



int main( int argc, char** argv ) {

	efsw::FileWatcher* fileWatcher = new efsw::FileWatcher();

	UpdateListener* ul = new UpdateListener();

	fileWatcher->addWatch( ".", ul );

	fileWatcher->watch();

	delete fileWatcher;

	delete ul;

	return 0;

}

I'm missing something?

I just want to make someone getting main code path and why it crashed，which does not means i use filewatcherinotfiy directly.
You might catch this from stack filewatcher was used.

Lingxuan Zuo · Answer 3 · Fri Mar 10 2023 01:56:19 GMT+0800 (China Standard Time)

In my usage scenario, it will appear once about an hour running. Especially in the case of high load machine, pthread_create returns does not mean that the thread has been created and run entry point function successfully. Actually you can deduce it from the way the code runs, which is indeed possible.
@SpartanJ

Martín Lucas Golini · Answer 4 · Fri Mar 10 2023 03:24:30 GMT+0800 (China Standard Time)

Oh, I understand. I thought that your code snipped was a literal example.
If I understand well mIsActive should be only true when entryPoint is called / running and it should be atomic. Do you agree?

Martín Lucas Golini · Answer 5 · Fri Mar 10 2023 04:01:46 GMT+0800 (China Standard Time)

Let me know if the last commit helps with your issue.

Lingxuan Zuo · Answer 6 · Fri Mar 10 2023 11:56:26 GMT+0800 (China Standard Time)

Let me know if the last commit helps with your issue.

I pack this commit to my program but still crashed. So perhas it not works yet.

Martín Lucas Golini · Answer 7 · Fri Mar 10 2023 12:57:23 GMT+0800 (China Standard Time)

Sorry, the fix I sent was for another kind of error and also was worst than before. I misinterpreted the crash completely. I think that now I have a clear picture of the issue but I have no idea how I can avoid it in 100% of the cases.
Reading the Linux documentation says basically that the thread can start at any time, I'm assuming that in your case is taking enough time to be called so late that the owner (the Thread instance and the ThreadImpl instance) does not exist anymore. I assumed that the call to pthread_join should prevent that (the Thread destructor calls to pthread_join before destroying ThreadImpl and its own instance), but it seems that pthread_join does nothing if the thread did not start. I sent another change that will check if the join worked, and if that didn't work will call to a pthread_cancel. If that doesn't work, I actually have no idea how this could be fixed.
In your use case, you're spawning many efsw::FileWatcher and destroying them very quickly or something alike? Otherwise, I don't see how this could happen at all.

Martín Lucas Golini · Answer 8 · Fri Mar 10 2023 13:02:20 GMT+0800 (China Standard Time)

Also: I tried to replicate it by doing something stupid like:

int main( int argc, char** argv ) {
	int c = 0;
	UpdateListener* ul = new UpdateListener();
	while ( c++ < 10000 ) {
		efsw::FileWatcher* fileWatcher = new efsw::FileWatcher();
		fileWatcher->addWatch( ".", ul );
		fileWatcher->watch();
		delete fileWatcher;
	}

	delete ul;
	return 0;
}

And pthread_join is doing its job, it always waits for the entryPoint to run at completion.
That confuses me more.

Lingxuan Zuo · Answer 9 · Sat Mar 11 2023 22:39:53 GMT+0800 (China Standard Time)

I'm going to figure out the root cause of this issue and upload lastest clues ASAP.

Daniel Oberhoff · Answer 10 · Wed Nov 22 2023 19:50:53 GMT+0800 (China Standard Time)

Maybe using c++11 threads helps?

Daniel Oberhoff · Answer 11 · Thu Nov 23 2023 05:25:48 GMT+0800 (China Standard Time)

maybe try my PR that uses c++11 primitives for threading and locking:
#170

Martín Lucas Golini · Answer 12 · Thu Nov 23 2023 12:02:09 GMT+0800 (China Standard Time)

Thanks for the collaboration. It seems that still needs some work to compile properly. You had this same issue? Because I wasn't able to reproduce it yet.

Daniel Oberhoff · Answer 13 · Thu Nov 23 2023 15:08:57 GMT+0800 (China Standard Time)

what needs work? I have no problem comliling?

Daniel Oberhoff · Answer 14 · Thu Nov 23 2023 15:09:37 GMT+0800 (China Standard Time)

no i did not, but the report somewhat lowered my trust in the hand rolled thread management, thus I pulled in the c++11 stuff, which I think is the best approach

Lingxuan Zuo · Answer 15 · Sat Dec 30 2023 23:42:19 GMT+0800 (China Standard Time)

no i did not, but the report somewhat lowered my trust in the hand rolled thread management, thus I pulled in the c++11 stuff, which I think is the best approach

Perphas it's better way to use standard thread library. Actually it's weird and no more detail information or core dump can be digged out.
I notice that filewatherinotify and some objects might not be threadsafe in high frequency creation or destruction.

Martín Lucas Golini · Answer 16 · Tue Jan 02 2024 21:47:17 GMT+0800 (China Standard Time)

I agree, efsw was using custom thread implementation because it was created actually before C++11 and I kept compatibility for older compilers until just recently. But know it's possible to migrate everything to C++11 standard headers. The PR looks OK but I'll not merge it until I have enough time to test it on all platforms.