Crash in ThreadPool during shutdown

[hgroenenboom]

Description

When exiting my unit test application that uses libdatachannel, the ThreadPool crashes inside a mutex lock call. This only happens if there are pending tasks during shutdown. I am running in Debug mode, so this mutex seems to be a mutex around the debug structure of the deque iterator.

Currently it seems the threads are still running after/during static deinitialization, which causes a crash when the threads are accessing members members of ThreadPool.

Also the destructor of ThreadPool is never ran, because it is never deleted unless cleanup() is actively called

Stacktrace

_Mtxlock(_RTL_CRITICAL_SECTION * _Mtx) Line 25	C++
std::_Lockit::_Lockit(int kind) Line 76	C++
std::_Iterator_base12::_Adopt_locked(const std::_Container_base12 * _Parent) Line 1355	C++
std::_Iterator_base12::_Adopt(const std::_Container_base12 * _Parent) Line 1295	C++
std::_Deque_const_iterator<std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>>>::_Setcont(const std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>> * _Pdeque) Line 393	C++
std::_Deque_const_iterator<std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>>>::_Deque_const_iterator<std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>>>(unsigned __int64 _Off, const std::_Container_base12 * _Pdeque) Line 253	C++
std::_Deque_iterator<std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>>>::_Deque_iterator<std::_Deque_val<std::_Deque_simple_types<rtc::impl::ThreadPool::Task>>>(unsigned __int64)	C++
std::deque<rtc::impl::ThreadPool::Task,std::allocator<rtc::impl::ThreadPool::Task>>::end() Line 911	C++
std::priority_queue<rtc::impl::ThreadPool::Task,std::deque<rtc::impl::ThreadPool::Task,std::allocator<rtc::impl::ThreadPool::Task>>,std::greater<rtc::impl::ThreadPool::Task>>::pop() Line 424	C++
rtc::impl::ThreadPool::dequeue() Line 82	C++
rtc::impl::ThreadPool::runOne() Line 66	C++
rtc::impl::ThreadPool::run() Line 61	C++

Environment

OS: Windows 11
IDE: Visual Studio 2022
libdatachannel: 0.23.2
Build flags: OPENSSL_USE_STATIC_LIBS, NO_MEDIA

Workarounds

Explicitly join the threads in the ThreadPool::~ThreadPool to avoid std::terminate and ensure ThreadPool destructor is called (missing delete):

ThreadPool &ThreadPool::Instance() {
	static ThreadPool instance;
	return instance;
}

ThreadPool::ThreadPool() {}

ThreadPool::~ThreadPool() 
{ 
	join();
}

[paullouisageneau]

You indeed need to call rtc::Cleanup().wait() to ensure global resources are deallocated and threads are stopped.

Currently it seems the threads are still running after/during static deinitialization, which causes a crash when the threads are accessing members members of ThreadPool.

Also the destructor of ThreadPool is never ran, because it is never deleted unless cleanup() is actively called

I don't understand, if the destructor of ThreadPool is never run, then it should not crash when accessing members, right?

Explicitly join the threads in the ThreadPool::~ThreadPool to avoid std::terminate and ensure ThreadPool destructor is called (missing delete):

This is already handled by Init::doCleanup(), because global deinitialization order is unknown.

[hgroenenboom]

I don't understand, if the destructor of ThreadPool is never run, then it should not crash when accessing members, right?

It was a quite tricky bug, because container iterator debugging in visual studio apparently uses a static mutex, which is deallocated while the worker threads are still using it. So it is not a mutex from libdatachannel but a mutex from msvcs C++ code where the crash occurs.

My application is an audioplugin (VST3/AU/AAX), so it runs inside a hosts memory context. I need to avoid memory leaks when my plugin is unloaded. Does that mean I need to invoke rtc::Cleanup() when the plugin is unloaded?

[paullouisageneau]

My application is an audioplugin (VST3/AU/AAX), so it runs inside a hosts memory context. I need to avoid memory leaks when my plugin is unloaded. Does that mean I need to invoke rtc::Cleanup() when the plugin is unloaded?

You need to call rtc::Cleanup().wait() after destroying all objects. It ensures threads are stopped and global resources are deallocated.