<div dir="auto"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>Od: <strong class="gmail_sendername" dir="auto">Damijan Skvarc</strong> <span dir="auto"><<a href="mailto:damjan.skvarc@gmail.com">damjan.skvarc@gmail.com</a>></span><br>Date: pon., 5. jul. 2021 15:09<br>Subject: deadlock in omniORB<br>To: <<a href="mailto:omniorb-list@omniorb-support.com">omniorb-list@omniorb-support.com</a>>, damjan.skvarc <<a href="mailto:Damjan.Skvarc@gmail.com">Damjan.Skvarc@gmail.com</a>><br></div><br><br><div dir="ltr"><div>Hi.</div><div><br></div><div>With the omniORB4.2.2 version (delivered by Ubunto 18.04) we have started facing a deadlock issue. Once I was able to attach with gdb into the process where such a deadlock happened and found the following situation. There was 21 threads, where one of the thread was found in:<br></div><div><pre>#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f691f213025 in __GI___pthread_mutex_lock (mutex=0x55c31d6f4e00) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007f691fc1d6c0 in omni::giopStream::errorOnReceive(int, char const*, unsigned int, omni::giopStream_Buffer*, bool, char const*) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#3 0x00007f691fc1da4b in omni::giopStream::sleepOnRdLockAlways() () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#4 0x00007f691fc369ac in omni::giopImpl12::inputReplyBegin(omni::giopStream*, void (*)(omni::giopStream*)) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#5 0x00007f691fc37c17 in omni::giopImpl12::inputMessageBegin(omni::giopStream*, void (*)(omni::giopStream*)) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#6 0x00007f691fc252fc in omni::GIOP_C::ReceiveReply() () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#7 0x00007f691fc047e8 in omniRemoteIdentity::dispatch(omniCallDescriptor&) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2<br><br></pre><pre><span style="font-family:arial,sans-serif">while the other 20 threads were stuck in </span><br><br>#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f691f214fb3 in __pthread_mutex_cond_lock (mutex=mutex@entry=0x55c31d6f4e00) at ../nptl/pthread_mutex_lock.c:80
#2 0x00007f691f216e7d in __pthread_cond_wait_common (abstime=0x7f68c5613eb0, mutex=0x55c31d6f4e00, cond=0x55c31d82b9d8) at pthread_cond_wait.c:645
#3 __pthread_cond_timedwait (cond=0x55c31d82b9d8, mutex=0x55c31d6f4e00, abstime=0x7f68c5613eb0) at pthread_cond_wait.c:667
#4 0x00007f691f90ca84 in omni_condition::timedwait(unsigned long, unsigned long) () from /usr/lib/x86_64-linux-gnu/libomnithread.so.4
#5 0x00007f691fc1d9f1 in omni::giopStream::sleepOnRdLockAlways() () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#6 0x00007f691fc369ac in omni::giopImpl12::inputReplyBegin(omni::giopStream*, void (*)(omni::giopStream*)) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#7 0x00007f691fc37c17 in omni::giopImpl12::inputMessageBegin(omni::giopStream*, void (*)(omni::giopStream*)) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#8 0x00007f691fc252fc in omni::GIOP_C::ReceiveReply() () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#9 0x00007f691fc047e8 in omniRemoteIdentity::dispatch(omniCallDescriptor&) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2
#10 0x00007f691fbe7b3c in omniObjRef::_invoke(omniCallDescriptor&, bool) () from /usr/lib/x86_64-linux-gnu/libomniORB4.so.2utex, but the call stack<br></pre><div>By inspecting pthread mutex at address0x55c31d6f4e00 I found the first thread already owned this mutex, but the call stack depicts the thread is trying to lock it again. It looks the thread for some reason "forget" to unlock it in one of the previous steps.<br><br>Since I couldn't get any additional data I have started to instrument the code with helgrind. Unfortunately, due to different timing conditions, I was not able to come into the same obstacle again. I have instrumented all versions from 4.2.0 up to 4.2.4 and in all these versions I've found one another raise condition which happened in the following case:</div><div><br></div><div>Possible<span style="font-family:monospace"> data race during read of size 1 at 0x67C9275 by thread #1<br>==9765== Locks held: none<br>==9765== at 0xB5ACBA: omni::SocketHolder::setNonBlocking() (SocketCollection.h:108)<br>==9765== by 0xB5A39D: omni::tcpConnection::Send(void*, unsigned long, omni_time_t const&) (tcpConnection.cc:71)<br>==9765== by 0xB16D19: omni::giopStream::sendChunk(omni::giopStream_Buffer*) (giopStream.cc:1129)<br>==9765== by 0xB3391A: omni::giopImpl12::outputMessageEnd(omni::giopStream*) (giopImpl12.cc:1320)<br>==9765== by 0xB1E881: omni::GIOP_C::InitialiseRequest() (GIOP_C.cc:109)<br>==9765== by 0xAF9363: omniRemoteIdentity::dispatch(omniCallDescriptor&) (remoteIdentity.cc:97)<br>==9765== by 0xADAF14: omniObjRef::_invoke(omniCallDescriptor&, bool) (omniObjRef.cc:675)<br>==9765== by 0x873DCD: i_NET::_objref_RemoteNode::GetFlags(i_NET::NodeNames const&) (i_remote_node.cc:1418)<br>==9765== by 0x82B49C: mci::RemoteNode::OnGetFlags(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_<br>traits<char>, std::allocator<char> > > > const&) const (remote_node.cpp:333)<br>==9765== by 0x822E00: mci::TreeNode::GetFlags(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_trai<br>ts<char>, std::allocator<char> > > > const&) const (tree_node.cpp:94)<br>==9765== by 0x7A6190: mci::Node::GetFlags() const (node.cpp:312)<br>==9765== by 0x68C0B2: MciProxy::Connect(mci::Root, unsigned int) (mciProxy.cpp:56)<br>==9765== <br>==9765== This conflicts with a previous write of size 1 by thread #8<br>==9765== Locks held: none<br>==9765== at 0xB5ACA2: omni::SocketHolder::setBlocking() (SocketCollection.h:101)<br>==9765== by 0xB5A55D: omni::tcpConnection::Recv(void*, unsigned long, omni_time_t const&) (tcpConnection.cc:155)<br>==9765== by 0xB15CA6: omni::giopStream::inputMessage() (giopStream.cc:844)<br>==9765== by 0xB31149: omni::giopImpl12::inputNewServerMessage(omni::giopStream*) (giopImpl12.cc:438)<br>==9765== by 0xB319CB: omni::giopImpl12::inputMessageBegin(omni::giopStream*, void (*)(omni::giopStream*)) (giopImpl12.cc:646)<br>==9765== by 0xB209EF: omni::GIOP_S::dispatcher() (GIOP_S.cc:137)<br>==9765== by 0xB1DE1E: omni::giopWorker::execute() (giopWorker.cc:77)<br>==9765== by 0xAC43A1: omniAsyncWorker::real_run() (invoker.cc:578)</span><br><br></div><div>This raise condition happens between write & read operation, however there are also cases where raise happens also between write & write operation what is more problematic. For example:</div><div><br></div><div><span style="font-family:monospace">==9765== Possible data race during write of size 1 at 0x67C9275 by thread #7<br>==9765== Locks held: none<br>==9765== at 0xB5ACD7: omni::SocketHolder::setNonBlocking() (SocketCollection.h:110)<br>==9765== by 0xB5A39D: omni::tcpConnection::Send(void*, unsigned long, omni_time_t const&) (tcpConnection.cc:71)<br>==9765== by 0xB10FC6: omni::sendCloseConnection(omni::giopStrand*) (giopStrand.cc:121)<br>==9765== by 0xB12AE4: omni::Scavenger::execute() (giopStrand.cc:698)<br>==9765== by 0xAC43A1: omniAsyncWorker::real_run() (invoker.cc:578)<br>==9765== by 0xAC5885: omniAsyncPoolGeneral::workerRun(omniAsyncWorker*) (invoker.cc:297)<br>==9765== by 0xAC3F45: omniAsyncWorker::mid_run() (invoker.cc:511)<br>==9765== by 0xAC4614: omniAsyncWorkerInfo::run() (invoker.cc:662)<br>==9765== by 0xAC55C1: omniAsyncWorker::run(void*) (invoker.cc:126)<br>==9765== by 0xB67F33: omni_thread_wrapper (posix.cc:459)<br>==9765== by 0x4C38C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)<br>==9765== by 0x4E4B6DA: start_thread (pthread_create.c:463)<br>==9765== <br>==9765== This conflicts with a previous write of size 1 by thread #8<br>==9765== Locks held: none<br>==9765== at 0xB5ACA2: omni::SocketHolder::setBlocking() (SocketCollection.h:101)<br>==9765== by 0xB5A55D: omni::tcpConnection::Recv(void*, unsigned long, omni_time_t const&) (tcpConnection.cc:155)<br>==9765== by 0xB15CA6: omni::giopStream::inputMessage() (giopStream.cc:844)<br>==9765== by 0xB31149: omni::giopImpl12::inputNewServerMessage(omni::giopStream*) (giopImpl12.cc:438)<br>==9765== by 0xB319CB: omni::giopImpl12::inputMessageBegin(omni::giopStream*, void (*)(omni::giopStream*)) (giopImpl12.cc:646)<br>==9765== by 0xB209EF: omni::GIOP_S::dispatcher() (GIOP_S.cc:137)<br>==9765== by 0xB1DE1E: omni::giopWorker::execute() (giopWorker.cc:77)<br>==9765== by 0xAC43A1: omniAsyncWorker::real_run() (invoker.cc:578)</span><br></div><div><br></div><div></div><div>Obviously, the tcpConnection entity is not thread safe. <br><br>I don't know if this problem leads to the aforementioned deadlock issue, however I am a bit confused, since I don't know if omniOrb library is responsible for such kind of locking or me, as a user of omniOrb library, should handle this manually somehow.<br><br></div><div>Would you mind to give me some advice.<br></div><div><br></div><div>Thanks, <br>Damijan Skvarc,<br></div><div>Senior Software Engineer<br></div><div>Instrumentation Technologies, Slovenia<br></div><pre><span style="font-family:arial,sans-serif"></span></pre></div></div>
</div>