[omniORB] Omni threads keep waiting on omni::internalLock mutex, as a resul t Server hangs !!!

[Kamaldeep Singh Khanuja]

Hi All,

We are facing a problem in which our application hangs where in our server
accepts the calls but it could not 'dispatch' the call up to the respective
servants. This is followed by the following error message at the server
console:
"omniORB: ERROR -- A servant has been deleted that is still activated."

The cause for the hang seems to be due to the fact that omni::internal mutex
remains locked forever and new calls to the server wait on this mutex. This
happens when the servant gets deleted (in our application we have a
dedicated thread which performs the servant deactivation and deletion) when
there is a client who is currently waiting for the response from server i.e.
within the IDL call itself. So when the omni thread which was handling the
request returns from the IDL call it follows the following stack:
 
 
omniORB401_rtd.dll!omni::omniOrbPOA::lastInvocationHasCompleted(omniLocalIde
ntity * id=0x03e833b8)  Line 1902 C++
 
omniORB401_rtd.dll!omni::omniLocalIdentity_RefHolder::~omniLocalIdentity_Ref
Holder()  Line 115 C++
  omniORB401_rtd.dll!omniLocalIdentity::dispatch(omniCallHandle &
handle={...})  Line 203 + 0xf C++
  omniORB401_rtd.dll!omni::GIOP_S::handleRequest()  Line 302 C++
  omniORB401_rtd.dll!omni::GIOP_S::dispatcher()  Line 230 + 0x8 C++
  omniORB401_rtd.dll!omni::giopWorker::real_execute()  Line 212 + 0x8 C++
  omniORB401_rtd.dll!omni::giopWorkerInfo::run()  Line 101 C++
  omniORB401_rtd.dll!omni::giopWorker::execute()  Line 115 C++
  omniORB401_rtd.dll!omniAsyncWorker::real_run()  Line 182 + 0x10 C++
  omniORB401_rtd.dll!omniAsyncWorkerInfo::run()  Line 230 C++
  omniORB401_rtd.dll!omniAsyncWorker::run(void * __formal=0x00000000)  Line
134 C++
  omnithread30_rtd.dll!omni_thread_wrapper(void * ptr=0x03e7e8a0)  Line 502
C++

 
Within the omniOrbPOA::lastInvocationHasCompleted method (ref. poa.cc)
exception is thrown at setEtherealising() and 'lastInvocationHasCompleted'
returns without unlocking the omni::internalLock. This exception is caught
by CORBA::Boolean GIOP_S::handleRequest() (ref. giop_s.cc). Please see the
code below for the same with commets inline:
 
void
omniOrbPOA::lastInvocationHasCompleted(omniLocalIdentity* id)
{
...
...
// Exception will be thrown and this method would return without unlocking
the omni::internalLock!!!!
entry->setEtherealising();
 
// This would remain locked !!!!!!!!!!!
omni::internalLock->unlock();
 
....
...
}
 
Wouldn't it be good that omni just deactivates the deleted servant by simply
removing its reference from object table, probably prints the same error
message and does not hang. We are currently working from two sides to solve
this problem, firstly we correct our server code to prevent the deletion of
servants whose activation count is more than one and second make changes in
the omniORB code so that it should not hang i.e leave the omni::internalLock
locked!!.
 
Thanks,
 
--Kamal
kkhanuja at quark.co.in