[omniORB] omniORB 3_0_4 server crashes under heavy workload

[Jochen Behrens]

Hi,

We're using omniORB in a mission critical project with strong
requirements regarding workload and reliability. The platform we are
using is SUSE linux 6.2 (Kernel 2.2, glibc 2.1, gcc 2.95.2). The
application is CORBA-based on top of the C++ omniORB V3.0.4.

Since we have detected a non-determistic behaviour in our server under
very heavy workload (including crashes), we tried to find out the
problem by using the mcheck library which comes with the glibc 2.1.
With mcheck the server throws a pile of `memory clobbered before
allocated block' messages on stderr before aborting.

After a hard working week, we erased 99% functionality from our server,
but the problem remains.
So we implemented a very simple ORB stress test scenario containing a
server that provides one interface with only one method:

module foo
{
    interface bar 
    {
	IsAlive(in string str);	
    }; 
};

and a client that tries to request this method permanently:

while(1)
{
     barRef->IsAlive("holla");
};

Launching only one client, the server works robust. But several clients
lead to an abortion of the server.

We are not really shure whether this is a problem with mcheck, with the
ORB, or with wrong g++ compilerflag/library settings. A have attached
both the make outputs as well as the server backtrace as achieved by
setting  -ORBabortOnInternalError = 1.


Make process:
-------------

/common/sw/common/omni/omniORB_3_0_4/bin/i586_linux_2.0_glibc2.1/omniidl
-bcxx -Wbs=Skel.C foo.idl

g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
-I/common/sw/common/omni/omniORB_3_0_4/include -c fooSkel.C

g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
-I/common/sw/common/omni/omniORB_3_0_4/include -c barImpl.C

g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
-I/common/sw/common/omni/omniORB_3_0_4/include -c server.C

g++ -o server -lmcheck fooSkel.o barImpl.o server.o
-L/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1
-lomniORB3 -lCOS3 -lomnithread -ltcpwrapGK -lpthread


Backtrace:
----------

(gdb) bt
#0  0x4033e9e3 in mallochook (size=8, caller=0x805a6ac) at mcheck.c:130
#1  0x4033b6bc in __libc_malloc (bytes=8) at malloc.c:2621
#2  0x805a6ac in __builtin_new (sz=8) at ./cp/new1.cc:77
#3  0x805a8a3 in __builtin_vec_new (sz=8) at ./cp/new2.cc:39
#4  0x4008f158 in _CORBA_String_member::operator<<= ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0#5 
0x805209c in Foo::_impl_Bar::_dispatch ()
#6  0x40070013 in omniOrbPOA::dispatch ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#7  0x40066a89 in omniLocalIdentity::dispatch ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#8  0x4009337f in GIOP_S::HandleRequest ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#9  0x40092a69 in GIOP_S::dispatcher ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#10 0x400af907 in tcpSocketWorker::_realRun ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#11 0x400c39e0 in omniORB::giopServerThreadWrapper::run ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
#12 0x400af8aa in tcpSocketWorker::run ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3.so.0
---Type <return> to continue, or q <return> to quit---
#13 0x4029a864 in omni_thread_wrapper ()
   from
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomnithread.so.2
#14 0x402b618a in pthread_start_thread (arg=0xbf1ffe78) at
manager.c:214       


Best Regards,
Jochen

--
+------------------------ Jochen Behrens -------------------------+
|    Orthogon GmbH, Hastedter Osterdeich 222, D-28207 Bremen      |
|           E-Mail: Jochen Behrens <behrens@orthogon.de>          |
+---- Tel.: +49 (0)421 20122-0/-47, Fax: +49 (0)421 20122-999 ----+

ORTHOGON and ODS Toolbox are registered trademarks of Orthogon GmbH