[omniORB] omniORB 3_0_4 server crashes under heavy workload

[Sai-Lai Lo]

Your stack trace does point to memory corruption. We do the kind of test as
your test program all the time and do not see the problem.

Could you try njamd to pin-point where memory corruption happen? Run it
under gdb, it will abort as soon as the program try to access beyond the
allocated block.

At this stage, I'm more inclined to suspect your platform+compiler.

Regards,

Sai-Lai

----- Original Message -----
From: "Jochen Behrens" <Jochen.Behrens@orthogon.de>
To: <omniorb-list@uk.research.att.com>
Sent: Wednesday, August 15, 2001 4:16 PM
Subject: [omniORB] omniORB 3_0_4 server crashes under heavy workload


> Hi,
>
> We're using omniORB in a mission critical project with strong
> requirements regarding workload and reliability. The platform we are
> using is SUSE linux 6.2 (Kernel 2.2, glibc 2.1, gcc 2.95.2). The
> application is CORBA-based on top of the C++ omniORB V3.0.4.
>
> Since we have detected a non-determistic behaviour in our server under
> very heavy workload (including crashes), we tried to find out the
> problem by using the mcheck library which comes with the glibc 2.1.
> With mcheck the server throws a pile of `memory clobbered before
> allocated block' messages on stderr before aborting.
>
> After a hard working week, we erased 99% functionality from our server,
> but the problem remains.
> So we implemented a very simple ORB stress test scenario containing a
> server that provides one interface with only one method:
>
> module foo
> {
>     interface bar
>     {
> IsAlive(in string str);
>     };
> };
>
> and a client that tries to request this method permanently:
>
> while(1)
> {
>      barRef->IsAlive("holla");
> };
>
> Launching only one client, the server works robust. But several clients
> lead to an abortion of the server.
>
> We are not really shure whether this is a problem with mcheck, with the
> ORB, or with wrong g++ compilerflag/library settings. A have attached
> both the make outputs as well as the server backtrace as achieved by
> setting  -ORBabortOnInternalError = 1.
>
>
> Make process:
> -------------
>
> /common/sw/common/omni/omniORB_3_0_4/bin/i586_linux_2.0_glibc2.1/omniidl
> -bcxx -Wbs=Skel.C foo.idl
>
> g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
> -I/common/sw/common/omni/omniORB_3_0_4/include -c fooSkel.C
>
> g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
> -I/common/sw/common/omni/omniORB_3_0_4/include -c barImpl.C
>
> g++ -D__x86__ -D__linux__ -D__OSVERSION__=2 -D_REENTRANT
> -I/common/sw/common/omni/omniORB_3_0_4/include -c server.C
>
> g++ -o server -lmcheck fooSkel.o barImpl.o server.o
> -L/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1
> -lomniORB3 -lCOS3 -lomnithread -ltcpwrapGK -lpthread
>
>
> Backtrace:
> ----------
>
> (gdb) bt
> #0  0x4033e9e3 in mallochook (size=8, caller=0x805a6ac) at mcheck.c:130
> #1  0x4033b6bc in __libc_malloc (bytes=8) at malloc.c:2621
> #2  0x805a6ac in __builtin_new (sz=8) at ./cp/new1.cc:77
> #3  0x805a8a3 in __builtin_vec_new (sz=8) at ./cp/new2.cc:39
> #4  0x4008f158 in _CORBA_String_member::operator<<= ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0#5
> 0x805209c in Foo::_impl_Bar::_dispatch ()
> #6  0x40070013 in omniOrbPOA::dispatch ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #7  0x40066a89 in omniLocalIdentity::dispatch ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #8  0x4009337f in GIOP_S::HandleRequest ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #9  0x40092a69 in GIOP_S::dispatcher ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #10 0x400af907 in tcpSocketWorker::_realRun ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #11 0x400c39e0 in omniORB::giopServerThreadWrapper::run ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> #12 0x400af8aa in tcpSocketWorker::run ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomniORB3
.so.0
> ---Type <return> to continue, or q <return> to quit---
> #13 0x4029a864 in omni_thread_wrapper ()
>    from
>
/common/sw/common/omni/omniORB_3_0_4/lib/i586_linux_2.0_glibc2.1/libomnithre
ad.so.2
> #14 0x402b618a in pthread_start_thread (arg=0xbf1ffe78) at
> manager.c:214
>
>
> Best Regards,
> Jochen
>
> --
> +------------------------ Jochen Behrens -------------------------+
> |    Orthogon GmbH, Hastedter Osterdeich 222, D-28207 Bremen      |
> |           E-Mail: Jochen Behrens <behrens@orthogon.de>          |
> +---- Tel.: +49 (0)421 20122-0/-47, Fax: +49 (0)421 20122-999 ----+
>
> ORTHOGON and ODS Toolbox are registered trademarks of Orthogon GmbH
>
>