[omniORB] Core dump while pushing structured event

[Mark Zimmerman]

Greetings:

I am cross posting this because my crash occurs in an omniORB library
while calling an omniNotify function.

I have been having a nagging problem with a program that core dumps
after several days of operation. I have not found a deterministic way
to trigger the problem in less time so it has been difficult to track
down. Finally, I compiled all of omniORB and omniNotify with debug and
ran with orbtracelevel=10. After a three day run I finally got some
useful information.

All of the core dumps have occurred from the same point in my code.
However, this code gets executed many (>12000) times before the crash
occurs.  

I have not yet dug deeply into the omniORB code to track this down; I
wanted to post this information first in case someone with a greater
understanding of the internals can give me a clue. The relevant
information follows.

The last orb trace message is:
----------------------------------------------------------------------
omniORB: throw UNKNOWN from giopClient.cc:495
----------------------------------------------------------------------

The source code listing around this point (with numbers) is:
----------------------------------------------------------------------
485   RequestCompleted();
486   switch (s) {
487   case CORBA::COMPLETED_YES:
488   case CORBA::COMPLETED_NO:
489   case CORBA::COMPLETED_MAYBE:
490     break;
491   default:
492     OMNIORB_THROW(UNKNOWN,0,CORBA::COMPLETED_MAYBE);
493   };
494 
495   OMNIORB_FOR_EACH_SYS_EXCEPTION(CHECK_AND_IF_MATCH_THROW_SYSTEM_EXCEPTION)
496 
497   // If none of the above matched
498   OMNIORB_THROW(UNKNOWN,0,CORBA::COMPLETED_MAYBE);
499 
500 #undef CHECK_AND_IF_MATCH_THROW_SYSTEM_EXCEPTION
----------------------------------------------------------------------

Here is the stack trace just in case it's helpful:
----------------------------------------------------------------------
(/opt/SUNWspro/bin/../WS5.0/bin/sparcv9/dbx) where -h
current thread: t@1
  [1] __sigprocmask(0x0, 0xffbec9a8, 0x0, 0x0, 0x0, 0x0), at 0xfe659bf0
  [2] _resetsig(0xfe65c510, 0x0, 0x0, 0x1305e0, 0xfe66e000, 0x0), at 0xfe64e620
  [3] _sigon(0x1305e0, 0xfe675990, 0x6, 0xffbeca7c, 0x1305e0, 0xff200c90), at 0xfe64dd10
  [4] _thrp_kill(0x0, 0x1, 0x6, 0xfe66e000, 0x1, 0xfe5ba428), at 0xfe650e84
  [5] raise(0x6, 0x0, 0x0, 0xffffffff, 0xfe5ba394, 0xfebd4b74), at 0xfe549b08
  [6] abort(0xfe5b6000, 0xfebea940, 0xfebea118, 0xea468, 0xff3e0000, 0xff20013c), at 0xfe535124
  [7] __Cimpl::ex_terminate(0x0, 0x0, 0xffbedec0, 0x0, 0xffbedec0, 0xeac88), at 0xfebd7284
  [8] rethrow(0x0, 0x1, 0xfebea118, 0xfebea940, 0x0, 0xfebea940), at 0xfebd5070
  [9] omniObjRef::_invoke(0x148ce0, 0xfe94f0e8, 0xfe907aac, 0xfe955f0c, 0x0, 0xfe95613c), at 0xfe8c3ea0
  [10] CosNotifyComm::_objref_StructuredPushConsumer::push_structured_event(0x148d14, 0xffbecebc, 0xffbecee8, 0xffbecf18, 0x133240, 0xffbece00), at 0xfea985ec
=>[11] EventSource::push_structured_event(this = 0x11e2c8, se = STRUCT), line 149 in "eventsource.cc"
  [12] SpwLog::Speak(this = 0x11e2c8, severity = 0, pid = 495, progName = 0x13c9a8 "Ingest", msg = 0xffbed203 "INGEST moved file: /export/swafs/dih_acct/SPWDIH_destination_dir/12291429.tls"), line 165 in "spwlog.cc"
  [13] SpwLog::errlog(this = 0x11e2c8, severity = Routine, message = 0xffbed203 "INGEST moved file: /export/swafs/dih_acct/SPWDIH_destination_dir/12291429.tls"), line 117 in "spwlog.cc"
  [14] check_dir(config = 0x13048c), line 236 in "check_dir.cc"
  [15] main(), line 74 in "ingest_control.cc"
----------------------------------------------------------------------

Any insights would be greatly appreciated.
-- Mark