[omniORB] Callback loop stalling: sample code

[Duncan Grisby]

On Monday 30 October, Kevin Bailey wrote:

> Attached is a sample program which is much simpler but displays
> the same problem as mine. Again, this is omniORB 3.0.2 on
> FreeBSD 4.1.

The problem is due to an interaction with the way omniORB manages TCP
connections. On the first call to a particular address space, omniORB
opens a new TCP connection to it. This causes a new thread to start on
the server. After the first call, the connection is kept open, ready
for future requests. The thread therefore stays active on the server.
Connections which have been idle for some time are collected by a
scavenger thread.

However, when the last object reference to an address space is
deleted, any connections to that address space are closed immediately.
In some circumstances, that is precisely what you need, since
otherwise you may run out of file descriptors for the TCP sockets.

Now, in your situation, you have two machines, A and B. A passes a
reference to a call-back object to B. B performs the call-back,
causing a connection to be opened back to A, and a new thread to start
on A. B then drops the reference to the call-back object; it is the
only reference to an object in A's address space, so the connection is
closed, and A's thread exits. Then the whole thing repeats.

The key to the problem is that A is repeatedly creating and deleting
threads. FreeBSD obviously isn't coping very well with that. Linux
also gets upset, although only after a few thousand calls, when your
program receives a COMM_FAILURE.

There are a few things we could do in omniORB to help with this
situation, and we might think about it for future versions. In the
mean-time, a work-around is for B to keep a reference to an object in
A's address space -- for example the first callback object received.
That prevents the closure of the connection, so all calls are
dispatched on the same thread.

Cheers,

Duncan.

-- 
 -- Duncan Grisby  \  Research Engineer  --
  -- AT&T Laboratories Cambridge          --
   -- http://www.uk.research.att.com/~dpg1 --