[omniORB] thread pool vs. thread per connection

[baileyk@schneider.com]

Part III (or is it IV)...

I'm studying the performance impact to omniNames of the different thread
policies.  I think I've got a pretty good handle on what's going on now.
I'm using omniNames as a test server since it is available to everyone here
and can be a critical piece in the performance of a distributed system.
See some past posts (last couple weeks) from myself and steinhoffc for
background.

Thread pool:
     I found that the omniAsyncInvoker has a 10 second idle timeout for the
threads in the pool.  Meaning that any thread not used for 10 seconds will
die.  This explains the excessive thread creation.  I'd run a test, say 5
concurrent calls, and see 5 threads created.  Then I'd look over the
results (for more than 10 seconds) and run again, and again see 5 threads
created.  Under continuous load, the thread creation may not be so bad, but
I think the 10 second timeout should be configurable.  In reality usage
spikes may be expected ever few minutes, not seconds.

upcall delay:
     omniORB4's wiki site describes a 50-100ms delay when calls are
multiplexed on a single connection.  All of my tests are omniORB -> omniORB
and using default client connection semantics (no multiplexing).  But I
still see a 50-100ms delay when using a thread pool, without connection
watching.  I think I've tracked it down to the logic in
SocketCollection::setSelectable().  There's a comment

    // XXX poke the thread doing accept to look at the fdset immediately.

But no code follows this to do such a 'poke', which is evidently why
there's an XXX in the comment.  A future planned enhancement?  I think the
thread doing the upcall is attempting to notify the rendezvouser thread to
monitor the socket once the upcall starts.  But the rendezvouser is not
poked and doesn't wake up to include this socket for 50-100ms.  So even if
the upcall completes in 2ms, and the client sends another request
immediately on the same connection, the ORB isn't watching that connection
for a span of time.  This was _killing_ us during sequences of calls to
omniNames ( list(), next_n(), destroy(), resolve(), narrow() ).

I did my best to implement the 'poke' feature in the SocketCollection.
I've added a pipe as a member of the SocketCollection and make sure the
read side is in the rdfs set during a select() call.  Then in the
setSelectable() function I write 1 byte into the other end of the pipe.
After the select(), the read end of the pipe is checked and if readable the
1 byte is read and I set pd_abs_sec = pd_abs_nsec = 0, which I think forces
the next loop to pick up all selectable sockets in the next select() call.
Preliminary tests on Solaris 2.8 are very good.  The upcall delay average
(compared to using thread-per-connection) in omniNames is down from 72ms to
0.3ms.  Testing is ongoing.  I hope to show that the patch provides best
performance in a multiuser Java(with request multiplexing) -> Unix/C++
environment by minimizing thread creation (I upped the 10 seconds to 15
minutes) and eliminating the 50-100ms delays associated with multiplexed
calls.

My research into the possibility of porting this to Windows (which we don't
use with omniORB at all) is not good.  On Solaris and Linux (and other
Unices?) a select() can monitor a mix of file descriptors.  On Windows it
can not (only sockets).  I don't know how to poke the rendezvouser in that
case.  I had hoped to submit a patch that was portable at least to
Solaris/Linux/Windows after testing is completed.

If anyone wants code for the patch prior to the conclusion of my testing,
just ask.

Kendall