[omniORB] Re: Bug? Calls serialization in omni4.0.1

[baileyk at schneider.com]

The documentation discusses this, I think.  When omniORB dispatches a
thread to service a request, the connection will not be monitored for other
incoming requests during a short period of time (the 5ms you found I
guess).  Basically, when using thread pooling, the sequence goes like this

1. Thread A notices incoming request, passes the connection to thread B
2. Thread A blocks waiting for activity on other connections
3. Thread B reads the request from the connection.  It may also read part
or all of other requests that are multiplexed.  It is interested only in
the one request, but the others are not lost.  These others will be queued.
4. Thread B makes the upcall for the request.  The connection is not
monitored at all at this point.

Two things can happen here.  Thread B may finish the upcall and start
working on another request on the same connection if the upcall was short.
If the upcall takes a while, then thread A will wake up first and take over
the connection.  At this point thread A can then dispatch the next request
on another thread (thread C).  If A takes over the connection, then when B
finishes the upcall, it will simply go back to the thread pool.  A thread
in the thread pool will self destruct if not used within some timeout.

If you turn off connection watching (e.g. thread B checking for more
requests on the connection if A hasn't alread reclaimed it), then B will go
to the thread pool no matter what.  In this case the connection remains
un-monitored until A wakes up.

I ran into problems with this, oddly enough without getting Java into the
equation.  I had connection watching off and a series of calls to omniNames
from a single thread in a client was taking much longer than it should.
The problem was that each call had to wait for thread A to wake up and
dispatch the next request.  The fixed overhead killed the performance.

In researching that, I developed a patch to awaken thread A just before
thread B started it's upcall.  Unfortunately the patch only works on a
Unix-like OS.  Duncan requested a patch that would work on Windows too.  I
didn't have the time or resources to attempt that.  Later, Duncan indicated
that something equivalent to my patch was included in v4.0.1 (the above
discussion should be fairly accurate for v4.0.0).  I don't know if he found
a way to get thread A to wake up on Windows too, or just went with the fix
that works for Unix.

Thread A is blocked on a call to poll (or select?) on all of the
connections except those that have a request beeing read by a pooled
thread.  My approach was to add a pipe to the list of connections and have
thread B write a byte into the other end of the pipe.  That wakes thread A
up, but it knows the pipe is just a dummy file descriptor so it goes on to
check all the other connections for pending requests and to reclaim the
connection B has finished reading from in case another request comes in
during B's upcall.  Windows does not have a pipe equivalent that I could
find to do something similar with.

Hopefully I've not mucked up the description too much.  If things are as
I've described still in v4.0.1, then I'm not sure how to help you.
Fundamentally, some thread needs to watch the connection at all times or
else there will be added latencies.  The thread doing the upcall needs to
pass off responsibility to either thread A (as my patch did), or to yet
another thread.  But then the problem is how does that other thread hand it
back to A?  Perhaps there can be some overlap and if two threads both
monitor a single connection, they can realize this when the connection has
a request and only one of them reads from it?  I've never written a server
that can handle multiplexed overlapping requests, so I don't know how to do
it for sure.


Kendall




                                                                                                                             
                      Dmitry Davidovich                                                                                      
                      <DmitryD at enigma.c        To:       "'baileyk at schneider.com'" <baileyk at schneider.com>                   
                      om>                      cc:       "'omniorb-list at omniorb-support.com'"                                
                                                <omniorb-list at omniorb-support.com>                                           
                      07/01/2003 12:48         Fax to:                                                                       
                      PM                       Subject:  [omniORB] Re: Bug? Calls serialization  in omni4.0.1                
                                                                                                                             
                                                                                                                             




Thread id tracing discovered next behavior:
long call sequences (20 calls) dispatched to the same thread with only 2
threads in use (on 4 CPUs),
BUT !!!! it's true for relatively short calls only (5 ms). If I make a
longer busy loop (100 ms) all CPUs will be utilised.

Any idea?

Dmitry