[omniORB] BOA vs POA Performance

[Craig A. Roloff]

We have an HP-UX 10.20 client/server application we want to
upgrade from omniORB 2.8.0 (BOA) to omniORB 3.0.2 (POA).  The
conversion was relatively simple but the resulting performance
is perplexing -- the POA version runs an order of magnitude
slower.

The server and the client run on the same machine (indeed they
are parent and child processes).  When we try all four of the
BOA/POA combinations only the POA server combinations are slow.
When using either the BOA or POA client with the BOA server, the
application runs in 11 seconds.  When using either client with
the POA server, the application runs in 140 seconds.

We've tried tracing the ORB calls and they are identical between
BOA and POA.  We've tried setting a large object hash table and
disabling locate requests, without any improvements.

The server is passing a total of about 300K bytes in 4000 calls
(including locate requests).  We ran puma on the server.
Both the BOA and the POA servers have 5 threads.  They spend
essentially all of their time as follows:

Main)  Blocked in select
2)     Running in cma___null_thread/cma__io_available/_select_sys
3)     Blocked in omniORB_Ripper::run_undetached/omni_condition::wait
4)     Blocked in omniORB_Scavenger::run_undetached/omni_condition::timedwait
5)     Blocked in tcpSocketRendezvouser::run_undetached/accept

The POA server takes 256M user cycles and 1.7G system cycles,
whereas the BOA server takes 140M user cycles and 220M system
cycles.  Thus it is appears the POA server is spending more
of its time just waiting.

We're at a loss on what to investigate next.  Any ideas?