[omniORB] CORBA performance on InfiniBand network

[Soundararajan, Krishna]

Thanks Denis.
I tried to use the IPoverIB.
Will try SDP.

When we tried MPI in our homogenous cluster, memory alignment gave improvement in performance. But I did not think of memory alignment with CORBA. Will keep you posted

-----Original Message-----
From: Alexandre DENIS [mailto:Alexandre.Denis at labri.fr] 
Sent: Tuesday, August 04, 2015 6:52 PM
To: omniorb-list at omniorb-support.com
Subject: Re: [omniORB] CORBA performance on InfiniBand network

On Mon, 3 Aug 2015 05:01:22 +0000
"Soundararajan, Krishna" <Krishna.Soundararajan at kla-tencor.com> wrote:

> Hi,
> I am performing a bandwidth comparison study on CORBA using Infiniband 
> and 10GeT link.
> (...)
> With this results, I have following questions.
> 1)      I read in several old articles in internet that ORB kills the
> throughput because of inefficient memory copies and 
> marshalling/unmarshalling. In 10Gb Ethernet and 1 Gb Ethernet, I do 
> see almost  90% utilization(CORBA compared to ethernet). But IB card 
> give less than 25% of network utilization. What is causing this
> overhead? 2)      Though CORBA is meant to be used for remote method
> calls, we are amazed with its superiority in serializing. Can we use 
> it as alternative for socket data transfer with amnual serializing 
> code? Is CORBA suitable in high bandwidth  networks for data transfer?

Hi,

I worked on CORBA for high-performance computing in the past, and nowadays I work with InfiniBand. I can share my experience.

CORBA is not slow by itself. It depends on the ORB used. You can read in this paper https://hal.inria.fr/inria-00000136 some past results.
omniORB is the best performing CORBA implementation, and reaches most of the time the nominal network performance (in case of homogeneous hosts).

The use of InfiniBand may be tricky. What may prevent to reach nominal network speed is data alignment and memory registration. Bandwidth is low when buffers are not aligned on words boundaries. I don't know whether omniORB allocates aligned buffers or not. Memory registration is needed to program the hardware from user-space, so that the NIC knows the mapping for virtual addresses. This operation has a high cost. MPI implementations usually have a cache in case the same buffer is sent multiple times (which happens a lot, for MPI applications). I guess socket layers do the same. However, with CORBA you always have registration cache misses since memory at the receiver side is dynamically allocated for each invocation. You can get some more details here: https://hal.inria.fr/inria-00586015

What socket layer do you use? IPoIB or SDP?

Regards,
-Alexandre DENIS.