[omniNotify] omniNotify Performance Tuning Question / subscri ption bug

Fri Aug 8 11:48:30 BST 2003

Robert, (sorry about duplicate email. Forgot to cc to omniNotify list)

I am using the default channel.cfg.

I don't quite do it the way you have interpreted, sorry about the lack of
clarity.

Method 1: (older and slower approach for our project)
Application creates 512 proxies to match the 512 StructuredPushConsumer's,
each StructuredPushConsumer creates a single Filter (each Filter has a
single constraint) on it's associated Proxy. The StructuredPushConsumer's
are created in a loop, one after the other with the filter being added as
soon as the StructuredPushConsumer is created. After all the
StructuredPushConsumer's are created the same application will send events
that will only match one of the created constraints.

Method 2: (faster approach for our project)
Application creates 1 proxy to match a single StructuredPushConsumer, the
StructuredPushConsumer creates a single filter with  an empty constraint.
512 subscribers are then added - each subscriber adds its own constraint to
the existing Filter, one at a time, using add_constraints() (The
constraintExpSeq being passed to the filter only contains one constraint). I
know it would be faster to add all the constraints in one go with 512
constraints in the constraintExpSeq, but that approach does not suit our
project. Once all the subscribers have been added the application starts
sending events as in method 1. (Obviously some client side filtering based
on EventType needs to be applied in this scenario)

The bug occurs only in Method 2:
Start one instance of the application, once all constraints have been
updated on the single filter the application starts to send and receive
events. Now that the events are flowing start another instance of the
application, as soon as the constraints start being updated on the Filter
the NS will lock up, as described in my earlier email.

Does that clear it up a little?

Cheers
Cameron

-----Original Message-----
From: Robert E. Gruber [mailto:gruber at research.att.com]
Sent: Thursday, August 07, 2003 10:01 PM
To: 'Cameron Rochester'; omninotify-list at omniorb-support.com
Cc: Robert E Gruber
Subject: RE: [omniNotify] omniNotify Performance Tuning Question /
subscription bug

Cameron, let me make sure I understand your test setup.
Is this true:  You have a single client program that
connects through a single proxy.  Then it either adds
512 filters, where each filter subscribes to one event
type, or it adds one filter with 512 constraints, where
each constraint matches a single event type.
The same client then supplies events of all 512 types.

So you are not checking the scaling of omniNotify with
number of clients, but rather with either the number
of filters or the number of constraints per filter?

Also, the bug you reported occurs if you run two such
clients?

Also, what config do you use?  (The default channel.cfg?)

-- Bob

> One other thing - does omniNotify clean up invalid
> suppliers/consumers/filters if a client application crashes?

Once a proxy is marked as a problem (exception occurred when
using to its connected client) the proxy no longer holds onto
events queued for the client, but the proxy itself is not
destroyed right away.  To do a final cleanup of such 'dead'
proxies and their related admins and filters, you need to
use 'cleanup both' on the server using the do_command API.

-- Bob

-----Original Message-----
From: omninotify-list-bounces at omniorb-support.com
[mailto:omninotify-list-bounces at omniorb-support.com] On Behalf Of
Cameron Rochester
Sent: Thursday, August 07, 2003 2:13 AM
To: 'Robert E. Gruber'; omninotify-list at omniorb-support.com
Subject: RE: [omniNotify] omniNotify Performance Tuning Question

Robert,

I grabbed the latest omniNotify snapshot (and the latest omniORB
snapshot)
and checked the update.log, it contained the following entry:

"Changed things so event lock is not held across push 
(required different approach for omniORB3 vs. omniORB4).
Also remove all events from outgoing queue of a proxy
that gets _pxstatus set to RDI_Exception."

I assume this is the fix you have mentioned? After a few hours of
building I
was ready to go again.

One other thing - does omniNotify clean up invalid
suppliers/consumers/filters if a client application crashes?

On to the testing....

environment:
- Solaris 2.8 (Sunfire V480 - 2 x 1.1G with 4,096MB RAM. GCC 3.2.1)
running
omniNotify
- Win2k (P4 2.4G with 512MB RAM. VS 6.0) running test client

I performed the test again and found performance to be a little better.
For
the implementation of the client please refer to the "omniNotify
subscription bug" email I also sent today, I use the same application
for
gathering performance data.

Performance for unique subscriptions had improved but still exhibited
very
poor perfomance with > 32 unique subscribers. Performance for unique
subscriptions was generally double the last build with > 16 unique
subscribers. Shared subscriptions again show what approaches a flat
performance rate.

For a graph of this data please see http://ii.net/~rochest/graph.gif.
The
graph shows performance with up to 1024 subscribed types. Our system can
easily require in excess of 1000 subscribed event types.

Regards
Cameron Rochester

-----Original Message-----
From: Robert E. Gruber [mailto:gruber at research.att.com]
Sent: Wednesday, August 06, 2003 11:34 PM
To: 'Cameron Rochester'; omninotify-list at omniorb-support.com
Cc: Robert E. Gruber
Subject: RE: [omniNotify] omniNotify Performance Tuning Question

omniNotify is supposed to scale with clients much better than Cameron
describes.

The poor scaling he reports may be due to a bug that was recently fixed
(a
lock was held
across push calls where it should not be).  

Perhaps Cameron would you be willing to get the latest omniORB and
omniNotify from
sourceforge and try benchmarking again?   

For those interesting in getting stuff from sourceforge, I will send
another
email
describing how to do this.

A new omniORB 4 release will happen soon, and this will be followed by
the
long-awaited
(!) omniNotify 2.0 official release (which depends on the upcoming
omniORB
4, at least for
the Windows port).  

Next email: getting sources.

-- Bob

-----Original Message-----
From: omninotify-list-bounces at omniorb-support.com
[mailto:omninotify-list-bounces at omniorb-support.com] On Behalf Of
Cameron
Rochester
Sent: Tuesday, August 05, 2003 8:27 PM
To: omninotify-list at omniorb-support.com
Subject: RE: [omniNotify] omniNotify Performance Tuning Question

Patrick,

How many consumers are connected to the notification service? What is
the
complexity of the filters that you have in place?

I have found with testing that performance drops off considerably as
consumers increase (it is much worse than a linear degradation).
However,
evaluating more complex filters and having less consumers is actually a
lot
faster. Our system used to have multiple connections to the notification
service from the same client. I changed the approach and reduced the
number
of connections while increasing the filter complexity and found
performance
to be much better while reducing load on the service.

Having said that, I never saw the kind of lag that you are experiencing.
Running the service on a (relatively) slow dual processor solaris box (2
x
400MHz, 512Mb RAM) and sending/receiving events from a seperate windows
client (P4 2.4G, 512Mb RAM) the notification service would still manage
approx 10 events/sec with 1700 unique subscribers. The faster approach
was
to create a filter with a constraint expression seq with 1700 individual
constraints (but only one connection) and then perform simple filtering
on
the client side to ensure the correct subscriber receives the correct
event.
With this approach the notification service sustains ~500 events/sec
with
1700 subscribed types. With 1-2 subscribed types it would manage upwards
of
600 events/sec.

So, the service should be able to handle significantly more than 15-20
events minute, but we probably need a bit more information regarding the
way
in which you are consuming events.

Cheers
Cameron Rochester

-----Original Message-----
From: Patrick Vaughan [mailto:plvaugh at sandia.gov]
Sent: Wednesday, August 06, 2003 6:55 AM
To: omninotify-list at omniorb-support.com
Cc: omniorb-list at omniorb-support.com
Subject: [omniNotify] omniNotify Performance Tuning Question

Greetings,

I've been running an application which uses omniORB/omniORBpy with 
omniNotify.  Everything works, but the performance in respect to event 
passing seems quite slow.  The load on the system is about 15-20 events 
per minute.  I'm seeing delays of around a full minute from the time an 
event is sent to the time the event arrives.  I've run the application 
on both a Mac OS X 10.2.6 system and a Red Hat Linux 7.3.2 system.  The 
delay is about the same regardless of the system I run the application 
on.

Any suggestions on how to decrease the delay in event delivery would be 
appreciated.

Thanks.

_______________________________________________
omninotify-list mailing list
omninotify-list at omniorb-support.com
http://www.omniorb-support.com/mailman/listinfo/omninotify-list

_______________________________________________
omninotify-list mailing list
omninotify-list at omniorb-support.com
http://www.omniorb-support.com/mailman/listinfo/omninotify-list