[omniORB] Fundamental problems with support for "smart" proxies (long)

[Chris Newbold]

We're currently using omniORB 3.0.3 and have implemented some "smart"
proxies in our application. The "smart" proxies cache the values of some
interface attributes locally to clients, improving performance.

We create our own proxy object factories which return instances of
classes which derive from the _objref_XXXX generated proxies. These
"smart" proxies contain members for caching the attribute values.

Since the amount of data we cache is rather large and (more
significantly) there are a large number of unique CORBA object
instances, the proxy object factory keeps one proxy object instance per
unique object key (as extracted from the TCP profile). This means that
our POF will return the _same_ proxy object multiple times.

Right about now, I'm sure a few of the omniORB developers are going "Uh
oh." It's taken me a while to track all of this down, but it looks like
it was never designed to support this...

Here are the problems I've found:

1) After calling the POF, omniORB calls gainObjRef() on the identity; if
the identity is a localIdentity, gainObjRef() attempts to build a
linked-list of the references for that identity. However, if we return
the same proxy object multiple times, this results in a circularly
linked list of one element.

(Now, it really doesn't make sense to have "smart" proxies for local
objects anyway, but this problem breaks local-remote transparency.)


2) Immediately after calling the POF, omniORB attempts to clear the
type_verified flag in the new proxy object (depending on context).
Normally, modifications to these flags must be done while holding the
internal mutex, but it looks like the assuption is that no other thread
has access here.

When our POF returns the same proxy, however, this is no longer true and
the flags in omniObjRef may be corrupted.


3) It looks like the profiles and identities passed into the POF must be
consumed. In our POF, we can delete the profiles if we return the same
proxy object, but we cannot delete the indentity, since we don't have
access to the destructor (in fact, the lifecycle of identities would
seem to prohibit such destruction).


4) There is a nasty race condition between the thread which first
creates a proxy object through our factory and threads which access that
object subsequently. When the proxy is constructed, it has a pointer to
the identity with which it is assocaited. However, we must wait to gain
the internal lock before calling gainObjRef() and incrementing the
reference count on the indentity.

The race is between this thread trying to grab the lock and increment
the identity's reference count and any other thread which may be
attempting an invocation on the _same_ proxy object. This is due in part
to the fact that each trip through createObjRef() results in a new
remote identity, even if the POF returns the same proxy object as before
(which already has an identity).

Hopefully this thread diagram illustrates the problem:


        tid #1                     tid #2
           |                          |
     createObjRef()                   |
           |                          |
           |                    createObjRef()
<create remoteIdentity #1>            |
           |                          |
           |               <create remoteIdentity #2>
           |                          |
      newObjRef()                     |
           |                          |
           |                     newObjRef()      <- returns same proxy
           |                          |              object as on tid #1
           |                          |
           |                 <take internal lock>
           |                          |
           |                      gainObjRef()
           |                          |
           |                 <drop internal lock>
           |                          |
           |                 <take internal lock>
           |                          |
           |                     locateRequest()
           |                          |
           |            ++remoteIdentity::pd_refCount   <- on id #1,
           |                          |                    NOT id #2!
           |                          |
           |                 <drop internal lock>
           |                          |
           |                    <remote call>
           |                          |
           |                 <take internal lock>
           |                          |
           |            --remoteIdentity::pd_refCount  <- on id #1
           |                          |                  again NOT #2
           |                          |
           |                  ~remoteIdentity          <- id #1 OOPS!
           |                          |
           |                 <drop internal lock>
           |                          |
   <take internal lock>               |
           |                          |
    gainObjRef()        <- called on  |
           |               id #1      |
           |                          |
   <drop internal lock>               |
           |                          |
           |                   omniObjRef::invoke
           |                          |
           |                   omniIdentity::dispatch  <- OOPS!
           |                          |
           |                   < B O O M ! ! ! >


After tracking this down, I came to the conclusion that POFs were simply
never intended to return the same proxy object over and over. Is that
true?

I can work around this by having my POF return a new instance of some
wrapper class on each call, where all of the wrappers for a given key
point to the same cache object.

However, this seems more cumbersome than what I was hoping for....

-Chris Newbold
Laurel Networks, Inc.