[omniORB] Python and omniORB thread issues

[Duncan Grisby]

On Friday 6 October, Thomas Nugent wrote:

>   Python threads spawned from an omniORB created thread are being
>   trashed by the omnipyThreadScavenger (pyThreadCache.cc).

The root of your problem is that you are mixing threads created with
the thread module with primitives from the threading module. You
should not do this. If you are going to use things from threading, you
should start all your threads with threading.Thread. I'm not sure it
can be considered a bug in omniORB that it masks this incorrect use,
and causes a later error. There is in fact a bug which is related to
this, but it is not what is affecting you.

To explain what's going wrong, I need to explain a few things about
how Python and omniORB manage threads.

For each thread, Python maintains two separate bits of "thread
state". One is a C structure called PyThreadState which contains
various "global" things like the Python exception status, and so on.
Whenever a particular thread is running Python code, the global
interpreter lock must be held, and a suitable PyThreadState must be
installed for that thread.

The second piece of state is a Python object, derived from
threading.Thread, created by the threading module. It is used by the
primitives in threading, such as RLock, to track what each thread is
doing. The threading module maintains a dictionary, called _active,
which maps thread ids to these Thread objects. This state only exists
if a thread is created with the threading module. If you try to use
one of threading's primitives on a non-threading thread, it creates a
DummyThread object and prints the "currentThread(): no current thread
for n" message. The DummyThread object is never deleted, so you get a
memory leak.

Whenever omniORB makes a call from C++ into Python code, it has to
install a suitable PyThreadState struct. Often, it is the case that
the thread in question was created by omniORB, and therefore has never
had a PyThreadState. A suitable struct is therefore created. In other
cases, the call into Python has resulted from an earlier call _from_
Python, in meaning there is already a suitable PyThreadState.
However, in some circumstances omniORB has no way of getting hold of
the original PyThreadState, so it creates another one. As far as
Python is concerned, a different thread makes the call into Python,
but that's OK since CORBA doesn't make any guarantees about how
threads are used. Once omniORB has created a PyThreadState, it caches
it to save the overhead of creating it again. Cached thread states are
removed after a minute with no use.

Now the problematic bit. People expect to be able to use the
primitives from the threading module within threads created by
omniORB. Therefore, when omniORB creates a new PyThreadState for a
thread, it also looks to see if it needs to create a threading.Thread
object to put in threading._active. In the case of threads created by
omniORB or by the thread module, it does; in the case of threads
created by threading, it does not. If a threading.Thread object is
added to _active, then it is removed again when the cached thread
state is deleted.

So, the problem that you have is that your thread is created with the
thread module, so it has no entry in threading._active. Now you make a
call into some omniORB function which performs an up-call into Python
(a call to _this(), for example). omniORB is unable to get hold of the
existing PyThreadState for the thread, so it creates a new one. It
discovers that there is no entry in threading._active, so it creates
one. This would not be a problem if you did not use anything from the
threading module. However, since you _do_ use threading.RLock, things
go wrong when the thread state scavenger does its thing. I don't think
I can do anything to detect this situation, but I'll think about it.

If you create your thread with threading.Thread, rather than
thread.start_new_thread(), then there will already be an entry in
threading._active when you call into omniORB, and everything will be
fine.

The related bug is that the thread state would sometimes be deleted
prematurely for a thread created by omniORB. This would occur if you
did some call which re-entered Python, then spent more than a minute
doing other things. It's fixed in CVS.

Cheers,

Duncan.

-- 
 -- Duncan Grisby  \  Research Engineer  --
  -- AT&T Laboratories Cambridge          --
   -- http://www.uk.research.att.com/~dpg1 --