[omniORB] std::terminate in omniORBpy

Fri May 27 13:36:08 UTC 2022

It certainly looks as though the thread exit code in omniORBpy should
check _Py_IsFinalizing(), for versions that have that. That won't help
you if you are stuck on Python 3.6, though.

The interesting question is why these threads are exiting during
Python's finalization, and why it has changed since omniORBpy 4.2. I
suspect the change may actually not be one that is unique to omniORBpy
4.3, but rather a change that was made in 4.2.5:

https://sourceforge.net/p/omniorb/svn/6539/

Do you call orb.destroy() before exit?

Duncan.

On Mon, 2022-05-23 at 14:26 +0200, Michael Teske via omniORB-list
wrote:
> Hi,
> 
> I think I found it in https://docs.python.org/3/c-api/init.html :
> 
> "void PyEval_RestoreThread(PyThreadState *tstate)
>      Part of the Stable ABI.
> 
>      Acquire the global interpreter lock (if it has been created) and
> set the thread state to tstate, which must not be 
> NULL. If the lock has been created, the current thread must not have
> acquired it, otherwise deadlock ensues.
> 
>      Note
> 
>      Calling this function from a thread when the runtime is
> finalizing will terminate the thread, even if the thread 
> was not created by Python. You can use _Py_IsFinalizing() or
> sys.is_finalizing() to check if the interpreter is in 
> process of being finalized before calling this function to avoid
> unwanted termination."
> 
> Unfortunately _Py_IsFinalizing() is not available on python 3.6,
> which I'm currently stuck with.
> Is there a way to wait for all omniOrbPy threads before exiting (in
> the python script)? Maybe this would fix the 
> problem, too.
> 
> Regards,
>    Michael
> 
> Am 12/05/2022 um 13:59 schrieb Michael Teske via omniORB-list:
> > Hi,
> > 
> > since upgrading from 4.2.4, I sometimes get aborts on python
> > shutdown.
> > Must be a race condition somewhere as it does not happen reliably.
> > 
> > 
> > Thread 1 (Thread 0x7f6949e7b700 (LWP 2933428) (Exiting)):
> > #0  0x00007f69ca7bc37f in raise () from /usr/lib64/libc.so.6
> > #1  0x00007f69ca7a6db5 in abort () from /usr/lib64/libc.so.6
> > #2  0x00007f69c75207ba in __gnu_cxx::__verbose_terminate_handler()
> > () from /usr/lib64/libstdc++.so.6
> > --Type <RET> for more, q to quit, c to continue without paging--
> > #3  0x00007f69c751e53c in __cxxabiv1::__terminate(void (*)()) ()
> > from /usr/lib64/libstdc++.so.6
> > #4  0x00007f69c751e597 in std::terminate() () from
> > /usr/lib64/libstdc++.so.6
> > #5  0x00007f69c751df0a in __gxx_personality_v0 () from
> > /usr/lib64/libstdc++.so.6
> > #6  0x00007f69c7280bf8 in _Unwind_ForcedUnwind_Phase2 () from
> > /usr/lib64/libgcc_s.so.1
> > #7  0x00007f69c7281240 in _Unwind_ForcedUnwind () from
> > /usr/lib64/libgcc_s.so.1
> > #8  0x00007f69cb2e5556 in __pthread_unwind () from
> > /usr/lib64/libpthread.so.0
> > #9  0x00007f69cb2dd40b in pthread_exit () from
> > /usr/lib64/libpthread.so.0
> > #10 0x00007f69cb55c3fd in PyThread_exit_thread () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #11 0x00007f69cb5ed322 in PyEval_RestoreThread () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #12 0x00007f69c7e68ece in omnipyThreadCache::threadExit
> > (cn=0x7f6938004de0) at pyThreadCache.cc:234
> > #13 0x00007f69c7e690eb in omnipyThreadData::~omnipyThreadData
> > (this=0x7f6938005710, __in_chrg=<optimized out>) at 
> > pyThreadCache.cc:78
> > #14 omnipyThreadData::~omnipyThreadData (this=0x7f6938005710,
> > __in_chrg=<optimized out>) at pyThreadCache.cc:79
> > #15 0x00007f69c781fc98 in omni_thread::~omni_thread
> > (this=0x7f694c002710, __in_chrg=<optimized out>) at posix.cc:536
> > #16 0x00007f69c7af6c94 in omniAsyncWorker::~omniAsyncWorker
> > (this=0x7f694c002710, __in_chrg=<optimized out>) at 
> > invoker.cc:109
> > #17 omniAsyncWorker::~omniAsyncWorker (this=0x7f694c002710,
> > __in_chrg=<optimized out>) at invoker.cc:117
> > #18 0x00007f69c78205df in omni_thread::exit
> > (return_value=return_value at entry=0x0) at posix.cc:774
> > #19 0x00007f69c7820724 in omni_thread_wrapper (ptr=0x7f694c002710)
> > at posix.cc:460
> > #20 0x00007f69cb2dc17a in start_thread () from
> > /usr/lib64/libpthread.so.0
> > #21 0x00007f69ca881dc3 in clone () from /usr/lib64/libc.so.6
> > 
> > there are several such threads and
> > the main thread is here:
> > 
> > Thread 5 (Thread 0x7f69cc13ab80 (LWP 2933363)):
> > #0  0x00007f69cb2e274a in pthread_cond_timedwait@@GLIBC_2.3.2 ()
> > from /usr/lib64/libpthread.so.0
> > #1  0x00007f69cb5ed0a4 in take_gil () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #2  0x00007f69cb5ed2dd in PyEval_RestoreThread () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #3  0x00007f69c7e3eb06 in
> > omniPy::InterpreterUnlocker::~InterpreterUnlocker (this=<synthetic
> > pointer>, 
> > __in_chrg=<optimized out>) at ./omnipy.h:404
> > #4  pyObjRef_dealloc (self=0x7f699f0d1ee8) at pyObjectRef.cc:604
> > #5  0x00007f69cb5deb2c in dict_dealloc () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #6  0x00007f69cb67b351 in subtype_dealloc () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #7  0x00007f69cb5deb87 in dict_dealloc () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #8  0x00007f69cb5dafcf in free_keys_object () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #9  0x00007f69cb6aa806 in type_clear () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #10 0x00007f69cb66c276 in collect () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #11 0x00007f69cb6e424b in _PyGC_CollectNoFail () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #12 0x00007f69cb6fe169 in PyImport_Cleanup () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #13 0x00007f69cb701772 in Py_FinalizeEx () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #14 0x00007f69cb702544 in Py_Main () from
> > /usr/lib64/libpython3.6m.so.1.0
> > #15 0x000055e0d2a45b96 in main ()
> > 
> > When I set ORBtraceLevel=40 I see this:
> > 
> > omniORB: (9) 2022-05-12 13:44:38.662928: ORB shutdown is complete.
> > omniORB: (9) 2022-05-12 13:44:38.662932: Shut down AsyncInvoker...
> > omniORB: (9) 2022-05-12 13:44:38.662937: Shut down dedicated thread
> > queue.
> > omniORB: (9) 2022-05-12 13:44:38.662942: AsyncInvoker shut down.
> > omniORB: (9) 2022-05-12 13:44:38.662955: Deleting Python state for
> > thread id 140552133670656 (thread exit)
> > omniORB: (3) 2022-05-12 13:44:39.149144: Scan for idle connections
> > (1652355879,149046000)
> > omniORB: (3) 2022-05-12 13:44:39.149201: Scavenger reduce idle
> > count for strand 0x558342c168e0 to 23
> > omniORB: (3) 2022-05-12 13:44:39.149211: Scavenger reduce idle
> > count for strand 0x558342d11100 to 23
> > omniORB: (3) 2022-05-12 13:44:39.149216: Scavenger reduce idle
> > count for strand 0x5583424957f0 to 23
> > omniORB: (3) 2022-05-12 13:44:39.149221: Scavenger reduce idle
> > count for strand 0x558342920020 to 23
> > omniORB: (3) 2022-05-12 13:44:39.149226: Scavenger reduce idle
> > count for strand 0x558342ada5e0 to 23
> > omniORB: (3) 2022-05-12 13:44:39.149233: Scan for idle connections
> > done (1652355879,149046000).
> > omniORB: (0) 2022-05-12 13:44:39.661452: Destroy ORB...
> > omniORB: (1) 2022-05-12 13:44:39.661536: Remove Python state for
> > thread id 140552643036928 from cache (shutdown)
> > omniORB: (1) 2022-05-12 13:44:39.661570: Remove Python state for
> > thread id 140552617858816 from cache (shutdown)
> > omniORB: (1) 2022-05-12 13:44:39.661582: Remove Python state for
> > thread id 140552626251520 from cache (shutdown)
> > omniORB: (1) 2022-05-12 13:44:39.661585: Remove Python state for
> > thread id 140553558406912 from cache (shutdown)
> > omniORB: (1) 2022-05-12 13:44:39.661590: Remove Python state for
> > thread id 140552634644224 from cache (shutdown)
> > omniORB: (1) 2022-05-12 13:44:39.661711: Python thread state
> > scavenger exit.
> > omniORB: (0) 2022-05-12 13:44:39.661818: Terminate strand
> > scavenger.
> > omniORB: (0) 2022-05-12 13:44:39.661838: Close remaining strands.
> > omniORB: (0) 2022-05-12 13:44:39.661844: Shutdown close connection
> > to giop:tcp:[::1]:15001
> > omniORB: (3) 2022-05-12 13:44:39.661885: Scan for idle connections
> > (1652355884,149246000)
> > omniORB: (0) 2022-05-12 13:44:39.661910: Client connection
> > giop:tcp:[::1]:15001 refcount (forced) = 0
> > omniORB: (0) 2022-05-12 13:44:39.661915: Client close connection to
> > giop:tcp:[::1]:15001
> > omniORB: (0) 2022-05-12 13:44:39.661921: Shutdown close connection
> > to giop:tcp:10.49.64.73:15001
> > omniORB: (0) 2022-05-12 13:44:39.661925: sendCloseConnection: to
> > giop:tcp:10.49.64.73:15001 12 bytes
> > omniORB: (0) 2022-05-12 13:44:39.661928:
> > 4749 4f50 0102 0105 0000 0000           GIOP........
> > omniORB: (0) 2022-05-12 13:44:39.661971: Client connection
> > giop:tcp:10.49.64.73:15001 refcount (forced) = 0
> > omniORB: (0) 2022-05-12 13:44:39.661976: Client close connection to
> > giop:tcp:10.49.64.73:15001
> > omniORB: (0) 2022-05-12 13:44:39.661980: Shutdown close connection
> > to giop:tcp:10.49.64.73:34493
> > omniORB: (0) 2022-05-12 13:44:39.661983: sendCloseConnection: to
> > giop:tcp:10.49.64.73:34493 12 bytes
> > omniORB: (0) 2022-05-12 13:44:39.661985:
> > 4749 4f50 0102 0105 0000 0000           GIOP........
> > omniORB: (0) 2022-05-12 13:44:39.662016: Client connection
> > giop:tcp:10.49.64.73:34493 refcount (forced) = 0
> > omniORB: (0) 2022-05-12 13:44:39.662020: Client close connection to
> > giop:tcp:10.49.64.73:34493
> > omniORB: (0) 2022-05-12 13:44:39.662024: Shutdown close connection
> > to giop:tcp:10.49.64.73:48261
> > omniORB: (0) 2022-05-12 13:44:39.662027: sendCloseConnection: to
> > giop:tcp:10.49.64.73:48261 12 bytes
> > omniORB: (0) 2022-05-12 13:44:39.662029:
> > 4749 4f50 0102 0105 0000 0000           GIOP........
> > omniORB: (0) 2022-05-12 13:44:39.662059: Client connection
> > giop:tcp:10.49.64.73:48261 refcount (forced) = 0
> > omniORB: (0) 2022-05-12 13:44:39.662067: Client close connection to
> > giop:tcp:10.49.64.73:48261
> > omniORB: (0) 2022-05-12 13:44:39.662071: Shutdown close connection
> > to giop:tcp:10.49.64.73:39675
> > omniORB: (0) 2022-05-12 13:44:39.662074: sendCloseConnection: to
> > giop:tcp:10.49.64.73:39675 12 bytes
> > omniORB: (0) 2022-05-12 13:44:39.662077:
> > 4749 4f50 0102 0105 0000 0000           GIOP........
> > omniORB: (0) 2022-05-12 13:44:39.662109: Client connection
> > giop:tcp:10.49.64.73:39675 refcount (forced) = 0
> > omniORB: (0) 2022-05-12 13:44:39.662114: Client close connection to
> > giop:tcp:10.49.64.73:39675
> > omniORB: (0) 2022-05-12 13:44:39.662147: 0 remaining bidir ropes
> > deleted.
> > omniORB: (0) 2022-05-12 13:44:39.662157: 5 remaining ropes deleted.
> > omniORB: (0) 2022-05-12 13:44:39.662172: Clear endPoint options.
> > omniORB: (3) 2022-05-12 13:44:39.662184: Scan for idle connections
> > done (1652355884,149246000).
> > omniORB: (0) 2022-05-12 13:44:39.662215: Wait for 7 invoker threads
> > to finish.
> > omniORB: (3) 2022-05-12 13:44:39.662235: AsyncInvoker: thread id 3
> > finished immediate general task.
> > omniORB: (3) 2022-05-12 13:44:39.662264: AsyncInvoker: thread id 3
> > released from general tasks. Total general threads = 1.
> > omniORB: (3) 2022-05-12 13:44:39.662274: AsyncInvoker: thread id 3
> > has exited. Total threads = 6.
> > omniORB: (2) 2022-05-12 13:44:39.662281: AsyncInvoker: thread id 2
> > released from general tasks. Total general threads = 0.
> > omniORB: (2) 2022-05-12 13:44:39.662298: AsyncInvoker: thread id 2
> > has exited. Total threads = 5.
> > omniORB: (7) 2022-05-12 13:44:39.662309: AsyncInvoker: thread id 7
> > released from server tasks. Total server threads = 4.
> > omniORB: (8) 2022-05-12 13:44:39.662324: AsyncInvoker: thread id 8
> > released from server tasks. Total server threads = 3.
> > omniORB: (7) 2022-05-12 13:44:39.662330: AsyncInvoker: thread id 7
> > has exited. Total threads = 4.
> > omniORB: (8) 2022-05-12 13:44:39.662336: AsyncInvoker: thread id 8
> > has exited. Total threads = 3.
> > omniORB: (7) 2022-05-12 13:44:39.662360: Deleting Python state for
> > thread id 140552626251520 (thread exit)
> > omniORB: (8) 2022-05-12 13:44:39.662362: Deleting Python state for
> > thread id 140552617858816 (thread exit)
> > omniORB: (5) 2022-05-12 13:44:39.662393: AsyncInvoker: thread id 5
> > released from server tasks. Total server threads = 1.
> > omniORB: (5) 2022-05-12 13:44:39.662432: AsyncInvoker: thread id 5
> > has exited. Total threads = 2.
> > omniORB: (5) 2022-05-12 13:44:39.662447: Deleting Python state for
> > thread id 140552643036928 (thread exit)
> > omniORB: (4) 2022-05-12 13:44:39.662371: AsyncInvoker: thread id 4
> > released from server tasks. Total server threads = 2.
> > omniORB: (4) 2022-05-12 13:44:39.662505: AsyncInvoker: thread id 4
> > has exited. Total threads = 1.
> > omniORB: (4) 2022-05-12 13:44:39.662551: Deleting Python state for
> > thread id 140553558406912 (thread exit)
> > omniORB: (6) 2022-05-12 13:44:39.662561: AsyncInvoker: thread id 6
> > released from server tasks. Total server threads = 0.
> > omniORB: (6) 2022-05-12 13:44:39.662586: AsyncInvoker: thread id 6
> > has exited. Total threads = 0.
> > omniORB: (6) 2022-05-12 13:44:39.662601: Deleting Python state for
> > thread id 140552634644224 (thread exit)
> > omniORB: (0) 2022-05-12 13:44:39.662625: Invoker threads finished.
> > omniORB: (0) 2022-05-12 13:44:39.662640: AsyncInvoker: deleted.
> > omniORB: (0) 2022-05-12 13:44:39.662654: ORB destroyed.
> > omniORB: (0) 2022-05-12 13:44:39.662792: No more references to the
> > ORB -- deleted.
> > omniORB: (0) 2022-05-12 13:44:39.743916:
> > ObjRef(IDL:fundamental/Server:1.0) -- deleted.
> > terminate called without an active exception
> > 
> > It's not critical because it's on exit anyway, but annoying enough
> > to ask for help here. Is it maybe because there 
> > somehow are object references after ORB destruction which may be
> > throw exceptions (similar to 
> > https://stackoverflow.com/questions/59082973/terminate-called-without-an-active-exception-after-pthread-cancel
> >  although 
> > no cancel involved?)
> > 
> > I'll do some more tests to find out what exactly changed. The
> > python code was changed as well so maybe it's just 
> > coincidence that this happened after the update...
> > 
> > Greetings,
> >    Michael
> > 
> > _______________________________________________
> > omniORB-list mailing list
> > omniORB-list at omniorb-support.com
> > https://www.omniorb-support.com/mailman/listinfo/omniorb-list
> 
> 
> _______________________________________________
> omniORB-list mailing list
> omniORB-list at omniorb-support.com
> https://www.omniorb-support.com/mailman/listinfo/omniorb-list

-- 
-- Duncan Grisby --
-- duncan at grisby.org --
-- http://www.grisby.org --

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.omniorb-support.com/pipermail/omniorb-list/attachments/20220527/22518183/attachment-0001.html>