[omniORB] std::terminate in omniORBpy

Michael Teske subscribe at teskor.de
Fri May 27 14:15:31 UTC 2022 

Hi,

Am 27/05/2022 um 15:36 schrieb Duncan Grisby:
> It certainly looks as though the thread exit code in omniORBpy should check _Py_IsFinalizing(), for versions that have 
> that. That won't help you if you are stuck on Python 3.6, though.

yes, untortunately.

> The interesting question is why these threads are exiting during Python's finalization, and why it has changed since 
> omniORBpy 4.2. I suspect the change may actually not be one that is unique to omniORBpy 4.3, but rather a change that 
> was made in 4.2.5:
>
> https://sourceforge.net/p/omniorb/svn/6539/ <https://sourceforge.net/p/omniorb/svn/6539/>

This could be, I tested 4.2.4 and 4.3.0 only.

> Do you call orb.destroy() before exit?

Yes, in my first post I appended the ORBtraceLevel=40 log. There are all log messages from

omniORB: (0) 2022-05-12 13:44:39.661452: Destroy ORB...
to
omniORB: (0) 2022-05-12 13:44:39.662654: ORB destroyed.
omniORB: (0) 2022-05-12 13:44:39.662792: No more references to the ORB -- deleted.

Unfortunately one thread seems to survive this.

Greetings,
   Michael

> Duncan.
> 
> 
> On Mon, 2022-05-23 at 14:26 +0200, Michael Teske via omniORB-list wrote:
>> Hi,
>>
>> I think I found it in https://docs.python.org/3/c-api/init.html <https://docs.python.org/3/c-api/init.html> :
>>
>> "void PyEval_RestoreThread(PyThreadState *tstate)
>>      Part of the Stable ABI.
>>
>>      Acquire the global interpreter lock (if it has been created) and set the thread state to tstate, which must not be
>> NULL. If the lock has been created, the current thread must not have acquired it, otherwise deadlock ensues.
>>
>>      Note
>>
>>      Calling this function from a thread when the runtime is finalizing will terminate the thread, even if the thread
>> was not created by Python. You can use _Py_IsFinalizing() or sys.is_finalizing() to check if the interpreter is in
>> process of being finalized before calling this function to avoid unwanted termination."
>>
>> Unfortunately _Py_IsFinalizing() is not available on python 3.6, which I'm currently stuck with.
>> Is there a way to wait for all omniOrbPy threads before exiting (in the python script)? Maybe this would fix the
>> problem, too.
>>
>> Regards,
>>    Michael
>>
>> Am 12/05/2022 um 13:59 schrieb Michael Teske via omniORB-list:
>>> Hi,
>>>
>>> since upgrading from 4.2.4, I sometimes get aborts on python shutdown.
>>> Must be a race condition somewhere as it does not happen reliably.
>>>
>>>
>>> Thread 1 (Thread 0x7f6949e7b700 (LWP 2933428) (Exiting)):
>>> #0  0x00007f69ca7bc37f in raise () from /usr/lib64/libc.so.6
>>> #1  0x00007f69ca7a6db5 in abort () from /usr/lib64/libc.so.6
>>> #2  0x00007f69c75207ba in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
>>> --Type <RET> for more, q to quit, c to continue without paging--
>>> #3  0x00007f69c751e53c in __cxxabiv1::__terminate(void (*)()) () from /usr/lib64/libstdc++.so.6
>>> #4  0x00007f69c751e597 in std::terminate() () from /usr/lib64/libstdc++.so.6
>>> #5  0x00007f69c751df0a in __gxx_personality_v0 () from /usr/lib64/libstdc++.so.6
>>> #6  0x00007f69c7280bf8 in _Unwind_ForcedUnwind_Phase2 () from /usr/lib64/libgcc_s.so.1
>>> #7  0x00007f69c7281240 in _Unwind_ForcedUnwind () from /usr/lib64/libgcc_s.so.1
>>> #8  0x00007f69cb2e5556 in __pthread_unwind () from /usr/lib64/libpthread.so.0
>>> #9  0x00007f69cb2dd40b in pthread_exit () from /usr/lib64/libpthread.so.0
>>> #10 0x00007f69cb55c3fd in PyThread_exit_thread () from /usr/lib64/libpython3.6m.so.1.0
>>> #11 0x00007f69cb5ed322 in PyEval_RestoreThread () from /usr/lib64/libpython3.6m.so.1.0
>>> #12 0x00007f69c7e68ece in omnipyThreadCache::threadExit (cn=0x7f6938004de0) at pyThreadCache.cc:234
>>> #13 0x00007f69c7e690eb in omnipyThreadData::~omnipyThreadData (this=0x7f6938005710, __in_chrg=<optimized out>) at
>>> pyThreadCache.cc:78
>>> #14 omnipyThreadData::~omnipyThreadData (this=0x7f6938005710, __in_chrg=<optimized out>) at pyThreadCache.cc:79
>>> #15 0x00007f69c781fc98 in omni_thread::~omni_thread (this=0x7f694c002710, __in_chrg=<optimized out>) at posix.cc:536
>>> #16 0x00007f69c7af6c94 in omniAsyncWorker::~omniAsyncWorker (this=0x7f694c002710, __in_chrg=<optimized out>) at
>>> invoker.cc:109
>>> #17 omniAsyncWorker::~omniAsyncWorker (this=0x7f694c002710, __in_chrg=<optimized out>) at invoker.cc:117
>>> #18 0x00007f69c78205df in omni_thread::exit (return_value=return_value at entry=0x0) at posix.cc:774
>>> #19 0x00007f69c7820724 in omni_thread_wrapper (ptr=0x7f694c002710) at posix.cc:460
>>> #20 0x00007f69cb2dc17a in start_thread () from /usr/lib64/libpthread.so.0
>>> #21 0x00007f69ca881dc3 in clone () from /usr/lib64/libc.so.6
>>>
>>> there are several such threads and
>>> the main thread is here:
>>>
>>> Thread 5 (Thread 0x7f69cc13ab80 (LWP 2933363)):
>>> #0  0x00007f69cb2e274a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
>>> #1  0x00007f69cb5ed0a4 in take_gil () from /usr/lib64/libpython3.6m.so.1.0
>>> #2  0x00007f69cb5ed2dd in PyEval_RestoreThread () from /usr/lib64/libpython3.6m.so.1.0
>>> #3  0x00007f69c7e3eb06 in omniPy::InterpreterUnlocker::~InterpreterUnlocker (this=<synthetic pointer>,
>>> __in_chrg=<optimized out>) at ./omnipy.h:404
>>> #4  pyObjRef_dealloc (self=0x7f699f0d1ee8) at pyObjectRef.cc:604
>>> #5  0x00007f69cb5deb2c in dict_dealloc () from /usr/lib64/libpython3.6m.so.1.0
>>> #6  0x00007f69cb67b351 in subtype_dealloc () from /usr/lib64/libpython3.6m.so.1.0
>>> #7  0x00007f69cb5deb87 in dict_dealloc () from /usr/lib64/libpython3.6m.so.1.0
>>> #8  0x00007f69cb5dafcf in free_keys_object () from /usr/lib64/libpython3.6m.so.1.0
>>> #9  0x00007f69cb6aa806 in type_clear () from /usr/lib64/libpython3.6m.so.1.0
>>> #10 0x00007f69cb66c276 in collect () from /usr/lib64/libpython3.6m.so.1.0
>>> #11 0x00007f69cb6e424b in _PyGC_CollectNoFail () from /usr/lib64/libpython3.6m.so.1.0
>>> #12 0x00007f69cb6fe169 in PyImport_Cleanup () from /usr/lib64/libpython3.6m.so.1.0
>>> #13 0x00007f69cb701772 in Py_FinalizeEx () from /usr/lib64/libpython3.6m.so.1.0
>>> #14 0x00007f69cb702544 in Py_Main () from /usr/lib64/libpython3.6m.so.1.0
>>> #15 0x000055e0d2a45b96 in main ()
>>>
>>> When I set ORBtraceLevel=40 I see this:
>>>
>>> omniORB: (9) 2022-05-12 13:44:38.662928: ORB shutdown is complete.
>>> omniORB: (9) 2022-05-12 13:44:38.662932: Shut down AsyncInvoker...
>>> omniORB: (9) 2022-05-12 13:44:38.662937: Shut down dedicated thread queue.
>>> omniORB: (9) 2022-05-12 13:44:38.662942: AsyncInvoker shut down.
>>> omniORB: (9) 2022-05-12 13:44:38.662955: Deleting Python state for thread id 140552133670656 (thread exit)
>>> omniORB: (3) 2022-05-12 13:44:39.149144: Scan for idle connections (1652355879,149046000)
>>> omniORB: (3) 2022-05-12 13:44:39.149201: Scavenger reduce idle count for strand 0x558342c168e0 to 23
>>> omniORB: (3) 2022-05-12 13:44:39.149211: Scavenger reduce idle count for strand 0x558342d11100 to 23
>>> omniORB: (3) 2022-05-12 13:44:39.149216: Scavenger reduce idle count for strand 0x5583424957f0 to 23
>>> omniORB: (3) 2022-05-12 13:44:39.149221: Scavenger reduce idle count for strand 0x558342920020 to 23
>>> omniORB: (3) 2022-05-12 13:44:39.149226: Scavenger reduce idle count for strand 0x558342ada5e0 to 23
>>> omniORB: (3) 2022-05-12 13:44:39.149233: Scan for idle connections done (1652355879,149046000).
>>> omniORB: (0) 2022-05-12 13:44:39.661452: Destroy ORB...
>>> omniORB: (1) 2022-05-12 13:44:39.661536: Remove Python state for thread id 140552643036928 from cache (shutdown)
>>> omniORB: (1) 2022-05-12 13:44:39.661570: Remove Python state for thread id 140552617858816 from cache (shutdown)
>>> omniORB: (1) 2022-05-12 13:44:39.661582: Remove Python state for thread id 140552626251520 from cache (shutdown)
>>> omniORB: (1) 2022-05-12 13:44:39.661585: Remove Python state for thread id 140553558406912 from cache (shutdown)
>>> omniORB: (1) 2022-05-12 13:44:39.661590: Remove Python state for thread id 140552634644224 from cache (shutdown)
>>> omniORB: (1) 2022-05-12 13:44:39.661711: Python thread state scavenger exit.
>>> omniORB: (0) 2022-05-12 13:44:39.661818: Terminate strand scavenger.
>>> omniORB: (0) 2022-05-12 13:44:39.661838: Close remaining strands.
>>> omniORB: (0) 2022-05-12 13:44:39.661844: Shutdown close connection to giop:tcp:[::1]:15001
>>> omniORB: (3) 2022-05-12 13:44:39.661885: Scan for idle connections (1652355884,149246000)
>>> omniORB: (0) 2022-05-12 13:44:39.661910: Client connection giop:tcp:[::1]:15001 refcount (forced) = 0
>>> omniORB: (0) 2022-05-12 13:44:39.661915: Client close connection to giop:tcp:[::1]:15001
>>> omniORB: (0) 2022-05-12 13:44:39.661921: Shutdown close connection to giop:tcp:10.49.64.73:15001
>>> omniORB: (0) 2022-05-12 13:44:39.661925: sendCloseConnection: to giop:tcp:10.49.64.73:15001 12 bytes
>>> omniORB: (0) 2022-05-12 13:44:39.661928:
>>> 4749 4f50 0102 0105 0000 0000           GIOP........
>>> omniORB: (0) 2022-05-12 13:44:39.661971: Client connection giop:tcp:10.49.64.73:15001 refcount (forced) = 0
>>> omniORB: (0) 2022-05-12 13:44:39.661976: Client close connection to giop:tcp:10.49.64.73:15001
>>> omniORB: (0) 2022-05-12 13:44:39.661980: Shutdown close connection to giop:tcp:10.49.64.73:34493
>>> omniORB: (0) 2022-05-12 13:44:39.661983: sendCloseConnection: to giop:tcp:10.49.64.73:34493 12 bytes
>>> omniORB: (0) 2022-05-12 13:44:39.661985:
>>> 4749 4f50 0102 0105 0000 0000           GIOP........
>>> omniORB: (0) 2022-05-12 13:44:39.662016: Client connection giop:tcp:10.49.64.73:34493 refcount (forced) = 0
>>> omniORB: (0) 2022-05-12 13:44:39.662020: Client close connection to giop:tcp:10.49.64.73:34493
>>> omniORB: (0) 2022-05-12 13:44:39.662024: Shutdown close connection to giop:tcp:10.49.64.73:48261
>>> omniORB: (0) 2022-05-12 13:44:39.662027: sendCloseConnection: to giop:tcp:10.49.64.73:48261 12 bytes
>>> omniORB: (0) 2022-05-12 13:44:39.662029:
>>> 4749 4f50 0102 0105 0000 0000           GIOP........
>>> omniORB: (0) 2022-05-12 13:44:39.662059: Client connection giop:tcp:10.49.64.73:48261 refcount (forced) = 0
>>> omniORB: (0) 2022-05-12 13:44:39.662067: Client close connection to giop:tcp:10.49.64.73:48261
>>> omniORB: (0) 2022-05-12 13:44:39.662071: Shutdown close connection to giop:tcp:10.49.64.73:39675
>>> omniORB: (0) 2022-05-12 13:44:39.662074: sendCloseConnection: to giop:tcp:10.49.64.73:39675 12 bytes
>>> omniORB: (0) 2022-05-12 13:44:39.662077:
>>> 4749 4f50 0102 0105 0000 0000           GIOP........
>>> omniORB: (0) 2022-05-12 13:44:39.662109: Client connection giop:tcp:10.49.64.73:39675 refcount (forced) = 0
>>> omniORB: (0) 2022-05-12 13:44:39.662114: Client close connection to giop:tcp:10.49.64.73:39675
>>> omniORB: (0) 2022-05-12 13:44:39.662147: 0 remaining bidir ropes deleted.
>>> omniORB: (0) 2022-05-12 13:44:39.662157: 5 remaining ropes deleted.
>>> omniORB: (0) 2022-05-12 13:44:39.662172: Clear endPoint options.
>>> omniORB: (3) 2022-05-12 13:44:39.662184: Scan for idle connections done (1652355884,149246000).
>>> omniORB: (0) 2022-05-12 13:44:39.662215: Wait for 7 invoker threads to finish.
>>> omniORB: (3) 2022-05-12 13:44:39.662235: AsyncInvoker: thread id 3 finished immediate general task.
>>> omniORB: (3) 2022-05-12 13:44:39.662264: AsyncInvoker: thread id 3 released from general tasks. Total general threads 
>>> = 1.
>>> omniORB: (3) 2022-05-12 13:44:39.662274: AsyncInvoker: thread id 3 has exited. Total threads = 6.
>>> omniORB: (2) 2022-05-12 13:44:39.662281: AsyncInvoker: thread id 2 released from general tasks. Total general threads 
>>> = 0.
>>> omniORB: (2) 2022-05-12 13:44:39.662298: AsyncInvoker: thread id 2 has exited. Total threads = 5.
>>> omniORB: (7) 2022-05-12 13:44:39.662309: AsyncInvoker: thread id 7 released from server tasks. Total server threads = 4.
>>> omniORB: (8) 2022-05-12 13:44:39.662324: AsyncInvoker: thread id 8 released from server tasks. Total server threads = 3.
>>> omniORB: (7) 2022-05-12 13:44:39.662330: AsyncInvoker: thread id 7 has exited. Total threads = 4.
>>> omniORB: (8) 2022-05-12 13:44:39.662336: AsyncInvoker: thread id 8 has exited. Total threads = 3.
>>> omniORB: (7) 2022-05-12 13:44:39.662360: Deleting Python state for thread id 140552626251520 (thread exit)
>>> omniORB: (8) 2022-05-12 13:44:39.662362: Deleting Python state for thread id 140552617858816 (thread exit)
>>> omniORB: (5) 2022-05-12 13:44:39.662393: AsyncInvoker: thread id 5 released from server tasks. Total server threads = 1.
>>> omniORB: (5) 2022-05-12 13:44:39.662432: AsyncInvoker: thread id 5 has exited. Total threads = 2.
>>> omniORB: (5) 2022-05-12 13:44:39.662447: Deleting Python state for thread id 140552643036928 (thread exit)
>>> omniORB: (4) 2022-05-12 13:44:39.662371: AsyncInvoker: thread id 4 released from server tasks. Total server threads = 2.
>>> omniORB: (4) 2022-05-12 13:44:39.662505: AsyncInvoker: thread id 4 has exited. Total threads = 1.
>>> omniORB: (4) 2022-05-12 13:44:39.662551: Deleting Python state for thread id 140553558406912 (thread exit)
>>> omniORB: (6) 2022-05-12 13:44:39.662561: AsyncInvoker: thread id 6 released from server tasks. Total server threads = 0.
>>> omniORB: (6) 2022-05-12 13:44:39.662586: AsyncInvoker: thread id 6 has exited. Total threads = 0.
>>> omniORB: (6) 2022-05-12 13:44:39.662601: Deleting Python state for thread id 140552634644224 (thread exit)
>>> omniORB: (0) 2022-05-12 13:44:39.662625: Invoker threads finished.
>>> omniORB: (0) 2022-05-12 13:44:39.662640: AsyncInvoker: deleted.
>>> omniORB: (0) 2022-05-12 13:44:39.662654: ORB destroyed.
>>> omniORB: (0) 2022-05-12 13:44:39.662792: No more references to the ORB -- deleted.
>>> omniORB: (0) 2022-05-12 13:44:39.743916: ObjRef(IDL:fundamental/Server:1.0) -- deleted.
>>> terminate called without an active exception
>>>
>>> It's not critical because it's on exit anyway, but annoying enough to ask for help here. Is it maybe because there
>>> somehow are object references after ORB destruction which may be throw exceptions (similar to
>>> https://stackoverflow.com/questions/59082973/terminate-called-without-an-active-exception-after-pthread-cancel 
>>> <https://stackoverflow.com/questions/59082973/terminate-called-without-an-active-exception-after-pthread-cancel> although 
>>>
>>> no cancel involved?)
>>>
>>> I'll do some more tests to find out what exactly changed. The python code was changed as well so maybe it's just
>>> coincidence that this happened after the update...
>>>
>>> Greetings,
>>>    Michael
>>>
>>> _______________________________________________
>>> omniORB-list mailing list
>>> omniORB-list at omniorb-support.com <mailto:omniORB-list at omniorb-support.com>
>>> https://www.omniorb-support.com/mailman/listinfo/omniorb-list 
>>> <https://www.omniorb-support.com/mailman/listinfo/omniorb-list>
>>
>>
>> _______________________________________________
>> omniORB-list mailing list
>> omniORB-list at omniorb-support.com <mailto:omniORB-list at omniorb-support.com>
>> https://www.omniorb-support.com/mailman/listinfo/omniorb-list 
>> <https://www.omniorb-support.com/mailman/listinfo/omniorb-list>
> 
> -- 
> 
>   -- Duncan Grisby --
>    -- duncan at grisby.org --
>     -- http://www.grisby.org --
>

More information about the omniORB-list mailing list