[omniORB] crash in openssl code

Wed Feb 21 13:13:00 UTC 2024

Hi,

I've created an issue

https://github.com/openssl/openssl/issues/23650

but now the question arose if omniORB does ensure to use a ssl connection only from one thread. I looks like it's so, but maybe there's a bug somewhere?

Regards,

   Michael

On 2/21/24 10:40, Michael Teske via omniORB-list wrote:
>
> Hi,
>
> as required by our customers, we switched to use omniorb with ssl. This works fine so far,
>
> but on our machine with openssl3 (running RHEL9), we get crashes from time to time looking like this:
>
> (gdb) where
> #0  0x00007f26bb0e14df in tls_get_message_header (mt=<synthetic pointer>, s=0x7f2684001940) at ssl/statem/statem_lib.c:1167
> #1  read_state_machine (s=0x7f2684001940) at ssl/statem/statem.c:587
> #2  state_machine (s=0x7f2684001940, server=<optimized out>) at ssl/statem/statem.c:442
> #3  0x00007f26bb0d2969 in ssl3_read_bytes (s=<optimized out>, type=23, recvd_type=0x0, buf=0x7f25b0000d48 "", len=8192, peek=0, readbytes=0x7f25e69f39f0) at 
> ssl/record/rec_layer_s3.c:1711
> #4  0x00007f26bb0ab7fc in ssl3_read_internal (s=0x7f2684001940, buf=0x7f25b0000d48, len=8192, peek=0, readbytes=0x7f25e69f39f0) at ssl/s3_lib.c:4462
> #5  0x00007f26bb0b2137 in SSL_read (s=<optimized out>, buf=<optimized out>, num=<optimized out>) at ssl/ssl_lib.c:1885
> #6  0x00007f26bbfbd1bf in omni::sslConnection::Recv (this=0x7f2684016b78, buf=0x7f25b0000d48, sz=<optimized out>, deadline=...)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/ssl/sslConnection.cc:246
> #7  0x00007f26bbf04b88 in omni::giopStream::inputMessage (this=this at entry=0x7f25b0000b68) at 
> /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:841
> #8  0x00007f26bbf175a3 in omni::giopImpl12::inputNewServerMessage (g=0x7f25b0000b68) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:438
> #9  0x00007f26bbf184a8 in omni::giopImpl12::inputMessageBegin (g=0x7f25b0000b68, unmarshalHeader=<optimized out>)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:646
> #10 0x00007f26bbf0bc81 in omni::GIOP_S::dispatcher (this=0x7f25b0000b60) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/internal/giopStream.h:143
> #11 0x00007f26bbf096b5 in omni::giopWorker::execute (this=0x7f2684017db0) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopWorker.cc:77
> #12 0x00007f26bbec38d0 in omniAsyncWorker::real_run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:578
> #13 0x00007f26bbec487f in omniAsyncPoolServer::workerRun (this=<optimized out>, worker=<optimized out>)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:328
> #14 0x00007f26bbec3499 in omniAsyncWorker::mid_run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:511
> #15 0x00007f26bbec47de in omniAsyncWorker::run (this=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:126
> #16 0x00007f26bbfcd1d5 in omni_thread_wrapper (ptr=0x7f268401a490) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:459
> #17 0x00007f26baa9f802 in start_thread (arg=<optimized out>) at pthread_create.c:443
> #18 0x00007f26baa3f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>
> I did dnf debuginfo-install openssl-libs-3.0.7-24.el9.x86_64 and am thus able to look into openssl source. It looks as if it gets wrong data in ssl3_read_bytes
>
> and crashes then because s->init_buf is NULL, probably already resetted in tls_finish_handshake().
>
> In the source code around ssl/record/rec_layer_s3.c:1711 it says:
>
> /*
>
> * Unexpected handshake message (ClientHello, NewSessionTicket (TLS1.3) or
>
> * protocol violation)
>
> */
>
> if((s->rlayer.handshake_fragment_len>= 4)
>
> && !ossl_statem_get_in_handshake(s)) {
>
> intined = (s->early_data_state== SSL_EARLY_DATA_READING);
>
> /* We found handshake data, so we're going back into init*/
>
> ossl_statem_set_in_init(s, 1);
>
> i = s->handshake_func(s);
>
> (s->handshake_func is ossl_statem_connect which calls state_machine(s,0)) . While this looks like an openssl bug to me (it should call SSL_Fatal or re-create the buffer),
>
> the question remains, why does this happen?
>
> The thread causing this is here:
>
> (gdb) where
> #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f26a9a24730, op=393, expected=0, futex_word=0x7f26840016d0) at futex-internal.c:57
> #1  __futex_abstimed_wait_common (futex_word=futex_word at entry=0x7f26840016d0, expected=expected at entry=0, clockid=clockid at entry=0, abstime=abstime at entry=0x7f26a9a24730,
>     private=private at entry=0, cancel=cancel at entry=true) at futex-internal.c:87
> #2  0x00007f26baa9c3ff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word at entry=0x7f26840016d0, expected=expected at entry=0, clockid=clockid at entry=0,
>     abstime=abstime at entry=0x7f26a9a24730, private=private at entry=0) at futex-internal.c:139
> #3  0x00007f26baa9eea4 in __pthread_cond_wait_common (abstime=0x7f26a9a24730, clockid=0, mutex=0x8349e0, cond=0x7f26840016a8) at pthread_cond_wait.c:504
> #4  ___pthread_cond_timedwait64 (cond=0x7f26840016a8, mutex=0x8349e0, abstime=0x7f26a9a24730) at pthread_cond_wait.c:644
> #5  0x00007f26bbfcc9a4 in omni_condition::timedwait (this=0x7f26840016a0, secs=<optimized out>, nanosecs=<optimized out>)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:175
> #6  0x00007f26bbf03ec5 in omni_condition::timedwait (t=..., this=<optimized out>) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omnithread.h:363
> #7  omni::giopStream::sleepOnRdLockAlways (this=0x7f2684001748) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:392
> #8  0x00007f26bbf17334 in omni::giopImpl12::inputReplyBegin (g=0x7f2684001748, unmarshalHeader=0x7f26bbf1aa90 <omni::giopImpl12::unmarshalReplyHeader(omni::giopStream*)>)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:557
> #9  0x00007f26bbf0a85c in omni::GIOP_C::ReceiveReply (this=0x7f2684001740) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/GIOP_C.cc:126
> #10 0x00007f26bbeef2a6 in omniRemoteIdentity::dispatch (this=0x7f2684000d70, call_desc=...) at /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/IOP_C.h:137
> #11 0x00007f26bbed83ec in omniObjRef::_invoke (this=0x7f2684001208, call_desc=..., do_assert=false) at 
> /home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/omniObjRef.h:202
> #12 0x00007f26bbed8593 in omniObjRef::_remote_is_a (this=<optimized out>, a_repoId=<optimized out>)
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:343
> #13 0x00007f26bbed87e3 in omniObjRef::_realNarrow (this=0x7f2684001208, repoId=0x6d88c0 "IDL:exchangeAgent/eaServer:1.0")
>     at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:174
> #14 0x00000000005eae9b in exchangeAgent::eaServer::_narrow (obj=<optimized out>) at libidl/EAGeneral_skel.cpp:845
> #15 0x0000000000526687 in LS::narrow_with_timeout<exchangeAgent::eaServer> (timeout=5000, x=0x7f2684001260) at 
> /home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:112
> #16 LS::UsedEA::run (this=0x89e120) at /home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:576
> #17 0x00000000006ca225 in omniJTCThread::entrance_hook (this=0x89e330) at /home/build/Builds/Trader-build1605/Trader/src/libomniJTC/Thread.cpp:1037
> #18 0x00007f26bbfcd14e in omni_thread_wrapper (ptr=0x89e7e0) at /home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:449
> #19 0x00007f26baa9f802 in start_thread (arg=<optimized out>) at pthread_create.c:443
> #20 0x00007f26baa3f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>
>
> The object reference it tries to narrow is from an url like this
>
> corbaloc:ssliop:rhel9build:33344/ExchangeAgent
>
>
> The process providing this url was in the middle of starting up. It creates its main server object using a POA with
>
> omniPolicy::PLAIN_OBJECT_KEYS_ENABLE and BiDirPolicy::BIDIRECTIONAL_POLICY_TYPE
>
> and a fixed port so it can be used as URL.
>
> There must be some small window  where the port is already open and ssl handshake can done,  but ssl  malfunctioning and sending wrong data.
>
> Special omniorb options used:
>
> "serverTransportRule", "* unix,tcp,ssl,bidir", "acceptBiDirectionalGIOP", "1",
>
> "clientTransportRule", "* unix,tcp,ssl,bidir", "offerBiDirectionalGIOP", "1",
>
> "connectionWatchImmediate", "1",
>
> My questions are, is this really an openssl bug? Can something bad happen in omniORB initialization as well?
>
> Any ideas on how to debug this further are welcome. I'll try raising traceLevel, but it might be a while until it happens again.
>
>
> Regards,
>
>   Michael
>
>
>
>
>
>
>
> _______________________________________________
> omniORB-list mailing list
> omniORB-list at omniorb.net
> https://www.omniorb-support.com/mailman/listinfo/omniorb-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.omniorb-support.com/pipermail/omniorb-list/attachments/20240221/4575a75d/attachment.html>