<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi again,</p>
<p><br>
</p>
<p>unfortunately this is an omniORB problem with bidirectional and
ssl. By using a mutex around all calls using <span
style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;"></span><span
style="color:#0000c0;background-color:#d4d4d4;">pd_ssl</span></span></span></p>
<p>in sslConnection.cc I found out that Recv snd Send are then used
at the same time by different threads. Recv is used</p>
<div class="moz-cite-prefix">sometimes in blocking mode so simply
using a mutex is not the solution as omniorb will than hang
completely.</div>
<div class="moz-cite-prefix">The only option for us to use ssl now
is to switch bidirectional CORBA off. <br>
</div>
<div class="moz-cite-prefix">It's just strange that this does not
happen more often...</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Greetings,</div>
<div class="moz-cite-prefix"> Michael<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2/21/24 14:13, Michael Teske via
omniORB-list wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8842bd53-534f-47d6-bf35-45d46f2cd080@teskor.de">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hi,<br>
</p>
<p><br>
</p>
<p>I've created an issue <br>
</p>
<p><a class="moz-txt-link-freetext"
href="https://github.com/openssl/openssl/issues/23650"
moz-do-not-send="true">https://github.com/openssl/openssl/issues/23650</a></p>
<p>but now the question arose if omniORB does ensure to use a ssl
connection only from one thread. I looks like it's so, but maybe
there's a bug somewhere?</p>
<p><br>
</p>
<p>Regards,</p>
<p> Michael<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 2/21/24 10:40, Michael Teske via
omniORB-list wrote:<br>
</div>
<blockquote type="cite"
cite="mid:85c50f40-d441-44e3-a6ea-6531067c6426@teskor.de">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<p>Hi,</p>
<p>as required by our customers, we switched to use omniorb with
ssl. This works fine so far,</p>
<p>but on our machine with openssl3 (running RHEL9), we get
crashes from time to time looking like this:</p>
<p>(gdb) where<br>
#0 0x00007f26bb0e14df in tls_get_message_header
(mt=<synthetic pointer>, s=0x7f2684001940) at
ssl/statem/statem_lib.c:1167<br>
#1 read_state_machine (s=0x7f2684001940) at
ssl/statem/statem.c:587<br>
#2 state_machine (s=0x7f2684001940, server=<optimized
out>) at ssl/statem/statem.c:442<br>
#3 0x00007f26bb0d2969 in ssl3_read_bytes (s=<optimized
out>, type=23, recvd_type=0x0, buf=0x7f25b0000d48 "",
len=8192, peek=0, readbytes=0x7f25e69f39f0) at
ssl/record/rec_layer_s3.c:1711<br>
#4 0x00007f26bb0ab7fc in ssl3_read_internal
(s=0x7f2684001940, buf=0x7f25b0000d48, len=8192, peek=0,
readbytes=0x7f25e69f39f0) at ssl/s3_lib.c:4462<br>
#5 0x00007f26bb0b2137 in SSL_read (s=<optimized out>,
buf=<optimized out>, num=<optimized out>) at
ssl/ssl_lib.c:1885<br>
#6 0x00007f26bbfbd1bf in omni::sslConnection::Recv
(this=0x7f2684016b78, buf=0x7f25b0000d48, sz=<optimized
out>, deadline=...)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/ssl/sslConnection.cc:246<br>
#7 0x00007f26bbf04b88 in omni::giopStream::inputMessage
(this=this@entry=0x7f25b0000b68) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:841<br>
#8 0x00007f26bbf175a3 in
omni::giopImpl12::inputNewServerMessage (g=0x7f25b0000b68) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:438<br>
#9 0x00007f26bbf184a8 in omni::giopImpl12::inputMessageBegin
(g=0x7f25b0000b68, unmarshalHeader=<optimized out>)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:646<br>
#10 0x00007f26bbf0bc81 in omni::GIOP_S::dispatcher
(this=0x7f25b0000b60) at
/home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/internal/giopStream.h:143<br>
#11 0x00007f26bbf096b5 in omni::giopWorker::execute
(this=0x7f2684017db0) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopWorker.cc:77<br>
#12 0x00007f26bbec38d0 in omniAsyncWorker::real_run
(this=0x7f268401a490) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:578<br>
#13 0x00007f26bbec487f in omniAsyncPoolServer::workerRun
(this=<optimized out>, worker=<optimized out>)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:328<br>
#14 0x00007f26bbec3499 in omniAsyncWorker::mid_run
(this=0x7f268401a490) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:511<br>
#15 0x00007f26bbec47de in omniAsyncWorker::run
(this=0x7f268401a490) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/invoker.cc:126<br>
#16 0x00007f26bbfcd1d5 in omni_thread_wrapper
(ptr=0x7f268401a490) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:459<br>
#17 0x00007f26baa9f802 in start_thread (arg=<optimized
out>) at pthread_create.c:443<br>
#18 0x00007f26baa3f450 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
</p>
<p>I did dnf debuginfo-install openssl-libs-3.0.7-24.el9.x86_64
and am thus able to look into openssl source. It looks as if
it gets wrong data in ssl3_read_bytes <br>
</p>
<p>and crashes then because s->init_buf is NULL, probably
already resetted in <span><span
class="ui-provider ed axc axd axe axf axg axh axi axj axk axl axm axn axo axp axq axr axs axt axu axv axw axx axy axz aya ayb ayc ayd aye ayf ayg ayh ayi ayj"
dir="ltr">tls_finish_handshake().</span></span></p>
<p>In the source code around ssl/record/rec_layer_s3.c:1711 it
says:</p>
<div style="background-color:#f7f7f7;padding:0px 0px 0px 2px;">
<div
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><p
style="margin:0;"><span style="color:#000000;"> </span><span
style="color:#3f7f5f;">/*</span></p><p style="margin:0;"><span
style="color:#3f7f5f;"> * Unexpected handshake message (ClientHello, NewSessionTicket (TLS1.3) or</span></p><p
style="margin:0;"><span style="color:#3f7f5f;"> * protocol violation)</span></p><p
style="margin:0;"><span style="color:#3f7f5f;"> */</span></p><p
style="margin:0;"><span style="color:#000000;"> </span><span
style="color:#7f0055;font-weight:bold;">if</span><span
style="color:#000000;"> ((s-></span><span
style="color:#0000c0;">rlayer</span><span
style="color:#000000;">.</span><span style="color:#0000c0;">handshake_fragment_len</span><span
style="color:#000000;"> >= 4)</span></p><p
style="margin:0;"><span style="color:#000000;"> && !ossl_statem_get_in_handshake(s)) {</span></p><p
style="margin:0;"><span style="color:#000000;"> </span><span
style="color:#7f0055;font-weight:bold;">int</span><span
style="color:#000000;"> ined = (s-></span><span
style="color:#0000c0;">early_data_state</span><span
style="color:#000000;"> == </span><span
style="color:#0000c0;font-style:italic;">SSL_EARLY_DATA_READING</span><span
style="color:#000000;">);</span></p><p style="margin:0;">
</p><p style="margin:0;"><span style="color:#000000;"> </span><span
style="color:#3f7f5f;">/* We found handshake data, so we're going back into </span><span
style="color:#3f7f5f;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">init</span><span
style="color:#3f7f5f;"> */</span></p><p style="margin:0;"><span
style="color:#000000;"> </span><span
style="color:#000000;background-color:#d4d4d4;">ossl_statem_set_in_init</span><span
style="color:#000000;">(s, 1);</span></p><p style="margin:0;">
</p><p style="margin:0;"><span style="color:#000000;"> i = s-></span><span
style="color:#0000c0;">handshake_func</span><span
style="color:#000000;">(s);</span></p><p style="margin:0;">
</p></div>
</div>
<p>(s->handshake_func is ossl_statem_connect which calls
state_machine(s,0)) . While this looks like an openssl bug to
me (it should call SSL_Fatal or re-create the buffer),</p>
<p>the question remains, why does this happen?</p>
<p>The thread causing this is here:</p>
<p>(gdb) where<br>
#0 __futex_abstimed_wait_common64 (private=0, cancel=true,
abstime=0x7f26a9a24730, op=393, expected=0,
futex_word=0x7f26840016d0) at futex-internal.c:57<br>
#1 __futex_abstimed_wait_common
(futex_word=futex_word@entry=0x7f26840016d0,
expected=expected@entry=0, clockid=clockid@entry=0,
abstime=abstime@entry=0x7f26a9a24730, <br>
private=private@entry=0, cancel=cancel@entry=true) at
futex-internal.c:87<br>
#2 0x00007f26baa9c3ff in
__GI___futex_abstimed_wait_cancelable64
(futex_word=futex_word@entry=0x7f26840016d0,
expected=expected@entry=0, clockid=clockid@entry=0, <br>
abstime=abstime@entry=0x7f26a9a24730,
private=private@entry=0) at futex-internal.c:139<br>
#3 0x00007f26baa9eea4 in __pthread_cond_wait_common
(abstime=0x7f26a9a24730, clockid=0, mutex=0x8349e0,
cond=0x7f26840016a8) at pthread_cond_wait.c:504<br>
#4 ___pthread_cond_timedwait64 (cond=0x7f26840016a8,
mutex=0x8349e0, abstime=0x7f26a9a24730) at
pthread_cond_wait.c:644<br>
#5 0x00007f26bbfcc9a4 in omni_condition::timedwait
(this=0x7f26840016a0, secs=<optimized out>,
nanosecs=<optimized out>)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:175<br>
#6 0x00007f26bbf03ec5 in omni_condition::timedwait (t=...,
this=<optimized out>) at
/home/michael.teske/build/thirdparty/trader/omniORB/include/omnithread.h:363<br>
#7 omni::giopStream::sleepOnRdLockAlways
(this=0x7f2684001748) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopStream.cc:392<br>
#8 0x00007f26bbf17334 in omni::giopImpl12::inputReplyBegin
(g=0x7f2684001748, unmarshalHeader=0x7f26bbf1aa90
<omni::giopImpl12::unmarshalReplyHeader(omni::giopStream*)>)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/giopImpl12.cc:557<br>
#9 0x00007f26bbf0a85c in omni::GIOP_C::ReceiveReply
(this=0x7f2684001740) at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/GIOP_C.cc:126<br>
#10 0x00007f26bbeef2a6 in omniRemoteIdentity::dispatch
(this=0x7f2684000d70, call_desc=...) at
/home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/IOP_C.h:137<br>
#11 0x00007f26bbed83ec in omniObjRef::_invoke
(this=0x7f2684001208, call_desc=..., do_assert=false) at
/home/michael.teske/build/thirdparty/trader/omniORB/include/omniORB4/omniObjRef.h:202<br>
#12 0x00007f26bbed8593 in omniObjRef::_remote_is_a
(this=<optimized out>, a_repoId=<optimized out>)<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:343<br>
#13 0x00007f26bbed87e3 in omniObjRef::_realNarrow
(this=0x7f2684001208, repoId=0x6d88c0
"IDL:exchangeAgent/eaServer:1.0")<br>
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omniORB/orbcore/omniObjRef.cc:174<br>
#14 0x00000000005eae9b in exchangeAgent::eaServer::_narrow
(obj=<optimized out>) at libidl/EAGeneral_skel.cpp:845<br>
#15 0x0000000000526687 in
LS::narrow_with_timeout<exchangeAgent::eaServer>
(timeout=5000, x=0x7f2684001260) at
/home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:112<br>
#16 LS::UsedEA::run (this=0x89e120) at
/home/build/Builds/Trader-build1605/Trader/src/LoginServer/EAInfo.cpp:576<br>
#17 0x00000000006ca225 in omniJTCThread::entrance_hook
(this=0x89e330) at
/home/build/Builds/Trader-build1605/Trader/src/libomniJTC/Thread.cpp:1037<br>
#18 0x00007f26bbfcd14e in omni_thread_wrapper (ptr=0x89e7e0)
at
/home/michael.teske/build/thirdparty/trader/omniORB/src/lib/omnithread/posix.cc:449<br>
#19 0x00007f26baa9f802 in start_thread (arg=<optimized
out>) at pthread_create.c:443<br>
#20 0x00007f26baa3f450 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
</p>
<p><br>
</p>
<p>The object reference it tries to narrow is from an url like
this</p>
<p>corbaloc:ssliop:rhel9build:33344/ExchangeAgent</p>
<p><br>
</p>
<p>The process providing this url was in the middle of starting
up. It creates its main server object using a POA with</p>
<p><span
style="background-color:#f5f5b5;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f5f5b5;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;">omniPolicy::PLAIN_OBJECT_KEYS_ENABLE and </span></span></span><span
style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;">BiDirPolicy::</span><span
style="color:#000000;background-color:#f0d8a8;">BIDIRECTIONAL_POLICY_TYPE</span></span></span></p>
<p>and a fixed port so it can be used as URL. <br>
</p>
<p>There must be some small window where the port is already
open and ssl handshake can done, but ssl malfunctioning and
sending wrong data.</p>
<p>Special omniorb options used:</p>
<div style="background-color:#f7f7f7;padding:0px 0px 0px 2px;">
<div
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><p
style="margin:0;"><span style="color:#000000;"></span><span
style="color:#000000;"> </span><span style="color:#2a00ff;">"serverTransportRule"</span><span
style="color:#000000;">, </span><span
style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;"></span><span style="color:#2a00ff;">"* </span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">unix</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">tcp</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">ssl</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">bidir</span><span
style="color:#2a00ff;">"</span></span></span><span
style="color:#000000;">, </span><span style="color:#2a00ff;">"acceptBiDirectionalGIOP"</span><span
style="color:#000000;">, </span><span style="color:#2a00ff;">"1"</span><span
style="color:#000000;">,</span></p><p style="margin:0;"><span
style="color:#000000;"> </span><span style="color:#2a00ff;">"clientTransportRule"</span><span
style="color:#000000;">, </span><span
style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;"></span><span style="color:#2a00ff;">"* </span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">unix</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">tcp</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">ssl</span><span
style="color:#2a00ff;">,</span><span
style="color:#2a00ff;text-decoration:underline;text-decoration-color:#ff8040;text-decoration-style:wavy;">bidir</span><span
style="color:#2a00ff;">"</span></span></span><span
style="color:#000000;">, </span><span style="color:#2a00ff;">"offerBiDirectionalGIOP"</span><span
style="color:#000000;">, </span><span style="color:#2a00ff;">"1"</span><span
style="color:#000000;">,</span></p><p style="margin:0;"><span
style="color:#000000;"> </span><span style="color:#2a00ff;">"connectionWatchImmediate"</span><span
style="color:#000000;">, </span><span style="color:#2a00ff;">"1"</span><span
style="color:#000000;">,</span></p><p style="margin:0;"><span
style="color:#000000;"> </span></p></div>
</div>
<p>My questions are, is this really an openssl bug? Can
something bad happen in omniORB initialization as well? <br>
</p>
<p>Any ideas on how to debug this further are welcome. I'll try
raising traceLevel, but it might be a while until it happens
again.</p>
<p><br>
</p>
<p>Regards,</p>
<p> Michael<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
omniORB-list mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:omniORB-list@omniorb.net" moz-do-not-send="true">omniORB-list@omniorb.net</a>
<a class="moz-txt-link-freetext"
href="https://www.omniorb-support.com/mailman/listinfo/omniorb-list"
moz-do-not-send="true">https://www.omniorb-support.com/mailman/listinfo/omniorb-list</a>
</pre>
</blockquote>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
omniORB-list mailing list
<a class="moz-txt-link-abbreviated" href="mailto:omniORB-list@omniorb.net">omniORB-list@omniorb.net</a>
<a class="moz-txt-link-freetext" href="https://www.omniorb-support.com/mailman/listinfo/omniorb-list">https://www.omniorb-support.com/mailman/listinfo/omniorb-list</a>
</pre>
</blockquote>
</body>
</html>