<div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div>Hello Experts</div><div><br></div><div>Looking for some help on a problem that I am facing with detecting a peer loss in OmniORB4.1.6, which was not an issue in ORB4.1.3.</div>
<div><br></div><div>Here is my setup :</div>
<div>-----------------------------</div><div>-----------</div><div>Setup : </div><div>-----------</div><div><br></div><div>Have 2 systems ( A and B ) that are talking over IP in point to point wired link.</div><div><br></div>
<div>Have identical daemon on both sides that is hosting its services using OmniORB4.1.6 infrastructure.</div><div><br></div><div>Have a Ping code that is a part of this daemon itself that will see if the other identical daemon is running or not by trying to reset some of remote daemon's vailables.</div>
<div><br></div><div>----------</div><div>Code ::</div><div>------------</div><div><br></div><div>1/. Ping system A from B.</div><div>2/. Sleep 15 seconds</div><div>3/. Goto step-1 and ping again and this goes on in a loop..</div>
<div><br></div><div>This functionality is working perfecet.</div><div><br></div><div>However, if we have certain condition like :</div><div><br></div><div>1/. Ping system A from B. --> Lets say this goes okay..</div><div>
2/. Started sleeping for 15 seconds.</div><div>3/. system-B goes away ( say it did a reset ) and since its a point to point link, the link on this A is also unloaded.</div><div>4/. Sleep of 15 seconds finished</div><div>
5/. Now the ping call that is destined for system B must return immediately since the network itself is unloaded.</div>
<div><br></div><div>OR</div><div><br></div><div>1/. Ping system A from B. --> Lets say this goes okay..</div><div>2/. Sleep for 15 seconds.</div><div>3/. System-B started resetting and is dumping.</div><div>4/. Attempt to Ping system-B</div>
<div>5/. Now the ping call that is destined for system B must return immediately or sooner since the network itself is unloaded.</div><div><br></div><div><br></div><div>==============</div>
<div><br></div><div>I see this is working perfect in ORB4.1.3 -but- in Orb4.1.6, I see A's ORB takes a lot of time ( in the order of 200+ seconds ) to actually return to the ping code saying it failed.</div><div>It even waits for B to come up and then fail the ping</div>
<div><br></div><div>================</div><div><br></div><div>The Message that are displayed when traces enabled at 40 are as follows</div><div><br></div><div>============</div>
<div>**************</div><div>In 4.1.3 :</div><div>**************</div><div><br></div><div><br></div><div>00001509.838454| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.804590: Invoke 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div>
<div>00001509.838454| 1776|TRACE | 110|</div><div>00001509.838844| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.804912: sendChunk: to giop:tcp:x.y.z.a:90000 80 bytes</div><div>00001509.838844| 1776|TRACE | 110|</div>
<div>00001509.839234| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.805266:</div><div>00001509.839234| 1776|TRACE | 110|4749 4f50 0102 0000 0000 0044 0000 0496 GIOP.......D....</div><div>00001509.839234| 1776|TRACE | 110|0300 0000 0000 0000 0000 001a ff46 6970 .............Fip</div>
<div>00001509.839234| 1776|TRACE | 110|7320 4d54 2050 4f41 004c 6f63 616c 2053 s MT POA.Local S</div><div>00001509.839234| 1776|TRACE | 110|6572 7669 6365 697a 0000 000f 6861 6e64 erviceiz....hand</div><div>
00001509.839234| 1776|TRACE | 110|6c65 5375 7276 5069 6e67 000f 0000 0000 leSurvPing......</div><div>00001509.839234| 1776|TRACE | 110|</div><div>00001509.846890| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.812898: inputMessage: from giop:tcp:x.y.z.a:90000 32 bytes</div>
<div>00001509.846890| 1776|TRACE | 110|</div><div>00001509.847166| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.813301:</div><div>00001509.847166| 1776|TRACE | 110|4749 4f50 0102 0001 0000 0014 0000 0496 GIOP............</div>
<div>00001509.847166| 1776|TRACE | 110|0000 0000 0000 0000 0000 0000 0fbe 3fa3 ..............?.</div><div>00001509.847166| 1776|TRACE | 110|</div><div>00001509.847479| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:43.813619: Return 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div>
<div>00001509.847479| 1776|TRACE | 110|</div><div><br></div><div>00001509.847548| 1776|TRACE | 267| Surveillance success, Communication IDs - Local: 0xFBE3FA3 Remote 0xFBE3FA3</div><div><br></div><div>##>> This says the ping went okay <<##</div>
<div><br></div><div>00001524.850535| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:58.816821: Invoke 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div><div>00001524.850535| 1776|TRACE | 110|</div>
<div>00001524.851322| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:58.817619: sendChunk: to giop:tcp:x.y.z.a:90000 80 bytes</div><div>00001524.851322| 1776|TRACE | 110|</div><div>00001524.851676| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:07:58.817899:</div>
<div>00001524.851676| 1776|TRACE | 110|4749 4f50 0102 0000 0000 0044 0000 049c GIOP.......D....</div><div>00001524.851676| 1776|TRACE | 110|0300 0000 0000 0000 0000 001a ff46 6970 .............Fip</div><div>
00001524.851676| 1776|TRACE | 110|7320 4d54 2050 4f41 004c 6f63 616c 2053 s MT POA.Local S</div><div>00001524.851676| 1776|TRACE | 110|6572 7669 6365 697a 0000 000f 6861 6e64 erviceiz....hand</div><div>00001524.851676| 1776|TRACE | 110|6c65 5375 7276 5069 6e67 000f 0000 0000 leSurvPing......</div>
<div>00001524.851676| 1776|TRACE | 110|</div><div><br></div><div>00001537.998148| 699| |1078|INFO: device0 removed because buddy clock was lost!</div><div>00001538.038406| 699| |1078|INFO: device2 removed because buddy clock was lost!</div>
<div>00001538.039077| 699| |1091|ERR: device_release[-18]</div><div><br></div><div>##>> Point to point link was unloaded since remote was reset <<##</div><div><br></div><div>00001540.542412| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:08:14.508872: Error in network receive (start of message): giop:tcp:x.y.z.a:90000</div>
<div><br></div><div>##>> Comm failure detected immediately <<##</div><div><br></div><div>00001540.542412| 1776|TRACE | 110|</div><div>00001540.542652| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:08:14.509137: throw giopStream::CommFailure from giopStream.cc:874(0,MAYBE,COMM_FAILURE_WaitingForReply)</div>
<div>00001540.542652| 1776|TRACE | 110|</div><div>00001540.543409| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:08:14.509867: Client connection refcount = 0</div><div>00001540.543409| 1776|TRACE | 110|</div>
<div>00001540.543624| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:08:14.510131: Client close connection to giop:tcp:x.y.z.a:90000</div><div>00001540.543624| 1776|TRACE | 110|</div><div>00001540.544152| 1776|TRACE | 110|INF> ORB: omniORB: (?) 2006-09-29 16:08:14.510594: throw COMM_FAILURE from omniObjRef.cc:792 (MAYBE,COMM_FAILURE_WaitingForReply)</div>
<div><br></div><div>=====================================================================================================================</div><div><br></div><div>--------------</div><div>In 4.1.6 :</div><div>-------------------</div>
<div><br></div><div>00000214.088587390| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.198456: Invoke 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div>
<div>00000214.088587390| 3124|TRACE | 114|</div><div>00000214.088659147| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.198588: sendChunk: to giop:tcp:x.y.z.a:90000 80 bytes</div><div>00000214.088659147| 3124|TRACE | 114|</div>
<div>00000214.088964359| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.198836:</div><div>00000214.088964359| 3124|TRACE | 114|4749 4f50 0102 0000 0000 0044 0000 0040 GIOP.......D...@</div><div>
00000214.088964359| 3124|TRACE | 114|0300 0000 0000 0000 0000 001a ff46 6970 .............Fip</div><div>00000214.088964359| 3124|TRACE | 114|7320 4d54 2050 4f41 004c 6f63 616c 2053 s MT POA.Local S</div><div>
00000214.088964359| 3124|TRACE | 114|6572 7669 6365 697a 0000 000f 6861 6e64 erviceiz....hand</div><div>00000214.088964359| 3124|TRACE | 114|6c65 5375 7276 5069 6e67 000f 0000 0000 leSurvPing......</div><div>
00000214.088964359| 3124|TRACE | 114|</div><div>00000214.092186617| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.202080: inputMessage: from giop:tcp:x.y.z.a:90000 32 bytes</div><div>00000214.092186617| 3124|TRACE | 114|</div>
<div>00000214.092253468| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.202168:</div><div>00000214.092253468| 3124|TRACE | 114|4749 4f50 0102 0001 0000 0014 0000 0040 GIOP...........@</div><div>
00000214.092253468| 3124|TRACE | 114|0000 0000 0000 0000 0000 0000 417a 5503 ............AzU.</div><div>00000214.092253468| 3124|TRACE | 114|</div><div>00000214.092315155| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:36:54.202246: Return 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div>
<div>00000214.092315155| 3124|TRACE | 114|</div><div>00000214.092336833| 3124|TRACE | 271| Surveillance success, Communication IDs - Local: 0x417A5503 Remote 0x417A5503</div><div><br></div><div>##>> Showing the Ping has gone okay <<##</div>
<div><br></div><div>00000229.092562595| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:37:09.202431: Invoke 'handleSurvPing' on remote: root/Fips MT POA<Local.Service></div><div>00000229.092562595| 3124|TRACE | 114|</div>
<div>00000229.092634882| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:37:09.202564: sendChunk: to giop:tcp:x.y.z.a:90000 80 bytes</div><div>00000229.092634882| 3124|TRACE | 114|</div><div>00000229.092733758| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:37:09.202613:</div>
<div>00000229.092733758| 3124|TRACE | 114|4749 4f50 0102 0000 0000 0044 0000 0042 GIOP.......D...B</div><div>00000229.092733758| 3124|TRACE | 114|0300 0000 0000 0000 0000 001a ff46 6970 .............Fip</div>
<div>00000229.092733758| 3124|TRACE | 114|7320 4d54 2050 4f41 004c 6f63 616c 2053 s MT POA.Local S</div><div>00000229.092733758| 3124|TRACE | 114|6572 7669 6365 697a 0000 000f 6861 6e64 erviceiz....hand</div>
<div>00000229.092733758| 3124|TRACE | 114|6c65 5375 7276 5069 6e67 000f 0000 0000 leSurvPing......</div><div>00000229.092733758| 3124|TRACE | 114|</div><div><br></div><div>##>> Another command in flight but the remote has started resetting so the call is not honored yet <<##</div>
<div><br></div><div>00000256.696070735| 738| |1104|INFO: device0 removed because buddy clock was lost!</div><div>00000256.917866762| 2991| | 709|ERR: device_write[-18]</div><div>00000257.208085276| 738| |1104|INFO: device2 removed because buddy clock was lost!</div>
<div><br></div><div>##>> At 257 seconds is when the point ti point module as unloaded but ORB did not detect the loss <<##</div><div><br></div><div>00000257.208401978| 738| 1117|ERR: device_release[-18]</div>
<div>00000272.826223240| 3453| |1037|INFO: device2 open, ip (ifconfig up)</div><div>00000272.833337367| 3452| |1037|INFO: device0 open, ip (ifconfig up)</div><div><br></div><div>##>> Remote system even came back on and we loaded the point to point module since we detected the link again.. <<##</div>
<div><br></div><div>00000441.880148020| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:40:41.990036: Error in network receive (start of message): giop:tcp:x.y.z.a:90000</div><div>00000441.880148020| 3124|TRACE | 114|</div>
<div><br></div><div>##> It did wait really a long long time to detect the failure since the loss of remote <<##</div><div><br></div><div>00000441.880201124| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:40:41.990128: throw giopStream::CommFailure from giopStream.cc:879(0,MAYBE,COMM_FAILURE_WaitingForReply)</div>
<div>00000441.880201124| 3124|TRACE | 114|</div><div>00000441.880471357| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:40:41.990401: Client connection refcount = 0</div><div>00000441.880471357| 3124|TRACE | 114|</div>
<div>00000441.880511072| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:40:41.990448: Client close connection to giop:tcp:x.y.z.a:90000</div><div>00000441.880511072| 3124|TRACE | 114|</div><div>00000441.880628325| 3124|TRACE | 114|INF> ORB: omniORB: (?) 2009-02-19 06:40:41.990557: throw COMM_FAILURE from omniObjRef.cc:804 (MAYBE,COMM_FAILURE_WaitingForReply)</div>
<div>00000441.880628325| 3124|TRACE | 114|</div><div><br></div><div><br></div><div>Interpreting :: 00000441.880628325 ==> 441 seconds -and- 880628325 microseconds since the boot</div><div><br></div><div>==========</div>
<div><br></div><div>Its Very easy to recreate this issue at will.</div><div><br></div><div>==============</div><div><br></div><div>Here is the Config files :</div><div><br></div><div>*********************</div>
<div>CONFIG for 4.1.6 :</div><div>*********************</div><div><br></div><div>maxGIOPVersion = 1.2</div><div><br></div><div>strictIIOP = 1<br></div>
<div><br></div><div>lcdMode = 0</div><div><br></div><div>tcAliasExpand = 0</div><div><br></div><div>useTypeCodeIndirections = 1</div><div><br></div><div>acceptMisalignedTcIndirections = 0</div><div><br></div><div>scanGranularity = 5</div>
<div><br></div><div>nativeCharCodeSet = ISO-8859-1</div><div><br></div><div>nativeWCharCodeSet = UTF-16</div><div><br></div><div>abortOnInternalError = 0</div><div><br></div><div>abortOnNativeException = 0</div><div><br>
</div>
<div>InitRef = NameService_LOCAL=corbaname:omniunix:/tmp/forb.socket.fsp.local.omninames</div><div>#deleted some of the internal IPS</div><div><br></div><div>clientTransportRule = * unix,tcp</div>
<div><br></div><div>clientCallTimeOutPeriod = 0</div><div><br></div><div>clientConnectTimeOutPeriod = 15000</div><div><br></div><div>supportPerThreadTimeOut = 0</div><div><br></div><div>resetTimeOutOnRetries = 0</div><div>
<br></div><div>outConScanPeriod = 30</div><div><br></div><div>maxGIOPConnectionPerServer = 5</div><div><br></div><div>oneCallPerConnection = 0</div><div><br></div><div>offerBiDirectionalGIOP = 0</div><div><br></div><div>
diiThrowsSysExceptions = 0</div>
<div><br></div><div>verifyObjectExistsAndType = 0</div><div><br></div><div>giopTargetAddressMode = 0</div><div><br></div><div>immediateAddressSwitch = 0</div><div><br></div><div>serverTransportRule = * unix,tcp</div>
<div><br></div><div>serverCallTimeOutPeriod = 0</div><div><br></div><div>inConScanPeriod = 0</div><div><br></div><div>threadPerConnectionPolicy = 0</div><div><br></div><div>maxServerThreadPerConnection = 100</div><div><br>
</div><div><br></div><div>maxServerThreadPoolSize = 250</div><div><br></div><div>threadPerConnectionUpperLimit = 10000</div><div><br></div><div>threadPerConnectionLowerLimit = 9000</div><div><br></div><div><br></div><div>
threadPoolWatchConnection = 0</div><div><br></div><div><br></div><div>connectionWatchPeriod = 1</div><div><br></div><div>connectionWatchImmediate = 0</div><div><br></div><div>acceptBiDirectionalGIOP = 0</div><div><br></div>
<div>unixTransportDirectory = /tmp/</div><div><br></div><div>unixTransportPermission = 0777</div><div><br></div><div>supportCurrent = 0</div><div><br></div><div>copyValuesInLocalCalls = 1</div><div><br></div><div>objectTableSize = 0</div>
<div><br></div><div>poaHoldRequestTimeout = 0</div><div><br></div><div>poaUniquePersistentSystemIds = 1</div><div><br></div><div>supportBootstrapAgent = 0</div><div><br></div><div>===========================</div>
<div><br></div><div>*********************</div><div>CONFIG for 4.1.6 :</div><div>*********************</div><div><br></div><div>maxGIOPVersion = 1.2<br></div><div><br></div><div>strictIIOP = 1</div>
<div><br></div><div>lcdMode = 0</div><div><br></div><div>tcAliasExpand = 0</div><div><br></div><div>useTypeCodeIndirections = 1</div>
<div><br></div><div>acceptMisalignedTcIndirections = 0</div><div><br></div><div>scanGranularity = 5</div><div><br></div><div>nativeCharCodeSet = ISO-8859-1</div><div><br></div><div>nativeWCharCodeSet = UTF-16</div><div><br>
</div><div>abortOnInternalError = 0</div><div><br></div><div>abortOnNativeException = 0</div><div><br></div><div>InitRef = NameService_LOCAL=corbaname:omniunix:/tmp/forb.socket.fsp.local.omninames</div><div><br></div><div>
clientTransportRule = *unix,tcp<br></div><div><br></div><div>clientCallTimeOutPeriod = 0</div><div><br></div><div>clientConnectTimeOutPeriod = 15000</div><div><br></div><div>supportPerThreadTimeOut = 0</div><div>
<br></div><div>resetTimeOutOnRetries = 0</div><div><br></div><div>outConScanPeriod = 30</div><div><br></div><div>maxGIOPConnectionPerServer = 5</div><div><br></div><div>oneCallPerConnection = 0</div><div><br></div><div>offerBiDirectionalGIOP = 0</div>
<div><br></div><div>diiThrowsSysExceptions = 0</div><div><br></div><div>verifyObjectExistsAndType = 0</div><div><br></div><div>giopTargetAddressMode = 0</div><div><br></div><div>immediateAddressSwitch = 0</div><div><br></div>
<div>serverTransportRule = *unix,tcp</div><div><br></div><div>serverCallTimeOutPeriod = 0</div><div><br></div><div>inConScanPeriod = 0</div><div><br></div><div>threadPerConnectionPolicy = 0</div><div><br></div><div>maxServerThreadPerConnection = 100</div>
<div><br></div><div>maxServerThreadPoolSize = 50</div><div><br></div><div>threadPerConnectionUpperLimit = 10000</div><div><br></div><div>threadPerConnectionLowerLimit = 9000</div><div><br></div><div>threadPoolWatchConnection = 0<br>
</div><div><br></div><div>connectionWatchPeriod = 1</div><div><br></div><div>connectionWatchImmediate = 0</div><div><br></div><div>acceptBiDirectionalGIOP = 0</div><div><br></div><div>unixTransportDirectory = /tmp/</div>
<div><br></div><div>unixTransportPermission = 0777</div><div><br></div><div>supportCurrent = 0</div><div><br></div><div>copyValuesInLocalCalls = 1</div><div><br></div><div>objectTableSize = 0</div><div><br></div><div>poaHoldRequestTimeout = 0</div>
<div><br></div><div>poaUniquePersistentSystemIds = 1</div><div><br></div><div>supportBootstrapAgent = 0</div><div><br></div><div>===================</div><div><br></div><div>!! THANK YOU VERY MUCH !!</div>
<div><br></div><div>Really appreciate any help in resolving this.</div></div>
</div><br></div>