[omniORB] location forward to stale reference - strangeness
baileyk at schneider.com
baileyk at schneider.com
Fri Mar 12 13:57:34 GMT 2004
I captured several stack traces. One before the client makes the call, a
few while the client was waiting and another after the client timed out.
All the traces look almost identical. I'm using pstack on Solaris 8 to
grab the stack trace of the server (the one doing the redirection to a
stale reference). This server is using omniORBpy.
I couldn't decipher what the ORB was up to, but I'll include the first
stack traces captured after the client started waiting for a response.
The behavior seems like omniORB just stopped monitoring the connection
altogether. If you could point me to the location in omniORB first
executed in response to an incoming request, I can set a breakpoint there
and see if it get's triggered at all. As before, there's no trace data on
the server to suggest the client's second attempt is picked up by the
server. I'll attempt to manually trace what happens to a connection just
after a ForwardRequest exception is translated into a LOCATION_FORWARD
response...
Thanks,
Kendall
----------------- lwp# 1 / thread# 1 --------------------
fee9f054 lwp_sema_wait (dffa8)
ff249ac4 _park (dffa8, ff26e000, 0, dfee8, 24d84, fdb05d70) + 114
ff24978c _swtch (dfee8, 0, ff26e000, 5, 1000, 0) + 424
ff247e10 cond_reltimedwait (0, 0, 0, 1, 0, 0) + 1f8
ff247c08 cond_timedwait (fe3ec4e8, fe3ec4a8, ffbee8d8, fe3ec4e8, 0, 0) +
2c
ff247b70 pthread_cond_timedwait (fe3ec4e8, fe3ec4a8, ffbee8d8, 0,
fecb43b4, 0) + c
feca1fb8 __1cOomni_conditionJtimedwait6MLL_i_ (fe3ec4e0, 405210c6,
317314fe, 25c, 2800, 29c0) + 40
fe2eb008 __1cEomniPORBAsyncInvokerHperform6MLL_v_ (fe3ec4a8, 405210c6,
317314fe, 0, 25c8, fe3ec504) + ac
fe2e9fb8 __1cKomniOrbORBLrun_timeout6MLL_b_ (25a7c0, 405210c6, 317314fe,
0, 0, 317314fe) + dc
fed04394 pyORB_run_timeout (0, 0, fed543f0, d0, fed3e711, 0) + 16c
0004553c eval_frame (231b90, 0, 0, 1bae70, 0, 2) + 3b44
00046c78 PyEval_EvalCodeEx (0, 3, 231b90, 1129b8, 0, 0) + 960
00048208 fast_function (1e6308, ffbeec30, 1, 1, 0, 1) + 58
00045690 eval_frame (112830, 0, 0, 14b440, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, b899c, 112830, c, 0, 0) + 960
00048208 fast_function (1b2d00, ffbeedb0, 1, 1, 0, cc400) + 58
00045690 eval_frame (13b0e0, 0, 0, 130940, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, a, 13b0e0, f3620, 3, 0) + 960
00048208 fast_function (f27b8, ffbeef30, 0, 0, 0, cc400) + 58
00045690 eval_frame (f34d0, 0, 0, e8068, 0, 0) + 3c98
00046c78 PyEval_EvalCodeEx (0, 0, f34d0, 0, 0, 0) + 960
00041784 PyEval_EvalCode (135ac8, f1a28, f1a28, 62a30, 0, 0) + 2c
00062a50 run_node (e5258, ffbef35b, f1a28, f1a28, ffbef154, 1) + 38
00061abc PyRun_SimpleFileExFlags (dfd90, ffbef35b, 1, ffbef154, 1, 1) +
100
0001ba48 Py_Main (2, ffbef1bc, ffbef1c8, 0, dfd90, dfc00) + 748
0001b158 _start (0, 0, 0, 0, 0, 0) + 108
----------------- lwp# 2 / thread# 2 --------------------
fee9e9a4 signotifywait ()
ff24ed54 _dynamiclwps (ff26e000, 0, 0, feebf500, 0, 0) + 1c
ff252030 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 3 / thread# 6 --------------------
fee9d1e0 poll (fdc07988, 2, 32)
fee4d1a0 select (7, 0, 0, fdc07998, feebf1a4, fdc07988) + 348
ff25b134 select (228898, 50, fe3caea8, 0, 40, 1) + 34
fe3910a8
__1cEomniLtcpEndpointQAcceptAndMonitor6MpFpvpn0AOgiopConnection__v2_4_
(228890, fe358860, 32fb00, fe3ee364, 137b28, 0) + 64
fe358938 __1cEomniQgiopRendezvouserHexecute6M_v_ (32fb00, ff26e000,
32fb00, 800, a10, 0) + b4
fe307d58 __1cPomniAsyncWorkerIreal_run6M_v_ (330428, 1, 0, 247308, 32fb00,
fe3f0090) + 21c
fe307f40 __1cPomniAsyncWorkerDrun6Mpv_v_ (330428, 0, ff26e000, 0, 330428,
0) + 40
feca2658 omni_thread_wrapper (330428, fecb3f94, 0, 5, 1, fe401000) + d4
ff25b728 _thread_start (330428, 0, 0, 0, 0, 0) + 40
----------------- lwp# 4 --------------------------------
ff259770 lwp_cond_wait (ff275548, ff275558, ff26edb0)
ff2490ac _age (3e, ff26ed9c, ff26e000, ff27ad8c, 0, fe401000) + 74
ff249030 _qswtch (fde0bd70, feb93d10, 0, 5, 1, fe401000) + 118
----------------- lwp# 5 / thread# 5 --------------------
fee9f054 lwp_sema_wait (ff26fa08)
ff248d04 _co_timerset (ff26ed30, ff26e000, 1, 3, ff26e000, 0) + f4
ff25b728 _thread_start (0, 0, 0, 0, 0, 0) + 40
----------------- lwp# 6 / thread# 8 --------------------
fee9d1e0 poll (fda02ff0, 0, 32)
fee4d1a0 select (0, 0, 0, fda02ff0, feebf1a4, fda02ff0) + 348
ff25b134 select (3fa99999, 9999999a, fda030e8, dd000, 1, fedf2c98) + 34
fede1768 time_sleep (0, 3241b8, 0, 14a, 1860b0, 0) + 30
0004553c eval_frame (185f38, 0, 0, 147138, 0, 1) + 3b44
00046c78 PyEval_EvalCodeEx (0, 9, 185f38, 186274, 0, 0) + 960
00048208 fast_function (158a90, fda03300, 2, 2, 0, 2) + 58
00045690 eval_frame (186108, 0, 0, 10b068, 0, 2) + 3c98
00046c78 PyEval_EvalCodeEx (0, 8, 186108, 1730a0, 0, 0) + 960
00048208 fast_function (3145f8, fda03480, 1, 1, 0, 1) + 58
00045690 eval_frame (172f50, 0, 0, 31b720, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, 1, 172f50, 3243cc, 0, 0) + 960
0008c16c function_call (314470, 3243c0, 159500, 8c010, d3000, 5f5f636c) +
15c
0007b9b4 PyObject_Call (314470, 3243c0, 159500, dd000, 1, 4) + 1c
00082354 instancemethod_call (322a08, e01f0, 159500, 314470, c1000,
14bfb8) + 180
0007b9b4 PyObject_Call (322a08, e01f0, 159500, fda03810, fda0380c,
5f5f636c) + 1c
00081938 instance_call (34c2d0, e01f0, 159500, 818e4, fda03650, 3a) + 54
0007b9b4 PyObject_Call (34c2d0, e01f0, 159500, 3, 324b44, dbe9d) + 1c
00047e3c PyEval_CallObjectWithKeywords (34c2d0, e01f0, 159500, fda03824,
cc000, cc3bc) + e8
0009afac builtin_apply (dbc00, 324b38, 0, 28, 20d978, 0) + 13c
0004553c eval_frame (20d818, 0, 0, 14bfb8, 0, 3) + 3b44
00046c78 PyEval_EvalCodeEx (0, 1, 20d818, 16b704, 0, 0) + 960
00048208 fast_function (1597a0, fda03a38, 1, 1, 0, 1) + 58
00045690 eval_frame (16b5b0, 0, 0, 14b440, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, 2, 16b5b0, 323f94, 0, 0) + 960
0008c16c function_call (1597e0, 323f88, 0, 8c010, d3000, 40) + 15c
0007b9b4 PyObject_Call (1597e0, 323f88, 0, dd000, 1, 5) + 1c
00082354 instancemethod_call (32d8e0, e01f0, 0, 1597e0, 1, 0) + 180
0007b9b4 PyObject_Call (32d8e0, e01f0, 0, 0, 0, 0) + 1c
00047e3c PyEval_CallObjectWithKeywords (32d8e0, e01f0, 0, 0, 0, 0) + e8
0006814c t_bootstrap (274b38, ff26f688, 1, 1, ff26e000, 0) + 1c
ff25b728 _thread_start (274b38, 0, 0, 0, 0, 0) + 40
----------------- lwp# 7 / thread# 9 --------------------
fee9f054 lwp_sema_wait (fd901e30)
ff249ac4 _park (fd901e30, ff26e000, 0, fd901d70, 0, 0) + 114
ff2494c0 _swtch (fd901d70, 0, ff26e000, 5, 1000, 0) + 158
ff24826c cond_wait (fd901d70, 0, 0, ff26e000, 0, 351c48) + 13c
ff248110 pthread_cond_wait (351c38, 351c48, cc5a8, 351c48, cc400, 1) + 8
00065db0 PyThread_acquire_lock (351c30, 1, 136fbc, 0, ba198, 15eb) + d0
00067fe4 lock_PyThread_acquire_lock (274b68, 0, 0, 8a, 1aa1cc, 0) + 4c
0004558c eval_frame (1aa058, 0, 0, 136fa8, 0, 0) + 3b94
00046c78 PyEval_EvalCodeEx (0, b899c, 1aa058, 4, 0, 0) + 960
00048208 fast_function (158a90, fd901540, 1, 1, 0, 1) + 58
00045690 eval_frame (1a8918, 0, 0, 10b068, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, 3, 1a8918, 34ab04, 0, 0) + 960
0008c16c function_call (1b3050, 34aaf8, 159500, 8c010, d3000, 14bfb8) +
15c
0007b9b4 PyObject_Call (1b3050, 34aaf8, 159500, dd000, 1, 3) + 1c
00082354 instancemethod_call (324c60, e01f0, 159500, 1b3050, fd901650, 3a)
+ 180
0007b9b4 PyObject_Call (324c60, e01f0, 159500, 3, 26a3b4, dbe9d) + 1c
00047e3c PyEval_CallObjectWithKeywords (324c60, e01f0, 159500, fd901824,
cc000, cc3bc) + e8
0009afac builtin_apply (dbc00, 26a3a8, 0, 28, 1a95e0, 0) + 13c
0004553c eval_frame (1a9480, 0, 0, 14bfb8, 0, 3) + 3b44
00046c78 PyEval_EvalCodeEx (0, 1, 1a9480, 13d1b4, 0, 0) + 960
00048208 fast_function (1597a0, fd901a38, 1, 1, 0, 1) + 58
00045690 eval_frame (13d060, 0, 0, 14b440, 0, 1) + 3c98
00046c78 PyEval_EvalCodeEx (0, 2, 13d060, 34a94c, 0, 0) + 960
0008c16c function_call (1597e0, 34a940, 0, 8c010, d3000, 40) + 15c
0007b9b4 PyObject_Call (1597e0, 34a940, 0, dd000, 1, 4) + 1c
00082354 instancemethod_call (324948, e01f0, 0, 1597e0, 1, 0) + 180
0007b9b4 PyObject_Call (324948, e01f0, 0, 0, 0, 0) + 1c
00047e3c PyEval_CallObjectWithKeywords (324948, e01f0, 0, 0, 0, 0) + e8
0006814c t_bootstrap (274b98, ff26f688, 1, 1, ff26e000, 0) + 1c
ff25b728 _thread_start (274b98, 0, 0, 0, 0, 0) + 40
----------------- lwp# 8 / thread# 7 --------------------
fee9f054 lwp_sema_wait (fdb05e30)
ff249ac4 _park (fdb05e30, ff26e000, 0, fdb05d70, 24d84, dfee8) + 114
ff24978c _swtch (fdb05d70, 0, ff26e000, 5, 1000, 0) + 424
ff247e10 cond_reltimedwait (0, 0, 0, 1, 0, 0) + 1f8
ff247c08 cond_timedwait (e52e0, e5338, fdb05ad0, e52e0, 0, 0) + 2c
ff247b70 pthread_cond_timedwait (e52e0, e5338, fdb05ad0, 0, fecb43b4, 0) +
c
feca1fb8 __1cOomni_conditionJtimedwait6MLL_i_ (e52d8, 405210ca, 21583b0,
0, 0, 21583b0) + 40
fe34fa80 __1cEomniJScavengerHexecute6M_v_ (274ad8, 2894, 2800, 0,
fe3e5b80, 0) + cc
fe307d58 __1cPomniAsyncWorkerIreal_run6M_v_ (34cd50, 0, 1, 247308, 274ad8,
fe3efd4c) + 21c
fe307f40 __1cPomniAsyncWorkerDrun6Mpv_v_ (34cd50, 0, ff26e000, 0, 34cd50,
0) + 40
feca2658 omni_thread_wrapper (34cd50, fecb3f94, 1, ff27ad8c, 0, 2) + d4
ff25b728 _thread_start (34cd50, 0, 0, 0, 0, 0) + 40
-------------------------- thread# 3 --------------------
ff24ddbc _reap_wait (ff2729e0, ff26e000, 0, 1a, 0, 1a) + 38
ff24db14 _reaper (ff26ee30, fec15d10, ff2729e0, ff26ee08, 0, fe400000) +
38
ff25b728 _thread_start (0, 0, 0, 0, 0, 0) + 40
-------------------------- thread# 4 --------------------
ff247e10 cond_reltimedwait (0, 0, 0, 1, 0, 0) + 1f8
ff247c08 cond_timedwait (2474b8, e5698, fde0bc10, 2474b8, 0, 0) + 2c
ff247b70 pthread_cond_timedwait (2474b8, e5698, fde0bc10, 0, fecb43b4, 0)
+ c
feca1fb8 __1cOomni_conditionJtimedwait6MLL_i_ (2474b0, 405210dc, d6e1ab0,
0, 0, d6e1ab0) + 40
fed2e668 __1cVomnipyThreadScavengerOrun_undetached6Mpv_1_ (247458, 1e, 43,
fed43722, 247458, fe3ee364) + 11c
feca2678 omni_thread_wrapper (247458, fecb3f94, 0, 5, 1, fe400000) + f4
ff25b728 _thread_start (247458, 0, 0, 0, 0, 0) + 40
-------------------------- thread# 27 --------------------
ff247e10 cond_reltimedwait (0, 0, 0, 1, 0, 0) + 1f8
ff247c08 cond_timedwait (e55e0, e5198, fd30dbb8, e55e0, fe3efe70,
fe3e5b80) + 2c
ff247b70 pthread_cond_timedwait (e55e0, e5198, fd30dbb8, 0, fecb43b4, 0) +
c
feca1fb8 __1cOomni_conditionJtimedwait6MLL_i_ (e55d8, 405213e0, 306c322c,
0, 0, 306c322c) + 40
fe307cac __1cPomniAsyncWorkerIreal_run6M_v_ (3d42e0, fe3ed7a0, 1, 247308,
3c1860, fe3f0060) + 170
fe307f40 __1cPomniAsyncWorkerDrun6Mpv_v_ (3d42e0, 0, ff26e000, 0, 3d42e0,
0) + 40
feca2658 omni_thread_wrapper (3d42e0, fecb3f94, 0, 5, 1, fe401000) + d4
ff25b728 _thread_start (3d42e0, 0, 0, 0, 0, 0) + 40
Duncan Grisby
<duncan at grisby.org> To: baileyk at schneider.com
Sent by: cc: omniorb-list at omniorb-support.com
omniorb-list-bounces at omniorb- Fax to:
support.com Subject: Re: [omniORB] location forward to stale reference -
strangeness
03/12/2004 10:51 AM
On Thursday 4 March, baileyk at schneider.com wrote:
> I have a server which does location forwarding. It sometimes forwards to
a
> stale reference and I don't understand the resulting behavior. It
appears
> that the client reverts to the original profile and tries the call again,
> which makes sense, but then the server appears to never receive the
second
> call. The client then times out. The server ORB later hangs when
> attempting to shutdown.
>
> Traces are below. Two questions
>
> 1) Why do the traces show a second call to _non_existent() sent by the
> client, and so matching input message on the server side?
>
> 2) What should happen? If the server didn't lose the second try, and
> continues to forward to the same stale reference, how many times should a
> client retry?
The client is doing what I'd expect. When a forwarded reference fails
(whether on the first contact or subsequent ones), the client goes
back to the original location to have another go. That's required to
support migration of objects if they migrate more than once.
I really don't know why the server doesn't see the second request.
What should happen is that it keeps retrying, with exponential
back-off delays. You might try attaching a debugger to the server
process when the client is waiting for the second call to return, to
see what the server is up to.
Cheers,
Duncan.
--
-- Duncan Grisby --
-- duncan at grisby.org --
-- http://www.grisby.org --
_______________________________________________
omniORB-list mailing list
omniORB-list at omniorb-support.com
http://www.omniorb-support.com/mailman/listinfo/omniorb-list
More information about the omniORB-list
mailing list