[omniORB] Access violation in ~Strand_iterator

[Gavin Matthews]

/Resending, I got a bounce on the original send/
Hi, 
 I'm experiencing a problem with omniORB2.6.1 (unfortunately we're tied into
2.6.1, so for the time being upgrading is not a solution). When under
extremely heavy load for a prolonged period of time (~24-48 hours) we
experience get an access violation in omniORB on WindowsNT. 

The information extracted from Dr.Watson and the dump file points to a race
condition in ~Strand_iterator

The faulting address is:

>State Dump for Thread Id 0x154
>
>eax=00000000 ebx=026c00b0 ecx=05ae0038 edx=05ae0038 esi=026c00b0
edi=00000000
>eip=77f67616 esp=02a1feb8 ebp=02a1fec4 iopl=0         nv up ei pl zr na po
nc
>cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000
efl=00000246
>
>
>function: RtlLeaveCriticalSection
>        77f67610 8b542404         mov     edx,[esp+0x4]
ss:0574e8bf=00000000
>        77f67614 33c0             xor     eax,eax
>FAULT ->77f67616 ff4a08           dec     dword ptr [edx+0x8]
ds:0880ea3e=????????
>        77f67619 7525             jnz     RtlLeaveCriticalSection+0x30
(77f67640)
>        77f6761b 89420c           mov     [edx+0xc],eax
ds:0880ea3e=????????
>        77f6761e f0ff4a04         lock   dec dword ptr [edx+0x4]
ds:0880ea3f=????????
>        77f67622 7d03             jge     RtlLeaveCriticalSection+0x17
(77f67627)
>        77f67624 c20400           ret     0x4
>        77f67627 52               push    edx
>        77f67628 e829590000       call    RtlpUnWaitCriticalSection
(77f6cf56)
>        77f6762d 33c0             xor     eax,eax
>        77f6762f c20400           ret     0x4
>        77f67632 2e8bc0           mov     eax,cs:eax
>        77f67635 2e8bc0           mov     eax,cs:eax

And the stack for the faulting thread (154) is:

>.  2  id: 146.154   Suspend: -1 Teb 7ffdc000 Unfrozen
>ChildEBP RetAddr  Args to Child              
>02a1feb4 002312f1 05ae0038 05ae0038 02a1fed0
ntdll!RtlLeaveCriticalSection+0x6
>02a1fec4 10025641 02a1fef0 02a1ff18 10026317
omnithread2_rt!omni_mutex__unlock+0x11
>02a1fed0 10026317 05ae0034 05ae0034 05ae0034
omniORB260_rt!Strand_iterator__~Strand_iterator+0x21
>02a1ff18 10026116 026f0038 1009d4a4 02830034
omniORB260_rt!outScavenger_t__run_undetached+0x387
>02a1ff60 00232078 00000000 00000000 00000000
omniORB260_rt!outScavenger_t__run_undetached+0x186
>02a1ff84 7800248d 02830034 00000000 00000000
omnithread2_rt!omni_thread__init_t__init_t+0x2b8
>02a1ffb8 77f04ede 026c00b0 00000000 00000000 MSVCRT!beginthreadex+0xb2
>02a1ffec 00000000 78002432 026c00b0 00000000 KERNEL32!BaseThreadStart+0x51

 I've implemented all the patches I could find that appeared to be related,
none of them have solved the problem.

Relevant Patches Implemented: 
http://www.uk.research.att.com/omniORB/bugs/40.html
http://www.uk.research.att.com/omniORB/archives/1999-07/0230.html

Are there any other patches that I should be looking into? Has anyone seen
this problem or something similar? Any help would be much appreciated.

thanks,
gavin

---------------------------------
Gavin Matthews
Software Engineer II
Tumbleweed Communications Corp