[omniORB] Byte order inversion on NT/MSVC++

[Dietmar May]

I've run into a problem with byte order inversion within a server on Windows
NT 4 (sp3) compiling with MS VC++ 5.0 (sp3).

Enums are causing marshalling errors, because the op<== (unsigned long) is
reversing the byte ordering when unmarshalling values - 0x00000004 becomes
0x04000000, etc. Same thing happens for doubles, 1.0 becoming 3.2e-316 or
some such value.

The unmarshalling code (NetBufferedStream::operator<==()) performs a
comparison of RdMessageByteOrder() with omni::myByteOrder. In my case,
RdMessageByteOrder() yields 1, while omni::myByteOrder is 255. There seems
to be a flaw in the code here that should be fixed, since it is performing a
boolean comparison using two unsigned char values; hence, if both of these
are non-zero, but not equivalent, omniORB inverts the byte ordering of data
in the input stream.

Another strange behavior I've noticed is that the address (and data) for
omni::myByteOrder is different when I set a breakpoint in my application
code than when I put the breakpoint in omniORB DLL code. If I create a
static initializer object within constants.cc and break in the constructor
(during the runtime initialization sequence), I can view the address and
data of omni::myByteOrder, and it looks reasonable (including being
initialized to 1).

If I debug through the rest of the runtime initialization, it looks fine
UNTIL it returns from executing DllMain(). The address of omni::myByteOrder
changes at this instant (as viewed in the DevStudio watch window), and its
value changes to 255. However, the data at the original (omniORB DLL)
address of myByteOrder remains unchanged (at 1). The new address of
omni::myByteOrder (with value 255) is the address inside the map file of the
application.

I've tried compiling and linking with 2.5.1, 2.6.1, creating both stub-DLL
and stub-linked versions. An earlier version of the application was working.
The problem is obviously NOT the application randomly overwriting memory,
since the original memory contents remain unchanged, and since the address
of myByteOrder suddenly changes when the DLL initialization code completes.
I suspect some issue with the way that MSVC links static data from DLLs and
performs its fix-ups, since the results of the link seem to think the
myByteOrder address is somewhere other than where the omniORB runtime DLL
thinks it is.

Any ideas or suggestions would be greatly appreciated.

Thanks,
Dietmar May

Software Architect
Object Workshops, Inc.
dcmay@object-workshops.com