[omniORB] IDL to Python language mapping update?

[Luke Deller]

Duncan Grisby wrote:
> I think the biggest challenge with Python 3 support is how to 
> structure the source. The changes will be sufficiently pervasive that 
> I don't think it will be sensible to have a single source for both 
> Python 2 and Python 3 support, which will lead to the risk of 
> divergence between the implementations. I'm open to suggestions if
> anyone's looked into it.

Brian Neal wrote:
> I haven't looked at the source, but are you talking about the Python
> code, or the C++ or both? I've seen many recommendations these days to
> make all new development in Python 3, and then use the 3to2 tool to
> automatically create a Python 2.x code-base.

I looked into this question with the C++ codebase a few months ago.  It
appears to be quite possible to share the same codebase for Python 2.6+ and
Python 3.  I got as far as having the same tree compile with 2.6.5 and
3.1.2, though not working right as I didn't work through the str issue
properly yet.

(Perhaps Python 2.4 and 2.5 could be supported too if we added a few
forward-compatibility macros which are included with Python 2.6+, but I'm
not sure how valuable this would be).

Required changes that I encountered were:

- PyInt_* are gone in Python 3. There were actually some macros defined in
  3.0 mapping these to PyLong_*, but these were removed in 3.1.  These are
  not really appropriate for many cases in omniORBpy anyway.
  
  Most omniORBpy code that I came across accepts both int objects and
  long objects, handled as two separate cases, so all we need here is
  to #ifdef out the int handling code for Python 3.

- PyString_* functions are renamed to PyBytes_* in Python 3 (and both work
  in Python 2.6+ where PyBytes_* are defined as macros for PyString_* for
  forwards compatibility).

  My first attempt was to change all PyString_* calls to PyBytes_*, but
  this is not appropriate for text where we actually want to use a str
  object in Python 2 but a unicode object in Python 3.  What is really
  needed here is to sort through every use of PyString_ in the code,
  categorising them into three cases:
  1) Binary data: rename PyString_* to PyBytes_*
  2) Text: use PyString_*, and add macros for Python 3 to map these to
     PyUnicode_*.  This works in many but not all cases as the PyUnicode_*
     interface is a little different from the PyString_* interface.
  3) Text where the macro approach does not suffice: need to use #ifdef
     with separate code for Python 2 and Python 3.

  This is the highest-impact change in Python 3, and may be what makes
  a fork for Python 3 become more attractive.

- Module initialisation is different in Python 3, notably Py_InitModule is
  gone.  The pattern suggested in the porting HOWTO for supporting both
  Python 2 and 3 works great:

  http://docs.python.org/py3k/howto/cporting.html#module-initialization-and-state

- PyObject_HEAD_INIT() is gone in Python 3; use PyVarObject_HEAD_INIT
  instead for both Python 2 and 3.

- tp_compare in the PyTypeObject struct is no longer used in Python 3; use
  tp_richcompare instead for both Python 2 and 3 (requires rewriting
  comparison function)

- ob_type in PyObject has moved in Python 3; use Py_TYPE() macro instead, 
  which works in both Python 2 and Python 3.

- Old style classes (PyInstance_*) are gone.  There are a few instances in
  the omniORBpy code where it grabs the instance dictionary directly out
  of the PyInstanceObject struct, purportedly for performance reasons. 
  This needs to be changed to use the generic object interface such as
  PyObject_GetAttrString.  This works for both Python 2 and 3, though
  perhaps with a performance degradation.

  Some places actually fall back to using generic code if PyInstance_Check
  fails (eg validateTypeStruct in modules/pyMarshal.cc).  In those cases
  we could potentially leave the PyInstance optimization in for Python 2
  only, protected with an #ifdef.

- PyArg_NoArgs is gone in Python 3; use METH_NOARGS instead which works
  in both Python 2 and Python 3.

- The PyNumberMethods struct has changed in Python 3, eg nb_divide has gone.
  This requires some #ifdef's in pyFixed.cc (though ultimately this code
  should be removed in favour of using decimal.Decimal)

- the DL_IMPORT macro was deprecated in Python 2.3 and removed in Python 3.0
  so we need to declare it ourselves for use in modules/pyFixed.h

- print is now a function; the Makefiles actually invoke python with a print
  statement on the command line, so they need to be adjusted to put
  parenthesis around the print arguments

The .py files need changes too.  I actually changed them to work with both
Python 2 and 3, though I'm not sure if this is the best idea.  Changes were:
- print is a function in Python 3; add parentheses around arguments
- replace use of types.TupleType/types.BooleanType/etc with tuple/bool/etc
- "except" syntax for accessing the exception object is conflicting, so use
  sys.exc_info() to access it instead
- dict.has_key method is gone; use "in" operator instead
- exceptions module is gone; all the exception types are built-in anyway
  in Python 3 and recent Python 2.x

Hopefully this gives an initial idea of things needed to port omniORBpy to
Python 3, and what may be required to support both Python 3 and more recent
Python 2.x versions in the same codebase.

Regards,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege.  If you are not the intended recipient you must not use, distribute or copy this email.  If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of IRESS Market Technology Limited.

It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
IRESS Market Technology Limited does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************