Last modified 5 years ago Last modified on 01/31/2014 12:42:41 PM

Write Python code in a way that looks ahead to Python 3

This proposal (#2967) was accepted by the TCT on 30 Jan2014 as follows:

  • Henceforth, all new code going forward will use the Python 3 idioms presented below.


It is possible to write much of our Python code in a way that will run well under Python 2.7 and Python 3.x, without harming readability (and in some cases, improving it). There are other cases where we can write our code in a way that helps the 2to3 code converter produce more efficient code. Towards this end, I propose the following additions and/or changes to our Python coding standards:

Python 3 Idioms

  • Use from __future__ import division so that / is floating-point division and // is truncated integer division, regardless of the type of numbers being divided. This gives more predictable behavior than the old operators, avoiding a common source of obscure bugs. It also makes intent of the code more obvious.

  • Use from __future__ import absolute_import and import local modules using relative imports (e.g. from . import foo or from .foo import bar). This results in clearer code and avoids shadowing global modules with local modules. It also makes 2to3 conversion more reliable.
  • Avoid dict.keys() and dict.iterkeys(). For iterating over keys, iterate over the dictionary itself, e.g.: for x in mydict:. To test for inclusion use in, e.g. if key in myDict:. This is preferred over keys() and iterkeys() and avoids the issues mentioned in the previous item.

  • Replace file with open. This is preferred and file is gone in Python 3.

  • Use as when catching an exception, e.g. except Exception as e or except (LookupError, TypeError) as e. The new syntax is clearer, especially when catching multiple exception classes, and the old syntax does not work in Python 3.
  • Use from __future__ import print_function. Minor, but provides forward compatibility. This will affect very little code since we rarely use print.

  • Use next(myIter) instead of This is preferred, and the special method next has been renamed to __next__ in Python 3.

For more information see, among several useful references.

Comments Prior and During to TCT Review

RHL's Comment

If we go this route, then I think we should follow the advice of e.g. and run 2to3 and fix the results to run with python 2.7 and python 3; the swig interface is claimed to build with either 2 or 3.

I guess that the one I hate most is the treatment of iterdict. Using iterdict in python2 is almost always a micro-optimisation, and one that goes away in python3! The argument is:

  • People should change idiomatic python2 and python3 to non-idiomatic python2 (with iterdict) so that 2to3 can convert it back.

Fortunately you can disable this when you run 2to3:

	2to3 -x dict

and I think we should, followed by a hand conversion as needed (or removal of iterdict before running 2to3)

MJ's Comment

To RHL: To offer a different viewpoint on iter*() methods (and xrange()): one's view of these strongly depends on when they began learning python. To those who've come to the party late (e.g. >= Python 2.2, circa ~2001), this *is* idiomatic python2 (and .values(), .keys(), etc... are "methods to be used when I explicitly want a list back"). Same for xrange(). They're warts, for sure, but idiomatic warts.

N.b., I see these as being about robustness, less than optimization: though the initial assumption that some dict is small and .keys() are safe to use may be true, that can change in the future w/o nobody noticing (causing difficult-to-track-down performance regressions).

TCT Meeting Notes

See Ticket #2967 for a summary of the TCT's discussion on this proposal.