Last modified 6 years ago Last modified on 10/08/2013 05:39:21 PM

SWIG vs. Boost.Python

This page is a starting point for Jim Bosch's long-term campaign to rid the LSST world of SWIG and replace it with Boost.Python. The intent is to collect all the arguments in favor of and against such a change. Finding all of the arguments against such a switch and finding an answer for all of them is the most important part of this effort.

NOTE: This page was written as of Swig 1.34; some features mentioned here may have improved since then.

Interface Libraries vs. Code Generators

SWIG is a code generator; Boost.Python is an interface library. Code generators require less maintenance, as changes in the C++ code should be automatically reflected in the Python wrappers. An interface library forces the user to explicitly declare the mapping between C++ and Python, by writing additional code (in Boost.Python's case, C++ code). This code may require updates when the C++ interface changes.

Because the mapping between C++ and Python is not obvious in all cases, the mapping generated by a code generator like SWIG is the result of a large, complex set of rules, modified by explicit user intervention where necessary. Because this ruleset is (necessarily) complex, determining the correctness of the resulting Python interface becomes a concern for unit tests, increasing the amount of test code that must be maintained. In contrast, while Boost.Python requires this mapping to be made explicitly, resulting in additional code that must be maintained, mapping failures that require unit tests in SWIG will translate to compile-time failures in Boost.Python, decreasing the amount of test code that must be maintained.

Code generators also require a parser that is capable of reading C++ code, and SWIG's simply isn't up to the task - many of our problems with SWIG concern its inability to deal with advanced C++ constructs. And as the lists below show, by "advanced C++" I really mean "not C": some of the most basic C++ constructs don't work reliably with SWIG. So far, our workarounds have been to "dumb-down" our C++ code, and make it more like the C code SWIG understands.

Finally, a code generation approach puts an extra "black box" in the middle of any debugging attempt - the code generator itself. When compiling Boost.Python code, long, template-based error messages are a concern, but the entire process involves nothing more than compiling C++ code. You can step through it, and you can read the Boost.Python source you've included and linked against. It's not always easy, but it's very straightforward where to look next. While one can of course read the SWIG source as well, that's a much more daunting task.

Boost.Python Code Generation

The current code generator for Boost.Python is Py++, which uses GCCXML to process C++ code into an XML tree it then parses and converts into Boost.Python wrapper code. The code generation is strongly rule-based, and very flexible - Py++ is pure-Python, and configured with Python, and you can attach essentially any kind of code-generation rule to the parser. In my opinion, it's a very good design for a code generation tool. Because it outputs to Boost.Python, the output code is highly readable, and Py++ makes an effort (with indentation and other formatting) to make sure that's the case. GCCXML is the weak link - while it's a much better C++ parser than SWIG (because it's based on GCC internals), it tends to lag GCC releases by quite a bit (last I checked, it was using GCC 4.2 internals). And it's not clear how much it's still being developed (the last changelog on the website is 2007, though it seems they may just not be updating the website - though that is itself worrying).

In the same vein, the creator of Py++ (Roman Yakovenko) has recently stopped hosting the Py++ website with online documentation and examples. The full project and documentation are still available in slightly less-convenient form at sourceforge, and he has asserted on the Boost.Python email list that he continues to support Py++ and the Python bindings for GCCXML. It does highlight the fact that Py++ is essentially still a one-man operation, however.

As nice as Py++ is, I still prefer to use Boost.Python without a code generator - except for extremely homogeneous libraries, I find that it's actually more difficult to come up with a complete set of rules that can cover all the exceptional cases rather than simply explicitly wrap everything.

With that in mind, I have created a Python script I call "bpdox" that parses Doxygen's XML output and provides a few macros for generating boilerplate-heavy Boost.Python code and inserting formatted Doxygen comments as Python docstrings. Unlike a true code generator, bpdox does not attempt to parse and wrap a complete library given its source - but it does make it possible to wrap most libraries (including documentation!) by providing little more than the names of classes and [member] functions to wrap. And because it's a macro processor, not a full code generator, difficult cases are easily handled by writing the Boost.Python wrappers directly rather than using the macros.

Stuff Boost.Python does unambiguously better than SWIG

  • SWIG ignores all namespaces, requiring all names to be fully-qualified in header files for generated code to compile. For instance, the following does not generate compilable code:
    namespace a {
    class A;
    namespace b {
    class B {
        void apply(A &);
    Instead, one has to explicitly write apply(a::A&) - this requires changes to C++ source to make them compatible with Python, and makes highly nested namespaces (which are very common in the LSST stack) much less useful. While Boost.Python still doesn't automatically set up Python packages to match C++ namespaces, it never has any trouble understanding C++ namespaces.
  • SWIG cannot parse inner classes. Any inner class must be explicitly removed with #ifdef SWIG, and there's no way to get SWIG to wrap it. SWIG generally fails with a non-informative syntax error upon encountering an inner class.
  • SWIG imports and includes are (occasionally) non-recursive, requiring high-level packages to sometimes include headers from lower-level packages. This may be related to how the lower-level package was wrapped, but no one seems to know why it happens or know how to fix it.
  • SWIG is unsafe when wrapping C++ functions that return by reference, including member functions that return this. This makes a naive wrapping of typical C++ getters generally unsafe (with no warning), and makes C++ method-chaining by returning this impossible to wrap correctly for Python. Boost.Python solves this problem with call policies, which force the user to define how to handle the reference counts of raw pointers and reference returns at wrapper-compilation time.
  • Boost.Python handles returns by base-class reference or pointer (or smart pointer) automatically, casting them to the true derived class type upon conversion to Python. This matches the native behavior of a duck-typed language, in which one always only has derived class objects. In SWIG, such returns result in base-class types that have to be explicitly downcasted, using explicitly wrapped downcasting functions.
  • Boost.Python handles shared_ptr and other smart pointers extremely gracefully, even to the point of constructing shared_ptrs from wrapped Python objects that don't even have shared_ptrs inside them, by using Python's own reference count in a custom deleter. While this promises to get better in SWIG 2.0, I can't imagine it will come anywhere near to Boost.Python's handling.
  • Boost.Python allows what are effectively templated typemaps. These aren't for the faint of heart, but they allow a power-user (like me) to, for instance, create automatic conversions for templated array or matrix classes that don't require any boilerplate or declarations of what template instantiations will be used. These are simply impossible with SWIG.
  • Boost.Python's "call policies" are much more powerful than similar features in SWIG, allowing users to modify the default behaviors when wrapping functions. These can include changes to ownership (so one can return a member variable by reference and have its Python lifetime linked to the lifetime of the owning Python object) or to completely convert the result (to return a std::vector<int> directly as a Python tuple or list instead of as a wrapped object).
  • All the SWIG code for a single Python module has to go in a single source file. That makes source files many tens of thousands of lines long. Even worse, much of the code from imported or included modules has to be repeated as well, meaning the highest-level wrapper code grows to an extremely large size. For example, a SWIG .i file that does nothing other than %import afw and meas_multifit is over 10000 lines long, and takes approximately two minutes to generate and compile. Boost.Python code is heavily templated and slower to compile line-by-line, but you can split it up into multiple source files just as easily as you could any C++ code. Most importantly, the size of a Boost.Python modules's wrapper code does not depend at all on the size of its dependencies' wrappers.
  • SWIG assumes that all classes have default constructors, assignment operators, and copy constructors that are equivalent to default construction + assignment when wrapping functions that return classes by value. It can avoid making this assumption when %feature{"valuewrapper"} is used, but it often isn't apparent that this is necessary until a dependent package does a return by value, at which point the SWIG wrappers for the lower-level package must be changed.
  • SWIG's %template does not accept full signature specifications, making it is impossible to wrap functions with the same name and template argument list but different signatures unless you want to wrap all of them.
  • SWIG's name lookup and typedef resolution doesn't work with templates. This makes, for instance, the Pixel typedef in Kernel and Psf very confusing: to support Numpy conversions to an array of this type, it's necessary to instantiate typemaps for Array<Kernel::Pixel,N>, Array<Psf::Pixel,N>, and Array<LocalPsf::Pixel,N> even though these are all defined to be the exact same type.
  • Boost.Python correctly converts public const data members to Python properties, raising an exception when one tries to set them. With SWIG, setting these is a silent failure.
  • C++ functions that return built-in types like bool or double by reference return opaque objects in Python (admittedly, there's no obvious correct conversion here, because conversion to Python requires a copy, but Boost.Python refuses to compile unless you explicitly say you want to copy).

Stuff SWIG does unambiguously better than Boost.Python

  • Boost.Python doesn't support general typemaps for C++ lvalues (non-const references or pointers). Getting such a value from a Python object that doesn't actually contain such an object requires the ability to create a temporary of the appropriate type in the wrapper layer, which SWIG has but Boost.Python does not.
  • SWIG is awesome for wrapping simple C files quickly.
  • SWIG typemaps can be used to change the conversion and matching behavior for built-in C++ and Python types. These generally cannot be changed in Boost.Python.
  • Some Boost.Python wrapper invocations are very verbose and repetitive, because it's limited by C++'s introspection ability. The typical declaration for a function requires writing the function's name twice (once for the Python name, once for the C++ function pointer), repeating all of the keyword argument names, and possibly listing all of the argument types to cast the function pointer to the proper type to resolve overloads. Enum wrappers are even worse: all values associated with an enum have to be re-listed in the wrappers. Using Py++ or bpdox fixes this problem, bug it's very annoying in vanilla Boost.Python.
  • There is very little active development of new features for Boost.Python. This is not to say the library is unmaintained; bugs are still fixed, the library is kept compatible with the latest versions of Python, and it still qualifies as a more "finished product" than SWIG in terms of support for C++ and Python features. But the Boost development model doesn't really encourage the development of new major versions of existing libraries or foster the sort of open-source collaborations needed to create them. SWIG has that, and Boost.Python doesn't. As a result, some of the most useful user-contributed Boost.Python code is scattered around the internet in the form of extension projects (disclaimer: some of these are mine) that are not officially affiliated with Boost. The best code for handling C++ standard library containers and Numpy fits into this category.

Stuff that's just different

  • All SWIG code goes in .i files. Boost.Python puts all C++ code in .cc files and all Python code in .py files. That makes emacs happier, but it does separate the interface for a single class into multiple files when custom Python code is needed.
  • Boost.Python's documentation of advanced features is not great; you really have to read the code a little to understand how the type conversion works, for instance. SWIG has much more documentation, but there's so much more to be documented that the sense of what's missing is (in my opinion) about the same.

Real Examples of Swig Problems

  • Source::getFootprint returns type boost::shared_ptr<Footprint const> which swig does not know how to handle. No tests were done in this revision of afw to test proper wrapping, so the problem was not detected until a dependent package tried to make use of the method. Additionally, note the overload of setFootprint to make swig happy.
  • Even with the correct %template invocations, SWIG couldn't figure out the CRTP and typedefs in meas::multifit::definition::ParameterComponent?, requiring most functions to be rewritten in the .i file without the typedefs.