Glom 1.13/14 using boost::python

Finally Figured Out boost::python

After lots of experimentation and two previous failed attempts, Glom now uses boost::python, with very little use of the nasty Python C API remaining. This should make it easier to add Python API to easily access and set field/record/table/database details from the Python that’s used in calculated fields or button scripts.

Useful Things I Now Know about boost::python

boost::python is no fun to get started with, but it’s now far easier for me to use than the Python C API.

I’ve mentioned the awful boost::python documentation before. Here are some essential things that I figured out, which are not really documented. This is thanks to helpful people on the boost::python mailing list. Corrections welcome – there’s so much here that some of it must be wrong.

None of this is very obvious or pleasant. If anyone had their first real C++ experience with boost::python then I’d forgive them for being put off C++ for good. I love C++ so that would be unfortunate.

“Converting” between C and C++ object types

C++ to C: To get the underlying PyObject* from a boost::python::object (awful docs), when you need to use a C function:

PyObject* cobject = theobject.ptr();

To test for a boost::python::object with a null underlying PyObject*, do:

if(cppobject.ptr())

Do not do if(cppobject). That tests if the python object is actually a boolean that is PyTrue.

C to C++: To get a boost::python::object for a PyObject*, when you received one from a C function, but you then need to use the result in a C++ function, or just want the improved memory management:

  • If the C function gave you a reference that you should later unreference:
    boost::python::object cppobject( (boost::python::handle<>(cobject)) );

    (You need those extra brackets, for “interesting” compiler reasons.)

  • Or, if you need to take a reference.
    boost::python::object cppobject(boost::python::borrowed(cobject));

    (Yes. that’s horrible too. I see no reason to expose boost::python::handle in the API.)

  • However, you’ll need to use allow_null too to avoid exceptions if the PyObject* might be null. Well, any pointer could be null, so say hello to:
    boost::python::object cppobject(  (boost::python::handle<>(boost::python::allow_null(cobject))) );

    or

    boost::python::object(boost::python::borrowed(boost::python::allow_null(cobject)));

    (Shoot me now. No, reducing it to b::p::whatever is not a significant improvement.)
    I understand that a null PyObject* may sometimes be an exceptional unexpected event, but forcing the use of a try/catch by default just for a null pointer check is annoying. Explicit functions such as wrap() and wrap_not_null() would be so much easier.
    See the equivalent for gtkmm (plus calling reference() when necessary with non-widgets).

Using boost::python with your own wrapped C++ classes.

  • To get a boost::python::object for an instance of your C++ class that you’ve wrapped for Python with boost::python::class, just do:
    boost::python::object obj(new YourWrappedClass);
  • To get a C++ instance of your wrapped class from a boost::python::object, use boost::python::extract (as also mentioned generically below):
    boost::python::extract<MyClass*> extractor(cppobject);
    if(extractor.check()) myobject = extractor;

Others

  • To get a C++ value out of a boost::python::object do, for instance:
    boost::python::extract<std::string> extractor(cppobject);
    if(extractor.check()) mystring = extractor;

    You can do

    mystring = boost::python::extract<std::string>(cppobject)

    without the check() but that will throw an exception if the underlying type is not really what you expect.

  • boost::python likes to throw exceptions. I think it only ever throws boost::python::error_already_set, though the (Python) error is often not already set when it’s thrown. When the error is set, you’ll need to use Python C API to discover what it is.
  • To provide [] syntax in python for your wrapped class, you’ll need to know how the C API works. Add this voodoo to your boot::python::class declaration:
    .def("__getitem__", &MyClass::getitem)
    .def("__len__", &MyClass::len)

    Those methods can then have signatures like this:

    boost::python::object getitem(const boost::python::object& cppitem);
    long len() const;
  • To use Python date or time values, you will need to use C Python functions. For instance:
    PyObject* cobject = cppobject.ptr();
    int day = PyDateTime_GET_DAY(cobject);
    int month = PyDateTime_GET_MONTH(cobject);
    int year = PyDateTime_GET_YEAR(cobject) );

Boost has no .pc files

boost is a complete pain as a dependency. I understand that they don’t want to freeze API or ABI, because it’s a place for gradually improving API, though I think they should just have regular stable/devel phases with parallel installs. But I can’t forgive how difficult it is to get the header and linker options to use boost libraries. There are some m4 macros out there but they are hacky and fragile, and don’t actually work for boost::python. It shouldn’t be hard to provide pkg-config .pc files, so you wouldn’t need to do any compilation or linker checks in configure at all. I hacked some m4 code together based on some existing stuff, but I couldn’t recommend it.

So distro packagers won’t enjoy this new dependency. Sorry.

9 thoughts on “Glom 1.13/14 using boost::python

  1. did you bring up your issues on their mailing list and/or proposed some API changes? Or is this just another pointless vomit into the interweb?

  2. Pavel, yes, all of this is based on the helpful replies of people on the mailing list (I forgot to make that clear – I’ll edit). I haven’t strongly suggested API changes because the current state of boost::python makes me think that the maintainers have massively different ideas to me about good API and good documentation. I also don’t have the unpaid time right now that I could promise to overhauling things even if it was wanted.

  3. Murray: The link to your website in any comment (checked another post is wrong), it points to http:://www.murrayc.com (extra :).

  4. So what’s the benefit of this? I’m hearing a lot of negatives from you – poorly documented, badly designed API, and a project that generally doesn’t make things easy to use. So why are you doing it?

  5. Boost allows you to have parallel installs, but you need to change the default options, do some patching and provide custom symlinks.
    In Gentoo there are slotted boost installs since boost 1.35 AFAIK. However, you can have only one system version (for gentoo “eselect boost list|set”) active for compiling and linking, which is the (unfortunate) default for the boost install on a nix system.

  6. Could you clarify why you need .pc files? With newer versions of Boost you should be able to just use -lboost_python, and while I can implement .pc generation for 1.43, this won’t help with older versions of Boost anyway. And if a current distro manages to specifically alter build of Boost to not provide -lboost_python but rather -lboost_python-whatever, then they are likely to mangle .pc files the same way, making ‘pkg-config … boost_python’ nonportable.

  7. At the risk of telling you something you already know, you can write python converters that can be registered with the system, and are called automatically and transparently when it is necessary to convert across the Python/C++ interface from Python -> C++ or vice versa. While this is a killer feature, it has ironically like zero official documentation as far as I know. This can be used for classes, functions etc. So it is not necessarily to manually wrap your objects to convert them.

    The best documentation available for someone starting with this is the blog post
    http://misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/
    which deservedly is the first Google hit for “boost python converters”.

    Like everyone else, I agree the Boost Python documentation is (unnecessarily) abysmal. But that is actually not so uncommon for free software, and C++ experts seem to suffer from a strange kind of meta-programming machismo.

    On the other hand, once you get Boost Python working, it is fantastic.

    Regards, Faheem

Comments are closed.