Glom: Backups and coping with PostgreSQL upgrades

General,

Over the last few days, I added a Backup feature to Glom. It’s in Glom 1.14.4 that I released today. I don’t usually add features to stable releases, but this time it might help some people.

The Export Backup menu item saves everything in a .tar.gz tarball, using PostgresSQL’s pg_dump command and tar. (I hate calling command-line utilities from code, instead of calling functions in libraries.) The Restore Backup command uses that .tar.gz tarball and recreates a Glom system, either locally or on a central server if you choose.

The backup feature is obviously useful in general, but it’s particularly important since we noticed that use of pg_dump and pg_restore is necessary for migration when upgrading PostgreSQL, for instance from version 8.3 to 8.4.That’s not so awful when you just have one central database server, and Debian/Ubuntu even have a script that does the pg_dump/pg_restore dance on your central data when you upgrade. But Glom’s self-hosting feature (the default) has a separate set of database files for each .glom file, starting the PostgreSQL instance on demand. Ubuntu’s upgrade system obviously can’t know about all those files, which might even be on a USB stick.

The Backup feature can be used to work around this problem, if you remember to use it before upgrading your distro, but it’s not a nice solution.

Before anybody tells me to use SQLite instead, please remember that it still doesn’t have the features Glom needs, such as access-control and multi-user network sharing.

Qt’s Open Bug Tracker

General, ,

Qt has had an open bug-tracker for a few months now. I am very happy about that. It has made life far more pleasant for Qt developers compared to the past. However, there are some problems. In summary, bug reporters are often treated like the enemy instead of contributors, and this could be fixed easily.

My company, Openismus, provides bugmasters to corporate open source projects such as maemo.org. We know how to make the best use of outside contributions, so enthusiasts stay loyal, and we know how to manage bug databases for the long term. See the overwhelmingly positive feedback for Andre and Karsten, maemo.org’s bugmasters. They acquired these skills while establishing  GNOME’s bug squad, working on GNOME’s huge bug database.

I have linked to some Qt bugs as anecdotal examples of poor interaction with bug reporters. I am not interested in a discussion here of the technical aspects of these bugs. They are not meant to be technically critical. Also, half of them are for QtMobility, whose developers are generally very helpful, but even they haven’t learned the best bug-tracker habits. I don’t think I have seen enough Qt bug reports to have a fully representative sample, but my short experience does suggest that there are serious problems. I don’t mean to exaggerate or be unfair.

Awkward UI

Let’s look at a random bug. This view (not this particular bug) will be the first look at the bug-tracker for many people, as they follow a link to an existing bug. But it does not seem designed for humans. There are some particular problems:

How do I add a comment?

Typically, I’ll read the comments on a bug report to see what’s happening. But then there’s no obvious way to add my own comment. If I look around then I finally see a “Comment on this issue” link nestled among other less important action at the left hand side. I believe that many people won’t find that link, or will be discouraged by the awkwardness. I much prefer just writing a comment right into a bugzilla page.

Noise

The “all” page is too complicated, with lots of machine-generated noise. Instead of fixing that, there are tabs that show reduced views: All, Comments, Work Log, Change History, Transitions. This makes the whole page seem even more complicated.

The all page is so nasty that many people will switch to the comments page. But then comments lose their context. For instance, here (bug 7303) the Qt employee seems to be blaming the reporter though he is really just talking to someone that he has assigned the bug to. I’d rather just see a single page, as in bugzilla, without so much noise. The notification emails are similarly padded with irrelevant details.

Rush to close

We notice a strong tendency for the Qt employees to close bugs at any cost for any arbitrary reason as quickly as possible. Presumably they are under pressure to reduce the simple number of open bugs. But a bug-tracker exists to collect information that can gradually be used to improve software. If you think of everything as short-term and rush to ignore that information then you will not make the improvements and customers will assume that you don’t care.

Hard to check the fix

When I close bugs, I mention at least the commit message that I used. Sometimes I link to the commit’s web page on our gitweb or gitorious. But Qt bugs are often closed with a mention of the git commit ID (such as d65d3becc292523f0962aac9d2bf7b96d9c47c04), with no link. For instance, bug 8865 and bug 5729. I am thankful for the fixes but, particularly for documentation fixes, it’s best if I can verify it and reply quickly.

Viewing the change for a particular commit ID is awkward, requiring lengthy use of the command line outside of the browser. Even the gitorious web site doesn’t let me search for a commit ID, so I have to browse through pages of recent commits using the browsers find-on-page feature.

Actually, the person fixing the bug often cannot provide a gitorious web link at that time because that commit isn’t even public yet, because Qt’s gitorious repository is just a mirror, updated approximately daily from a secret repository. Things would be easier if they just worked in an open repository. In the meantime, Qt’s bug tracker should be hacked to recognize and link commit IDs, showing a “try tomorrow” page if the commit is not yet public.

Not supported. Not public

If the Qt developers decide (after the fact) that something in Qt is not supported then the bug may be closed, instead of leaving it open for someone to fix. But if something is not supported then it shouldn’t be in the release. For instance, bug 7334.

There’s also lots of Qt API that is used widely but not documented. Bug reports about the lack of documentation may be closed instead of leaving it open for someone to fix. But all API should be documented for the sake of implementation quality, regardless of whether it is public. And if API shouldn’t be used then its documentation should say that it shouldn’t be used.

Closing bugs because they require work

Bugs are sometimes closed because they would require effort to fix them. This makes no sense to me. Even if no Qt employee plans to fix them, they should stay open so others can fix them, and so that the information is kept in one bug report, with repeated reports being marked as duplicates. For instance, bug 5915.

Sometimes the idea of necessary work seems to be invented just as a way to close a bug. For instance, in bug 5729, a simple patch is not enough. The reporter is told that he must test it on multiple platforms, ignoring the possibility to test the harmless change widely during regular unstable releases.

For non-employees, the page even has some probably-unintentional machine-generated “You don’t have permission to work on this issue” at the left, just in case the reader didn’t feel unwanted enough already.

Out Of Scope

Likewise, if the Qt developers don’t personally see a problem as their priority then they will sometimes close it as “Out Of Scope”, regardless of whether the reporter cares about it or whether he might provide a patch.

Sometimes (bug 5729 again) this happens without the Qt employee even writing a comment explaining why. This is the bug-tracker equivalent of “Go away. We don’t want you to use our software”. Stopping this should be a top priority.

I’ve also seen one case (bug 6074) where this was used when the Qt developer meant “already fixed”. Until I asked for clarification (most people don’t) I assumed that they just didn’t care.

Can’t reopen bugs.

If a fix is not good enough, or the Qt employee has misunderstood the problem, I can’t reopen the bug. That is very frustrating. Most people will not beg for the bug to be reopened or even bother continuing to comment on the closed bug. Luckily we do have the option to submit gitorious merge requests when our bug has been incorrectly closed. For instance, bug 11496.

Can’t close bugs

There’s also no way for external contributors to close bugs. So Qt gets no outside help with bug triaging. This arbitrary separation between employees and external contributors is an obstacle to open development.

Tristan Van Berkom joined Openismus

General,

Tristan has been doing some challenging work for us recently, working on Glade, the GtkToolPallette, and finally finishing off the extended-layout improvements to GTK+ that Mathias Hasselmann started for Google’s Summer of Code in the days when he had free time. The extended layout fixes several user-noticeable problems in applications, making it easier for UIs to adjust properly to different screen sizes without hacks. It’s hard detailed work in lots of code, requiring Tristan’s methodical approach.

I hope that this will also allow some new types of container widgets. He might work on that when he’s back from Holiday in August.

Anyway, this post is just to say that Tristan has agreed to work with us for the foreseeable future so we always have someone with time to work on things like this.

Gitorious Merge Requests are Awkward for Small Changes

General,

Some projects on gitorious want all patches via git merge requests. This is very tedious, particularly for small changes. It’s therefore likely that many small improvements will never be submitted. Projects must care about that if they care about quality.

As far as I can tell, you must do it like this:

Create a remote clone

You can do that via the project’s top level web page. For instance. Then you need to “checkout” that clone on the command line.

Or reuse an existing clone, though you’ll have to rebase against the original, probably fixing some awkward conflicts from previous commits to your clone if you haven’t used branches for everything as I suggest below. It looks like people often create new clones for each merge request.

Create a branch in your remote clone

For instance:

  • git branch mr_dosomethingtofooforgoo master
  • git checkout mr_dosomethingtofooforgoo
  • Make your changes, add and commit them.
  • git push origin mr_dosomethingtofooforgoo

Request a Merge

Click the “Request Merge” button at the right, when looking at your clone’s top-level. For instance. There is no “Merge Request” button when looking at an individual commit, or even when looking at a branch. This would make it far easier to submit small changes.

Then you must copy/paste the first-line and details from your commit message into the two text fields on the Request Merge form.

You must also choose your branch name from a drop-down list and choose the commit from that branch that you want to use.

Then you can finally click “Create Merge Request”.

All gtkmm documentation moved to library.gnome.org

General,

*mm documentation on library.gnome.org

We’ve finally moved the API documentation for gtkmm and the other C++ bindings to library.gnome.org’s C++ Reference section. These are all created with Doxygen. Using library.gnome.org means that it’s updated every time we do a tarball release and we don’t lose old versions when it’s updated. We also have both the stable (on distros) and unstable (in progress) API documentation available.

We’ve also moved the gtkmm book, libsigc++ manual, and libxml++ tutorial to the Guides section. These all use DocBook XML.

library.gnome.org is a wonderful system, giving developers what they need, thanks to Frederic Peters’ hard work and responsiveness. I like it so much that Openismus might soon offer to set up library instances for other projects or companies.

Main Pages

Moving the API documentation to library.gnome.org made it obvious that we needed proper introductory text for each module. So each one now lists:

  • What it is
  • Where in the API to start looking, with links
  • How to #include the headers
  • How to get the include and linker flags with pkg-config

For instance, see the glibmm API reference’s main page.  I regularly complain that non-GNOME libraries don’t provide clues about what include or linker flags should be used, or how to include them, let alone API documentation, causing application builds to be inconsistent and fragile. It’s the least that should be done, so it’s good that I can now point to something consistent.

Lingering Problems

There are some small problems still to work out:

Autotools Things

General, ,

We are comfortably fond of autotools here at Openismus. It’s not perfect but we can use it, it does what we need, and it doesn’t have the problems that we see in other systems.

Autotools Tutorials

Years ago I wrote some popular introductory web pages showing how to do the obvious important stuff. David King has updated them and put the examples in git. They are now even simpler, because autotools got even better in the meantime. I think they will be useful.

Autotools with Qt Creator

Some of us like the Qt Creator IDE. So we asked Peter Penz to create a qt-creator plugin that provides some autotools support making it easier to use qt-creator with lots more existing projects. For instance, you can now have some code-completion in qt-creator with autotools-based projects, and you can run the whole build cycle from inside the IDE. Looking at the code, it seems fairly easy to create these plugins once you know how.

It currently works with the stable qt-creator 1.3 (ignore the uneven version number) though you’ll need to build it from source yourself for now. There is some interest from the qt-creator maintainers, so we hope to find time to update it for qt-creator’s master branch, which uses Qt 4.7 (currently unstable, but eventually stable with the same uneven version number), and then make a proper gitorious merge request.

By default, qt-creator uses qmake, though I think there is some CMake support. qmake is a time-wasting copy/paste-encouraging feature-lacking broken failure, so anything that helps people to avoid it must be good.

Jens Georg and Jon Nordby joining Openismus

General, ,

The Openismus family is expanding again. Jens Georg will join us as a developer in July, bringing both his open GNOME experience and closed/commercial Qt experience. His name is probably familiar to you from his from his work on Rygel or Anjuta.

Around the same time, Jon Nordby will become our third trainee, joining Chris and Patricia. I suspect they may not need much training, but that’s all good. Jon already works on MyPaint, which I keep hearing about.

I like them and I like having them in the company.

Openismus 2010 Trainees Chosen

General, , ,

We have chosen two new Openismus trainees who will start at the beginning of June: Patricia Santana Cruz and Chris Kühl. We are looking forward to the new life in our Berlin office and I bet they are looking forward to life in Berlin. I’m thinking of hiring a third trainee, so email me if you are interested.

After they have settled down, they will be studying hard and probably looking for easy ways to help real-world projects such as gnome-love, with our assistance. It won’t be long before we start thinking of them as junior developers instead. I don’t envy them the hard work but I do wish I’d had the same opportunity.

Friedrich Kossebau joining Openismus

General, ,

Last week we had the pleasure of meeting Friedrich Kossebau at our Berlin office and he accepted my offer to work for Openismus, starting at the beginning of May. Friedrich contacted me because we need more Qt expertise. His long KDE involvement fills that requirement perfectly and I think he has the temperament and attention to detail that we value. Unusually, we must send him right off to Helsinki, so we can’t immediately enjoy his company in Berlin. But I think he likes the adventure.

This means we are still looking for an extra Qt developer to work in our Berlin office, so please do email me.

Glom Python Documentation

General, ,

I’ve done some work to document Glom’s Python API. Here is a temporary snapshot, though I will try to get it onto library.gnome.org for the future. I used Sphinx, which is now used for Python itself. I’m fairly happy with it, though it’s not everything I hoped for.

What is the Glom Python API?

Glom lets you use Python to write field calculations (so field values can be based on other field values) and buttons scripts (stuff that happens when you press a button). That Python can code use the record and ui objects that Glom provides.

That Glom API is implemented by the pyglom (actually pyglom_1_14 currently) Python module. We use boost::python, writing the docstring comments in our C++ code, like so.

Bad Tools. Bad Syntax

Because it’s just a Python module, with docstrings, we can use standard Python documentation tools. However, I am not impressed by those tools. I am not impressed by Python documentation in general, and I suspect that the tools and conventions are the problem.

I understand that many developers don’t see the need for Python documentation because the arguments and return types for Python functions could be of any type and behaviour at runtime, but I believe in a deterministic universe, I know that most Python functions actually behave in a predictable way, I think that should be documented, and nobody can seriously write an application if they are really meant to doubt the behavior of the APIs they use. So let’s assume that Python APIs should be documented properly.

In the end, I had pretty good results with Sphinx, particularly after writing lots of documentation in its reStructuredText format. The output from pydoc was quite awful, though that’s partly because Glom’s python module is created with boost::python, which has some docstring issues and limitations.

But the main problem is that I can’t find any convention for documenting the arguments, return types and possible exceptions of Python functions. Various things (PEP 257, PEP 287,  reStructuredText, Sphinx’s reStructuredText primer) have been written about docstrings, but none of them mention this fundamental issue. So, these things are typically not documented, or are documented in a freeform unreliable way.

For comparison, here’s an example with doxygen for C++, using the simple Javadoc-style syntax:

/** A brief description in the first sentence.
* A more detailed description. This can be multiple sentences.
* Parameters such as @a boo can be mentioned here too, and
* mentions of an other_method() or an OtherClass will show up
* as links in the HTML. The parameter types and return type
* will be linked too.
*
* @param boo The goo's foo.
* @param hoo The voodoo that you do.
* @result The current moo.
*
* @throws FooError
* @deprecated Use some_other_method() instead, which does the necessary foo and bar.
*/
Moo some_method(const Boo& foo, Hoo hoo);

gtk-doc uses very similar conventions, aiming to document the same aspects of APIs, for C.

doxygen will even automatically create links to classes and functions that you mention. sphinx’s reStructuredText requires awkward syntax such :class:`Record`, which also unnecessarily shows the module name in the resulting HTML.

Note also that Sphinx doesn’t default to using docstrings anyway. It seems to want us to write API documentation in separate files. I’m convinced that it’s a mistake – I’ve only known APIs to be properly documented when the documentation is directly in the code as comments, for instance via doxygen or gtk-doc. Luckily I could use docstrings via Sphinx’s autodoc module, though that doesn’t support all of reStructuredText, such as section headings.

I also like how doyxgen or gtk-doc documentation generally lists the functions at the top, with brief descriptions, allowing me to click on a function’s link to jump to the detailed description. Instead, with the spinx-generated HTML, we have to scroll through all the details of the whole class. And all the module’s classes are on one HTML page, obscuring things even more. Maybe there’s some way to fix these issues via Sphinx configuration.