The new "concurrent.futures" module from Python 3.2 will now make it even easier to execute tasks concurrently. We never miss an opportunity to remove some code so we had to try it, some functionality being similar to the contents of the Runner module.

First, we could not remove a much code as we hoped: fewer than 20 lines, most of the code being in the synchronization and the exception handling anyway. Then we met a surprising performance degradation linked to the creation of circular references.

Finally, the Python executor appeared to increase the builds by a few seconds on our build benchmarks, so we finally gave up.

The executor API looks good though, and many developers will be tempted to try to create new python-based build systems from it. Executing tasks represents only a very small part of what one expects in a build system though:

  • A system for handling commands and command-line options
  • A system of (task) order and dependencies
  • A handler for exceptions and user errors
  • A usable system (a very difficult problem)
  • An extension system for new programming languages
  • A portable system (not limited to Python 3.2)
  • Adequate performance (it is so easy to over synchronize or to write bad algorithms)

Developers seem to underestimate the amount of work required to make a system work for more than one or two projects (Waf has had nearly 11000 revisions in 6 years). Specialized and incomplete build scripts are still popping up. Just to name a few python-based ones:

The fragmentation seems unavoidable, and the situation is probably going to be similar to that of operating systems or programming languages, with a few big players and tons of niche projects.

If it must happen, then let it happen properly, at least. Starting in version 1.6.2, the Waf framework features a new build system kit which enables the creation of new build systems. By re-using the tested and maintained components from the Waf framework, much more time can be spent on more interesting problems such as creating an intuitive XML/YAML/JSON schema (if you believe that IDEs should edit your scripts) or creating a domain-specific programming language (make-like, cmake-like, ... ?), or extracting commands and dependencies to create derivated files (Makefiles, Visual studio, ...)

The few examples illustrate the possibilities offered by the Waf framework:

  • overview: Create a file using the Waf components to perform simple builds that do not use wscript files
  • parser: Add a parser for a domain-specific language, and create a file able to build targets from specific files
  • noscript: Create a file able to build targets without using a script file, finding the source files from header dependencies
  • nostate: Create a make-like build system that does not use build files (timestamps only)
  • makefile_dumper: Use the Waf API to dump the build decription into a makefile
Have a look and think of the kittens before trying to start a build system from scratch.