wscript file headers
#! /usr/bin/env python
# encoding: utf-8Several benefits have been observed over the time, and it appears that this header is a best practice:
- Most text editors - especially on unix-like systems - will read the first line beginning by #! to apply proper syntax highlighting. Your text editor may well display the file properly, but others may not.
- If the wscript files are ever meant to be executed, then:
- The process will not hang when called as ./wscript (the 'import' program takes screenshots)
- The /usr/bin/env will redirect the interpreter to the adequate python executable (python can be installed in /usr/local or /opt for example)
- The encoding declaration will prevent special characters such as €, é or í from causing errors
When a wscript file is created for the first time, it may have to be reloaded to force the text editor to take the file type into account. In Vim at least, a mapping can be added to save the hassle of re-opening the file:
au BufNewFile,BufRead wscript* set filetype=python
Running Waf on different Python interpreters
A few of you may be curious to know how well Waf performs on different python interpreters. Different propjects may have different requirements, so the tests are always biased in one way or another. Among all the tests considered so far, the genbench script provides a good balance between project size, process execution, regexps and parsing. The numbers collected represent the best run in in five executions:
./genbench.py /tmp/build 50 100 15 5
cd /tmp/build
waf clean; time waf build -p -j3
On Windows XP, the IronPython runtime is fairly close to the Pypy one. And Pypy 1.5 did not complete normally (file descriptor leak).
| cPython 2.7 | IronPython 2.7 | Pypy 1.5 |
|---|---|---|
| 1m54 | 2m14 | 2m05 |
IronPython requires .NET 4, and it would not run on Linux using Mono 2.10.2. It is no surprize that the test completes faster as spawning new processes (fork) is slower on Windows. Pypy 1.5 could not even complete normally once in about 10 runs.
| python 2.6.5 | python 3.2 | jython 2.5.2 |
|---|---|---|
| 1m41 | 1m41 | 5m30 |
For the moment, the best interpreter remains cPython for both portability and performance.
Debugging tools
The module "why" provides some more output to help understanding why files are being compiled. It requires a fresh recompilation though, and inflates the cache files significantly as it stores some data between the builds. The output will display the task signatures and the reason of the execution:
12:26:20 /comp/waf/demos/c> ./waf build --zones=task
Waf: Entering directory `/disk/comp/waf/demos/c/build'
12:26:21 task -> Task must run : c3490b5e8498da72aef5a9f645d72276 39da3e173d11cdbb273cda14856dbb6a
12:26:21 task -> * Source file or manual dependency : 3d3fde4f5fc0124efdba8e94e1d216f8 5e4ef70cc1618412ef9975baab0157c1
[12/15] c: program/main.c -> build/program/main.c.1.o
Waf: Leaving directory `/disk/comp/waf/demos/c/build'
'build' finished successfully (0.089s)
For performance debugging, the module parallel_debug creates pictures representing when tasks are executed during the build. The diagrams show how well the build can be distributed, and therefore how fast the build can be. Generating the diagrams incurs a minor performance penalty, which is the reason why this tool is not included by default in the waf file.
It has been updated recently to improve the layout, now tooltips are present in the svg output file. Follow the links to consult the diagrams of the Samba 4 developer build for 2 processors and for 8 processors.
Colors in cmd
The progress bar, which was broken for a long time, is even displayed properly (waf 1.6.4):
The display in msys it also known to work in waf 1.6:
Right now, the progress bar cannot display the very last characters on Windows Vista (issue 931). The platform-specific and underdocumented functions from the module windll make it very difficult to understand and fix the problems though
windll.kernel32.FillConsoleOutputCharacterA
New python projects resembling ansiterm.py have appeared recently, here are for example:
* Colorama
* Colorconsole
Both also split the small amount of code into several files, which make the re-use only more difficult, and neither seems to support clearing single lines, which is mandatory for displaying progress bars properly.
Python 3.2, and the build system kit
The new "concurrent.futures" module from Python 3.2 will now make it even easier to execute tasks concurrently. We never miss an opportunity to remove some code so we had to try it, some functionality being similar to the contents of the Runner module.
First, we could not remove a much code as we hoped: fewer than 20 lines, most of the code being in the synchronization and the exception handling anyway. Then we met a surprising performance degradation linked to the creation of circular references.
Finally, the Python executor appeared to increase the builds by a few seconds on our build benchmarks, so we finally gave up.
The executor API looks good though, and many developers will be tempted to try to create new python-based build systems from it. Executing tasks represents only a very small part of what one expects in a build system though:
- A system for handling commands and command-line options
- A system of (task) order and dependencies
- A handler for exceptions and user errors
- A usable system (a very difficult problem)
- An extension system for new programming languages
- A portable system (not limited to Python 3.2)
- Adequate performance (it is so easy to over synchronize or to write bad algorithms)
Developers seem to underestimate the amount of work required to make a system work for more than one or two projects (Waf has had nearly 11000 revisions in 6 years). Specialized and incomplete build scripts are still popping up. Just to name a few python-based ones:
If it must happen, then let it happen properly, at least. Starting in version 1.6.2, the Waf framework features a new build system kit which enables the creation of new build systems. By re-using the tested and maintained components from the Waf framework, much more time can be spent on more interesting problems such as creating an intuitive XML/YAML/JSON schema (if you believe that IDEs should edit your scripts) or creating a domain-specific programming language (make-like, cmake-like, ... ?), or extracting commands and dependencies to create derivated files (Makefiles, Visual studio, ...)
The few examples illustrate the possibilities offered by the Waf framework:
- overview: Create a file using the Waf components to perform simple builds that do not use wscript files
- parser: Add a parser for a domain-specific language, and create a file able to build targets from specific files
- noscript: Create a file able to build targets without using a script file, finding the source files from header dependencies
- nostate: Create a make-like build system that does not use build files (timestamps only)
- makefile_dumper: Use the Waf API to dump the build decription into a makefile