An interesting idea to accelerate the builds is to cache already generated object files. The Waf library provides a simple cache system by intercepting the task execution and retrieving files from the cache. Extensions are even provided to limit directory growth or to share the files over the network

In practice, implementing a cache layer on the build system level will not work very well. The following points are conclusions of years of experimentation on both open and closed-source projects:

1. The task signatures used for identifying tasks make poor keys for accessing the cache. Platform-specific command-line flags, characters (/ or \), and absolute paths severely limit the cache re-use.
2. Implementing different task signatures to work around the previous limitations (overriding BuildContext.hash_env_vars for example) will cause at best only performance issues (long startup time), and at worst mysterious cache reuse errors.
3. Because of the two previous points, the build system can become too brittle and too complex.
4. The Python runtime is essentially single-threaded. The build process is therefore unable to launch more tasks when retrieving files from the cache.

The best system so far is to wrap the compilers or the commands in the manner of ccache. While this requires some more work up front, the resulting builds are faster and more robust.

The ccache application is limited to C/C++ compilations, but it is easy to write command-line wrappers. Such wrappers can then access custom low-latency tcp servers for example.

Is Pypy an option for running Waf builds now? While Pypy 2.0 beta 1 still hangs on simple parallel builds, Pypy nightly (59365-f2f4cb496c1c) seems to work much better now.

The numbers below represent the best times of 10 runs on a 64-bit Ubuntu 12.10 laptop. The typical benchmark project was used for this purpose (./utils/genbench.py /tmp/build 50 100 15 5):

	cPython 2.7.3	pypy-c-jit	pypy-c-nojit
no-op build	0.76s	6.5s	7.7s
full build	39s	45.4s	48.3s

The no-op build times represent the time taken to load the serialized Python data without executing any command. Pypy is still using a pure python implementation if pickle, which is likely to take much more time than the C extension present in cPython.

This can explain the time differences on the full build times. If we substract these values, we can imagine that the Pypy runtime is getting nearly as fast as cPython.

Waf was originally created to ease the creation of KDE applications, but it has not worked so well in practice. The first versions of KDE 4 were terrible, and I think they discouraged anyone from using it ever again.

Fortunately, the version 4.9 has changed for the best, and it finally provides a pleasant development environment. At least, after the stability fixes (the plasma desktop does not crash anymore, the network manager just works), there are fewer annoyances than on other desktop environments. In particular, the focus stealing prevention policy helps to concentrate, and the apps do not pop-up password/keyring windows all the time anymore.

If Qt5 and KDE5 do not break the API too much, we should see more applications for KDE appearing over time.

Since Pypy does not work too well for multithreaded applications at the moment, so I am now stuck with cPython.

Since Python 2.7.3 is about as fast as Python 3.2 for my applications, I wondered what Python 3 optimizations could be backported to 2.7. The computed gotos patch did not look too complicated to adapt, so I have created my own version. Here are two files to add to build a computed-gotos-enabled cPython 2.7.3 interpreter: Python/ceval.c and Python/opcode_targets.h.

The optimization does not seem to make a visible difference on my applications though, even after recompiling with -fno-gcse/-fno-crossjumping.

Listing files on Windows platforms is not particularly fast, but detecting if files are folders, or listing the last modification times is extremely slow (os.path.isfile, os.stat). Such function calls become major bottlenecks on very large Windows builds. An effective workaround is to use the functions FindFirstFile and FindNextFile to list files and their properties at the same time while listing folders. The results can then be added to a cache for later use.

Though cPython provides access to these functions though ctypes, finding a good example is fairly difficult. Here is a short code snippet that works with Python 2:

import ctypes, ctypes.wintypes
FILE_ATTRIBUTE_DIRECTORY = 0x10
INVALID_HANDLE_VALUE = -1
BAN = (u'.', u'..')

FindFirstFile = ctypes.windll.kernel32.FindFirstFileW
FindNextFile  = ctypes.windll.kernel32.FindNextFileW
FindClose     = ctypes.windll.kernel32.FindClose

out  = ctypes.wintypes.WIN32_FIND_DATAW()
fldr = FindFirstFile(u"C:\\Windows\\*", ctypes.byref(out))
if fldr == INVALID_HANDLE_VALUE:
    raise ValueError("invalid handle!")
try:
    while True:
       if out.cFileName not in ban:
           isdir = out.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY
           ts = out.ftLastWriteTime
           timestamp = (ts.dwLowDateTime << 32) | ts.dwHighDateTime
           print str(out.cFileName), isdir, timestamp
       if not FindNextFile(fldr, ctypes.byref(out)):
           break
finally:
    FindClose(fldr)

To learn more about the attributes available on the "out" object, consult the msdn documentation on WIN32_FIND_DATAW