wafcache

Filesystem-based cache system to share and re-use build artifacts

Cache access operations (copy to and from) are delegated to independent pre-forked worker subprocesses.

Usage:

def build(bld):
        bld.load('wafcache')
        ...

To troubleshoot:

waf clean build --zone=wafcache

General parameters

Use the following environment variables

• WAFCACHE, which can be

  • An absolute path to the waf cache folder, by default:

    ~/.cache/wafcache_user # `user` represents the currently logged-in user
    

  • A URL to a cache server, for example:

    export WAFCACHE=http://localhost:8080/files/
    #in that case, `GET/POST` requests are made to urls of the form
    #`http://localhost:8080/files/000000000/0` (cache management is delegated to the server)
    

  • A GCS, S3 or MINIO bucket:

    gs://my-bucket/    # (uses gsutil command line tool or WAFCACHE_CMD)
    s3://my-bucket/    # (uses aws command line tool or WAFCACHE_CMD)
    minio://my-bucket/ # (uses mc command line tool or WAFCACHE_CMD)
    

• WAFCACHE_CMD: bucket upload/download command, for example:

  WAFCACHE_CMD="gsutil cp %{SRC} %{TGT}"
  

• WAFCACHE_NO_PUSH: if set, disables pushing to the cache

• WAFCACHE_VERBOSITY: if set, displays more detailed cache operations

• WAFCACHE_STATS: if set, displays cache usage statistics on exit

Remote buckets

The WAFCACHE bucket value is used for the source or destination depending on the operation (upload or download). For example, with:

WAFCACHE="gs://mybucket/"

the following commands may be run:

gsutil cp build/myprogram  gs://mybucket/aa/aaaaa/1
gsutil cp gs://mybucket/bb/bbbbb/2 build/somefile

File cache

Files are copied using hard links by default; if the cache is located onto another partition, the system switches to file copies instead.

Additional environments can be set:

• WAFCACHE_TRIM_MAX_FOLDER: maximum amount of tasks to cache (1M)

• WAFCACHE_EVICT_MAX_BYTES: maximum amount of cache size in bytes (10GB)

• WAFCACHE_EVICT_INTERVAL_MINUTES: minimum time interval to try and trim the cache (3 minutes)

Asynchronous transfers

• WAFCACHE_ASYNC_WORKERS: define a number of workers to upload results asynchronously

  this may improve build performance with many/long file uploads the default is unset (synchronous uploads)

• WAFCACHE_ASYNC_NOWAIT: do not wait for uploads to complete (default: False)

  this requires asynchonous uploads to have an effect

waflib.Tools.wafcache.can_retrieve_cache(self)

New method for waf Task classes

waflib.Tools.wafcache.put_files_cache(self)

New method for waf Task classes

waflib.Tools.wafcache.hash_env_vars(self, env, vars_lst)

Reimplement BuildContext.hash_env_vars so that the resulting hash does not depend on local paths

waflib.Tools.wafcache.uid(self)

Reimplement Task.uid() so that the signature does not depend on local paths

waflib.Tools.wafcache.make_cached(cls)

Enable the waf cache for a given task class

waflib.Tools.wafcache.get_process()

Returns a worker process that can process waf cache commands The worker process is assumed to be returned to the process pool when unused

waflib.Tools.wafcache.build(bld)

Called during the build process to enable file caching

waflib.Tools.wafcache.cache_command(proc, sig, files_from, files_to)

Create a command for cache worker processes, returns a pickled base64-encoded tuple containing the task signature, a list of files to cache and a list of files files to get from cache (one of the lists is assumed to be empty)

waflib.Tools.wafcache.copyfun(src, dst, *, src_dir_fd=None, dst_dir_fd=None, follow_symlinks=True)

Create a hard link to a file.

If either src_dir_fd or dst_dir_fd is not None, it should be a file

descriptor open to a directory, and the respective path string (src or dst) should be relative; the path will then be relative to that directory.

If follow_symlinks is False, and the last element of src is a symbolic

link, link will create a link to the symbolic link itself instead of the file the link points to.

src_dir_fd, dst_dir_fd, and follow_symlinks may not be implemented on your

platform. If they are unavailable, using them will raise a NotImplementedError.

waflib.Tools.wafcache.atomic_copy(orig, dest)

Copy files to the cache, the operation is atomic for a given file

waflib.Tools.wafcache.lru_trim()

the cache folders take the form: CACHE_DIR/0b/0b180f82246d726ece37c8ccd0fb1cde2650d7bfcf122ec1f169079a3bfc0ab9 they are listed in order of last access, and then removed until the amount of folders is within TRIM_MAX_FOLDERS and the total space taken by files is less than EVICT_MAX_BYTES

waflib.Tools.wafcache.lru_evict()

Reduce the cache size

waflib.Tools.wafcache.loop(service)

This function is run when this file is run as a standalone python script, it assumes a parent process that will communicate the commands to it as pickled-encoded tuples (one line per command)

The commands are to copy files to the cache or copy files from the cache to a target destination