C/C++ preprocessor for finding dependencies

Reasons for using the Waf preprocessor by default

  1. Some c/c++ extensions (Qt) require a custom preprocessor for obtaining the dependencies (.moc files)
  2. Not all compilers provide .d files for obtaining the dependencies (portability)
  3. A naive file scanner will not catch the constructs such as “#include foo()”
  4. A naive file scanner will catch unnecessary dependencies (change an unused header -> recompile everything)

Regarding the speed concerns:

  • the preprocessing is performed only when files must be compiled
  • the macros are evaluated only for #if/#elif/#include
  • system headers are not scanned by default

Now if you do not want the Waf preprocessor, the tool +gccdeps* uses the .d files produced during the compilation to track the dependencies (useful when used with the boost libraries). It only works with gcc >= 4.4 though.

A dumb preprocessor is also available in the tool c_dumbpreproc

exception waflib.Tools.c_preproc.PreprocError(msg='', ex=None)[source]

Bases: waflib.Errors.WafError

waflib.Tools.c_preproc.POPFILE = '-'

Constant representing a special token used in waflib.Tools.c_preproc.c_parser.start() iteration to switch to a header read previously

waflib.Tools.c_preproc.recursion_limit = 150

Limit on the amount of files to read in the dependency scanner

waflib.Tools.c_preproc.go_absolute = False

Set to True to track headers on files in /usr/include, else absolute paths are ignored (but it becomes very slow)

waflib.Tools.c_preproc.use_trigraphs = 0

Apply trigraph rules (False by default)

waflib.Tools.c_preproc.g_optrans = {'and': '&&', 'and_eq': '&=', 'bitand': '&', 'bitor': '|', 'compl': '~', 'not': '!', 'not_eq': '!', 'or': '||', 'or_eq': '|=', 'xor': '^', 'xor_eq': '^='}

Operators such as and/or/xor for c++. Set an empty dict to disable.

waflib.Tools.c_preproc.re_lines = re.compile('^[ \t]*(?:#|%:)[ \t]*(ifdef|ifndef|if|else|elif|endif|include|import|define|undef|pragma)[ \t]*(.*)\r*$', re.IGNORECASE|re.MULTILINE)

Match #include lines

waflib.Tools.c_preproc.re_mac = re.compile('^[a-zA-Z_]\\w*')

Match macro definitions

waflib.Tools.c_preproc.re_fun = re.compile('^[a-zA-Z_][a-zA-Z0-9_]*[(]')

Match macro functions

waflib.Tools.c_preproc.re_pragma_once = re.compile('^\\s*once\\s*', re.IGNORECASE)

Match #pragma once statements

waflib.Tools.c_preproc.re_nl = re.compile('\\\\\r*\n', re.MULTILINE)

Match newlines

waflib.Tools.c_preproc.re_cpp = re.compile('//.*?$|/\\*.*?\\*/|\\\'(?:\\\\.|[^\\\\\\\'])*\\\'|"(?:\\\\.|[^\\\\"])*"', re.MULTILINE|re.DOTALL)

Filter C/C++ comments

waflib.Tools.c_preproc.trig_def = [('??=', '#'), ('??-', '~'), ('??/', '\\'), ('??!', '|'), ("??'", '^'), ('??(', '['), ('??)', ']'), ('??<', '{'), ('??>', '}')]

Trigraph definitions

waflib.Tools.c_preproc.chr_esc = {"'": 39, '0': 0, '\\': 92, 'a': 7, 'b': 8, 'f': 11, 'n': 10, 'r': 13, 't': 9, 'v': 12}

Escape characters

waflib.Tools.c_preproc.NUM = 'i'

Number token

waflib.Tools.c_preproc.OP = 'O'

Operator token

waflib.Tools.c_preproc.IDENT = 'T'

Identifier token

waflib.Tools.c_preproc.STR = 's'

String token

waflib.Tools.c_preproc.CHAR = 'c'

Character token

waflib.Tools.c_preproc.tok_types = ['i', 's', 'T', 'O']

Token types

waflib.Tools.c_preproc.exp_types = ["0[xX](?P<hex>[a-fA-F0-9]+)(?P<qual1>[uUlL]*)|L*?'(?P<char>(\\\\.|[^\\\\'])+)'|(?P<n1>\\d+)[Ee](?P<exp0>[+-]*?\\d+)(?P<float0>[fFlL]*)|(?P<n2>\\d*\\.\\d+)([Ee](?P<exp1>[+-]*?\\d+))?(?P<float1>[fFlL]*)|(?P<n4>\\d+\\.\\d*)([Ee](?P<exp2>[+-]*?\\d+))?(?P<float2>[fFlL]*)|(?P<oct>0*)(?P<n0>\\d+)(?P<qual2>[uUlL]*)", 'L?"([^"\\\\]|\\\\.)*"', '[a-zA-Z_]\\w*', '%:%:|<<=|>>=|\\.\\.\\.|<<|<%|<:|<=|>>|>=|\\+\\+|\\+=|--|->|-=|\\*=|/=|%:|%=|%>|==|&&|&=|\\|\\||\\|=|\\^=|:>|!=|##|[\\(\\)\\{\\}\\[\\]<>\\?\\|\\^\\*\\+&=:!#;,%/\\-\\?\\~\\.]']

Expression types

waflib.Tools.c_preproc.re_clexer = re.compile('(?P<i>0[xX](?P<hex>[a-fA-F0-9]+)(?P<qual1>[uUlL]*)|L*?\'(?P<char>(\\\\.|[^\\\\\'])+)\'|(?P<n1>\\d+)[Ee](?P<exp0>[+-]*?\\d+)(?P<float0>[fFlL]*)|(?P<n2>\\d*\\.\\d+)([Ee](?P<exp1>[+-]*?\\d+))?(?P<float1, re.MULTILINE)

Match expressions into tokens

waflib.Tools.c_preproc.accepted = 'a'

Parser state is accepted

waflib.Tools.c_preproc.ignored = 'i'

Parser state is ignored, for example preprocessor lines in an #if 0 block

waflib.Tools.c_preproc.undefined = 'u'

Parser state is undefined at the moment

waflib.Tools.c_preproc.skipped = 's'

Parser state is skipped, for example preprocessor lines in a #elif 0 block


Replace function used with waflib.Tools.c_preproc.re_cpp

waflib.Tools.c_preproc.prec = {'!=': 4, '%': 0, '&': 5, '&&': 6, '*': 0, '+': 1, ',': 7, '-': 1, '/': 0, '<': 3, '<<': 2, '<=': 3, '==': 4, '>': 3, '>=': 3, '>>': 2, '^': 5, '|': 5, '||': 6}

Operator precedence rules required for parsing expressions of the form:

#if 1 && 2 != 0
waflib.Tools.c_preproc.reduce_nums(val_1, val_2, val_op)[source]

Apply arithmetic rules to compute a result

  • val1 (int or string) – input parameter
  • val2 (int or string) – input parameter
  • val_op (string) – C operator in +, /, -, etc
Return type:



Try to obtain a number from a list of tokens. The token types are defined in waflib.Tools.ccroot.tok_types.

Parameters:lst (list of tuple (tokentype, value)) – list of preprocessor tokens
Returns:a pair containing the number and the rest of the list
Return type:tuple(value, list)

Evaluate an expression recursively, for example:

1+1+1 -> 2+1 -> 3
Parameters:lst (list of tuple(token, value)) – list of tokens
Returns:the value and the remaining tokens
Return type:value, list

Take a list of tokens and output true or false for #if/#elif conditions.

Parameters:lst (list of tuple(token, value)) – a list of tokens
Returns:a token
Return type:tuple(NUM, int)

Merge a list of tokens into a string

Parameters:lst (list of tuple(token, value)) – a list of tokens
Return type:string
waflib.Tools.c_preproc.paste_tokens(t1, t2)[source]

Token pasting works between identifiers, particular operators, and identifiers and numbers:

a ## b  ->  ab
> ## =  ->  >=
a ## 2  ->  a2
  • t1 (tuple(type, value)) – token
  • t2 (tuple(type, value)) – token
waflib.Tools.c_preproc.reduce_tokens(lst, defs, ban=[])[source]

Replace the tokens in lst, using the macros provided in defs, and a list of macros that cannot be re-applied

  • lst (list of tuple(token, value)) – list of tokens
  • defs (dict) – macro definitions
  • ban (list of string) – macros that cannot be substituted (recursion is not allowed)

the new list of tokens

Return type:

value, list

waflib.Tools.c_preproc.eval_macro(lst, defs)[source]

Reduce the tokens by waflib.Tools.c_preproc.reduce_tokens() and try to return a 0/1 result by waflib.Tools.c_preproc.reduce_eval().

  • lst (list of tuple(token, value)) – list of tokens
  • defs (dict) – macro definitions
Return type:


Process a macro definition of the form::
#define f(x, y) x * y

into a function or a simple macro without arguments

Parameters:txt (string) – expression to exact a macro definition from
Returns:a tuple containing the name, the list of arguments and the replacement
Return type:tuple(string, [list, list])
waflib.Tools.c_preproc.extract_include(txt, defs)[source]

Process a line in the form:

#include foo
  • txt (string) – include line to process
  • defs (dict) – macro definitions

the file name

Return type:



Parse a c character

Parameters:txt (string) – character to parse
Returns:a character literal
Return type:string

Convert a string into a list of tokens (shlex.split does not apply to c/c++/d)

Parameters:s (string) – input to tokenize
Returns:a list of tokens
Return type:list of tuple(token, value)
class waflib.Tools.c_preproc.c_parser(nodepaths=None, defines=None)[source]

Bases: object

Used by waflib.Tools.c_preproc.scan() to parse c/h files. Note that by default, only project headers are parsed.

lines = None

list of lines read

nodepaths = None

Include paths

nodes = None

List of waflib.Node.Node found so far

names = None

List of file names that could not be matched by any file

curfile = None

Current file

ban_includes = None

Includes that must not be read (#pragma once)

listed = None

Include nodes/names already listed to avoid duplicates in self.nodes/self.names

cached_find_resource(node, filename)[source]

Find a file from the input directory


the node if found, or None

Return type:


tryfind(filename, kind='"', env=None)[source]

Try to obtain a node from the filename based from the include paths. Will add the node found to waflib.Tools.c_preproc.c_parser.nodes or the file name to waflib.Tools.c_preproc.c_parser.names if no corresponding file is found. Called by waflib.Tools.c_preproc.c_parser.start.

Parameters:filename (string) – header to find
Returns:the node if found
Return type:waflib.Node.Node

Filter the comments from a c/h file, and return the preprocessor lines. The regexps waflib.Tools.c_preproc.re_cpp, waflib.Tools.c_preproc.re_nl and waflib.Tools.c_preproc.re_lines are used internally.

Returns:the preprocessor directives as a list of (keyword, line)
Return type:a list of string pairs

Add the lines from a header in the list of preprocessor lines to parse

Parameters:node (waflib.Node.Node) – header
start(node, env)[source]

Preprocess a source file to obtain the dependencies, which are accumulated to waflib.Tools.c_preproc.c_parser.nodes and waflib.Tools.c_preproc.c_parser.names.

Parameters:line (string) – define line
Return type:string
Returns:the define name

Get the dependencies using a c/c++ preprocessor, this is required for finding dependencies of the kind:

#include some_macro()

This function is bound as a task method on waflib.Tools.c.c and waflib.Tools.cxx.cxx for example