c_preproc

C/C++ preprocessor for finding dependencies

Reasons for using the Waf preprocessor by default

  1. Some c/c++ extensions (Qt) require a custom preprocessor for obtaining the dependencies (.moc files)
  2. Not all compilers provide .d files for obtaining the dependencies (portability)
  3. A naive file scanner will not catch the constructs such as “#include foo()”
  4. A naive file scanner will catch unnecessary dependencies (change an unused header -> recompile everything)

Regarding the speed concerns:

  • the preprocessing is performed only when files must be compiled
  • the macros are evaluated only for #if/#elif/#include
  • system headers are not scanned by default

Now if you do not want the Waf preprocessor, the tool +gccdeps* uses the .d files produced during the compilation to track the dependencies (useful when used with the boost libraries). It only works with gcc >= 4.4 though.

A dumb preprocessor is also available in the tool c_dumbpreproc

waflib.Tools.c_preproc.POPFILE = '-'

Constant representing a special token used in waflib.Tools.c_preproc.c_parser.start() iteration to switch to a header read previously

waflib.Tools.c_preproc.recursion_limit = 150

Limit on the amount of files to read in the dependency scanner

waflib.Tools.c_preproc.go_absolute = False

Set to True to track headers on files in /usr/include, else absolute paths are ignored (but it becomes very slow)

waflib.Tools.c_preproc.use_trigraphs = 0

Apply trigraph rules (False by default)

waflib.Tools.c_preproc.g_optrans = {'and': '&&', 'and_eq': '&=', 'bitand': '&', 'bitor': '|', 'compl': '~', 'not': '!', 'not_eq': '!', 'or': '||', 'or_eq': '|=', 'xor': '^', 'xor_eq': '^='}

Operators such as and/or/xor for c++. Set an empty dict to disable.

waflib.Tools.c_preproc.re_lines = <_sre.SRE_Pattern object>

Match #include lines

waflib.Tools.c_preproc.re_mac = <_sre.SRE_Pattern object>

Match macro definitions

waflib.Tools.c_preproc.re_fun = <_sre.SRE_Pattern object>

Match macro functions

waflib.Tools.c_preproc.re_pragma_once = <_sre.SRE_Pattern object>

Match #pragma once statements

waflib.Tools.c_preproc.re_nl = <_sre.SRE_Pattern object>

Match newlines

waflib.Tools.c_preproc.re_cpp = <_sre.SRE_Pattern object>

Filter C/C++ comments

waflib.Tools.c_preproc.trig_def = [('??=', '#'), ('??-', '~'), ('??/', '\\'), ('??!', '|'), ("??'", '^'), ('??(', '['), ('??)', ']'), ('??<', '{'), ('??>', '}')]

Trigraph definitions

waflib.Tools.c_preproc.chr_esc = {"'": 39, '0': 0, '\\': 92, 'a': 7, 'b': 8, 'f': 11, 'n': 10, 'r': 13, 't': 9, 'v': 12}

Escape characters

waflib.Tools.c_preproc.NUM = 'i'

Number token

waflib.Tools.c_preproc.OP = 'O'

Operator token

waflib.Tools.c_preproc.IDENT = 'T'

Identifier token

waflib.Tools.c_preproc.STR = 's'

String token

waflib.Tools.c_preproc.CHAR = 'c'

Character token

waflib.Tools.c_preproc.tok_types = ['i', 's', 'T', 'O']

Token types

waflib.Tools.c_preproc.exp_types = ["0[xX](?P<hex>[a-fA-F0-9]+)(?P<qual1>[uUlL]*)|L*?'(?P<char>(\\\\.|[^\\\\'])+)'|(?P<n1>\\d+)[Ee](?P<exp0>[+-]*?\\d+)(?P<float0>[fFlL]*)|(?P<n2>\\d*\\.\\d+)([Ee](?P<exp1>[+-]*?\\d+))?(?P<float1>[fFlL]*)|(?P<n4>\\d+\\.\\d*)([Ee](?P<exp2>[+-]*?\\d+))?(?P<float2>[fFlL]*)|(?P<oct>0*)(?P<n0>\\d+)(?P<qual2>[uUlL]*)", 'L?"([^"\\\\]|\\\\.)*"', '[a-zA-Z_]\\w*', '%:%:|<<=|>>=|\\.\\.\\.|<<|<%|<:|<=|>>|>=|\\+\\+|\\+=|--|->|-=|\\*=|/=|%:|%=|%>|==|&&|&=|\\|\\||\\|=|\\^=|:>|!=|##|[\\(\\)\\{\\}\\[\\]<>\\?\\|\\^\\*\\+&=:!#;,%/\\-\\?\\~\\.]']

Expression types

waflib.Tools.c_preproc.re_clexer = <_sre.SRE_Pattern object>

Match expressions into tokens

waflib.Tools.c_preproc.accepted = 'a'

Parser state is accepted

waflib.Tools.c_preproc.ignored = 'i'

Parser state is ignored, for example preprocessor lines in an #if 0 block

waflib.Tools.c_preproc.undefined = 'u'

Parser state is undefined at the moment

waflib.Tools.c_preproc.skipped = 's'

Parser state is skipped, for example preprocessor lines in a #elif 0 block

waflib.Tools.c_preproc.repl(m)[source]

Replace function used with waflib.Tools.c_preproc.re_cpp

waflib.Tools.c_preproc.prec = {'!=': 4, '%': 0, '&': 5, '&&': 6, '*': 0, '+': 1, ',': 7, '-': 1, '/': 0, '<': 3, '<<': 2, '<=': 3, '==': 4, '>': 3, '>=': 3, '>>': 2, '^': 5, '|': 5, '||': 6}

Operator precedence rules required for parsing expressions of the form:

#if 1 && 2 != 0
waflib.Tools.c_preproc.reduce_nums(val_1, val_2, val_op)[source]

Apply arithmetic rules to compute a result

Parameters:
  • val1 (int or string) – input parameter
  • val2 (int or string) – input parameter
  • val_op (string) – C operator in +, /, -, etc
Return type:

int

waflib.Tools.c_preproc.get_num(lst)[source]

Try to obtain a number from a list of tokens. The token types are defined in waflib.Tools.ccroot.tok_types.

Parameters:lst (list of tuple (tokentype, value)) – list of preprocessor tokens
Returns:a pair containing the number and the rest of the list
Return type:tuple(value, list)
waflib.Tools.c_preproc.get_term(lst)[source]

Evaluate an expression recursively, for example:

1+1+1 -> 2+1 -> 3
Parameters:lst (list of tuple(token, value)) – list of tokens
Returns:the value and the remaining tokens
Return type:value, list
waflib.Tools.c_preproc.reduce_eval(lst)[source]

Take a list of tokens and output true or false for #if/#elif conditions.

Parameters:lst (list of tuple(token, value)) – a list of tokens
Returns:a token
Return type:tuple(NUM, int)
waflib.Tools.c_preproc.stringize(lst)[source]

Merge a list of tokens into a string

Parameters:lst (list of tuple(token, value)) – a list of tokens
Return type:string
waflib.Tools.c_preproc.paste_tokens(t1, t2)[source]

Token pasting works between identifiers, particular operators, and identifiers and numbers:

a ## b  ->  ab
> ## =  ->  >=
a ## 2  ->  a2
Parameters:
  • t1 (tuple(type, value)) – token
  • t2 (tuple(type, value)) – token
waflib.Tools.c_preproc.reduce_tokens(lst, defs, ban=[])[source]

Replace the tokens in lst, using the macros provided in defs, and a list of macros that cannot be re-applied

Parameters:
  • lst (list of tuple(token, value)) – list of tokens
  • defs (dict) – macro definitions
  • ban (list of string) – macros that cannot be substituted (recursion is not allowed)
Returns:

the new list of tokens

Return type:

value, list

waflib.Tools.c_preproc.eval_macro(lst, defs)[source]

Reduce the tokens by waflib.Tools.c_preproc.reduce_tokens() and try to return a 0/1 result by waflib.Tools.c_preproc.reduce_eval().

Parameters:
  • lst (list of tuple(token, value)) – list of tokens
  • defs (dict) – macro definitions
Return type:

int

waflib.Tools.c_preproc.extract_macro(txt)[source]
Process a macro definition of the form::
#define f(x, y) x * y

into a function or a simple macro without arguments

Parameters:txt (string) – expression to exact a macro definition from
Returns:a tuple containing the name, the list of arguments and the replacement
Return type:tuple(string, [list, list])
waflib.Tools.c_preproc.extract_include(txt, defs)[source]

Process a line in the form:

#include foo
Parameters:
  • txt (string) – include line to process
  • defs (dict) – macro definitions
Returns:

the file name

Return type:

string

waflib.Tools.c_preproc.parse_char(txt)[source]

Parse a c character

Parameters:txt (string) – character to parse
Returns:a character literal
Return type:string
waflib.Tools.c_preproc.tokenize(s)[source]

Convert a string into a list of tokens (shlex.split does not apply to c/c++/d)

Parameters:s (string) – input to tokenize
Returns:a list of tokens
Return type:list of tuple(token, value)
class waflib.Tools.c_preproc.c_parser(nodepaths=None, defines=None)[source]

Bases: object

Used by waflib.Tools.c_preproc.scan() to parse c/h files. Note that by default, only project headers are parsed.

lines = None

list of lines read

nodepaths = None

Include paths

nodes = None

List of waflib.Node.Node found so far

names = None

List of file names that could not be matched by any file

curfile = None

Current file

ban_includes = None

Includes that must not be read (#pragma once)

listed = None

Include nodes/names already listed to avoid duplicates in self.nodes/self.names

cached_find_resource(node, filename)[source]

Find a file from the input directory

Parameters:
Returns:

the node if found, or None

Return type:

waflib.Node.Node

tryfind(filename, kind='"', env=None)[source]

Try to obtain a node from the filename based from the include paths. Will add the node found to waflib.Tools.c_preproc.c_parser.nodes or the file name to waflib.Tools.c_preproc.c_parser.names if no corresponding file is found. Called by waflib.Tools.c_preproc.c_parser.start.

Parameters:filename (string) – header to find
Returns:the node if found
Return type:waflib.Node.Node
filter_comments(node)[source]

Filter the comments from a c/h file, and return the preprocessor lines. The regexps waflib.Tools.c_preproc.re_cpp, waflib.Tools.c_preproc.re_nl and waflib.Tools.c_preproc.re_lines are used internally.

Returns:the preprocessor directives as a list of (keyword, line)
Return type:a list of string pairs
addlines(node)[source]

Add the lines from a header in the list of preprocessor lines to parse

Parameters:node (waflib.Node.Node) – header
start(node, env)[source]

Preprocess a source file to obtain the dependencies, which are accumulated to waflib.Tools.c_preproc.c_parser.nodes and waflib.Tools.c_preproc.c_parser.names.

Parameters:
define_name(line)[source]
Parameters:line (string) – define line
Return type:string
Returns:the define name
waflib.Tools.c_preproc.scan(task)[source]

Get the dependencies using a c/c++ preprocessor, this is required for finding dependencies of the kind:

#include some_macro()

This function is bound as a task method on waflib.Tools.c.c and waflib.Tools.cxx.cxx for example