compdb will pull in headers that are behind `ifdef`s

[roddux]

Issue

Running compdb on the Linux kernel results in a compile_commands.json file that includes selftest headers and other files, even if they are not scheduled to be built in the current Kconfig.

The issue is that these headers are gated behind #ifdef CONFIG_XYZ options, but the logic in includedb.py doesn't take these ifdefs into account:

compdb/compdb/includedb.py

Lines 102 to 114 in c145d02

	include_pattern = re.compile(
	br'\s*#\s*include\s+(?P<quote>["<])(?P<filename>.+?)[">]')
	for b_line in istream:
	b_match = re.match(include_pattern, b_line)
	if not b_match:
	continue
	u_quote = b_match.group('quote').decode('ascii')
	b_filename = b_match.group('filename')
	try:
	u_filename = b_filename.decode('utf-8')
	except UnicodeDecodeError:
	u_filename = b_filename.decode('latin-1')
	yield (u_quote, u_filename)

Solution?

Solving this is likely difficult!

I tried running each compile command with the -E flag to see what files are included, and parsing the output. This... works? But incorporating this would incur A LOT of overhead, and you would need to reliably modify the commands from the compilation database without inducing other side-effects; such as removing the -o parameter so you don't overwrite object files, etc.

I am not suggesting compdb incorporate these changes— I am creating this issue to document the behaviour in case anyone else stumbles upon this. Perhaps this could be noted explicitly in the README?

Example

An edited and contrived example, to demonstrate what I'm talking about:

$ cat test.c
#include <stdio.h>
int main() {
    char *var = "hello, world\n";
#ifdef DEBUG_OPTION
#include "test.h"
#endif
    puts(var);
}

$ cat test.h
var = "this is never used";

$ clang test.c -MJ compile_commands.json && ./a.out
hello, world

$ cat compile_commands.json
{
  "file": "test.c",
  "output": "/var/folders/ny/5_1gj3sx15z1fbykpqj5zlpr0000gn/T/test-3e2d71.o",
  ...
}

$ python3 -m compdb -p . list
[
  {
    "file": "test.c",
    ...
  },
  {
    "file": "test.h",
    ...
  }
]

[roddux]

I wrote a separate tool to do what I described above: run each command to the preprocessor stage by modifying the compilation commands to add -E and remove -o: https://github.com/roddux/header_fixer