Add support for -Xclang -ast-dump to facilitate debugging.

clang/docs/checkedc/3C/CONTRIBUTING.md

@@ -49,128 +49,18 @@ working, so you may have to wait for us to address 5C-specific
 problems arising from your 3C pull request and/or we may ask you to
 make specific changes to your pull request to accommodate 5C's code.
 
-## Testing
-
-3C has a regression test suite located in `clang/test/3C`. At the
-appropriate time during development of a pull request, please run it
-and correct any failures. (For example, it may not make sense to run
-it on a draft pull request containing an unfinished demonstration of
-an idea.) The easiest way to run it is to run the following in your
-build directory:
-
-```
-ninja check-3c
-```
-
-This command will build everything needed that hasn't already been
-built, run the test suite, report success or failure (exit 0 or 1, so
-you can use it in scripts), and display some information about any
-failures, which may or may not be enough for you to understand what
-went wrong.
-
-For deeper troubleshooting, run the following in your build directory
-to build all dependencies of the test suite:
-
-```
-ninja check-3c-deps
-```
-
-Then run the following in the `clang/test/3C` directory:
-
-```
-llvm-lit -vv TEST.c
-```
-
-where `TEST.c` is the path of the test you want to run (you can also
-specify more than one test). This assumes you've put the `bin`
-subdirectory of your build directory on your `$PATH` or arranged some
-other means of running `llvm-lit` from there. The first `-v` makes
-`llvm-lit` display the stdout and stderr of failed tests; the second
-makes it display the `RUN` commands as they execute so you can tell
-which one failed.
-
-Every `.c` file under `clang/test/3C` is a test file. There are a few
-in subdirectories, so `*.c` will not pick up all of them; instead you
-can use `llvm-lit -vv .` to specify all test files under the current
-directory.
-
-### Diagnostic verification
-
-3C supports the standard Clang diagnostic verifier
-([`VerifyDiagnosticConsumer`](https://clang.llvm.org/doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details))
-for testing errors and warnings reported by 3C via its main `DiagnosticsEngine`.
-(Some 3C errors and warnings are reported via other means and cannot be tested
-this way; the best solution we have for them right now is to `grep` the stderr
-of 3C.) Diagnostic verification can be enabled via the usual `-Xclang -verify`
-compiler option; other diagnostic verification options (`-Xclang
--verify=PREFIX`, etc.) should also work as normal. These must be passed as
-_compiler_ options, not `3c` options; for example, if you are using `--` on the
-`3c` command line, these options must be passed _after_ the `--`.
-
-Some notes about diagnostic verification in the context of 3C:
-
-* Parsing of the source files uses some of the compiler logic and thus may
-  generate compiler warnings, just as if you ran `clang` on the code. These are
-  sent to the diagnostic verifier along with diagnostics generated by 3C's
-  analysis. If you find it distracting to have to include the compiler warnings
-  in the set of expected diagnostics for a test, you can turn them off via the
-  `-Wno-everything` compiler option (which does not affect diagnostics generated
-  by 3C's analysis).
-
-* The `3c` tool works in several passes, where each pass runs on all translation
-  units: first `3c` parses the source files, then it runs several passes of
-  analysis. If a pass encounters at least one error, `3c` exits at the end of
-  that pass. Diagnostic verification does not change the _point_ at which `3c`
-  exits, but it changes the exit _code_ to indicate the result of verification
-  rather than the presence of errors. The verification includes the diagnostics
-  from all passes up to the point at which `3c` exits (i.e., the same
-  diagnostics that would be displayed if verification were not used). However,
-  an error that doesn't go via the main `DiagnosticsEngine` will cause an
-  unsuccessful exit code regardless of diagnostic verification. (This is
-  typically the behavior you want for a test.)
-
-* Diagnostic verification is independent for each translation unit, so in tests
-  with multiple translation units, you'll have to be careful that preprocessing
-  of each translation unit sees the correct set of `expected-*` directives for
-  the diagnostics generated for that translation unit (or an
-  `expected-no-diagnostics` directive if that translation unit generates no
-  diagnostics, even if other translation units do generate diagnostics). Be
-  warned that since which translation unit generated a given diagnostic isn't
-  visible to a normal user, we don't put much work into coming up with sensible
-  rules for this, but it should at least be deterministic for testing.
-
-Note that some 3C tests use diagnostic verification on calls to `clang` rather
-than `3c`, so if you see `expected-*` directives in a test, you can look at the
-`RUN` commands to see which command has `-Xclang -verify` and is using the
-directives. If you want to verify diagnostics of more than one `RUN` command in
-the same test, you can use different directive prefixes (`-Xclang
--verify=PREFIX`).
-
-## Coding guidelines
-
-Please follow [LLVM coding
-standards](https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly)
-in your code. Specifically:
-
-* The maximum length of a line: 80 chars
-
-* All comments should start with a Capital letter.
-
-* All Local variables, including fields and parameters, should start
-  with a Capital letter (similar to PascalCase). Short names are
-  preferred.
-
-* A space between the conditional keyword and `(` i.e., `if (`,
-  `while (`, ``for (` etc.
-
-* Space after the type name, i.e., `Type *K` _not_ `Type* K`.
-
-* Space before and after `:` in iterators, i.e., `for (auto &k : List)`
-
-Our goal is that all files should be formatted with `clang-format` and
-pass `clang-tidy` ([more information](clang-tidy.md)), and nonempty
-files should have a final newline (surprisingly, `clang-format` cannot
-enforce this). However, until we have better automation, we decided it
-isn't reasonable to require contributors to manually run these tools
-and fix style nits in each change; instead, we periodically run the
-tools on the entire 3C codebase.
+At the appropriate time during development of a pull request, please
+run the [regression tests](development.md#regression-tests) and
+correct any failures. (For example, it may not make sense to do this
+on a draft pull request containing an unfinished demonstration of an
+idea.) All regression tests must pass (or be disabled if appropriate)
+before your pull request can be merged. If you're changing behavior
+(as opposed to just cleaning up the code), we'll typically require you
+to add or update enough tests to exercise the important behavior
+changes (i.e., those tests fail before your code change and pass after
+it). If there's a concern that your change might affect other cases
+that are not adequately tested yet, we may ask you to add tests for
+those cases as well.
+
+See the [developer's guide](development.md) for additional information
+that may be helpful as you work on 3C.

clang/docs/checkedc/3C/development.md

@@ -0,0 +1,144 @@
+# 3C Developer's Guide
+
+This file collects information that is important or useful to know
+when developing 3C as well as some standards we generally expect the
+code to follow. (Some other standards are covered in the [contribution
+guide](CONTRIBUTING.md)). If you could use help with something that's
+not documented here, feel free to ask.
+
+## Regression tests
+
+3C has a regression test suite located in `clang/test/3C`. The easiest
+way to run it is to run the following in your build directory:
+
+```
+ninja check-3c
+```
+
+This command will build everything needed that hasn't already been
+built, run the test suite, report success or failure (exit 0 or 1, so
+you can use it in scripts), and display some information about any
+failures, which may or may not be enough for you to understand what
+went wrong.
+
+For deeper troubleshooting, run the following in your build directory
+to build all dependencies of the test suite:
+
+```
+ninja check-3c-deps
+```
+
+Then run the following in the `clang/test/3C` directory:
+
+```
+llvm-lit -vv TEST.c
+```
+
+where `TEST.c` is the path of the test you want to run (you can also
+specify more than one test). This assumes you've put the `bin`
+subdirectory of your build directory on your `$PATH` or arranged some
+other means of running `llvm-lit` from there. The first `-v` makes
+`llvm-lit` display the stdout and stderr of failed tests; the second
+makes it display the `RUN` commands as they execute so you can tell
+which one failed.
+
+Every `.c` file under `clang/test/3C` is a test file. There are a few
+in subdirectories, so `*.c` will not pick up all of them; instead you
+can use `llvm-lit -vv .` to specify all test files under the current
+directory.
+
+### Diagnostic verification
+
+3C supports the standard Clang diagnostic verifier
+([`VerifyDiagnosticConsumer`](https://clang.llvm.org/doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details))
+for testing errors and warnings reported by 3C via its main `DiagnosticsEngine`.
+(Some 3C errors and warnings are reported via other means and cannot be tested
+this way; the best solution we have for them right now is to `grep` the stderr
+of 3C.) Diagnostic verification can be enabled via the usual `-Xclang -verify`
+compiler option; other diagnostic verification options (`-Xclang
+-verify=PREFIX`, etc.) should also work as normal. These must be passed as
+_compiler_ options, not `3c` options; for example, if you are using `--` on the
+`3c` command line, these options must be passed _after_ the `--`.
+
+Some notes about diagnostic verification in the context of 3C:
+
+* Parsing of the source files uses some of the compiler logic and thus may
+  generate compiler warnings, just as if you ran `clang` on the code. These are
+  sent to the diagnostic verifier along with diagnostics generated by 3C's
+  analysis. If you find it distracting to have to include the compiler warnings
+  in the set of expected diagnostics for a test, you can turn them off via the
+  `-Wno-everything` compiler option (which does not affect diagnostics generated
+  by 3C's analysis).
+
+* The `3c` tool works in several passes, where each pass runs on all translation
+  units: first `3c` parses the source files, then it runs several passes of
+  analysis. If a pass encounters at least one error, `3c` exits at the end of
+  that pass. Diagnostic verification does not change the _point_ at which `3c`
+  exits, but it changes the exit _code_ to indicate the result of verification
+  rather than the presence of errors. The verification includes the diagnostics
+  from all passes up to the point at which `3c` exits (i.e., the same
+  diagnostics that would be displayed if verification were not used). However,
+  an error that doesn't go via the main `DiagnosticsEngine` will cause an
+  unsuccessful exit code regardless of diagnostic verification. (This is
+  typically the behavior you want for a test.)
+
+* Diagnostic verification is independent for each translation unit, so in tests
+  with multiple translation units, you'll have to be careful that preprocessing
+  of each translation unit sees the correct set of `expected-*` directives for
+  the diagnostics generated for that translation unit (or an
+  `expected-no-diagnostics` directive if that translation unit generates no
+  diagnostics, even if other translation units do generate diagnostics). Be
+  warned that since which translation unit generated a given diagnostic isn't
+  visible to a normal user, we don't put much work into coming up with sensible
+  rules for this, but it should at least be deterministic for testing.
+
+Note that some 3C tests use diagnostic verification on calls to `clang` rather
+than `3c`, so if you see `expected-*` directives in a test, you can look at the
+`RUN` commands to see which command has `-Xclang -verify` and is using the
+directives. If you want to verify diagnostics of more than one `RUN` command in
+the same test, you can use different directive prefixes (`-Xclang
+-verify=PREFIX`).
+
+## Debugging
+
+### AST dumping
+
+`3c` honors the usual `-Xclang -ast-dump` compiler option to dump the
+AST to stdout before running its analysis. (This must be passed as a
+compiler option, not a `3c` option, e.g., _after_ the `--` on the
+command line if applicable.) The dump includes the AST node memory
+addresses (e.g., `VarDecl 0x5569e5ef3988 ...`), so if you see in the
+debugger that (for example) a `Decl *` variable in 3C has the value
+`0x5569e5ef3988`, you can look for that memory address in the AST dump
+to quickly see what AST node it is (modulo pointer adjustments for
+multiple inheritance; adding a debugger watch with a cast to the
+pointer type used in the AST dump can help with this).
+
+## Coding guidelines
+
+Please follow [LLVM coding
+standards](https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly)
+in your code. Specifically:
+
+* The maximum length of a line: 80 chars
+
+* All comments should start with a Capital letter.
+
+* All Local variables, including fields and parameters, should start
+  with a Capital letter (similar to PascalCase). Short names are
+  preferred.
+
+* A space between the conditional keyword and `(` i.e., `if (`,
+  `while (`, ``for (` etc.
+
+* Space after the type name, i.e., `Type *K` _not_ `Type* K`.
+
+* Space before and after `:` in iterators, i.e., `for (auto &k : List)`
+
+Our goal is that all files should be formatted with `clang-format` and
+pass `clang-tidy` ([more information](clang-tidy.md)), and nonempty
+files should have a final newline (surprisingly, `clang-format` cannot
+enforce this). However, until we have better automation, we decided it
+isn't reasonable to require contributors to manually run these tools
+and fix style nits in each change; instead, we periodically run the
+tools on the entire 3C codebase.

clang/lib/3C/3C.cpp

@@ -16,6 +16,7 @@
 #include "clang/3C/ConstraintBuilder.h"
 #include "clang/3C/IntermediateToolHook.h"
 #include "clang/3C/RewriteUtils.h"
+#include "clang/Frontend/ASTConsumers.h"
 #include "clang/Frontend/VerifyDiagnosticConsumer.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "llvm/Support/TargetSelect.h"
@@ -251,9 +252,57 @@ class _3CASTBuilderAction : public ToolAction {
     if (!AST)
       return false;
 
+    handleExtraProgramAction(Invocation->getFrontendOpts(),
+                             AST->getASTContext());
+
     ASTs.push_back(std::move(AST));
     return true;
   }
+
+private:
+  void handleExtraProgramAction(FrontendOptions &Opts,
+                                ASTContext &C) {
+    // The Opts.ProgramAction field is normally used only by `clang -cc1` to
+    // select a FrontendAction (see CreateFrontendBaseAction in
+    // ExecuteCompilerInvocation.cpp) and is ignored by LibTooling tools, which
+    // perform a custom FrontendAction. But we want to support at least AST
+    // dumping (as an addition to 3C's normal workflow) since it's useful for
+    // debugging 3C, and we prefer to honor the standard `-Xclang -ast-dump`
+    // option rather than define our own tool-level option like clang-check
+    // does. We could add support for more `-ast-*` options here if desired.
+    switch (Opts.ProgramAction) {
+    case frontend::ParseSyntaxOnly:
+      // Nothing extra to do.
+      break;
+    case frontend::ASTDump: {
+      // Code copied from ASTDumpAction::CreateASTConsumer since we don't have a
+      // good way to actually use ASTDumpAction from here. :/
+      //
+      // XXX: Maybe we'd prefer to output this somewhere other than stdout to
+      // separate it from the updated main file written to stdout? This doesn't
+      // look trivial because ASTPrinter requires ownership of the output
+      // stream, and it probably isn't important for the intended debugging use
+      // case.
+      std::unique_ptr<ASTConsumer> Dumper =
+          CreateASTDumper(nullptr /*Dump to stdout.*/, Opts.ASTDumpFilter,
+                          Opts.ASTDumpDecls, Opts.ASTDumpAll,
+                          Opts.ASTDumpLookups, Opts.ASTDumpDeclTypes,
+                          Opts.ASTDumpFormat);
+      // In principle, we should call all the ASTConsumer methods the same way
+      // the normal AST parsing process would, but there isn't an obvious way to
+      // do that when using ASTUnit. Instead, we rely on the assumption
+      // (apparently valid as of this writing) that the only ASTConsumer method
+      // that has a nonempty implementation in ASTPrinter is
+      // HandleTranslationUnit, and we just call HandleTranslationUnit manually.
+      Dumper->HandleTranslationUnit(C);
+      break;
+    }
+    default:
+      llvm::errs() << "Warning: The requested ProgramAction is not implemented "
+                      "by 3C and will be ignored.\n";
+      break;
+    }
+  }
 };
 
 void dumpConstraintOutputJson(const std::string &PostfixStr,

clang/test/3C/ast_dump.c

@@ -0,0 +1,11 @@
+// RUN: rm -rf %t*
+
+// Turn off color in the AST dump so that we can more easily match it
+// (https://stackoverflow.com/questions/32447542/how-do-i-get-clang-to-dump-the-ast-without-color).
+// RUN: 3c -base-dir=%S %s -- -Xclang -ast-dump -fno-color-diagnostics | FileCheck -match-full-lines %s
+
+// RUN: 3c -base-dir=%S %s -- -Xclang -ast-list 2>%t.stderr
+// RUN: grep 'The requested ProgramAction is not implemented by 3C' %t.stderr
+
+int *p;
+// CHECK: `-VarDecl {{.*}} p 'int *'