Merge conflicts

Author: frabert

Diff for: CMakeLists.txt

@@ -15,6 +15,7 @@ add_library(clangql
     src/Index.pb.cc
     src/Service.grpc.pb.cc
     src/Service.pb.cc
+    src/RefsTable.cc
     src/RelationsTable.cc
     src/SymbolsTable.cc
     src/Module.cc)

Diff for: README.md

@@ -17,6 +17,7 @@ Afterwards, you can connect to a codebase by instantiating the various virtual t
     sqlite> CREATE VIRTUAL TABLE llvm_symbols USING clangql (symbols, clangd-index.llvm.org:5900);
     sqlite> CREATE VIRTUAL TABLE llvm_base_of USING clangql (base_of, clangd-index.llvm.org:5900);
     sqlite> CREATE VIRTUAL TABLE llvm_overridden_by USING clangql (overridden_by, clangd-index.llvm.org:5900);
+    sqlite> CREATE VIRTUAL TABLE llvm_refs USING clangql (refs, clangd-index.llvm.org:5900);
 
 You can then query the codebase as if it was a regular table (some caveats apply, read the last point to learn more):
 
@@ -54,13 +55,25 @@ As another example, searching all the subclasses of a particular class:
     MCAsmInfoELF       llvm::  llvm/include/llvm/MC/MCAsmInfoELF.h
     MCAsmInfoCOFF      llvm::  llvm/include/llvm/MC/MCAsmInfoCOFF.h
 
-In general, for each codebase three different virtual tables can be queried: a `symbols` table will contain information about every symbol in the codebase, a `base_of` table will contain information about what symbols are base classes of what symbols, and a `overridden_by` table will contain information about what symbols are overridden by what symbols.
+Searching for all declarations inside of the `std` namespace:
+
+    sqlite> SELECT decl.Name FROM llvm_symbols AS decl INNER JOIN llvm_refs AS ref ON ref.SymbolId = decl.Id WHERE decl.Scope = "std::" AND ref.Declaration = 1;
+    Name
+    ---------------------
+    align_val_t
+    __unexpected
+    is_execution_policy
+    __terminate
+    is_execution_policy_v
+
+In general, for each codebase four different virtual tables can be queried: a `symbols` table will contain information about every symbol in the codebase, a `base_of` table will contain information about what symbols are base classes of what symbols, a `overridden_by` table will contain information about what symbols are overridden by what symbols, and a `refs` table will contain information about symbol references.
 
 The syntax for instantiating the tables is the following:
 
     CREATE VIRTUAL TABLE my_symbols USING clangql (symbols, host:port);
     CREATE VIRTUAL TABLE my_base_of USING clangql (base_of, host:port);
     CREATE VIRTUAL TABLE my_overridden_by USING clangql (overridden_by, host:port);
+    CREATE VIRTUAL TABLE my_refs USING clangql (refs, host:port);
 
 `my_*` names are not important and can be anything, the first parameter to the creation of the virtual tables is important and must be left as-is, the second parameter is the connection string. Currently, only unencrypted gRPC connections are supported.
 
@@ -82,6 +95,15 @@ The meaning is as follows: if a row `(S, O)` is present in `base_of`, then `S` i
 
 Please note that it is only possible to query these two tables by their `Subject`, querying by `Object` is not possible due to limitations in the clangd protocol.
 
+The schema of `refs` tables is equivalent to
+
+    CREATE TABLE vtable(SymbolId TEXT, Declaration INT,
+      Definition INT, Reference INT, Spelled INT,
+      Path TEXT, StartLine INT, StartCol INT,
+      EndLine INT, EndCol INT)
+
+Please note that querying `refs` without a `SymbolId` will return 0 rows.
+
 ## How do I build it?
 
 ClangQL uses CMake, Protocol Buffers and gRPC. On Windows I used vcpkg to manage the two dependencies. I'm afraid I'm not knowledgeable enough with Linux and/or macOS to give precise indications on how to build it there, but I'm guessing that as long as you have the correct development packages installed and visible on your system, CMake will be able to locate them.
@@ -90,7 +112,7 @@ Once the repository is cloned, run:
 
     cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DVCPKG_TARGET_TRIPLET=x64-windows-static-md -DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake
 
-to configure the build. Adjust `CMAKE_BUILD_TYPE`, `VCPKG_TARGET_TRIPLET`, `CMAKE_TOOLCHAIN_FILE` and the generator type to suit your system and needs the best. Please not that the `sqlite` CLI tool and the extension must have the same bitness, at least on Windows. A 32-bit CLI (such as the precompiled one from SQLite.org) _will not_ load a 64-bit extension.
+to configure the build. Adjust `CMAKE_BUILD_TYPE`, `VCPKG_TARGET_TRIPLET`, `CMAKE_TOOLCHAIN_FILE` and the generator type to suit your system and needs the best. Please note that the `sqlite` CLI tool and the extension must have the same bitness, at least on Windows. A 32-bit CLI (such as the precompiled one from SQLite.org) _will not_ load a 64-bit extension.
 
 Once configured, run:
 
@@ -108,6 +130,8 @@ Also, there is currently no way to i.e. obtain all possible relations between tw
 
 Not all queries are equally fast: querying on symbol id, name or scope is fast, everything else needs to happen client side and is potentially slow.
 
-Similarly, when querying the `base_of` or `overridden_by` relations, only one of the two directions is possible, the other is not currently possible due to protocol limitations.
+Similarly, when querying the `base_of` or `overridden_by` relations, only one of the two directions is possible, the other is not currently possible due to protocol limitations. Also, not specifying a `Subject` will result in 0 rows being returned.
+
+Querying `refs` without specifying a `SymbolId` will result in 0 rows being produced. Specifying any one of `Definition`, `Declaration`, `Reference` or `Spelled` will generate more specific requests to the clangd server, all other fields are scanned client side.
 
 Error checking is nonexistant. This is not ready for production use and was mostly made for fun, to explore to what extent the clangd interface was suitable for use with SQLite, and to learn about the SQLite virtual table system.

Diff for: src/ClangQLModule.cc

@@ -2,6 +2,7 @@
 SQLITE_EXTENSION_INIT3
 #include "SymbolsTable.hpp"
 #include "RelationsTable.hpp"
+#include "RefsTable.hpp"
 #include <grpcpp/grpcpp.h>
 
 #include <string>
@@ -20,6 +21,8 @@ static std::unique_ptr<VirtualTable> InitTable(sqlite3* db, int argc, const char
     return std::make_unique<RelationsTable>(db, SymbolIndex::NewStub(channel), BaseOf);
   } else if(table_type == "overridden_by") {
     return std::make_unique<RelationsTable>(db, SymbolIndex::NewStub(channel), OverriddenBy);
+  } else if(table_type == "refs") {
+    return std::make_unique<RefsTable>(db, SymbolIndex::NewStub(channel));
   } else {
     return nullptr;
   }

Diff for: src/RefsTable.cc

@@ -0,0 +1,234 @@
+#include "RefsTable.hpp"
+#include "IResultStream.hpp"
+#include "sqlite3ext.h"
+SQLITE_EXTENSION_INIT3
+#include "VirtualTableCursor.hpp"
+
+#include <string>
+#include <vector>
+
+using namespace clang::clangd::remote;
+using clang::clangd::remote::v1::SymbolIndex;
+
+enum RefKind {
+  Kind_Unknown = 0,
+  Kind_Declaration = 1 << 0,
+  Kind_Definition = 1 << 1,
+  Kind_Reference = 1 << 2,
+  Kind_Spelled = 1 << 3,
+  Kind_All = Kind_Declaration | Kind_Definition | Kind_Reference | Kind_Spelled
+};
+
+class RefStream final : public IResultStream<Ref> {
+  grpc::ClientContext m_ctx;
+  std::unique_ptr<grpc::ClientReader<RefsReply>> m_replyReader;
+  RefsRequest m_req;
+  RefsReply m_reply;
+public:
+  RefStream(SymbolIndex::Stub& stub, RefsRequest& req)
+    : m_replyReader(stub.Refs(&m_ctx, req))
+    , m_req(req) { }
+
+  const Ref& Current() override {
+    return m_reply.stream_result();
+  }
+
+  bool Next() override {
+    return m_replyReader->Read(&m_reply) && m_reply.has_stream_result();
+  }
+
+  const std::string& Id() {
+    return m_req.ids(0);
+  }
+};
+
+enum {
+  CONSTR_ID = 1,
+  CONSTR_DEF = 2,
+  CONSTR_DECL = 4,
+  CONSTR_REF = 8,
+  CONSTR_SPE = 16
+};
+
+class RefsCursor final : public VirtualTableCursor {
+  SymbolIndex::Stub& m_stub;
+  bool m_eof = false;
+  std::unique_ptr<RefStream> m_stream = nullptr;
+public:
+  RefsCursor(SymbolIndex::Stub& stub)
+    : m_stub(stub) {}
+
+  int Eof() override { return m_eof; }
+  int Next() override {
+    m_eof = !m_stream->Next();
+    return SQLITE_OK;
+  }
+  int Close() override {
+    return SQLITE_OK;
+  }
+  sqlite3_int64 RowId() override { return 0; }
+  int Column(sqlite3_context *ctx, int idxCol) override {
+    auto& Current = m_stream->Current();
+    switch(idxCol) {
+      #define SET_RES2(field1, field2) do { \
+        if(Current.has_ ## field1() && Current.field1().has_ ## field2()) { \
+          sqlite3_result_text(ctx, Current.field1().field2().c_str(), -1, SQLITE_TRANSIENT); \
+        } else { sqlite3_result_null(ctx); } } while(0)
+      #define SET_RES3(field1, field2, field3) do { \
+        if(Current.has_ ## field1() && Current.field1().has_ ## field2() && Current.field1().field2().has_ ## field3()) { \
+          sqlite3_result_int(ctx, Current.field1().field2().field3()); \
+        } else { sqlite3_result_null(ctx); } } while(0)
+
+      case 0: sqlite3_result_text(ctx, m_stream->Id().c_str(), -1, SQLITE_TRANSIENT); break;
+      case 1: sqlite3_result_int(ctx, (Current.kind() & Kind_Declaration) == Kind_Declaration); break;
+      case 2: sqlite3_result_int(ctx, (Current.kind() & Kind_Definition) == Kind_Definition); break;
+      case 3: sqlite3_result_int(ctx, (Current.kind() & Kind_Reference) == Kind_Reference); break;
+      case 4: sqlite3_result_int(ctx, (Current.kind() & Kind_Spelled) == Kind_Spelled); break;
+      case 5: SET_RES2(location, file_path); break;
+      case 6: SET_RES3(location, start, line); break;
+      case 7: SET_RES3(location, start, column); break;
+      case 8: SET_RES3(location, end, line); break;
+      case 9: SET_RES3(location, end, column); break;
+      #undef SET_RES2
+      #undef SET_RES3
+    }
+    return SQLITE_OK;
+  }
+  int Filter(int idxNum, const char *idxStr, int argc, sqlite3_value **argv) override {
+    if(idxNum) {
+      RefsRequest req;
+      int argvIndex = 0;
+      int kind = Kind_All;
+
+      if(idxNum & CONSTR_ID) {
+        req.add_ids((const char*)sqlite3_value_text(argv[argvIndex++]));
+      }
+
+      if(idxNum & CONSTR_DEF) {
+        if(!sqlite3_value_int(argv[argvIndex++])) {
+          kind &= !Kind_Definition;
+        }
+      }
+
+      if(idxNum & CONSTR_DECL) {
+        if(!sqlite3_value_int(argv[argvIndex++])) {
+          kind &= !Kind_Declaration;
+        }
+      }
+
+      if(idxNum & CONSTR_REF) {
+        if(!sqlite3_value_int(argv[argvIndex++])) {
+          kind &= !Kind_Reference;
+        }
+      }
+
+      if(idxNum & CONSTR_SPE) {
+        if(!sqlite3_value_int(argv[argvIndex++])) {
+          kind &= !Kind_Spelled;
+        }
+      }
+
+      req.set_filter(kind);
+      m_stream = std::make_unique<RefStream>(m_stub, req);
+      return Next();
+    } else {
+      m_eof = true;
+      return SQLITE_OK;
+    }
+  }
+};
+
+constexpr const char* schema = R"cpp(
+  CREATE TABLE vtable(SymbolId TEXT, Declaration INT,
+    Definition INT, Reference INT, Spelled INT,
+    Path TEXT, StartLine INT, StartCol INT,
+    EndLine INT, EndCol INT,
+    PRIMARY KEY (
+      SymbolId, Declaration, Definition, Reference, Spelled,
+      Path, StartLine, StartCol, EndLine, EndCol))
+  WITHOUT ROWID)cpp";
+
+RefsTable::RefsTable(sqlite3* db, std::unique_ptr<SymbolIndex::Stub> stub)
+  : m_stub(std::move(stub)) {
+  int err = sqlite3_declare_vtab(db, schema);
+  if(err != SQLITE_OK) {
+    auto errmsg = sqlite3_errmsg(db);
+    throw std::exception(errmsg);
+  }
+}
+
+int RefsTable::BestIndex(sqlite3_index_info* info) {
+  int argvIndex = 0;
+
+  // Check for id
+  for(int i = 0; i < info->nConstraint; i++) {
+    auto& constraint = info->aConstraint[i];
+    if(!constraint.usable || constraint.op != SQLITE_INDEX_CONSTRAINT_EQ) continue;
+    if(constraint.iColumn == 0) {
+      info->aConstraintUsage[i].argvIndex = ++argvIndex;
+      info->idxNum |= CONSTR_ID;
+      info->estimatedCost = 1;
+      break;
+    }
+  }
+
+  // Check for def
+  for(int i = 0; i < info->nConstraint; i++) {
+    auto& constraint = info->aConstraint[i];
+    if(!constraint.usable || constraint.op != SQLITE_INDEX_CONSTRAINT_EQ) continue;
+    if(constraint.iColumn == 1) {
+      info->aConstraintUsage[i].argvIndex = ++argvIndex;
+      info->idxNum |= CONSTR_DEF;
+      break;
+    }
+  }
+
+  // Check for decl
+  for(int i = 0; i < info->nConstraint; i++) {
+    auto& constraint = info->aConstraint[i];
+    if(!constraint.usable || constraint.op != SQLITE_INDEX_CONSTRAINT_EQ) continue;
+    if(constraint.iColumn == 2) {
+      info->aConstraintUsage[i].argvIndex = ++argvIndex;
+      info->idxNum |= CONSTR_DEF;
+      break;
+    }
+  }
+
+  // Check for ref
+  for(int i = 0; i < info->nConstraint; i++) {
+    auto& constraint = info->aConstraint[i];
+    if(!constraint.usable || constraint.op != SQLITE_INDEX_CONSTRAINT_EQ) continue;
+    if(constraint.iColumn == 3) {
+      info->aConstraintUsage[i].argvIndex = ++argvIndex;
+      info->idxNum |= CONSTR_REF;
+      break;
+    }
+  }
+
+  // Check for spelled
+  for(int i = 0; i < info->nConstraint; i++) {
+    auto& constraint = info->aConstraint[i];
+    if(!constraint.usable || constraint.op != SQLITE_INDEX_CONSTRAINT_EQ) continue;
+    if(constraint.iColumn == 4) {
+      info->aConstraintUsage[i].argvIndex = ++argvIndex;
+      info->idxNum |= CONSTR_SPE;
+      break;
+    }
+  }
+
+  return SQLITE_OK;
+}
+
+std::unique_ptr<VirtualTableCursor> RefsTable::Open() {
+  return std::make_unique<RefsCursor>(*m_stub);
+}
+
+int RefsTable::FindFunction(int nArg, const std::string& name, void (**pxFunc)(sqlite3_context*,int,sqlite3_value**), void **ppArg) { return 0; }
+
+int RefsTable::Disconnect() {
+  return SQLITE_OK;
+}
+
+int RefsTable::Destroy() {
+  return SQLITE_OK;
+}

Diff for: src/RefsTable.hpp

@@ -0,0 +1,20 @@
+#ifndef REFTABLE_HPP
+#define REFTABLE_HPP
+#include "sqlite3ext.h"
+#include "VirtualTable.hpp"
+#include "Service.grpc.pb.h"
+
+class RefsTable : public VirtualTable {
+  std::unique_ptr<clang::clangd::remote::v1::SymbolIndex::Stub> m_stub;
+
+public:
+  RefsTable(sqlite3* db, std::unique_ptr<clang::clangd::remote::v1::SymbolIndex::Stub> stub);
+
+  virtual int BestIndex(sqlite3_index_info* info) override;
+  virtual std::unique_ptr<VirtualTableCursor> Open() override;
+  virtual int FindFunction(int nArg, const std::string& name, void (**pxFunc)(sqlite3_context*,int,sqlite3_value**), void **ppArg) override;
+  virtual int Disconnect() override;
+  virtual int Destroy() override;
+};
+
+#endif

Diff for: src/RelationsTable.cc

@@ -6,6 +6,7 @@ SQLITE_EXTENSION_INIT3
 
 #include <string>
 #include <vector>
+#include <chrono>
 
 using namespace clang::clangd::remote;
 using clang::clangd::remote::v1::SymbolIndex;
@@ -81,8 +82,7 @@ class RelationsCursor final : public VirtualTableCursor {
     }
 
     m_stream = std::make_unique<RelationStream>(m_stub, req);
-    Next();
-    return SQLITE_OK;
+    return Next();
   }
 };