docs: document the architecture of the project

Signed-off-by: Miguel Molina <[email protected]>

Author: erizocosmico

ARCHITECTURE.md

@@ -0,0 +1,137 @@
+# Architecture overview
+
+This document provides an overview of all parts and pieces of the project as well as how they fit together. It is meant to help new contributors understand where things may be, and how changes in some components may interact with other components of the system.
+
+## Root package (`sqle`)
+
+This is where the engine lives. The engine is the piece that coordinates and makes all other pieces work together as well as the main API users of the system will use to create and configure an engine and perform queries.
+
+Because this is the point where all components fit together, it is also where integration tests are. Those integration tests can be found in `engine_test.go`.
+A test should be added here, plus in any specific place where the feature/issue belonged, if needed.
+
+**How to add integration tests**
+
+The test executing all integration test is `TestQueries`, which you can run with the following command:
+
+```
+go test -run=TestQueries
+```
+
+This test is just executing all the queries in a loop. New test cases should be added to the `queries` package variable at the top of `engine_test.go`.
+Simply add a new element to the slice with the query and the expected result.
+
+## `sql`
+
+This package is probably the most important of the project. It has several main roles:
+- Defines the main interfaces used in the rest of the packages `Node`, `Expression`, ...
+- Provides implementations of components used in the rest of the packages `Row`, `Context`, `ProcessList`, `Catalog`, ...
+- Defines the `information_schema` table, which is a special table available in all databases and contains some data about the schemas of other tables.
+
+### `sql/analyzer`
+
+The analyzer is the more complex component of the project. It contains a main component, which is the `Analyzer`, in charge of executing its registered rules on execution trees for resolving some parts, removing redundant data, optimizing things for performance, etc.
+
+There are several phases on the analyzer, because some rules need to be run before others, some need to be executed several times, other just once, etc.
+Inside `rules.go` are all the default rules and the phases in which they're executed.
+
+On top of that, all available rules are defined in this package. Each rule has a specific role in the analyzer. Rules should be as small and atomic as possible and try to do only one job and always produce a tree that is as resolved as the one it received or more.
+
+### `sql/expression`
+
+This package includes the implementation of all the SQL expressions available in go-mysql-server, except functions. Arithmetic operators, logic operators, conversions, etc are implemented here.
+
+Inside `registry.go` there is a registry of all the default functions, even if they're not defined here.
+
+`Inspect` and `Walk` utility functions are provided to inspect expressions.
+
+### `sql/expression/function`
+
+Implementation of all the functions available in go-mysql-server.
+
+### `sql/expression/function/aggregation`
+
+Implementation of all the aggregation functions available in go-mysql-server.
+
+### `sql/index`
+
+Contains the index driver configuration file implementation and other utilities for dealing with index drivers.
+
+### `sql/index/pilosa`
+
+Actual implementation of an index driver. Underneath, it's using a bitmap database called pilosa (hence the name) to implement bitmap indexes.
+
+### `sql/parse`
+
+This package exposes the `Parse` function, which parses a SQL query and translates it into an execution plan.
+
+Parsing is done using `vitess` parser, but sometimes there are queries vitess cannot parse. In this case, custom parsers are used. Otherwise, vitess is used to parse the query and then converted to a go-mysql-server execution plan.
+
+### `sql/plan`
+
+All the different nodes of the execution plan (except for very specific nodes used in some optimisation rules) are defined here.
+
+For example, `SELECT foo FROM bar` is translated into the following plan:
+
+```
+Project(foo)
+ |- Table(bar)
+```
+
+Which means, the execution plan is a `Project` node projecting `foo` and has a `ResolvedTable`, which is `bar` as its children.
+
+Each node inside this package implements at least the `sql.Node` interface, but it can implement more. `sql.Expressioner`, for example.
+
+Along with the nodes, `Inspect` and `Walk` functions are provided as utilities to inspect an execution tree.
+
+## `server`
+
+Contains all the code to turn an engine into a runnable server that can communicate using the MySQL wire protocol.
+
+## `auth`
+
+This package contains all the code related to the audit log, authentication and permission management in go-mysql-server.
+
+There are two authentication methods:
+- **None:** no authentication needed.
+- **Native:** authentication performed with user and password. Read, write or all permissions can be specified for those users. It can also be configured using a JSON file.
+
+## `internal/regex`
+
+go-mysql-server has multiple regular expression engines, such as oniguruma and the standard Go regexp engine. In this package, a common interface for regular expression engines is defined.
+This means, Go standard library `regexp` package should not be used in any user-facing feature, instead this package should be used.
+
+The default engine is oniguruma, but the Go standard library engine can be used using the `mysql_go_regex` build tag.
+
+## `test`
+
+Test contains pieces that are only used for tests, such as an opentracing tracer that stores spans in memory to be inspected later in the tests.
+
+## `_integration`
+
+To ensure compatibility with some clients, there is a small example connecting and querying a go-mysql-server server from those clients. Each folder corresponds to a different client.
+
+For more info about supported clients see [SUPPORTED_CLIENTS.md](/SUPPORTED_CLIENTS.md).
+
+These integrations tests can be run using this command:
+
+```
+make TEST=${CLIENT FOLDER NAME} integration
+```
+
+It will take care of setting up the test server and shutting it down.
+
+## `_example`
+
+A small example of how to use go-mysql-server to create a server and run it.
+
+# Connecting the dots
+
+`server` uses the engine defined in `sql`.
+
+Engine uses audit logs and authentication defined in `auth`, parses using `sql/parse` to convert a query into an execution plan, with nodes defined in `sql/plan` and expressions defined in `sql/expression`, `sql/expression/function` and `sql/expression/function/aggregation`.
+
+After parsing, the obtained execution plan is analyzed using the analyzer defined in `sql/analyzer` and its rules to resolve tables, fields, databases, apply optimisation rules, etc.
+
+If indexes can be used, the analyzer will transform the query so it uses indexes reading from the drivers in `sql/index` (in this case `sql/index/pilosa` because there is only one driver).
+
+Once the plan is analyzed, it will be executed recursively from the top of the tree to the bottom to obtain the results and they will be sent back to the client using the MySQL wire protocol.

CONTRIBUTING.md

@@ -17,8 +17,6 @@ the following line: `Signed-off-by: John Doe <[email protected]>`, using your
 
 This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d3aa82f6583b9aadfecc5f8fa1c74aca09/Documentation/git-commit.txt#L154-L161) flag on the `git commit`.
 
-
-
 ## Support Channels
 
 The official support channels, for both users and contributors, are:
@@ -30,7 +28,6 @@ The official support channels, for both users and contributors, are:
 search the project - it's likely that another user has already reported the
 issue you're facing, or it's a known issue that we're already aware of.
 
-
 ## How to Contribute
 
 Pull Requests (PRs) are the main and exclusive way to contribute code to source{d} projects.
@@ -45,6 +42,10 @@ In order for a PR to be accepted it needs to pass a list of requirements:
     - If the PR is a new feature, it has to come with a suite of unit tests, that tests the new functionality.
     - In any case, all the PRs have to pass the personal evaluation of at least one of the [maintainers](MAINTAINERS) of the project.
 
+### Getting started
+
+If you are a new contributor to the project, reading [ARCHITECTURE.md](/ARCHITECTURE.md) is highly recommended, as it contains all the details about the architecture of go-mysql-server and its components.
+
 
 ### Format of the commit message