Orchestrator. New seeding algorithm, support for different seed methods and agent redesign

.gitignore

@@ -21,3 +21,4 @@ main
 .vendor/go19
 /bin
 /build
+.vscode/

docs/agents.md

@@ -1,18 +1,56 @@
 # Agents
 
-You may optionally install [orchestrator-agent](https://github.com/github/orchestrator-agent) on your MySQL hosts.
-`orchestrator-agent`* is a service which registers with your `orchestrator` server and accepts requests by `orchestrator` via web API.
+You may optionally install [orchestrator-agent](https://github.com/opernark/orchestrator-agent) on your MySQL hosts.
+*`orchestrator-agent`* is a service which registers with your `orchestrator` server and accepts requests by `orchestrator` via web API.
 
-Supported requests relate to general, OS and LVM operations, such as:
-- Stopping/starting MySQL service on host
-- Getting MySQL OS info such as data directory, port, disk space usage
-- Performing various LVM operations such as finding LVM snapshots, mounting/unmounting a snapshot
-- Transferring data between hosts (e.g. via `netcat`)
+**orchestrator-agent** is capable of seeding new replicas using different seed methods, providing operating system, file system and LVM information to *orchestrator*, managing MySQL service as well as invoke certain commands and scripts.  
 
-`orchestrator-agent` is an ongoing effort in solving host-specific operations. It was originally developed to overcome cloning and restoring issues, and later expanded to other areas.
-
-The information and API exposed by `orchestrator-agent` to `orchestrator` allow `orchestrator` to coordinate and operate seeding of new or corrupted machines by getting data from freshly available snapshots. Moreover, it allows `orchestrator`
-to automatically suggest the source of data for a given MySQL machine, by looking up such hosts that actually have a
-recent snapshot available, preferably in the same datacenter.
+Generic functionality offered by **orchestrator-agent**:  
+- Detection of the MySQL service, starting and stopping (start/stop/status commands provided via configuration)
+- Detection of MySQL port, data directory, databases
+- Calculation of disk usage on data directory mount point, OS and RAM size
+- Tailing the error log file
+- Discovery (the mere existence of the *orchestrator-agent* service on a host may suggest the existence or need of existence of a MySQL service)
+
+Specialized functionality offered by **orchestrator-agent**:  
+- Seeding new slaves using different seed methods (LVM, Xtrabackup, Mydumper, Mysqldump, Clone plugin)
+- Detection of LVM snapshots on MySQL host (snapshots that are MySQL specific)
+- Creation of new snapshots
+- Mounting/umounting of LVM snapshots
+- Detection of DC-local and DC-agnostic snapshots available for a given cluster
 
 For security measures, an agent requires a token to operate all but the simplest requests. This token is randomly generated by the agent and negotiated with `orchestrator`. `orchestrator` does not expose the agent's token (right now some work needs to be done on obscuring the token on error messages).
+
+When using **orchestrator-agent** you will get access to new Agents UI, where you will be able to seed data to new hosts as well as monitor live seeding progress.
+
+In order to enable **orchestrator-agent** add following to orchestrator configuration file
+
+```json
+{
+  "ServeAgentsHttp": true,
+}
+```
+
+# Seeding
+## Supported seed methods
+Orchestrator-agent supports following seed methods:
+* Mysqldump
+* Mydumper
+* Xtrabackup
+* LVM
+* Clone plugin
+
+## Seed process
+Seed process consists of 5 stages, running sequentaly one after another:
+* Prepare. On this stage different preparations are performed. Depenging on seed method they can include creating specific directories, stopping MySQL, starting socat for data transfer
+* Backup. On this stage data is backuped on source host and transfered to target host
+* Restore. On this stage data is restored on source host
+* ConnectSlave. On this stage backup metadata is read and target host is connected as slave to source host
+* Cleanup. On this stage different cleanup activities are executed
+
+There are also additional parameters, which can help you with tunning **orchestrator-agent** and seed process:
+* `UnseenAgentForgetHours` - Number of hours after which an unseen agent is forgotten
+* `AgentPollMinutes` - Minutes between agent polling
+* `MaxRetriesForSeedStage` - Number of maximum retries for each of the seed stages, after which seed will be marked as failed
+* `SeedProcessIntervalSeconds` - Interval in seconds between processing active seeds
+* `SeedBackupStaleFailMinutes` - Number of minutes after which a stale (no progress) seed in Backup stage is considered failed