Add db indexes to Gorm config (#179)

ryanjohnsontv · abe garcia · web-flow · commit d6ce55f3c2b9 · 2024-04-29T15:30:03.000-04:00
* Add db indexes to Gorm config

* Update README

* Update sql tests

---------

Co-authored-by: abe garcia &lt;abraham_garcia-ortiz1@homedepot.com&gt;
diff --git a/README.md b/README.md
@@ -5,44 +5,50 @@
 Go Clouddriver is a rewrite of Spinnaker's [Clouddriver](https://github.com/spinnaker/clouddriver) microservice. It has an observed 95%+ decrease in CPU and memory load for Kubernetes operations.
 
 Go Clouddriver brings many features to the table which allow it to perform better than Clouddriver OSS for production loads:
+
 - it does not rely on `kubectl` and instead interfaces directly with the Kubernetes API for all operations
 - it utilizes an in-memory cache store for Kubernetes API discovery
 - it stores Kubernetes providers in a database, fronted by a simple CRUD API
 - it removes over-complicated strategies such as [Cache All The Stuff](https://github.com/spinnaker/clouddriver/tree/master/cats), instead making live calls for all operations
 
-Go Clouddriver is *not* an API complete implementation of Clouddriver OSS and only handles Kubernetes providers and operations. It is meant to be run in tandem with Clouddriver OSS. Visit [the wiki](https://github.com/homedepot/go-clouddriver/wiki) for feature support and intallation instructions.
+Go Clouddriver is _not_ an API complete implementation of Clouddriver OSS and only handles Kubernetes providers and operations. It is meant to be run in tandem with Clouddriver OSS. Visit [the wiki](https://github.com/homedepot/go-clouddriver/wiki) for feature support and intallation instructions.
 
 ## Getting Started
 
 ### Testing
 
 Run from the root directory
+
 ```bash
 make tools test
 ```
 
 ### Building
 
 Run from the root directory
+
 ```bash
 make build
 ```
 
 ### Running Locally
 
-1) Go Clouddriver generates its access tokens using [Arcade](https://github.com/homedepot/arcade) as a sidecar, so a working instance of Arcade will need to be running locally in order for Go Clouddriver to talk to Kubernetes clusters.
+1. Go Clouddriver generates its access tokens using [Arcade](https://github.com/homedepot/arcade) as a sidecar, so a working instance of Arcade will need to be running locally in order for Go Clouddriver to talk to Kubernetes clusters.
+
+2. Export the Arcade API key (the same one you set up in step 1).
 
-2) Export the Arcade API key (the same one you set up in step 1).
 ```bash
 export ARCADE_API_KEY=test
 ```
 
-3) Run Go Clouddriver.
+3. Run Go Clouddriver.
+
 ```bash
 make run
 ```
 
-4) Create your first Kubernetes provider! Go Clouddriver runs on port 7002, so you'll make a POST to `localhost:7002/v1/kubernetes/providers`.
+4. Create your first Kubernetes provider! Go Clouddriver runs on port 7002, so you'll make a POST to `localhost:7002/v1/kubernetes/providers`.
+
 ```bash
 curl -XPOST localhost:7002/v1/kubernetes/providers -d '{
   "name": "test-provider",
@@ -58,62 +64,70 @@ curl -XPOST localhost:7002/v1/kubernetes/providers -d '{
   }
 }' | jq
 ```
+
 And you should see the response
+
 ```json
 {
   "name": "test-provider",
   "host": "https://test-host",
   "caData": "test",
   "permissions": {
-    "read": [
-      "test-group"
-    ],
-    "write": [
-      "test-group"
-    ]
+    "read": ["test-group"],
+    "write": ["test-group"]
   }
 }
 ```
+
 Running the command again will return a `409 Conflict` unless you change the name of the provider.
 
-5) List your providers by calling the `/credentials` endpoint.
+5. List your providers by calling the `/credentials` endpoint.
+
 ```bash
 curl localhost:7002/credentials | jq
 ```
 
 ### Configuration
 
-| Environment Variable | Description | Notes | Default Value |
-|----------|:-------------:|-----------:|------------:|
-| `ARCADE_API_KEY` | Needed to talk to [Arcade](https://github.com/billiford/arcade). | Required for most operations. ||
-| `ARTIFACTS_CREDENTIALS_CONFIG_DIR` | Sets the directory for artifacts configuration. | Optional. Leave unset to use OSS Clouddriver's Artifacts API. ||
-| `KUBERNETES_USE_DISK_CACHE` | Stores Kubernetes API discovery on disk instead of in-memory. || `false` |
-| `DB_HOST` | Used to connect to MySQL database. | If not set will default to local SQLite database. ||
-| `DB_NAME` | Used to connect to MySQL database. | If not set will default to local SQLite database. ||
-| `DB_PASS` | Used to connect to MySQL database. | If not set will default to local SQLite database. ||
-| `DB_USER` | Used to connect to MySQL database. | If not set will default to local SQLite database. ||
-| `VERBOSE_REQUEST_LOGGING` | Logs all incoming request information. | Should only be used in non-production for testing. | `false` |
+| Environment Variable               |                           Description                            |                                                         Notes | Default Value |
+| ---------------------------------- | :--------------------------------------------------------------: | ------------------------------------------------------------: | ------------: |
+| `ARCADE_API_KEY`                   | Needed to talk to [Arcade](https://github.com/billiford/arcade). |                                 Required for most operations. |               |
+| `ARTIFACTS_CREDENTIALS_CONFIG_DIR` |         Sets the directory for artifacts configuration.          | Optional. Leave unset to use OSS Clouddriver's Artifacts API. |               |
+| `KUBERNETES_USE_DISK_CACHE`        |  Stores Kubernetes API discovery on disk instead of in-memory.   |                                                               |       `false` |
+| `DB_HOST`                          |                Used to connect to MySQL database.                |             If not set will default to local SQLite database. |               |
+| `DB_NAME`                          |                Used to connect to MySQL database.                |             If not set will default to local SQLite database. |               |
+| `DB_PASS`                          |                Used to connect to MySQL database.                |             If not set will default to local SQLite database. |               |
+| `DB_USER`                          |                Used to connect to MySQL database.                |             If not set will default to local SQLite database. |               |
+| `VERBOSE_REQUEST_LOGGING`          |              Logs all incoming request information.              |            Should only be used in non-production for testing. |       `false` |
 
 ### MySQL Indexes and Cleanup
 
 Go Clouddriver stores all deployed resource requests in its `kubernetes_resources` table, which needs to be cleaned up periodically. It also requires a few indexes
-to work efficiently and properly for continued deployments over long periods of time.
+to work efficiently and properly for continued deployments over long periods of time. These are defined by default when
+starting the application.
 
 #### Indexes
 
 First, an index to help the Applications API remain efficient.
+
 ```sql
-CREATE INDEX kind_account_name_kind_name_spinnaker_app_idx ON kubernetes_resources(account_name, kind, name, spinnaker_app);
+CREATE INDEX account_name_kind_name_spinnaker_app_idx ON kubernetes_resources(account_name, kind, name, spinnaker_app);
 ```
+
 Next, an index to assist in pulling "cluster" kinds from the `kubernetes_resources` table.
+
 ```sql
 CREATE INDEX kind_idx ON kubernetes_resources(kind);
 ```
+
 Next, an index to assist the Task API.
+
 ```sql
 CREATE INDEX task_id_idx ON kubernetes_resources(task_id);
 ```
+
 Finally, a couple of indexes on the provider read/write permissions tables to help the Credentials API and any queries to select providers from the database.
+
 ```sql
 CREATE INDEX account_name_idx ON provider_read_permissions(account_name);
 CREATE INDEX account_name_idx ON provider_write_permissions(account_name);
@@ -123,7 +137,8 @@ CREATE INDEX account_name_idx ON provider_write_permissions(account_name);
 
 The `kubernetes_resources` tables is incredibly important for storing the most-recent state of a cluster for the Kubernetes kinds we care about.
 
-It is recommended to run the following cleanup statements *daily* to maintain a healthy state of onboarded Kubernetes clusters.
+It is recommended to run the following cleanup statements _daily_ to maintain a healthy state of onboarded Kubernetes clusters.
+
 ```sql
 DELETE FROM kubernetes_resources where cluster = '' and timestamp < (NOW() - INTERVAL 6 HOUR);
 
@@ -141,5 +156,3 @@ WHERE
     t1.cluster = t2.cluster AND
     t1.timestamp < (NOW() - INTERVAL 6 HOUR);
 ```
-
-
diff --git a/internal/kubernetes/provider.go b/internal/kubernetes/provider.go
@@ -47,8 +47,8 @@ func (Provider) TableName() string {
 
 type ProviderNamespaces struct {
 	//ID          string `json:"-" gorm:"primary_key"`
-	AccountName string `json:"accountName"`
-	Namespace   string `json:"namespace,omitempty"`
+	AccountName string `json:"accountName" gorm:"index:account_name_namespace_idx,unique"`
+	Namespace   string `json:"namespace,omitempty" gorm:"index:account_name_namespace_idx,unique"`
 }
 
 func (ProviderNamespaces) TableName() string {
diff --git a/internal/kubernetes/resource.go b/internal/kubernetes/resource.go
@@ -3,19 +3,19 @@ package kubernetes
 import "time"
 
 type Resource struct {
-	AccountName  string    `json:"accountName"`
+	AccountName  string    `json:"accountName" gorm:"index:account_name_kind_name_spinnaker_app_idx,priority:1"`
 	ID           string    `json:"id" gorm:"primary_key"`
 	Timestamp    time.Time `json:"timestamp,omitempty" gorm:"type:timestamp;DEFAULT:current_timestamp"`
-	TaskID       string    `json:"taskId"`
+	TaskID       string    `json:"taskId" gorm:"index:task_id_idx"`
 	TaskType     string    `json:"-"`
 	APIGroup     string    `json:"apiGroup"`
-	Name         string    `json:"name"`
+	Name         string    `json:"name" gorm:"index:account_name_kind_name_spinnaker_app_idx,priority:3"`
 	ArtifactName string    `json:"-"`
 	Namespace    string    `json:"namespace"`
 	Resource     string    `json:"resource"`
 	Version      string    `json:"version"`
-	Kind         string    `json:"kind"`
-	SpinnakerApp string    `json:"spinnakerApp"`
+	Kind         string    `json:"kind" gorm:"index:account_name_kind_name_spinnaker_app_idx,priority:2;index:kind_idx"`
+	SpinnakerApp string    `json:"spinnakerApp" gorm:"index:account_name_kind_name_spinnaker_app_idx,priority:4"`
 	Cluster      string    `json:"-"`
 }
 
diff --git a/internal/sql/client_test.go b/internal/sql/client_test.go
@@ -52,7 +52,7 @@ var _ = Describe("Sql", func() {
 			"\\)$").
 			WillReturnResult(sqlmock.NewResult(1, 1))
 		mock.ExpectExec("(?i)^CREATE TABLE `kubernetes_resources` " +
-			"\\(`account_name`\\ varchar\\(256\\)," +
+			"\\(`account_name` varchar\\(256\\)," +
 			"`id` varchar\\(256\\)," +
 			"`timestamp` timestamp DEFAULT current_timestamp," +
 			"`task_id` varchar\\(256\\)," +
@@ -66,25 +66,28 @@ var _ = Describe("Sql", func() {
 			"`kind` varchar\\(256\\)," +
 			"`spinnaker_app` varchar\\(256\\)," +
 			"`cluster` varchar\\(256\\)," +
-			"PRIMARY KEY \\(`id`\\)" +
-			"\\)$").
-			WillReturnResult(sqlmock.NewResult(1, 1))
+			"PRIMARY KEY \\(`id`\\)," +
+			"INDEX `.*").WillReturnResult(sqlmock.NewResult(1, 1))
 		mock.ExpectExec("CREATE TABLE `kubernetes_providers_namespaces` " +
 			"\\(`account_name` varchar\\(256\\)," +
-			"`namespace` varchar\\(256\\)").
+			"`namespace` varchar\\(256\\)," +
+			"UNIQUE INDEX `account_name_namespace_idx` \\(`account_name`,`namespace`\\)" +
+			"\\)$").
 			WillReturnResult(sqlmock.NewResult(1, 1))
 		mock.ExpectExec("(?i)^CREATE TABLE `provider_read_permissions` " +
 			"\\(`id`\\ varchar\\(256\\)," +
 			"`account_name` varchar\\(256\\)," +
 			"`read_group` varchar\\(256\\)," +
-			"PRIMARY KEY \\(`id`\\)" +
+			"PRIMARY KEY \\(`id`\\)," +
+			"INDEX `account_name_idx` \\(`account_name`\\)" +
 			"\\)$").
 			WillReturnResult(sqlmock.NewResult(1, 1))
 		mock.ExpectExec("(?i)^CREATE TABLE `provider_write_permissions` " +
 			"\\(`id`\\ varchar\\(256\\)," +
 			"`account_name` varchar\\(256\\)," +
 			"`write_group` varchar\\(256\\)," +
-			"PRIMARY KEY \\(`id`\\)" +
+			"PRIMARY KEY \\(`id`\\)," +
+			"INDEX `account_name_idx` \\(`account_name`\\)" +
 			"\\)$").
 			WillReturnResult(sqlmock.NewResult(1, 1))
 
diff --git a/pkg/permissions.go b/pkg/permissions.go
@@ -7,7 +7,7 @@ type Permissions struct {
 
 type ReadPermission struct {
 	ID          string `json:"-" gorm:"primary_key"`
-	AccountName string `json:"accountName"`
+	AccountName string `json:"accountName" gorm:"index:account_name_idx"`
 	ReadGroup   string `json:"readGroup"`
 }
 
@@ -17,7 +17,7 @@ func (ReadPermission) TableName() string {
 
 type WritePermission struct {
 	ID          string `json:"-" gorm:"primary_key"`
-	AccountName string `json:"accountName"`
+	AccountName string `json:"accountName" gorm:"index:account_name_idx"`
 	WriteGroup  string `json:"writeGroup"`
 }
 

Original file line number	Diff line number	Diff line change
`@@ -47,8 +47,8 @@ func (Provider) TableName() string {`
`47`	`47`
`48`	`48`	`type ProviderNamespaces struct {`
`49`	`49`	//ID string `json:"-" gorm:"primary_key"`
`50`		- AccountName string `json:"accountName"`
`51`		- Namespace string `json:"namespace,omitempty"`
	`50`	+ AccountName string `json:"accountName" gorm:"index:account_name_namespace_idx,unique"`
	`51`	+ Namespace string `json:"namespace,omitempty" gorm:"index:account_name_namespace_idx,unique"`
`52`	`52`	`}`
`53`	`53`
`54`	`54`	`func (ProviderNamespaces) TableName() string {`
Original file line number	Diff line number	Diff line change
`@@ -7,7 +7,7 @@ type Permissions struct {`
`7`	`7`
`8`	`8`	`type ReadPermission struct {`
`9`	`9`	ID string `json:"-" gorm:"primary_key"`
`10`		- AccountName string `json:"accountName"`
	`10`	+ AccountName string `json:"accountName" gorm:"index:account_name_idx"`
`11`	`11`	ReadGroup string `json:"readGroup"`
`12`	`12`	`}`
`13`	`13`
`@@ -17,7 +17,7 @@ func (ReadPermission) TableName() string {`
`17`	`17`
`18`	`18`	`type WritePermission struct {`
`19`	`19`	ID string `json:"-" gorm:"primary_key"`
`20`		- AccountName string `json:"accountName"`
	`20`	+ AccountName string `json:"accountName" gorm:"index:account_name_idx"`
`21`	`21`	WriteGroup string `json:"writeGroup"`
`22`	`22`	`}`
`23`	`23`