Merge pull request #107 from codefresh-io/dind-storage

Dind storage

Author: kosta709

README.md

@@ -47,6 +47,7 @@
 | --skip-runtime-installation | boolean | Set flag if you already have a configured runtime-environment, add --runtime-environment flag with name |
 | --storage-class | string | Set a name of your custom storage class, note: this will not install volume provisioning components |
 | --tolerations | string | The kubernetes tolerations as JSON string to be used by venona resources (default is no tolerations). If prefixed with "@", loads from a file: @/tmp/tolerations.json |
+| --set-value | stringArray | Set values for templates, example: --set-value Storage.LocalVolumeParentDir=/mnt/disk/ssd0/codefresh-volumes |
 | --venona-version | string | Version of venona to install (default is the latest) |
 
 #### Install on cluster version < 1.10
@@ -72,6 +73,85 @@ rules:
   * Bind your user with cluster-admin kubernetes clusterrole
     > `kubectl create clusterrolebinding NAME --clusterrole cluster-admin --user YOUR_USER`
 
+#### Pipeline Storage with docker cache support
+
+##### **GKE LocalSSD**
+*Prerequisite:* [GKE custer with local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd)
+
+*Install venona for using GKE Local SSD:*
+```
+venonactl install [options] --set-value=Storage.LocalVolumeParentDir=/mnt/disks/ssd0/codefresh-volumes \
+                            --build-node-selector=cloud.google.com/gke-local-ssd=true
+```
+
+##### **GCE Disks** 
+*Prerequisite:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of google disks
+There are 3 options to provide cloud credentials on GCE:
+* run venona dind-volume-provisioniner on node with iam role which is allowed to create/delete/get of google disks
+* create Google Service Account with ComputeEngine.StorageAdmin, download its key and pass it to venona installed with `--set-file=Storage.GooogleServiceAccount=/path/to/google-service-account.json`
+* use [Google Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to assign iam role to `volume-provisioner-venona` service account 
+
+*Note*: Builds will be running in single availability zone, so you must to specify AvailabilityZone params
+
+
+*Install venona for using GKE Disks:*
+```
+venonactl install [options] --set-value=Storage.Backend=gcedisk \
+                            --set-value=Storage.AvailabilityZone=us-central1-a \
+                            --build-node-selector=failure-domain.beta.kubernetes.io/zone=us-central1-a \
+                            [--set-file=Storage.GoogleServiceAccount=/path/to/google-service-account.json]
+```
+
+##### **Amazon EBS**
+
+*Prerequisite:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of aws ebs
+Minimal iam policy for dind-volume-provisioner:
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "ec2:AttachVolume",
+        "ec2:CreateSnapshot",
+        "ec2:CreateTags",
+        "ec2:CreateVolume",
+        "ec2:DeleteSnapshot",
+        "ec2:DeleteTags",
+        "ec2:DeleteVolume",
+        "ec2:DescribeInstances",
+        "ec2:DescribeSnapshots",
+        "ec2:DescribeTags",
+        "ec2:DescribeVolumes",
+        "ec2:DetachVolume"
+      ],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+There are 3 options to provide cloud credentials on AWS:
+* run venona dind-volume-provisioniner on node with the iam role - use `--kube-node-selector=` option
+* create AWS IAM User, assign it the permissions above and suppy aws credentials to venona installer `--set-value=Storage.AwsAccessKeyId=ABCDF --set-value=Storage.AwsSecretAccessKey=ZYXWV`
+
+* use [Aws Identity for Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to assign iam role to `volume-provisioner-venona` service account
+
+*Notes*: 
+- Builds will be running in single availability zone, so you must specify AvailabilityZone parameter `--set-value=Storage.AvailabilityZone=<aws-az>` and build-node-selector `--build-node-selector=failure-domain.beta.kubernetes.io/zone=<aws-az>` in case of multizone cluster
+
+- We support both [in-tree ebs](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore) (`--set-value=Storage.Backend=ebs`) volumes and ebs-csi(https://github.com/kubernetes-sigs/aws-ebs-csi-driver) (`--set-value=Storage.Backend=ebs-csi`)
+
+*Install Command to run pipelines on ebs volumes*
+```
+venonactl install [options] --set-value=Storage.Backend=ebs \
+                            --set-value=Storage.AvailabilityZone=us-east-1d \
+                            --build-node-selector=failure-domain.beta.kubernetes.io/zone=us-east-1d \
+                            [--kube-node-selector=kubernetes.io/role=master] \
+                            [--set-value=Storage.AwsAccessKeyId=ABCDF --set-value=Storage.AwsSecretAccessKey=ZYXWV]
+```
+
 #### Kubernetes RBAC
 Installation of Venona on Kubernetes cluster installing 2 groups of objects,
 Each one has own RBAC needs and therefore, created roles(and cluster-roles)

package.json

@@ -1,6 +1,6 @@
 {
   "name": "venona",
-  "version": "0.30.5",
+  "version": "0.31.0",
   "description": "Codefresh agent to run on Codefresh's runtime environment and execute pipeline",
   "main": "index.js",
   "scripts": {

venonactl/VERSION

@@ -1 +1 @@
-0.30.5
+0.31.0

venonactl/cmd/install.go

@@ -31,6 +31,7 @@ import (
 	"github.com/codefresh-io/venona/venonactl/pkg/plugins"
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
+	"helm.sh/helm/v3/pkg/strvals"
 	k8sApi "k8s.io/api/core/v1"
 )
 
@@ -60,6 +61,8 @@ var installCmdOptions struct {
 	buildNodeSelector             string
 	buildAnnotations              []string
 	tolerations                   string
+	templateValues                []string
+	templateFileValues            []string
 }
 
 // installCmd represents the install command
@@ -197,6 +200,27 @@ var installCmd = &cobra.Command{
 		}
 
 		values := s.BuildValues()
+
+		// from https://github.com/helm/helm/blob/ec1d1a3d3eb672232f896f9d3b3d0797e4f519e3/pkg/cli/values/options.go#L41
+		base := map[string]interface{}{}
+		for _, value := range installCmdOptions.templateValues {
+			if err := strvals.ParseInto(value, base); err != nil {
+				dieOnError(fmt.Errorf("Cannot parse option --set-value %s", value))
+			}
+		}
+
+		for _, value := range installCmdOptions.templateFileValues {
+			reader := func(rs []rune) (interface{}, error) {
+				bytes, err := ioutil.ReadFile(string(rs))
+				return string(bytes), err
+			}
+			if err := strvals.ParseIntoFile(value, base, reader); err != nil {
+				dieOnError(fmt.Errorf("Cannot parse option --set-file %s", value))
+			}
+		}
+
+		values = mergeMaps(values, base)
+
 		for _, p := range builder.Get() {
 			values, err = p.Install(builderInstallOpt, values)
 			if err != nil {
@@ -207,6 +231,25 @@ var installCmd = &cobra.Command{
 	},
 }
 
+func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
+	out := make(map[string]interface{}, len(a))
+	for k, v := range a {
+		out[k] = v
+	}
+	for k, v := range b {
+		if v, ok := v.(map[string]interface{}); ok {
+			if bv, ok := out[k]; ok {
+				if bv, ok := bv.(map[string]interface{}); ok {
+					out[k] = mergeMaps(bv, v)
+					continue
+				}
+			}
+		}
+		out[k] = v
+	}
+	return out
+}
+
 func init() {
 	rootCmd.AddCommand(installCmd)
 
@@ -231,6 +274,9 @@ func init() {
 	installCmd.Flags().BoolVar(&installCmdOptions.setDefaultRuntime, "set-default", false, "Mark the install runtime-environment as default one after installation")
 	installCmd.Flags().BoolVar(&installCmdOptions.kubernetesRunnerType, "kubernetes-runner-type", false, "Set the runner type to kubernetes (alpha feature)")
 
+	installCmd.Flags().StringArrayVar(&installCmdOptions.templateValues, "set-value", []string{}, "Set values for templates, example: --set-value LocalVolumesDir=/mnt/disks/ssd0/codefresh-volumes")
+	installCmd.Flags().StringArrayVar(&installCmdOptions.templateFileValues, "set-file", []string{}, "Set values for templates from file, example: --set-value Storage.GoogleServiceAccount=/path/to/service-account.json")
+
 }
 
 type nodeSelector map[string]string

venonactl/go.mod

@@ -1,44 +1,29 @@
 module github.com/codefresh-io/venona/venonactl
 
+go 1.13
+
 require (
-	contrib.go.opencensus.io/exporter/ocagent v0.4.3 // indirect
-	github.com/Azure/go-autorest v11.4.0+incompatible // indirect
-	github.com/Masterminds/goutils v1.1.0 // indirect
+	github.com/Azure/go-autorest/autorest v0.10.0 // indirect
 	github.com/Masterminds/semver v1.5.0 // indirect
 	github.com/Masterminds/sprig v2.22.0+incompatible
-	github.com/codefresh-io/go-sdk v0.17.0
+	github.com/codefresh-io/go-sdk v0.18.0
 	github.com/dustin/go-humanize v1.0.0
-	github.com/google/go-github/v21 v21.0.0
-	github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367 // indirect
-	github.com/google/uuid v1.1.1 // indirect
-	github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d // indirect
-	github.com/gophercloud/gophercloud v0.0.0-20190206021053-df38e1611dbe // indirect
-	github.com/gregjones/httpcache v0.0.0-20170728041850-787624de3eb7 // indirect
-	github.com/hashicorp/go-version v1.1.0
+	github.com/google/go-github/v21 v21.0.1
+	github.com/gophercloud/gophercloud v0.8.0 // indirect
+	github.com/hashicorp/go-version v1.2.0
 	github.com/huandu/xstrings v1.3.0 // indirect
-	github.com/imdario/mergo v0.3.5 // indirect
-	github.com/inconshreveable/log15 v0.0.0-20180818164646-67afb5ed74ec
-	github.com/json-iterator/go v0.0.0-20180701071628-ab8a2e0c74be // indirect
-	github.com/mattn/go-colorable v0.1.0 // indirect
-	github.com/mattn/go-isatty v0.0.4 // indirect
-	github.com/mitchellh/copystructure v1.0.0 // indirect
-	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/imdario/mergo v0.3.8 // indirect
+	github.com/inconshreveable/log15 v0.0.0-20200109203555-b30bc20e4fd1
+	github.com/mattn/go-colorable v0.1.6 // indirect
 	github.com/olekukonko/tablewriter v0.0.4
-	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
-	github.com/spf13/cobra v0.0.5
-	github.com/spf13/viper v1.6.1
-	go.opencensus.io v0.19.0 // indirect
-	google.golang.org/appengine v1.4.0 // indirect
-	gopkg.in/inf.v0 v0.9.0 // indirect
-	gopkg.in/yaml.v2 v2.2.7
-	k8s.io/api v0.0.0-20181221193117-173ce66c1e39
-	k8s.io/apimachinery v0.0.0-20181127025237-2b1284ed4c93
-	k8s.io/client-go v10.0.0+incompatible
-	k8s.io/klog v0.0.0-20181108234604-8139d8cb77af // indirect
-	sigs.k8s.io/yaml v1.1.0 // indirect
+	github.com/spf13/cobra v0.0.6
+	github.com/spf13/viper v1.6.2
+	gopkg.in/yaml.v2 v2.2.8
+	helm.sh/helm/v3 v3.1.1
+	k8s.io/api v0.17.3
+	k8s.io/apimachinery v0.17.3
+	k8s.io/client-go v0.17.3
+	k8s.io/utils v0.0.0-20200229041039-0a110f9eb7ab // indirect
 )
 
-replace git.apache.org/thrift.git => github.com/apache/thrift v0.0.0-20181218151757-9b75e4fe745a
-
-go 1.13
+replace github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.2+incompatible