Merge pull request #169 from arms11/redis-as-coordination

Setting redis as coordination backend

Author: arm4b

.circleci/config.yml

@@ -50,7 +50,7 @@ jobs:
   # Spin up minikube K8s cluster and run Helm chart & e2e tests on it
   helm-e2e:
     # 'large' 4 vCPUs & 15GB RAM CircleCI machine executor
-    # required to deploy heavy 'stackstorm-ha' Helm release with RabbitMQ, MongoDB, etcd clusters and 25+ st2 Pods.
+    # required to deploy heavy 'stackstorm-ha' Helm release with RabbitMQ, MongoDB, Redis clusters and 25+ st2 Pods.
     # https://circleci.com/docs/2.0/configuration-reference/#machine-executor-linux
     resource_class: large
     machine:

CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## In Development
 
+## v0.51.0
+* Added Redis with Sentinel to replace etcd as a coordination backend (#169)
+
 ## v0.50.0
 * Drop Helm `v2` support and fully migrate to Helm `v3` (#163)
 * Switch dependencies from deprecated `helm/charts` to new Bitnami Subcharts (#163)

Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 # StackStorm version which refers to Docker images tag
 appVersion: 3.4dev
 name: stackstorm-ha
-version: 0.50.0
+version: 0.51.0
 description: StackStorm K8s Helm Chart, optimized for running StackStorm in HA environment.
 home: https://stackstorm.com/
 icon: https://landscape.cncf.io/logos/stack-storm.svg
@@ -40,7 +40,8 @@ dependencies:
     version: 4.0.0
     repository: https://charts.bitnami.com/bitnami
     condition: external-dns.enabled
-  - name: etcd
-    version: 5.1.0
+  - name: redis
+    version: 12.3.2
     repository: https://charts.bitnami.com/bitnami
-    condition: etcd.enabled
+    condition: redis.enabled
+

README.md

@@ -4,7 +4,7 @@
 K8s Helm Chart for running StackStorm cluster in HA mode.
 
 It will install 2 replicas for each component of StackStorm microservices for redundancy, as well as backends like
-RabbitMQ HA, MongoDB HA Replicaset and etcd cluster that st2 replies on for MQ, DB and distributed coordination respectively.
+RabbitMQ HA, MongoDB HA Replicaset and Redis cluster that st2 replies on for MQ, DB and distributed coordination respectively.
 
 It's more than welcome to fine-tune each component settings to fit specific availability/scalability demands.
 
@@ -116,7 +116,7 @@ All the workflow engine processes will share the load and pick up more work if o
 Multiple st2notifier processes can run in active-active mode, using connections to RabbitMQ and MongoDB and generating triggers based on
 action execution completion as well as doing action rescheduling.
 In an HA deployment there must be a minimum of `2` replicas of st2notifier running, requiring a coordination backend,
-which in our case is `etcd`.
+which in our case is `Redis`.
 
 ### [st2sensorcontainer](https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer)
 st2sensorcontainer manages StackStorm sensors: It starts, stops and restarts them as subprocesses.
@@ -143,7 +143,7 @@ st2:
 ### [st2actionrunner](https://docs.stackstorm.com/reference/ha.html#st2actionrunner)
 Stackstorm workers that actually execute actions.
 `5` replicas for K8s Deployment are configured by default to increase StackStorm ability to execute actions without excessive queuing.
-Relies on `etcd` for coordination. This is likely the first thing to lift if you have a lot of actions
+Relies on `redis` for coordination. This is likely the first thing to lift if you have a lot of actions
 to execute per time period in your StackStorm cluster.
 
 ### [st2garbagecollector](https://docs.stackstorm.com/reference/ha.html#st2garbagecollector)
@@ -174,9 +174,9 @@ Helm chart repository, - all settings could be overridden via `values.yaml`.
 
 The deployment of RabbitMQ to the k8s cluster can be disabled by setting the rabbitmq-ha.enabled key in values.yaml to false.  *Note: Stackstorm relies heavily on connections to a RabbitMQ instance.  If the in-cluster deployment of RabbitMQ is disabled, a connection to an external instance of RabbitMQ must be configured.  The st2.config key in values.yaml provides a way to configure stackstorm.  See [Configure RabbitMQ](https://docs.stackstorm.com/install/config/config.html#configure-rabbitmq) for configuration details.*
 
-### [etcd](https://docs.stackstorm.com/latest/reference/ha.html#zookeeper-redis)
-StackStorm employs etcd as a distributed coordination backend, required for st2 cluster components to work properly in HA scenario.
-`3` node Raft cluster is deployed via external bitnami Helm chart dependency [etcd](https://github.com/bitnami/charts/tree/master/bitnami/etcd).
+### [redis](https://docs.stackstorm.com/latest/reference/ha.html#zookeeper-redis)
+StackStorm employs redis sentinel as a distributed coordination backend, required for st2 cluster components to work properly in HA scenario.
+`3` node Redis cluster with Sentinel enabled is deployed via external bitnami Helm chart dependency [redis](https://github.com/bitnami/charts/tree/master/bitnami/redis).
 As any other Helm dependency, it's possible to further configure it for specific scaling needs via `values.yaml`.
 
 ## Install custom st2 packs in the cluster
@@ -220,7 +220,7 @@ Grab all logs for entire StackStorm cluster with dependent services in Helm rele
 kubectl logs -l release=<release-name>
 ```
 
-Grab all logs only for stackstorm backend services, excluding st2web and DB/MQ/etcd:
+Grab all logs only for stackstorm backend services, excluding st2web and DB/MQ/redis:
 ```
 kubectl logs -l release=<release-name>,tier=backend
 ```

templates/_helpers.tpl

@@ -76,3 +76,21 @@ Create the name of the stackstorm-ha service account to use
   {{- end -}}
 {{- end -}}
 {{- end -}}
+
+# Generate list of nodes for Redis with Sentinel connection string, based on number of replicas and service name
+{{- define "redis-nodes" -}}
+{{- if not .Values.redis.sentinel.enabled }}
+{{- fail "value for redis.sentinel.enabled MUST be true" }}
+{{- end }}
+{{- $replicas := (int (index .Values "redis" "cluster" "slaveCount")) }}
+{{- $master_name := (index .Values "redis" "sentinel" "masterSet") }}
+{{- $sentinel_port := (index .Values "redis" "sentinel" "port") }}
+{{- range $index0 := until $replicas -}}
+  {{- if eq $index0 0 -}}
+    {{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless:{{ $sentinel_port }}?sentinel={{ $master_name }}
+  {{- else -}}
+    &sentinel_fallback={{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless:{{ $sentinel_port }}
+  {{- end -}}
+{{- end -}}
+{{- end -}}
+    

templates/configmaps_st2-conf.yaml

@@ -20,9 +20,9 @@ data:
   st2.docker.conf: |
     [auth]
     api_url = http://{{ .Release.Name }}-st2api{{ template "enterpriseSuffix" . }}:9101/
-    {{- if index .Values "etcd" "enabled" }}
+    {{- if index .Values "redis" "enabled" }}
     [coordination]
-    url = etcd://{{ index .Values "etcd" "fullnameOverride" }}:2379
+    url = redis://{{ template "redis-nodes" $ }}
     {{- end }}
     {{- if index .Values "rabbitmq" "enabled" }}
     [messaging]

tests/st2tests.sh

@@ -57,3 +57,20 @@ load "${BATS_HELPERS_DIR}/bats-file/load.bash"
   assert_success
   assert_line --partial 'chatops.notify'
 }
+
+@test 'st2 key/value operations are functional' {
+  run st2 key set foo bar
+  assert_success
+
+  run st2 key get foo
+  assert_success
+  assert_line --partial 'bar'
+
+  run st2 key delete foo
+  assert_line --partial '"foo" has been successfully deleted'
+  assert_success
+
+  run st2 key get foo
+  assert_line --partial '"foo" is not found'
+  assert_failure
+}

values.yaml

@@ -509,18 +509,31 @@ rabbitmq:
     enabled: false
 
 ##
-## Etcd HA configuration (3rd party chart dependency)
+## Redis HA configuration (3rd party chart dependency)
 ##
 ## For values.yaml reference:
-## https://github.com/bitnami/charts/tree/master/bitnami/etcd
+## https://github.com/bitnami/charts/tree/master/bitnami/redis
 ##
-etcd:
-  # Change to `false` to disable in-cluster ectd deployment.
+redis:
+  # Change to `false` to disable in-cluster redis deployment.
   # Specify your external [coordination] connection parameters under st2.config
   enabled: true
-  statefulset:
-    replicaCount: 3
-  fullnameOverride: etcd
+  # By default the cluster is enabled for the subchart.
+  # We just need replica count here to ensure it is HA
+  cluster:
+    slaveCount: 3
+  # ## Sentinel settings. Sentinel is enabled for better resiliency.
+  # ## This is highly recommended as per tooz library documentation. 
+  # ## Hence, the chart requires the setting.
+  # ## https://docs.openstack.org/tooz/latest/user/drivers.html#redis
+  # ## https://github.com/bitnami/charts/tree/master/bitnami/redis#master-slave-with-sentinel
+  sentinel:
+    enabled: true
+  networkPolicy:
+    enabled: false
+  usePassword: false
+  metrics:
+    enabled: false
 
 ##
 ## External DNS configuration (3rd party chart dependency)