THREESCALE-11156 add tls env var to zync and system deployments

[austincunningham]

Issue

THREESCALE-11156

What

Add database client side tls to 3scale

Verification

• Setup Postgres clone this repo https://github.com/austincunningham/postgres and follow instructions to setup system-database and zync
• checkout this branch

# install crd's
make install
# create project
oc new-project 3scale-test
# get the wildcarddomain
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
# create s3 secret
oc apply -f - <<EOF
---
kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
data:
  AWS_ACCESS_KEY_ID: UkVQTEFDRV9NRQ==
  AWS_BUCKET: UkVQTEFDRV9NRQ==
  AWS_REGION: UkVQTEFDRV9NRQ==
  AWS_SECRET_ACCESS_KEY: UkVQTEFDRV9NRQ==
type: Opaque
EOF

kubectl apply -f - <<EOF                    
---                                    
apiVersion: apps.3scale.net/v1alpha1   
kind: APIManager                     
metadata:
  name: apimanager-sample
spec:
  system:
    systemDatabaseTLSEnabled: true
    database:
      postgresql: {}
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: s3-credentials
  zync:
    zyncDatabaseTLSEnabled: true
  wildcardDomain: $DOMAIN
  externalComponents:
    system:
      database: true
    zync:
      database: true
EOF

• Setup the system-database and zync secrets with sslmode disabled (need to be run from the postgres repo as thats where the certs are created)

oc create secret generic zync \
  --namespace=3scale-test \
  --from-literal=DATABASE_SSL_MODE=verify-ca \
  --from-literal=DATABASE_URL=postgresql://postgres:[email protected]/zync_production \
  --from-literal=ZYNC_DATABASE_PASSWORD=postgres \
  --from-file=DB_SSL_CA=rootCA.crt \
  --from-file=DB_SSL_CERT=client.crt \
  --from-file=DB_SSL_KEY=client.key 

oc create secret generic system-database \
  --from-literal=DATABASE_SSL_MODE=verify-ca \
  --from-literal=URL=postgresql://postgres:[email protected]/system \
  --from-literal=DB_USER=postgres \
  --from-literal=DB_PASSWORD=postgres \
  --from-file=DB_SSL_CA=rootCA.crt \
  --from-file=DB_SSL_CERT=client.crt \
  --from-file=DB_SSL_KEY=client.key 

• Create the default redis and wait for the deployments to be up

 make cluster/create/system-redis
 make cluster/create/backend-redis

• Run the operator locally

make run

• zync-que may crash as the database needs some bootstrapping, exec into the pod and run the bootstrap, Not tls related as happens without tls enabled.

bin/rails db:create

• wait for the install to complete
• exec into the zync and system pods to to confirm the env vars have been set e.g.

# get the pod name
 oc get po
NAME                                     READY   STATUS      RESTARTS   AGE
apicast-production-6d4d589848-shlls      1/1     Running     0          3h5m
apicast-staging-767df4cbd8-m2kf5         1/1     Running     0          3h5m
backend-cron-6bbd4d5846-hmc7r            1/1     Running     0          3h5m
backend-listener-9df5df447-842qk         1/1     Running     0          3h5m
backend-redis-b59dd94cb-pksk7            1/1     Running     0          3h5m
backend-worker-64d47694b8-89q2g          1/1     Running     0          3h5m
system-app-85c4d98dd-j9mtr               3/3     Running     0          4m40s
system-app-post-884dp                    0/1     Completed   0          3h3m
system-app-pre-wlzrd                     0/1     Completed   0          3h5m
system-memcache-6b4df58598-q6mzj         1/1     Running     0          3h5m
system-redis-bf8db486d-h5bq9             1/1     Running     0          3h5m
system-searchd-659bffb8f4-swf4g          1/1     Running     0          5m50s
system-searchd-manticore-reindex-5jjb4   0/1     Completed   0          3h5m
system-sidekiq-59cc7dd9ff-l6pxg          1/1     Running     0          5m53s
zync-6f49fb7c5c-2pltl                    0/1     Init:0/2    0          7m51s
zync-que-fdd4d69f7-7gxdx                 1/1     Running     0          7m50s
#e.g. exec and run printenv to check the envars
oc exec -it zync-que-fdd4d69f7-7gxdx -- printenv | grep DATABASE_SSL
Defaulted container "que" out of: que, set-permissions (init)
DATABASE_SSL_KEY=/tls/tls.key
DATABASE_SSL_MODE=verify-ca
DATABASE_SSL_CA=/tls/ca.crt
DATABASE_SSL_CERT=/tls/tls.crt

• Add that the watch-by label has been applied to zync and system-database secret

oc label secret zync apimanager.apps.3scale.net/watched-by=apimanager -n <operator-namespace>
oc label secret system-database apimanager.apps.3scale.net/watched-by=apimanager -n <operator-namespace>

• update the zync and system-database secrets key value DATABASE_SSL_MODE=disable

NOTE: side note the flag is different for MySQL which is disabled

• exec into the zync and system pods to to confirm the env vars have been set e.g.

# get the pod name
 oc get po
NAME                                     READY   STATUS      RESTARTS   AGE
apicast-production-6d4d589848-shlls      1/1     Running     0          3h5m
apicast-staging-767df4cbd8-m2kf5         1/1     Running     0          3h5m
backend-cron-6bbd4d5846-hmc7r            1/1     Running     0          3h5m
backend-listener-9df5df447-842qk         1/1     Running     0          3h5m
backend-redis-b59dd94cb-pksk7            1/1     Running     0          3h5m
backend-worker-64d47694b8-89q2g          1/1     Running     0          3h5m
system-app-85c4d98dd-j9mtr               3/3     Running     0          4m40s
system-app-post-884dp                    0/1     Completed   0          3h3m
system-app-pre-wlzrd                     0/1     Completed   0          3h5m
system-memcache-6b4df58598-q6mzj         1/1     Running     0          3h5m
system-redis-bf8db486d-h5bq9             1/1     Running     0          3h5m
system-searchd-659bffb8f4-swf4g          1/1     Running     0          5m50s
system-searchd-manticore-reindex-5jjb4   0/1     Completed   0          3h5m
system-sidekiq-59cc7dd9ff-l6pxg          1/1     Running     0          5m53s
zync-6f49fb7c5c-2pltl                    0/1     Init:0/2    0          7m51s
zync-que-fdd4d69f7-7gxdx                 1/1     Running     0          7m50s
#e.g. exec and run printenv to check the envars
oc exec -it zync-que-fdd4d69f7-7gxdx -- printenv | grep DATABASE_SSL
Defaulted container "que" out of: que, set-permissions (init)
DATABASE_SSL_KEY=/tls/tls.key
DATABASE_SSL_MODE=disable
DATABASE_SSL_CA=/tls/ca.crt
DATABASE_SSL_CERT=/tls/tls.crt

[austincunningham]

/test test-e2e

[austincunningham]

/test test-e2e

[MStokluska]

if it's not a problem, please move this to system spec and zync spec respectively. Not a big deal if you prefer this is a global.

[austincunningham]

Don't believe it would be a problem to move. It's preference really. And it doesn't matter to me where it lives .

[MStokluska]

as discussed, let's move the creation of empty keys + validation of keys + validation of values here: https://github.com/3scale/3scale-operator/blob/master/pkg/3scale/amp/operator/highavailability_options_provider.go#L29

[austincunningham]

Agree its better if the reconciler catches this.

[austincunningham]

Yes the secret validation should check if the secret values are populated in the highavailability reconcile and populate with an empty string if not set and return an error.

Two things here

• It doesn't look like highavailability_options_provider GetHighAvailabilityOptions() handles zync database at the moment (maybe because spec.highavaibility is deprecated). Can add the functionally but am now doubting this is the correct place to do so.
• And may still leave the helper to populate the env vars that are not available in the secret .
  i.e. the env vars in the secret have actual certs where as the env var that are used have paths to certs

[MStokluska]

is this required

[austincunningham]

Think it was commented during some debugging . So I would say its required.

[austincunningham]

having reviewed its not required.