Refactoring on Superset manifests (opendatahub-io#459)

ai-library/tests/setup/kfctl_openshift.yaml

@@ -28,10 +28,6 @@ spec:
     # Note:  In order to utilize ai-library, you also need to have Seldon installed
     name: ai-library-operator
   - kustomizeConfig:
-      parameters:
-      # Note: The admin username is admin
-      - name: SUPERSET_ADMIN_PASSWORD
-        value: admin
       repoRef:
         name: manifests
         path: superset

kfdef/kfctl_openshift.yaml

@@ -16,10 +16,6 @@ spec:
         path: odhseldon/cluster
     name: odhseldon
   - kustomizeConfig:
-      parameters:
-      # Note: The admin username is admin
-      - name: SUPERSET_ADMIN_PASSWORD
-        value: admin
       repoRef:
         name: manifests
         path: superset

kfdef/kfctl_openshift_distributed_training.yaml

@@ -28,10 +28,6 @@ spec:
     # Note:  In order to utilize ai-library, you also need to have Seldon installed
     name: ai-library-operator
   - kustomizeConfig:
-      parameters:
-      # Note: The admin username is admin
-      - name: SUPERSET_ADMIN_PASSWORD
-        value: admin
       repoRef:
         name: manifests
         path: superset

superset/README.md

@@ -11,17 +11,73 @@ To install Superset add the following to the `kfctl` yaml file.
 
 ```
   - kustomizeConfig:
-      parameters:
-      # Note: The admin username is admin
-      - name: SUPERSET_ADMIN_PASSWORD
-        value: admin
       repoRef:
         name: manifests
         path: superset
     name: superset
 ```
 
-By default the user and password to the Superset portal is admin/admin. You can change the password by changing the value of the `SUPERSET_ADMIN_PASSWORD`. To launch the portal, go to the routes in the namespace you installed Open Data Hub and click on the route with `superset` name.
+By default the user and password to the Superset portal is admin/admin. To launch the portal, go to the routes in the namespace you installed Open Data Hub and click on the route with `superset` name.
+
+### Parameters
+
+There are 11 parameters exposed via KfDef.
+
+#### storage_class
+
+Name of the storage class to be used for PVC created by Hue's database. This requires `storage-class` **overlay** to be enabled as well to work.
+#### superset_secret
+
+The secret containing the environment variables to set the admin credentials and the Superset app secret key used to encrypt information in the database. If not specified, environment variables from [`superset`](base/secret-superset.yaml) will be used.
+
+When creating a custom secret, the following information should be added:
+
+* **SUPERSET_ADMIN_USER**: The username of the Superset administrator. If using the `superset` secret, `admin` will be used
+* **SUPERSET_ADMIN_FNAME**: The First Name of the Superset administrator. If using the `superset` secret, `admin` will be used
+* **SUPERSET_ADMIN_LNAME**: The LastName of the Superset administrator. If using the `superset` secret, `admin` will be used
+* **SUPERSET_ADMIN_EMAIL**: The e-mail of the Superset administrator. If using the `superset` secret, `[email protected]` will be used
+* **SUPERSET_ADMIN_PASSWORD**: The password of the Superset administrator. If using the `superset` secret, `admin` will be used
+* **SUPERSET_SECRET_KEY**: The app secret key used by Superset to encrypt information in the database. If using the `superset` secret, `thisISaSECRET_1234` will be used
+
+#### superset_db_secret
+
+This parameter configures the Superset database. The secret of choice must contain `database-name`, `database-user`, and `database-password` keys. If not set, credentials from [`supersetdb-secret`](base/supersetdb-secret.yaml) will be used instead.
+
+#### superset_memory_requests
+
+This parameter will configure the Memory request for Superset. If not set, the default value `1Gi` will be used instead.
+
+#### superset_memory_limits
+
+This parameter will configure the Memory limits for Superset. If not set, the default value `2Gi` will be used instead.
+
+#### superset_cpu_requests
+
+This parameter will configure the CPU request for Superset. If not set, the default value `300m` will be used instead.
+
+#### superset_cpu_limits
+
+This parameter will configure the CPU limits for Superset. If not set, the default value `2` will be used instead.
+
+#### superset_db_memory_requests
+
+This parameter will configure the Memory request for Superset Database. If not set, the default value `300Mi` will be used instead.
+
+#### superset_db_memory_limits
+
+This parameter will configure the Memory limits for Superset Database. If not set, the default value `1Gi` will be used instead.
+
+#### superset_db_cpu_requests
+
+This parameter will configure the CPU request for Superset Database. If not set, the default value `300m` will be used instead.
+
+#### superset_db_cpu_limits
+
+This parameter will configure the CPU request for Superset Database. If not set, the default value `1` will be used instead.
+
+### Superset config file customization
+
+Superset manifests comes with a [`superset-config`](base/secret.yaml) secret, which will configure basic parameters for Superset like the database SQLAlchemy URL to connect to its database.
 
 ### Superset Database Initialization
 

superset/base/deployment.yaml

@@ -17,19 +17,58 @@ spec:
     spec:
       initContainers:
       - name: superset-init
-        image: $(SUPERSET_IMAGE)
-        command: ['sh', '-c', "sleep 30; superset-init --username $(SUPERSET_ADMIN_USER) --firstname $(SUPERSET_ADMIN_FNAME) --lastname $(SUPERSET_ADMIN_LNAME) --email $(SUPERSET_ADMIN_EMAIL) --password $(SUPERSET_ADMIN_PASSWORD)"]
-        imagePullPolicy: Always
+        env:
+        - name: SUPERSET_CONFIG_PATH
+          value: /etc/superset/superset_config.py
+        - name: POSTGRESQL_USERNAME
+          valueFrom:
+            secretKeyRef:
+              key: database-user
+              name: $(superset_db_secret)
+        - name: POSTGRESQL_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              key: database-password
+              name: $(superset_db_secret)
+        - name: POSTGRESQL_DATABASE
+          valueFrom:
+            secretKeyRef:
+              key: database-name
+              name: $(superset_db_secret)
+        envFrom:
+          - secretRef:
+              name: $(superset_secret)
+        image: superset
+        command: ['sh', '-c', "sleep 30; superset-init --username $SUPERSET_ADMIN_USER --firstname $SUPERSET_ADMIN_FNAME --lastname $SUPERSET_ADMIN_LNAME --email $SUPERSET_ADMIN_EMAIL --password $SUPERSET_ADMIN_PASSWORD"]
         volumeMounts:
           - mountPath: /etc/superset
             name: superset-config
       containers:
       - name: superset
-        envFrom:
-          - secretRef:
-              name: superset
-        image: $(SUPERSET_IMAGE)
-        imagePullPolicy: Always
+        env:
+        - name: SUPERSET_SECRET_KEY
+          valueFrom:
+            secretKeyRef:
+              key: SUPERSET_SECRET_KEY
+              name: $(superset_secret)
+        - name: POSTGRESQL_USERNAME
+          valueFrom:
+            secretKeyRef:
+              key: database-user
+              name: $(superset_db_secret)
+        - name: POSTGRESQL_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              key: database-password
+              name: $(superset_db_secret)
+        - name: POSTGRESQL_DATABASE
+          valueFrom:
+            secretKeyRef:
+              key: database-name
+              name: $(superset_db_secret)
+        - name: SUPERSET_CONFIG_PATH
+          value: /etc/superset/superset_config.py
+        image: superset
         ports:
           - containerPort: 8088
             protocol: TCP
@@ -38,11 +77,11 @@ spec:
             name: superset-config
         resources:
           requests:
-            cpu: $(SUPERSET_CPU_REQUESTS)
-            memory: $(SUPERSET_MEMORY_REQUESTS)
+            cpu: $(superset_cpu_requests)
+            memory: $(superset_memory_requests)
           limits:
-            cpu: $(SUPERSET_CPU_LIMITS)
-            memory: $(SUPERSET_MEMORY_LIMITS)
+            cpu: $(superset_cpu_limits)
+            memory: $(superset_memory_limits)
       volumes:
         - name: superset-config
           secret:

superset/base/kustomization.yaml

@@ -1,7 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-- service-account.yaml
 - pvc.yaml
 - secret-superset.yaml
 - deployment.yaml
@@ -25,143 +24,94 @@ configMapGenerator:
       - params.env
 
 vars:
-  - name: SUPERSET_ADMIN_PASSWORD
+  - name: superset_secret
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_ADMIN_PASSWORD
-  - name: SUPERSET_ADMIN_USER
+      fieldpath: data.superset_secret
+  - name: superset_db_secret
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_ADMIN_USER
-  - name: SUPERSET_ADMIN_FNAME
+      fieldpath: data.superset_db_secret
+  - name: superset_memory_requests
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_ADMIN_FNAME
-  - name: SUPERSET_ADMIN_LNAME
+      fieldpath: data.superset_memory_requests
+  - name: superset_memory_limits
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_ADMIN_LNAME
-  - name: SUPERSET_ADMIN_EMAIL
+      fieldpath: data.superset_memory_limits
+  - name: superset_cpu_requests
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_ADMIN_EMAIL
-  - name: SUPERSET_VOLUME_SIZE
+      fieldpath: data.superset_cpu_requests
+  - name: superset_cpu_limits
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_VOLUME_SIZE
-  - name: SUPERSET_SECRET_KEY
+      fieldpath: data.superset_cpu_limits
+  - name: superset_db_memory_requests
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_SECRET_KEY
-  - name: SUPERSET_DB_USERNAME
+      fieldpath: data.superset_db_memory_requests
+  - name: superset_db_memory_limits
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_DB_USERNAME
-  - name: SUPERSET_DB_PASSWORD
+      fieldpath: data.superset_db_memory_limits
+  - name: superset_db_cpu_requests
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_DB_PASSWORD
-  - name: LOG_LEVEL
+      fieldpath: data.superset_db_cpu_requests
+  - name: superset_db_cpu_limits
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.LOG_LEVEL
-  - name: SUPERSET_MEMORY_REQUESTS
+      fieldpath: data.superset_db_cpu_limits
+  - name: storage_class
     objref:
       name: superset-config
       kind: ConfigMap
       apiVersion: v1
     fieldref:
-      fieldpath: data.SUPERSET_MEMORY_REQUESTS
-  - name: SUPERSET_MEMORY_LIMITS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_MEMORY_LIMITS
-  - name: SUPERSET_CPU_REQUESTS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_CPU_REQUESTS
-  - name: SUPERSET_CPU_LIMITS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_CPU_LIMITS
-  - name: SUPERSET_DB_MEMORY_REQUESTS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_DB_MEMORY_REQUESTS
-  - name: SUPERSET_DB_MEMORY_LIMITS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_DB_MEMORY_LIMITS
-  - name: SUPERSET_DB_CPU_REQUESTS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_DB_CPU_REQUESTS
-  - name: SUPERSET_DB_CPU_LIMITS
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_DB_CPU_LIMITS
-  - name: SUPERSET_IMAGE
-    objref:
-      name: superset-config
-      kind: ConfigMap
-      apiVersion: v1
-    fieldref:
-      fieldpath: data.SUPERSET_IMAGE
-
+      fieldpath: data.storage_class
 
 configurations:
   - params.yaml
 
 generatorOptions:
   disableNameSuffixHash: true
+
+images:
+  - name: superset
+    newName: quay.io/opendatahub/superset
+    newTag: "1.1.0"
+  - name: supersetdb
+    newName: quay.io/internaldatahub/postgresql-96-centos7
+    newTag: "9.6"