Installation failed

[shutter-cp]

1. Script generated

sh customize_fusion_values.sh -c kubernetes -n lw5 --provider kubernetes  --num-solr 1 --node-pool "{}"

2. Execute the generated script

./kubernetes_kubernetes_lw5_upgrade_fusion.sh

The results of

namespace/lw5 created

Created namespace lw5 with owner label 

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Upgrading the 'lw5' release (Fusion chart: 5.2.0) in the 'lw5' namespace in the 'kubernetes' cluster using values:
      kubernetes_kubernetes_lw5_fusion_values.yaml

NOTE: If this will be a long-running cluster for production purposes, you should save the following file(s) in version control:
  kubernetes_kubernetes_lw5_fusion_values.yaml

Release "lw5" does not exist. Installing it now.
coalesce.go:199: warning: destination for client is a table. Ignoring non-table value 2181
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: lw5
LAST DEPLOYED: Sun Sep 13 17:53:04 2020
NAMESPACE: lw5
STATUS: deployed
REVISION: 1

Waiting up to 10 minutes to see the Fusion API Gateway deployment come online ...

Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "lw5-api-gateway" rollout to finish: 0 of 1 updated replicas are available...
error: timed out waiting for the condition

Waiting up to 5 minutes to see the Fusion Indexing deployment come online ...

Waiting for deployment "lw5-fusion-indexing" rollout to finish: 0 of 1 updated replicas are available...
error: deployment "lw5-fusion-indexing" exceeded its progress deadline
Context "kubernetes-admin@kubernetes" modified.

NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART       	APP VERSION
lw5 	lw5      	1       	2020-09-13 17:53:04.223112255 +0800 CST	deployed	fusion-5.2.0	5.2.0  

logs -> kubectl get all -n lw5

NAME                                             READY   STATUS                  RESTARTS   AGE
pod/lw5-admin-ui-fc648b4bc-df8k6                 1/1     Running                 0          16m
pod/lw5-ambassador-5c56f99f85-wb98r              1/1     Running                 0          16m
pod/lw5-api-gateway-56cb494746-w9xwt             0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-argo-ui-97688cdd5-ww6t5                  1/1     Running                 0          16m
pod/lw5-auth-ui-744bf58697-rqmnl                 1/1     Running                 0          16m
pod/lw5-classic-rest-service-0                   0/1     Pending                 0          16m
pod/lw5-devops-ui-84cf4bbb9f-d68gc               1/1     Running                 0          16m
pod/lw5-fusion-admin-7c66d87d99-dm75c            0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-fusion-indexing-fd7f886b-hsjl9           0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-fusion-log-forwarder-655f65c864-g2tmk    0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-insights-6c4c6f6464-k96wl                1/1     Running                 0          16m
pod/lw5-job-launcher-7bfc6d9878-49mn6            0/1     Running                 7          16m
pod/lw5-job-rest-server-575685b498-cggtd         0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-ml-model-service-858585f586-cr266        0/2     Init:CrashLoopBackOff   5          16m
pod/lw5-pm-ui-cf68fd6b6-w6s7g                    1/1     Running                 0          16m
pod/lw5-pulsar-bookkeeper-0                      0/1     Pending                 0          16m
pod/lw5-pulsar-broker-0                          0/1     Init:0/4                0          16m
pod/lw5-pulsar-broker-1                          0/1     Init:0/4                0          16m
pod/lw5-query-pipeline-5c44887974-2l5cw          0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-rest-service-6f87f5f488-sf58k            0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-rpc-service-59f4c7c5cb-xx7d4             0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-rules-ui-7d6cc45486-jd6dw                1/1     Running                 0          16m
pod/lw5-solr-0                                   0/1     Pending                 0          16m
pod/lw5-solr-exporter-74677cf947-t46nx           0/1     Init:0/1                0          16m
pod/lw5-templating-57c96d65fc-whq8c              0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-webapps-55b64587f8-gbspb                 0/1     Init:CrashLoopBackOff   5          16m
pod/lw5-workflow-controller-5b877d7c67-sx4km     1/1     Running                 0          16m
pod/lw5-zookeeper-0                              0/1     Pending                 0          16m
pod/seldon-controller-manager-7b855d7f5c-zs4cr   1/1     Running                 0          16m

NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
service/admin                      ClusterIP      10.109.175.238   <none>        8765/TCP                              16m
service/admin-ui                   ClusterIP      10.96.62.2       <none>        8080/TCP                              16m
service/auth-ui                    ClusterIP      10.103.125.149   <none>        8080/TCP                              16m
service/connector-plugin-service   ClusterIP      10.107.48.157    <none>        9020/TCP                              16m
service/connectors                 ClusterIP      10.97.196.223    <none>        9010/TCP                              16m
service/connectors-classic         ClusterIP      None             <none>        9000/TCP                              16m
service/connectors-rpc             ClusterIP      10.101.196.44    <none>        8771/TCP                              16m
service/devops-ui                  ClusterIP      10.107.250.159   <none>        8080/TCP                              16m
service/indexing                   ClusterIP      10.104.226.98    <none>        8765/TCP                              16m
service/insights                   ClusterIP      10.96.173.133    <none>        8080/TCP                              16m
service/job-launcher               ClusterIP      10.111.187.156   <none>        8083/TCP                              16m
service/job-rest-server            ClusterIP      10.97.153.182    <none>        8081/TCP                              16m
service/lw5-ambassador             ClusterIP      10.102.138.235   <none>        80/TCP,443/TCP                        16m
service/lw5-argo-ui                ClusterIP      10.96.18.81      <none>        2746/TCP                              16m
service/lw5-pulsar-bookkeeper      ClusterIP      None             <none>        3181/TCP,8000/TCP                     16m
service/lw5-pulsar-broker          ClusterIP      None             <none>        8080/TCP,6650/TCP                     16m
service/lw5-solr-exporter          ClusterIP      10.99.38.190     <none>        9983/TCP                              16m
service/lw5-solr-headless          ClusterIP      None             <none>        8983/TCP                              16m
service/lw5-solr-svc               ClusterIP      10.96.213.18     <none>        8983/TCP                              16m
service/lw5-zookeeper              ClusterIP      10.101.97.38     <none>        2181/TCP,2281/TCP                     16m
service/lw5-zookeeper-headless     ClusterIP      None             <none>        2181/TCP,3888/TCP,2888/TCP,2281/TCP   16m
service/ml-model-grpc              ClusterIP      10.101.108.200   <none>        6565/TCP                              16m
service/ml-model-service           ClusterIP      10.108.67.146    <none>        8086/TCP                              16m
service/pm-ui                      ClusterIP      10.100.61.168    <none>        8080/TCP                              16m
service/proxy                      LoadBalancer   10.108.80.194    <pending>     6764:30949/TCP                        16m
service/pulsar-broker              ClusterIP      None             <none>        8080/TCP,6650/TCP                     16m
service/query                      ClusterIP      10.110.49.50     <none>        8787/TCP                              16m
service/rules-ui                   ClusterIP      10.110.122.11    <none>        8080/TCP                              16m
service/seldon-webhook-service     ClusterIP      10.107.47.242    <none>        443/TCP                               16m
service/templating                 ClusterIP      10.108.50.41     <none>        5250/TCP                              16m
service/webapps                    ClusterIP      10.100.239.137   <none>        8780/TCP                              16m

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/lw5-admin-ui                   1/1     1            1           16m
deployment.apps/lw5-ambassador                 1/1     1            1           16m
deployment.apps/lw5-api-gateway                0/1     1            0           16m
deployment.apps/lw5-argo-ui                    1/1     1            1           16m
deployment.apps/lw5-auth-ui                    1/1     1            1           16m
deployment.apps/lw5-connector-plugin-service   0/0     0            0           16m
deployment.apps/lw5-devops-ui                  1/1     1            1           16m
deployment.apps/lw5-fusion-admin               0/1     1            0           16m
deployment.apps/lw5-fusion-indexing            0/1     1            0           16m
deployment.apps/lw5-fusion-log-forwarder       0/1     1            0           16m
deployment.apps/lw5-insights                   1/1     1            1           16m
deployment.apps/lw5-job-launcher               0/1     1            0           16m
deployment.apps/lw5-job-rest-server            0/1     1            0           16m
deployment.apps/lw5-ml-model-service           0/1     1            0           16m
deployment.apps/lw5-pm-ui                      1/1     1            1           16m
deployment.apps/lw5-query-pipeline             0/1     1            0           16m
deployment.apps/lw5-rest-service               0/1     1            0           16m
deployment.apps/lw5-rpc-service                0/1     1            0           16m
deployment.apps/lw5-rules-ui                   1/1     1            1           16m
deployment.apps/lw5-solr-exporter              0/1     1            0           16m
deployment.apps/lw5-templating                 0/1     1            0           16m
deployment.apps/lw5-webapps                    0/1     1            0           16m
deployment.apps/lw5-workflow-controller        1/1     1            1           16m
deployment.apps/seldon-controller-manager      1/1     1            1           16m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/lw5-admin-ui-fc648b4bc                    1         1         1       16m
replicaset.apps/lw5-ambassador-5c56f99f85                 1         1         1       16m
replicaset.apps/lw5-api-gateway-56cb494746                1         1         0       16m
replicaset.apps/lw5-argo-ui-97688cdd5                     1         1         1       16m
replicaset.apps/lw5-auth-ui-744bf58697                    1         1         1       16m
replicaset.apps/lw5-connector-plugin-service-6787cc8d46   0         0         0       16m
replicaset.apps/lw5-devops-ui-84cf4bbb9f                  1         1         1       16m
replicaset.apps/lw5-fusion-admin-7c66d87d99               1         1         0       16m
replicaset.apps/lw5-fusion-indexing-fd7f886b              1         1         0       16m
replicaset.apps/lw5-fusion-log-forwarder-655f65c864       1         1         0       16m
replicaset.apps/lw5-insights-6c4c6f6464                   1         1         1       16m
replicaset.apps/lw5-job-launcher-7bfc6d9878               1         1         0       16m
replicaset.apps/lw5-job-rest-server-575685b498            1         1         0       16m
replicaset.apps/lw5-ml-model-service-858585f586           1         1         0       16m
replicaset.apps/lw5-pm-ui-cf68fd6b6                       1         1         1       16m
replicaset.apps/lw5-query-pipeline-5c44887974             1         1         0       16m
replicaset.apps/lw5-rest-service-6f87f5f488               1         1         0       16m
replicaset.apps/lw5-rpc-service-59f4c7c5cb                1         1         0       16m
replicaset.apps/lw5-rules-ui-7d6cc45486                   1         1         1       16m
replicaset.apps/lw5-solr-exporter-74677cf947              1         1         0       16m
replicaset.apps/lw5-templating-57c96d65fc                 1         1         0       16m
replicaset.apps/lw5-webapps-55b64587f8                    1         1         0       16m
replicaset.apps/lw5-workflow-controller-5b877d7c67        1         1         1       16m
replicaset.apps/seldon-controller-manager-7b855d7f5c      1         1         1       16m

NAME                                        READY   AGE
statefulset.apps/lw5-classic-rest-service   0/1     16m
statefulset.apps/lw5-pulsar-bookkeeper      0/3     16m
statefulset.apps/lw5-pulsar-broker          0/2     16m
statefulset.apps/lw5-solr                   0/1     16m
statefulset.apps/lw5-zookeeper              0/3     16m

NAME                                           SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/lw5-job-launcher                 0 * * * *   False     0        10m             16m
cronjob.batch/lw5-job-launcher-spark-cleanup   0 * * * *   False     0        10m             16m

Wrong POD log

[root@p45460v chenpeng6]# kubectl get pods -n lw5 | grep CrashLoopBackOff |awk '{print "kubectl logs "  $1 " -n lw5"}'|sh -x
+ kubectl logs lw5-api-gateway-56cb494746-w9xwt -n lw5
Error from server (BadRequest): container "api-gateway" in pod "lw5-api-gateway-56cb494746-w9xwt" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-admin-7c66d87d99-dm75c -n lw5
Error from server (BadRequest): container "admin" in pod "lw5-fusion-admin-7c66d87d99-dm75c" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-indexing-fd7f886b-hsjl9 -n lw5
Error from server (BadRequest): container "fusion-indexing" in pod "lw5-fusion-indexing-fd7f886b-hsjl9" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-log-forwarder-655f65c864-g2tmk -n lw5
Error from server (BadRequest): container "fusion-log-forwarder" in pod "lw5-fusion-log-forwarder-655f65c864-g2tmk" is waiting to start: PodInitializing
+ kubectl logs lw5-job-launcher-7bfc6d9878-49mn6 -n lw5
Picked up JAVA_TOOL_OPTIONS: -XX:+ExitOnOutOfMemoryError -Dlogging.config=classpath:logback-kube.xml
Failed to connect to Pulsar topic persistent://lw5/_logs/system_logs at : pulsar://lw5-pulsar-broker:6650 due to: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'lw5-pulsar-broker' after 2 queries ; will re-try after brief wait ...
+ kubectl logs lw5-job-rest-server-575685b498-cggtd -n lw5
Error from server (BadRequest): container "job-rest-server" in pod "lw5-job-rest-server-575685b498-cggtd" is waiting to start: PodInitializing
+ kubectl logs lw5-ml-model-service-858585f586-cr266 -n lw5
error: a container name must be specified for pod lw5-ml-model-service-858585f586-cr266, choose one of: [java-service python-service] or one of the init containers: [check-admin]
+ kubectl logs lw5-query-pipeline-5c44887974-2l5cw -n lw5
Error from server (BadRequest): container "query-pipeline" in pod "lw5-query-pipeline-5c44887974-2l5cw" is waiting to start: PodInitializing
+ kubectl logs lw5-rest-service-6f87f5f488-sf58k -n lw5
Error from server (BadRequest): container "rest-service" in pod "lw5-rest-service-6f87f5f488-sf58k" is waiting to start: PodInitializing
+ kubectl logs lw5-rpc-service-59f4c7c5cb-xx7d4 -n lw5
Error from server (BadRequest): container "rpc-service" in pod "lw5-rpc-service-59f4c7c5cb-xx7d4" is waiting to start: PodInitializing
+ kubectl logs lw5-templating-57c96d65fc-whq8c -n lw5
Error from server (BadRequest): container "templating" in pod "lw5-templating-57c96d65fc-whq8c" is waiting to start: PodInitializing
+ kubectl logs lw5-webapps-55b64587f8-gbspb -n lw5
Error from server (BadRequest): container "webapps" in pod "lw5-webapps-55b64587f8-gbspb" is waiting to start: PodInitializing

Generate the file

file.zip

[shutter-cp]

Wrong POD describe

lw6-api-gateway-b756bbcb5-fskl7

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-api-gateway-b756bbcb5-fskl7 to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "jks" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-api-gateway-token-8jj5t" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff      3m25s (x29 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-fusion-admin-7cc7466458-zzw7n

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-fusion-admin-7cc7466458-zzw7n to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "logback-config" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "solr-autoscaling-config" : couldn't propagate object cache: timed out waiting for the condition
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-fusion-admin-token-jhwbj" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff      3m33s (x28 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-fusion-indexing-644568c789-n7lm4

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-fusion-indexing-644568c789-n7lm4 to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-fusion-indexing-token-m8pq9" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff      3m33s (x29 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-fusion-log-forwarder-68cc895959-dxvmx

Events:
  Type     Reason     Age                   From                                  Message
  ----     ------     ----                  ----                                  -------
  Normal   Scheduled  23m                   default-scheduler                     Successfully assigned lw6/lw6-fusion-log-forwarder-68cc895959-dxvmx to p45461v.hulk.shbt.qihoo.net
  Normal   Pulled     14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Created container check-pulsar
  Normal   Started    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Started container check-pulsar
  Warning  BackOff    3m25s (x29 over 19m)  kubelet, p45461v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-job-rest-server-b64f757c7-dz8kh

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-job-rest-server-b64f757c7-dz8kh to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-job-rest-server-token-cx5wt" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff      3m33s (x28 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-ml-model-service-54bbdd5659-pvcdq

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-ml-model-service-54bbdd5659-pvcdq to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-ml-model-service-token-xf8wd" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-admin
  Normal   Started      13m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-admin
  Warning  BackOff      3m34s (x27 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-pulsar-broker-0

Events:
  Type    Reason     Age   From                                  Message
  ----    ------     ----  ----                                  -------
  Normal  Scheduled  23m   default-scheduler                     Successfully assigned lw6/lw6-pulsar-broker-0 to p45460v.hulk.shbt.qihoo.net
  Normal  Pulled     23m   kubelet, p45460v.hulk.shbt.qihoo.net  Container image "apachepulsar/pulsar-all:2.5.2" already present on machine
  Normal  Created    23m   kubelet, p45460v.hulk.shbt.qihoo.net  Created container pulsar-bookkeeper-verify-clusterid
  Normal  Started    23m   kubelet, p45460v.hulk.shbt.qihoo.net  Started container pulsar-bookkeeper-verify-clusterid

lw6-pulsar-broker-1

Events:
  Type     Reason     Age                   From                                  Message
  ----     ------     ----                  ----                                  -------
  Normal   Scheduled  23m                   default-scheduler                     Successfully assigned lw6/lw6-query-pipeline-6c89476654-d9jhq to p45460v.hulk.shbt.qihoo.net
  Normal   Pulled     14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created    14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started    14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff    3m28s (x29 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-rest-service-58ccf8cbcf-rmhvm

Events:
  Type     Reason     Age                   From                                  Message
  ----     ------     ----                  ----                                  -------
  Normal   Scheduled  23m                   default-scheduler                     Successfully assigned lw6/lw6-rest-service-58ccf8cbcf-rmhvm to p45460v.hulk.shbt.qihoo.net
  Normal   Pulled     14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created    14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started    14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff    3m43s (x29 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-rpc-service-7847674ccb-5vs7v

Events:
  Type     Reason     Age                   From                                  Message
  ----     ------     ----                  ----                                  -------
  Normal   Scheduled  23m                   default-scheduler                     Successfully assigned lw6/lw6-rpc-service-7847674ccb-5vs7v to p45461v.hulk.shbt.qihoo.net
  Normal   Pulled     14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff    3m35s (x28 over 19m)  kubelet, p45461v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-solr-exporter-58d8579679-w88zh

Events:
  Type    Reason     Age   From                                  Message
  ----    ------     ----  ----                                  -------
  Normal  Scheduled  23m   default-scheduler                     Successfully assigned lw6/lw6-solr-exporter-58d8579679-w88zh to p45461v.hulk.shbt.qihoo.net
  Normal  Pulled     23m   kubelet, p45461v.hulk.shbt.qihoo.net  Container image "solr:8.4.1" already present on machine
  Normal  Created    23m   kubelet, p45461v.hulk.shbt.qihoo.net  Created container solr-init
  Normal  Started    23m   kubelet, p45461v.hulk.shbt.qihoo.net  Started container solr-init

lw6-templating-69cf4867d9-kvzcp

Events:
  Type     Reason     Age                   From                                  Message
  ----     ------     ----                  ----                                  -------
  Normal   Scheduled  23m                   default-scheduler                     Successfully assigned lw6/lw6-templating-69cf4867d9-kvzcp to p45461v.hulk.shbt.qihoo.net
  Normal   Pulled     14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started    14m (x5 over 23m)     kubelet, p45461v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff    3m28s (x29 over 19m)  kubelet, p45461v.hulk.shbt.qihoo.net  Back-off restarting failed container

lw6-webapps-849cfcf887-rf6dn

Events:
  Type     Reason       Age                   From                                  Message
  ----     ------       ----                  ----                                  -------
  Normal   Scheduled    23m                   default-scheduler                     Successfully assigned lw6/lw6-webapps-849cfcf887-rf6dn to p45460v.hulk.shbt.qihoo.net
  Warning  FailedMount  23m                   kubelet, p45460v.hulk.shbt.qihoo.net  MountVolume.SetUp failed for volume "lw6-webapps-token-t5rjd" : couldn't propagate object cache: timed out waiting for the condition
  Normal   Pulled       14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
  Normal   Created      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Created container check-zk
  Normal   Started      14m (x5 over 23m)     kubelet, p45460v.hulk.shbt.qihoo.net  Started container check-zk
  Warning  BackOff      3m42s (x28 over 19m)  kubelet, p45460v.hulk.shbt.qihoo.net  Back-off restarting failed container

describe.log

[ian-thebridge-lucidworks]

Hi @shutter-cp, it looks like the zookeeper pod isn't scheduling and is in a Pending state, can you describe the pod to see why it's stuck in this Pending State.