Spawners crash when deploying after cluster reboot #196
Description
We set up this workshop via RHPDS for a customer, then put the cluster to sleep so when can start it for the meeting the next day. Upon restarting the cluster, the Homeroom was available but upon clicking the individual language-specific lab links, these crashed. We manually re-created the project to fix it.
Having restarted it a second time so that the customer can continue to work on it, most labs worked and just the Java one didn't. We didn't look further into it the first time it happened and the details below are specific to the Java lab crashing upon cluster restart but we assume it's the same problem.
This is the Pod that crashes:
lab-getting-started-java-spawner-2-6jfk4
Logs:
+ . /opt/app-root/src/jupyterhub_config.sh
++ CONFIGURATION_TYPE=terminal-server
++ '[' -f /opt/app-root/src/configs/terminal-server.sh ']'
++ . /opt/app-root/src/configs/terminal-server.sh
+++ KUBERNETES_SERVER_URL=https://172.30.0.1:443
+++ OAUTH_METADATA_URL=https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ curl -ks https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ jq -r .issuer
+++ OAUTH_ISSUER_ADDRESS=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ export OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
++ '[' -f /opt/app-root/configs/jupyterhub_config.sh ']'
++ . /opt/app-root/configs/jupyterhub_config.sh
+ trap 'kill -TERM $PID' TERM INT
+ PID=32
+ wait 32
+ jupyterhub -f /opt/app-root/src/jupyterhub_config.py
INFO: Workshop name is 'lab-getting-started-java'.
INFO: Application name is 'lab-getting-started-java'.
INFO: Configuration type is 'terminal-server'.
INFO: Homeroom link is ''.
INFO: Homeroom name is 'homeroom'.
INFO: Service account name is 'lab-getting-started-java-spawner'.
INFO: Namespace is 'lab'.
INFO: Full service account name is 'system:serviceaccount:lab:lab-getting-started-java-spawner'.
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.30.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
INFO: Homeroom link set to http://homeroom-lab.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com.
[I 2021-05-12 10:49:19.642 JupyterHub app:2120] Using Authenticator: oauthenticator.openshift.OpenShiftOAuthenticator-0.9.0
[I 2021-05-12 10:49:19.643 JupyterHub app:2120] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2021-05-12 10:49:19.644 JupyterHub app:1257] Loading cookie_secret from /opt/app-root/data/cookie_secret
[I 2021-05-12 10:49:19.660 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-05-12 10:49:19.662 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-05-12 10:49:19.662 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-12 10:49:19.662 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2021-05-12 10:49:19.729 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace lab
[I 2021-05-12 10:49:19.788 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace lab
[W 2021-05-12 10:49:22.852 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:25.924 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:28.996 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:32.068 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:35.140 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:38.212 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:41.284 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
++ kill -TERM 32
+ trap - TERM INT
+ wait 32
/opt/app-root/etc/run.sh: line 18: 32 Terminated jupyterhub -f /opt/app-root/src/jupyterhub_config.py
+ STATUS=143
+ exit 143