A jupyter notebook docker swarm setup that consists of a customised JupyterHub service that spawns individual notebooks for individual users.
The stack is made of a 2 layered docker swarm stack, i.e. any external request is received by the JupyterHub proxy service, which redirects requests to the JupyterHub service where an authenticated user is allowed to start a notebook.
The authentication is defined by Authenticators where Jupyterhub allows for a custom
authenticator to be selected based on the local requirements.
Hence how a user should be authenticated before they are able to launch notebooks via the JupyterHub web interface.
The authenticator itself is selected by defining the authenticator_class variable as shown in
the example/non_ssl/basic_jupyterhub_config.py configuration file.
Beyond authentication, JupyterHub also allows for a custom Spawner
scheme to be overloaded.
The default spawner_class in the example/non_ssl/basic_jupyterhub_config.py configuration file
is defined with the jhub-swarmspawner which enables the deployment of
jupyter notebooks on a Docker Swarm Cluster
cluster whenever a user requests a new notebook.
As a prerequisite the JupyterHub service is able to launch separate notebook services,
JupyterHub needs access to the hosts docker daemon process. This access can
be gained in a number of ways, one of which is to mount the /var/run/docker
.sock file inside the JupyterHub service as a volume and then ensuring that
the user that executes the deploy command is part of the docker system
group. This is the default approach as defined in the docker-compose.yml file.
Another approach would be to expose the docker daemon remotely on port 2376 with TLS verification as explained @ Docker Docs under "Daemon socket option".
In addition it requires that the JupyterHub service is deployed on a swarm manager node. See Create a swarm. Hence the restriction set in the docker-compose file that the JupyterHub service is restricted to a manager node.
Upon launching the example/basic_docker-compose.yml stack also provides an docker-image-updater service.
This service provides a continuously monitor whether new versions of the specified notebook image is available,
and if so pulls it to every swarm node and prunes previous versions when no other running notebook depends on that particular version.
Before the stack can be deployed, the docker-compose.yml, jupyterhub_config.py, and .env files needs to be prepared. This can be achived by using the make init target to generate the nessesary environment for your deployment.
By default, the Makefile will select the non-SSL configuration in the example/non_ssl directory, and the defaults.env environment file will be produced and linked by the .env file.
This non-SSL version is configured to publish the JupyterHub service on port 8080. Furthermore, the example JupyterHub configuration file examples in
example/* uses an insecure default password ``dummy`` that should not be used in a production environment. This can be changed by altering the
following option in the default example/non_ssl/basic_jupyterhub_config.py file before it is copied into the hub/jupyterhub directory before calling the make init target.:
# Authenticator c.JupyterHub.authenticator_class = "jhubauthenticators.DummyAuthenticator" c.DummyAuthenticator.password = "dummy"
Once the selected jupyterhub_config.py file has been properly prepared, the configuration can be prepared by calling:
make init
An alternative to this is to setup the docker-compose.yml and hub/jupyterhub/jupyterhub_config.py files yourself before launching the stack.
Finally, make clean can be called if you want to remove the various configuration and environment files that have been prepared by the init call.
Once the Configuration has been completed, the service can be launched by executing the following command inside the repository directory:
make daemon
To verify that the stack is now deployed and the services are being spawned do:
docker stack ls docker service ls
The stack command should return that the jupyter-service stacks is running with 3 services, i.e. the JupyterHub, the proxy, and the image-updater services.