Get dev environment set up at CEDA

[agstephens]

Creating a reproducible development environment

We want to create a simple development environment that developers can get up and running with in minutes.

The best option for that is Docker, because:

• regardless of the host system, you can run Docker containers on your Desktop, laptop, etc.
• the container can be configured to pre-install all the dependencies
• you can connect to a Docker Container using various developer tools and IDEs (such as VSCode)
• we can create different Docker files for different installation / test modes
• the Docker recipe can mimic the existing Ansible recipes

Which types of environment do we want to support?

We want a system that allows us to toggle various settings in the container that we are building:

1. Install Slurm or not: include_slurm: True|False
2. Build environment with "environment.yml" or "spec-list.txt": conda_mode: environment_file|spec_file
3. Install "roocs" packages from PyPI or from GitHub in develop mode: roocs_mode: pypi|develop

How can we manage this?

It looks like we can use Docker Compose to provide a set of different configurations, e.g.:

https://docs.docker.com/compose/environment-variables/#using-the---env-file--option

Create multiple files, such as:

$ cat ./docker-configs/env.prod
TAG: 1.6
INCLUDE_SLURM: true
INSTALL_MODE: spec_file

$ cat ./docker-configs/env.dev
TAG: 1.6
INCLUDE_SLURM: false
INSTALL_MODE: environment_file

$ cat ./docker-compose.yml
version: '1'
services:
  web:
    images: "roocs/rook-prod:${TAG}"

Usage is:

$ docker compose --env-file ./docker-configs/env.dev up

The Dockerfile will need some conditional lines like this:

The env variables INSTALL_MODE ...comes in from the outside environment (file)...

RUN [ "${INSTALL_MODE}" == "environment_file" ]  && echo "conda install with env file" || echo "conda install spec file"

[agstephens]

@alaniwi and @cehbrecht: what are your thoughts on the above suggestion?

[agstephens]

Plan to set up ".devcontainer" for rook

• Create branch: dev-container
• Create .devcontainer directory with contents of here to start with: https://github.com/cedadev/esgf-stac-client/tree/dev-container/.devcontainer
• Modify contents of: devcontainer.json, Dockerfile and postCreateCommand.sh
• Merge in the existing top-level Dockerfile ingredients with following changes:
  • WORKDIR: /workspaces/rook (no need to COPY content elsewhere)
  • Install environment from: spec-file.txt instead of environment.yml - (e.g.: conda env create --file spec-file.txt) - this should be quicker than resolving environment.yml
• Instead of line installing WPS, we need:
  • Clone and install rooki and all dependencies in requirements.txt and requirements_[dev,test].txt
  • Clone and install roocs-utils and all dependencies in requirements.txt and requirements_[dev,test].txt
  • Clone and install clisops and all dependencies in requirements.txt and requirements_[dev,test].txt - but comment out roocs dependencies
  • Clone and install daops and all dependencies in requirements.txt and requirements_[dev,test].txt - but comment out roocs dependencies
  • Install rook (we are already in the rook repo at this point) and all dependencies in requirements.txt and requirements_[dev,test].txt - but comment out roocs dependencies