Skip to content

Commit

Permalink
Merge remote-tracking branch 'public/main' into rapid_evaluation_fram…
Browse files Browse the repository at this point in the history
…ework_tcre
  • Loading branch information
schlunma committed Nov 13, 2024
2 parents 7d4d214 + 7d8d72c commit 9c44219
Show file tree
Hide file tree
Showing 2,113 changed files with 255,438 additions and 145,013 deletions.
421 changes: 227 additions & 194 deletions .circleci/config.yml

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions .circleci/install_triggers
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
^\.circleci/
^environment\.yml$
^esmvaltool/install/
^pyproject.toml$
^setup\.py$
^setup\.cfg$
^MANIFEST.in$
10 changes: 10 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@

**/__pycache__
.*
doc
tests
ESMValTool.egg-info

!.git
!.zenodo.json

2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
esmvaltool/cmorizers @ESMValGroup/obs-maintainers
.github/workflows @valeriupredoi
76 changes: 76 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<!--
Thank you for contributing to our project!
Please do not delete this text completely, but read the text below and keep
items that seem relevant. If in doubt, just keep everything and add your
own text at the top, a reviewer will update the checklist for you.
While the checklist is intended to be filled in by the technical and scientific
reviewers, it is the responsibility of the author of the pull request to make
sure all items on it are properly implemented.
-->

## Description

<!--
Please describe your changes here, especially focusing on why this pull request makes
ESMValTool better and what problem it solves.
Before you start, please read our contribution guidelines: https://docs.esmvaltool.org/en/latest/community/
Please fill in the GitHub issue that is closed by this pull request, e.g. Closes #1903
-->
- Closes #issue_number
- Link to documentation:

* * *

## Before you get started

<!--
Please discuss your idea with the development team before getting started,
to avoid disappointment or unnecessary work later. The way to do this is
to open a new issue on GitHub.
-->

- [ ] [☝ Create an issue](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#contributing-code-and-documentation) to discuss what you are going to do

## Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the [🛠 Technical][1] or [🧪 Scientific][2] review.

<!-- The next two lines turn the 🛠 and 🧪 below into hyperlinks -->
[1]: https://docs.esmvaltool.org/en/latest/community/review.html#technical-review
[2]: https://docs.esmvaltool.org/en/latest/community/review.html#scientific-review

- [ ] [🛠][1] This pull request has a [descriptive title](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#pull-request-title)
- [ ] [🛠][1] Code is written according to the [code quality guidelines](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#code-quality)
- [ ] [🛠][1] [Documentation](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#documentation) is available
- [ ] [🛠][1] [Tests](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#tests) run successfully
- [ ] [🛠][1] The [list of authors](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#list-of-authors) is up to date
- [ ] [🛠][1] Any changed dependencies have been [added or removed correctly](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#dependencies)
- [ ] [🛠][1] All [checks below this pull request](https://docs.esmvaltool.org/en/latest/community/code_documentation.html#pull-request-checks) were successful

### [New or updated recipe/diagnostic](https://docs.esmvaltool.org/en/latest/community/diagnostic.html)

- [ ] [🧪][2] [Recipe runs successfully](https://docs.esmvaltool.org/en/latest/community/diagnostic.html#testing-recipes)
- [ ] [🧪][2] [Recipe is well documented](https://docs.esmvaltool.org/en/latest/community/diagnostic.html#recipe-and-diagnostic-documentation)
- [ ] [🧪][2] [Figure(s) and data](https://docs.esmvaltool.org/en/latest/community/diagnostic.html#diagnostic-output) look as expected from literature
- [ ] [🛠][1] [Provenance information](https://docs.esmvaltool.org/en/latest/community/diagnostic.html#recording-provenance) has been added

### [New or updated data reformatting script](https://docs.esmvaltool.org/en/latest/develop/dataset.html)

- [ ] [🛠][1] [Documentation](https://docs.esmvaltool.org/en/latest/community/dataset.html#dataset-documentation) is available
- [ ] [🛠][1] The dataset has been [added to the CMOR check recipe](https://docs.esmvaltool.org/en/latest/community/dataset.html#testing)
- [ ] [🛠][1] The dataset has been added to the shared [data pools](https://docs.esmvaltool.org/en/latest/community/dataset.html#cmorized-data) of DKRZ and Jasmin by the @ESMValGroup/OBS-maintainers team
- [ ] [🧪][2] Numbers and units of the data look [physically meaningful](https://docs.esmvaltool.org/en/latest/community/dataset.html#scientific-sanity-check)

***

To help with the number of pull requests:

- 🙏 We kindly ask you to [review](https://docs.esmvaltool.org/en/latest/community/review.html#review-of-pull-requests) two other [open pull requests](https://github.com/ESMValGroup/ESMValTool/pulls) in this repository

<!--
If you need help with any of the items on the checklists above, please do not hesitate to ask by commenting in the issue or pull request.
-->
24 changes: 24 additions & 0 deletions .github/workflows/citation_file_validator.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# workflow that performs a validity check of CITATION.cff file
# authors: Abel S. Siqueira, Faruk Diblen, Jurriaan Spaaks GH: @abelsiqueira, @fdiblen, @jspaaks

name: CFF File Validator

on:
push:
paths:
- CITATION.cff
schedule:
- cron: '0 0 1 * *'

jobs:
validate:
name: "validate"
runs-on: ubuntu-latest
steps:
- name: Check out a copy of the repository
uses: actions/checkout@v4

- name: Check whether the citation metadata from CITATION.cff is valid
uses: citation-file-format/[email protected]
with:
args: "--validate"
101 changes: 101 additions & 0 deletions .github/workflows/create-condalock-file.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: Conda lock file creation

on:
# Trigger on push on main or other branch for testing
# NOTE that push: main will create the file very often
# and hence lots of automated PRs
# push:
# branches:
# - main
schedule:
- cron: '0 4 */10 * *'

# Required shell entrypoint to have properly configured bash shell
defaults:
run:
shell: bash -l {0}

jobs:
create-lock-file:
name: Create conda lock file for latest Python
runs-on: 'ubuntu-latest'
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: conda-incubator/setup-miniconda@v3
with:
auto-update-conda: true
activate-environment: esmvaltool-fromlock
python-version: "3.12"
miniforge-version: "latest"
use-mamba: true
- name: Show conda config
run: |
conda update -n base -c conda-forge conda
conda --version
# setup-miniconda@v3 installs an old conda and mamba
# forcing a modern mamba updates both mamba and conda
# unpin mamba after conda-lock=3 release
# see github.com/ESMValGroup/ESMValTool/issues/3782
conda install -c conda-forge "mamba>=1.4.8,<2"
conda config --show-sources
conda config --show
conda --version
mamba --version
- name: Python info
run: |
which python
python --version
- name: Install conda-lock
run: mamba install -y -c conda-forge conda-lock
- name: Check version of conda-lock
run: conda-lock --version
- name: Create conda lock file for linux-64
run: conda-lock lock --platform linux-64 -f environment.yml --mamba --kind explicit
- name: Show conda version again
run: conda --version
- name: Show Python exec and version again
run: |
which python
python -V
- name: Create conda-lock environment
run: conda create --name esmvaltool-fromlock --file conda-linux-64.lock
- name: Install pip
run: mamba install -y pip
- name: Check Python and pip versions post pip-install
run: |
which python
pip --version
- name: Install ESMValTool
run: pip install -e .[develop]
- name: Check ESMValTool version
run: esmvaltool --help
- name: Check ESMValTool help
run: esmvaltool version
- name: Run flake8
run: flake8
- name: Run tests
run: pytest -n 2 -m "not installation"
# Automated PR
# see https://github.com/marketplace/actions/create-pull-request
- name: Create Automated PR if conda lock file has changed
uses: peter-evans/create-pull-request@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Updating Linux condalock file
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false
branch: condalock-update
delete-branch: true
title: '[Condalock] Update Linux condalock file'
body: |
Update condalock file
Automatic Pull Request.
labels: |
testing
condalock
automatedPR
assignees: valeriupredoi
reviewers: valeriupredoi
draft: false
87 changes: 87 additions & 0 deletions .github/workflows/install-from-conda.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
name: Install from Conda

# runs on a push on main and at the end of every day
on:
push:
branches:
- main
schedule:
- cron: '0 4 * * *'

# Required shell entrypoint to have properly configured bash shell
defaults:
run:
shell: bash -l {0}

jobs:
linux:
runs-on: "ubuntu-latest"
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
name: Linux Python ${{ matrix.python-version }}
steps:
- uses: conda-incubator/setup-miniconda@v3
with:
python-version: ${{ matrix.python-version }}
miniforge-version: "latest"
use-mamba: true
- run: mkdir -p conda_install_linux_artifacts_python_${{ matrix.python-version }}
- name: Record versions
run: |
mamba --version 2>&1 | tee conda_install_linux_artifacts_python_${{ matrix.python-version }}/conda_version.txt
which conda 2>&1 | tee conda_install_linux_artifacts_python_${{ matrix.python-version }}/conda_path.txt
which mamba 2>&1 | tee -a conda_install_linux_artifacts_python_${{ matrix.python-version }}/conda_path.txt
python -V 2>&1 | tee conda_install_linux_artifacts_python_${{ matrix.python-version }}/python_version.txt
- name: Install ESMValTool
run: mamba install esmvaltool 2>&1 | tee conda_install_linux_artifacts_python_${{ matrix.python-version }}/install.txt
- name: Verify installation
run: |
esmvaltool --help
esmvaltool version 2>&1 | tee conda_install_linux_artifacts_python_${{ matrix.python-version }}/version.txt
- name: Upload artifacts
if: ${{ always() }} # upload artifacts even if fail
uses: actions/upload-artifact@v4
with:
name: Conda_Install_Linux_python_${{ matrix.python-version }}
path: conda_install_linux_artifacts_python_${{ matrix.python-version }}

# uncomment from here when we have a testing environment on an OSX machine
# and we know that this should work
#
# osx:
# runs-on: "macos-latest"
# strategy:
# matrix:
# python-version: ["3.10", "3.11"]
# fail-fast: false
# name: OSX Python ${{ matrix.python-version }}
# steps:
# - uses: actions/checkout@v2
# - uses: conda-incubator/setup-miniconda@v3
# with:
# python-version: ${{ matrix.python-version }}
# miniconda-version: "latest"
# channels: conda-forge
# - run: mkdir -p conda_install_osx_artifacts_python_${{ matrix.python-version }}
# - run: conda --version 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/conda_version.txt
# - run: which conda 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/conda_path.txt
# - run: python -V 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/python_version.txt
# # ncurses needs to be from conda-forge and not main channel
# # for now it's turned off since we're not testing R/Julia installs
# # - run: conda uninstall -y ncurses
# # - run: conda list ncurses
# # - run: conda install -y conda-forge::ncurses
# # - run: conda list ncurses
# - run: conda install esmvaltool --no-update-deps 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/install.txt
# - run: conda install esmvaltool-python esmvaltool-ncl 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/install.txt
# - run: esmvaltool --help
# - run: esmvaltool version 2>&1 | tee conda_install_osx_artifacts_python_${{ matrix.python-version }}/version.txt
# - name: Upload artifacts
# if: ${{ always() }} # upload artifacts even if fail
# uses: actions/upload-artifact@v4
# with:
# name: Conda_Install_OSX_python_${{ matrix.python-version }}
# path: conda_install_osx_artifacts_python_${{ matrix.python-version }}
63 changes: 63 additions & 0 deletions .github/workflows/install-from-condalock-file.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Install esmvaltool from a conda lock file
# To build a conda lock file install conda-lock first then
# run conda-lock lock --platform linux-64 -f environment.yml --mamba
# (mamba activated for speed). Change platform for osx-64 or win-64.
# Env creation then happens as per normal use with
# conda create --name esmvaltool-fromlock --file conda-linux-64.lock
# note that pip and conda are NOT installed.

name: Conda-lock Install
on:
push:
branches:
- main
# - condalock-update
# run the test only if the PR is to main
# turn it on if required
#pull_request:
# branches:
# - main
schedule:
- cron: '0 0 * * *'

# Required shell entrypoint to have properly configured bash shell
defaults:
run:
shell: bash -l {0}

jobs:
linux:
runs-on: "ubuntu-latest"
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
fail-fast: false
name: Linux Python ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: conda-incubator/setup-miniconda@v3
with:
activate-environment: esmvaltool-fromlock
python-version: ${{ matrix.python-version }}
miniconda-version: "latest"
channels: conda-forge
- run: mkdir -p source_install_linux_artifacts_python_${{ matrix.python-version }}
- run: conda --version 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/conda_version.txt
- run: which python
- run: python -V 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/python_version.txt
- run: conda create --name esmvaltool-fromlock --file conda-linux-64.lock
- run: which python
- run: pip --version
- run: pip install -e .[develop]
- run: esmvaltool --help
- run: esmvaltool version 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/version.txt
- run: flake8
- run: pytest -n 2 -m "not installation"
- name: Upload artifacts
if: ${{ always() }} # upload artifacts even if fail
uses: actions/upload-artifact@v4
with:
name: Source_Install_Linux_python_${{ matrix.python-version }}
path: source_install_linux_artifacts_python_${{ matrix.python-version }}
Loading

0 comments on commit 9c44219

Please sign in to comment.