Skip to content

Commit

Permalink
squash merge update-gh-actions into fix-gh-actions
Browse files Browse the repository at this point in the history
* fix mkdocs deployment

* update path to python bin

* add trigger for current branch

* change path to python_bin for mac as well

* try to use setup-python@v4 instead of setting env

* remove setup conda action

* try to use $CONDA

* remove overseen action

* change branch from master to main

* sort out if then else for faster syntax

* remove more if functions

* add updates to create-caches as well

* eliminate the rest of if functions

* try to unpin pytorch and torchvision

* restore pinned versions

* try switching from set-output to use env

* update test-invoke-conda as well

* fix env var creation

* quote variable

* add second equal to compare

* try another way to use outputs

* fix outputs

* pip install for mac before creating conda env

* fix output variable

* fix python bin path

* remove pip install for before creating conda env

* unpin streamlit version in conda mac env

* try to make git-workflows better readable

* remove 4gotten trigger

* Update-gh-actions (invoke-ai#6)

* fix mkdocs deployment

* update path to python bin

* add trigger for current branch

* change path to python_bin for mac as well

* try to use setup-python@v4 instead of setting env

* remove setup conda action

* try to use $CONDA

* remove overseen action

* change branch from master to main

* sort out if then else for faster syntax

* remove more if functions

* add updates to create-caches as well

* eliminate the rest of if functions

* try to unpin pytorch and torchvision

* restore pinned versions

* try switching from set-output to use env

* update test-invoke-conda as well

* fix env var creation

* quote variable

* add second equal to compare

* try another way to use outputs

* fix outputs

* pip install for mac before creating conda env

* fix output variable

* fix python bin path

* remove pip install for before creating conda env

* unpin streamlit version in conda mac env

* try to make git-workflows better readable

* use macos-latest

* try to update conda before creating mac env

* better conda update trial

* re-pin streamlit version

* re-added trigger to run workflow in current branch

* try to find out if conda mac env could be updated

* install cmake, protobuf and rust b4 conda

* add yes to conda update

* lets try anaconda3-2022.05

* try environment.yml for mac as well

* reenable conda mac env, add pip install
also fix gitignore by changing from dream to invoke

* remove
- unecesary virtualenv creation
- conda update

change != macos back to == linux

* remove cmake from brew install since pre-installed

* disable opencv-python pip requirement

* fixed commands to find latest package versions

* update requirements for mac env

* back to the roots - only install conda env
depending on runner_os with or without extra env variables

* check out macOS in azure-pipelines
since becoming kind of tired of the GitHub Runner which is broken as ...

* let's try to setup python and update conda env

* initialize conda before using it

* add trigger in azure-pipelines.yml

* And another go for update first ....

* update azure-pipelines.yml
- add caching
- add checkpoint download
- add paths to trigger
and more

* unquote checkpoint-url

* fix chekpoint-url variable

* mkdir before downloading model

* set pr trigger to main, rename anaconda cache

* unique cacheHitVariables

* try to use macos-latest instead of macos-12

* update test-invoke-conda.yml:
- remove unecesarry echo step
- use s-weigand/setup-conda@v1
- remove conda update from install deps step since updated with action

* update test-invoke-conda.yml:
- rename conda env cache from ldm to invokeai
- reorder steps:
  1. checkout sources
  2. setup python
  3. setup conda
  4. keep order after set platform variables

* change macos back to 12 since also fails with 11

* update condition in run the tests
make difference between main or not main

* fix path to cache invokeai conda env

* fix invokeai conda env cache path

* update mkdocs-flow.yml

* change conda-channel priority

* update create-caches

* update conda env also when cache was used

* os dependend conda env cache path

* use existing CONDA env pointing to conda root

* create CONDA_ROOT output from $CONDA

* use output variable to define test prompts

* use setup-python v4, get rid of PYTHON_BIN env

* add runner.os to result artifacts name

* update test-invoke-conda.yml:
- reuse macos-latest
- disable setup python 3.9
- setup conda with default python version
- create or update conda environment depending on cache success
- remove name parameter from conda update since name is set in env yml

* improve mkdocs-flow.yml

* disable cache-hugginface-torch
since preload_models.py downloads to more than one location

* update mkdocs-flow.yml with new name

* rename mkdocs action to mkdocs-material

* try to ignore error when creating conda env
maybe it would still be usable, lets see ;P

* remove bloat

* update environment-mac.yml
to match dependencies of invoke-ai/InvokeAI's main branch

* disable conda update, tweak prompt condition

* try to set some env vars for macOS to fix conda

* stop ignoring error, use env instead of outputs

* tweak `[[` connditions

* update python and pip dependencie
makes a difference of 1 sec per itteration compared to 3.9!!!
also I see no reason why using a old pip version would be beneficial

* remove unecesarry env for macOS
everything was pre-tested on my MacBook Air 2020 with M1

* update conda env in setup step

* activate conda env after installation

* update test-invoke-conda.yml
- set conda env dependent on matrix.os
- set CONDA_ENV_NAME to prevent breaking action when renaming conda env
- fix conda env activation

* fix activate conda env

* set bash -l as default shell

* use action to activate conda env

* add conda env file to env activation

* try to replace s-weigeand with conda-incubator

* remove azure-pipelines.yml
funniest part is that the macos runner is the same as the one on github!

* include environment-file in matrix
- also disable auto-activate-base and auto-update-conda
- include macos-latest and macos-12 for debugging purpose
- set miniforge-version in matrix

* fix miniforge-variant, set fail-fast to false

* add step to setup miniconda
- make default shell a matrix variable
- remove bloat

* use a mac env yml without pinned versions

* unpin nomkl, pytorch and torchvision
also removed opencv-pyhton

* cache conda pkgs dir instead of conda env

* use python 3.10, exclude macos-12 from cache

* fix expression

* prepare for PR

* fix doubled id

* reuse pinned versions in mac conda env
- updated python pip version
- unpined pytorch and torchvision
- removed opencv-python
- updated versions to most recent (tested locally)

* fix classical copy/paste error

* remove unused env from shell-block comment

* fix hashFiles function to determine restore-keys

* reenable caching `~.cache`, update create-caches

* unpin all versions in mac conda env file
this was the only way I got it working in the action, also works locally
tested on MacBook Air 2020 M1
remove environment-mac-unpinned.yml

* prepare merge by removing this branch from trigger

* include pull_request trigger for main and dev

* remove pull_request trigger
  • Loading branch information
mauwii committed Oct 18, 2022
1 parent a8b794d commit 44ff8f8
Show file tree
Hide file tree
Showing 6 changed files with 226 additions and 160 deletions.
115 changes: 76 additions & 39 deletions .github/workflows/create-caches.yml
Original file line number Diff line number Diff line change
@@ -1,26 +1,43 @@
name: Create Caches
on:
workflow_dispatch

on: workflow_dispatch

jobs:
build:
os_matrix:
strategy:
matrix:
os: [ ubuntu-latest, macos-12 ]
name: Create Caches on ${{ matrix.os }} conda
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
environment-file: environment.yml
default-shell: bash -l {0}
- os: macos-latest
environment-file: environment-mac.yml
default-shell: bash -l {0}
name: Test invoke.py on ${{ matrix.os }} with conda
runs-on: ${{ matrix.os }}
defaults:
run:
shell: ${{ matrix.default-shell }}
steps:
- name: Set platform variables
id: vars
run: |
if [ "$RUNNER_OS" = "macOS" ]; then
echo "::set-output name=ENV_FILE::environment-mac.yml"
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
elif [ "$RUNNER_OS" = "Linux" ]; then
echo "::set-output name=ENV_FILE::environment.yml"
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
fi
- name: Checkout sources
uses: actions/checkout@v3

- name: setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-activate-base: false
auto-update-conda: false
miniconda-version: latest

- name: set environment
run: |
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
Expand All @@ -29,42 +46,62 @@ jobs:
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}

- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
env:
cache-name: cache-sd-v1-4
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}

- name: Download Stable Diffusion v1.4 Model
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
run: |
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
mkdir -p models/ldm/stable-diffusion-v1
fi
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
fi
- name: Use Cached Dependencies
id: cache-conda-env-ldm
[[ -d models/ldm/stable-diffusion-v1 ]] \
|| mkdir -p models/ldm/stable-diffusion-v1
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
- name: Use cached Conda Environment
uses: actions/cache@v3
env:
cache-name: cache-conda-env-ldm
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ~/.conda/envs/ldm
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
- name: Install Dependencies
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
run: |
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}

- name: Use cached Conda Packages
uses: actions/cache@v3
env:
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ${{ env.CONDA_PKGS_DIR }}
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}

- name: Activate Conda Env
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: ${{ env.CONDA_ENV_NAME }}
environment-file: ${{ matrix.environment-file }}

- name: Use Cached Huggingface and Torch models
id: cache-huggingface-torch
id: cache-hugginface-torch
uses: actions/cache@v3
env:
cache-name: cache-huggingface-torch
cache-name: cache-hugginface-torch
with:
path: ~/.cache
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
- name: Download Huggingface and Torch models
if: ${{ steps.cache-huggingface-torch.outputs.cache-hit != 'true' }}
run: |
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
- name: run preload_models.py
run: python scripts/preload_models.py
28 changes: 0 additions & 28 deletions .github/workflows/mkdocs-flow.yml

This file was deleted.

44 changes: 44 additions & 0 deletions .github/workflows/mkdocs-material.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: mkdocs-material
on:
push:
branches:
- 'main'
- 'development'
pull_request:
branches:
- 'main'
- 'development'

jobs:
mkdocs-material:
runs-on: ubuntu-latest
steps:
- name: checkout sources
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: setup python
uses: actions/setup-python@v4
with:
python-version: '3.10'

- name: install requirements
run: |
python -m \
pip install -r requirements-mkdocs.txt
- name: confirm buildability
run: |
python -m \
mkdocs build \
--clean \
--verbose
- name: deploy to gh-pages
if: ${{ github.ref == 'refs/heads/main' }}
run: |
python -m \
mkdocs gh-deploy \
--clean \
--force
115 changes: 65 additions & 50 deletions .github/workflows/test-invoke-conda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,43 @@ on:
branches:
- 'main'
- 'development'

jobs:
os_matrix:
strategy:
matrix:
os: [ ubuntu-latest, macos-12 ]
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
environment-file: environment.yml
default-shell: bash -l {0}
- os: macos-latest
environment-file: environment-mac.yml
default-shell: bash -l {0}
name: Test invoke.py on ${{ matrix.os }} with conda
runs-on: ${{ matrix.os }}
defaults:
run:
shell: ${{ matrix.default-shell }}
steps:
- run: |
echo The PR was merged
- name: Set platform variables
id: vars
run: |
# Note, can't "activate" via github action; specifying the env's python has the same effect
if [ "$RUNNER_OS" = "macOS" ]; then
echo "::set-output name=ENV_FILE::environment-mac.yml"
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
elif [ "$RUNNER_OS" = "Linux" ]; then
echo "::set-output name=ENV_FILE::environment.yml"
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
fi
- name: Checkout sources
uses: actions/checkout@v3

- name: setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-activate-base: false
auto-update-conda: false
miniconda-version: latest

- name: set environment
run: |
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
Expand All @@ -35,31 +49,42 @@ jobs:
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}

- name: Download Stable Diffusion v1.4 Model
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
run: |
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
mkdir -p models/ldm/stable-diffusion-v1
fi
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
fi
- name: Use Cached Dependencies
id: cache-conda-env-ldm
[[ -d models/ldm/stable-diffusion-v1 ]] \
|| mkdir -p models/ldm/stable-diffusion-v1
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
- name: Use cached Conda Environment
uses: actions/cache@v3
env:
cache-name: cache-conda-env-ldm
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ~/.conda/envs/ldm
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
- name: Install Dependencies
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
run: |
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}

- name: Use cached Conda Packages
uses: actions/cache@v3
env:
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ${{ env.CONDA_PKGS_DIR }}
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}

- name: Activate Conda Env
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: ${{ env.CONDA_ENV_NAME }}
environment-file: ${{ matrix.environment-file }}

- name: Use Cached Huggingface and Torch models
id: cache-hugginface-torch
uses: actions/cache@v3
Expand All @@ -70,28 +95,18 @@ jobs:
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
- name: Download Huggingface and Torch models
if: ${{ steps.cache-hugginface-torch.outputs.cache-hit != 'true' }}
run: |
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
# - name: Run tmate
# uses: mxschmitt/action-tmate@v3
# timeout-minutes: 30
- name: run preload_models.py
run: python scripts/preload_models.py

- name: Run the tests
run: |
# Note, can't "activate" via github action; specifying the env's python has the same effect
if [ $(uname) = "Darwin" ]; then
export PYTORCH_ENABLE_MPS_FALLBACK=1
fi
# Utterly hacky, but I don't know how else to do this
if [[ ${{ github.ref }} == 'refs/heads/master' ]]; then
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/preflight_prompts.txt
elif [[ ${{ github.ref }} == 'refs/heads/development' ]]; then
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/dev_prompts.txt
fi
time python scripts/invoke.py \
--from_file ${{ env.TEST_PROMPTS }}
mkdir -p outputs/img-samples
- name: Archive results
uses: actions/upload-artifact@v3
with:
name: results
name: results_${{ matrix.os }}
path: outputs/img-samples
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ src
**/__pycache__/
outputs

# Logs and associated folders
# Logs and associated folders
# created from generated embeddings.
logs
testtube
Expand Down
Loading

0 comments on commit 44ff8f8

Please sign in to comment.