Global refactoring (#617)

* Rename `bench-build` target to `bench` in bake file * Update bake file and break everything * Rename docker-compose.yml to compose.yml to avoid conflicting on `docker buildx bake` * Fix groups in bake file * Update frappe-worker * Update frappe-nginx, erpnext-nginx * Remove old erpnext images * Update frappe-socketio * Fix develop frappe-nginx build on linux/arm64 * Update dockerignore * Update gitignore * Update gitignore * Update .env files * Update installation (overrides) * Update tests * Fix image names * Update compose * Update get-latest-tags * Update CI * Setup and remove .env on tests * Add build bench workflow * Add triggers to main workflow * Add release helm job * Use reusable workflows * Rollback * Print configuration before running tests * Show tests/.env * Revert "Show tests/.env" This reverts commit 4bc3bde. * Fix ci image versions * Remove `frappe-` prefix in build directories * Move requirements-dev.txt * Fix image name in CI * Update gitignore * Update pre-commit config * Drop `version:` in compose files * Add push-backup * Fix postgres CI test * Change .yml to .yaml in compose file to follow compose-spec * Remove prettierignore * Fix dockerignore * Change .yml to .yaml in compose file to follow compose-spec * Don't depend on boto3 while testing (do it in backend) * Update erpnext example version * Don't fail ping on URLError * Move assets volume to main compose file * Fix type annotations for v12 * Fix postgres ci override in tests * Fix spaces in socketio * Reorder stages in nginx image, improve perfomance * Remove unused todo * Optimize worker build * Install Node in worker image * Add 502 error page * Remove unused quiet-pull in tests * Add configurator service to dynamically set common config * Remove unused compose.ci-postgres.yml * Use Python for configurator service: faster and more robust * Add TODO.md * Use python script to get latest tags in CI * Clean up nginx dockerfile * Remove VOLUME declaration https://stackoverflow.com/a/55052682 * Add custom app example * Remove pwd for now * Remove pwd for now * Use jq for parsing config in healthcheck * Take advantage of yaml lang: add defaults in compose file. Also require env vars * Fix CI * Use resusable workflow * Update * Move release_helm job to main.yml * Rename docker-build to docker-build-push * Rename main to build_stable * Rename bench targets * Remove quotes from docker-build-push inputs * Update build develop * Remove HELM_DEPLOY_KEY secret from docker-build-push * Add job names * Remove build_bench workflow * Update version input description in docker-build-push * Print .env in tests, if version is develop, change to latest (for tag) * Fix env setup * Uncomment tests * Parse and set short tags from git tag in bake file * Move devcontainer settings to devcontainer.json * Add db command notice * Fix CI? * Fix inconsistencies in development readme * Remove pwd for now * Remove custom apps for production instruction * Update todos * Add docs for images and compose files * Add variables docs and allow custom frappe site name header * Add notice about internal environment variables * Update site-operations docs * Update todos * Add Overrides header in images-and-compose-files * Update todos * Remove extra docs * Don't log requests in worker image (nginx already does that) * Remove default value of FRAPPE_SITE_NAME_HEADER in example.env * Use file that consistent in v12, v13 and develop to check /assets * Fix paths in CI * Update todos * Remove TODO.md * Update tests/_check_backup_files.py Co-authored-by: Revant Nandgaonkar <[email protected]> * Change variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY to S3_ACCESS_KEY, S3_SECRET_KEY in tests * Fix S3 test * Use `nginxinc/nginx-unprivileged` instead of `nginx` image Also use Ngnix 1.20 instead of unstable 1.21 * Fix https override * Update Dockerfile * Mount assets to backend service in read only mode * Touch .build (#307), use scripts from nginx image to generate config and touch .build * Update example env after building stable images * Touch `.build` on develop image (untill frappe/frappe#15396 is resolved) * Add `make` to worker build deps for linux/arm64 * Fix update example.env job * Fix .build creation on develop branch * Move bench CI to different file This way workflow runs only on PRs that relevant to bench build * Fix app name in custom app example * Update erpnext and frappe versions in example.env * Don't install `svg-sprite` and `sass` node modules in nginx image on linux/arm64 (frappe/frappe#15275) * docs: README and docs * docs: add link to site operations from docker swarm * ci: fix tests as per changes to compose.yaml * docs: move wiki articles to docs * docs: fix add custom domain * docs: fix patch code from images * fix: do not expose port 80 for old images * fix: custom domain labels to frontend container/service * Add missing descriptions to envs in example.env * Fix redis depends_on * Fix docker compose in tests when not running on TTY * Set -T flag in `docker compose exec` only if not tty * Run pre-commit on docs * Remove postgres healthcheck (it gets overriden by mariadb) * Refactor test * Update workflow names * Add pip to dependabot config * docs: backup and push (#19) * Beautify changes by @revant (#20) * feat: add gevent to worker image * feat: real_ip configuration for nginx * Return `healthcheck.sh` just for tests Co-authored-by: Lev Vereshchagin <[email protected]> * Make pretend bench catch unknown commands (closes #666) * Remove debug print in push-backup * Fix typing issues in push-backup * Update file keys in push-backups: from abs path to <site>/<file> * Refactor push-backup * Move gevent installation in Frappe step * Don't pin boto stubs requirement * Cache pip deps on build * Update example env versions * Refactor check backup files * Fix backup test * Fix backup test * Rename build/ dir to images/ * Rename build/ dir to images/ * Fix /build -> /images in docs * Update example.env * Use reusable workflow in frappe user instead of vrslev * Fix compose`s `project` option in docs (#617 (comment)) * Add note about project option in site-operations doc * Update example env * Rename build arg `USERNAME` to `REGISTRY_USER` * Allow https proxy to access Docker socket * Revert "Use reusable workflow in frappe user instead of vrslev" This reverts commit 6062500. * Revert "Revert "Use reusable workflow in frappe user instead of vrslev"" This reverts commit 4680d18. Co-authored-by: Revant Nandgaonkar <[email protected]>
frappe · Mar 14, 2022 · a9b6b75 · a9b6b75
1 parent 95aeb32
commit a9b6b75
Show file tree

Hide file tree

Showing 116 changed files with 2,562 additions and 4,963 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -1,7 +1,4 @@
-# frappe_docker .dockerignore file
-
-.travis.yml
 README.md
-LICENSE.md
+LICENSE
 .gitignore
-docker-*.yml
+compose*.yaml
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -6,36 +6,31 @@ updates:
       interval: daily
 
   - package-ecosystem: docker
-    directory: build/bench
+    directory: images/bench
     schedule:
       interval: daily
 
   - package-ecosystem: docker
-    directory: build/erpnext-nginx
+    directory: images/nginx
     schedule:
       interval: daily
 
   - package-ecosystem: docker
-    directory: build/erpnext-worker
+    directory: images/worker
     schedule:
       interval: daily
 
   - package-ecosystem: docker
-    directory: build/frappe-nginx
+    directory: images/socketio
     schedule:
       interval: daily
 
-  - package-ecosystem: docker
-    directory: build/frappe-socketio
-    schedule:
-      interval: daily
-
-  - package-ecosystem: docker
-    directory: build/frappe-worker
+  - package-ecosystem: npm
+    directory: images/socketio
     schedule:
       interval: daily
 
-  - package-ecosystem: npm
-    directory: build/frappe-socketio
+  - package-ecosystem: pip
+    directory: /
     schedule:
       interval: daily
diff --git a/.github/scripts/get-latest-tags.sh b/.github/scripts/get-latest-tags.sh
diff --git a/.github/scripts/get_latest_tags.py b/.github/scripts/get_latest_tags.py
@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+from typing import Literal
+
+Repo = Literal["frappe", "erpnext"]
+MajorVersion = Literal["12", "13", "develop"]
+
+
+def get_latest_tag(repo: Repo, version: MajorVersion) -> str:
+    if version == "develop":
+        return "develop"
+    regex = rf"v{version}.*"
+    refs = subprocess.check_output(
+        (
+            "git",
+            "ls-remote",
+            "--refs",
+            "--tags",
+            "--sort=v:refname",
+            f"https://github.com/frappe/{repo}",
+            str(regex),
+        ),
+        encoding="UTF-8",
+    ).split()[1::2]
+
+    if not refs:
+        raise RuntimeError(f'No tags found for version "{regex}"')
+    ref = refs[-1]
+    matches: list[str] = re.findall(regex, ref)
+    if not matches:
+        raise RuntimeError(f'Can\'t parse tag from ref "{ref}"')
+    return matches[0]
+
+
+def update_env(file_name: str, frappe_tag: str, erpnext_tag: str | None = None):
+    text = f"\nFRAPPE_VERSION={frappe_tag}"
+    if erpnext_tag:
+        text += f"\nERPNEXT_VERSION={erpnext_tag}"
+
+    with open(file_name, "a") as f:
+        f.write(text)
+
+
+def _print_resp(frappe_tag: str, erpnext_tag: str | None = None):
+    print(json.dumps({"frappe": frappe_tag, "erpnext": erpnext_tag}))
+
+
+def main(_args: list[str]) -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--repo", choices=["frappe", "erpnext"], required=True)
+    parser.add_argument("--version", choices=["12", "13", "develop"], required=True)
+    args = parser.parse_args(_args)
+
+    frappe_tag = get_latest_tag("frappe", args.version)
+    if args.repo == "erpnext":
+        erpnext_tag = get_latest_tag("erpnext", args.version)
+    else:
+        erpnext_tag = None
+
+    file_name = os.getenv("GITHUB_ENV")
+    if file_name:
+        update_env(file_name, frappe_tag, erpnext_tag)
+    _print_resp(frappe_tag, erpnext_tag)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))
diff --git a/.github/scripts/update_example_env.py b/.github/scripts/update_example_env.py
@@ -0,0 +1,32 @@
+import os
+import re
+
+
+def get_versions():
+    frappe_version = os.getenv("FRAPPE_VERSION")
+    erpnext_version = os.getenv("ERPNEXT_VERSION")
+    assert frappe_version, "No Frappe version set"
+    assert erpnext_version, "No ERPNext version set"
+    return frappe_version, erpnext_version
+
+
+def update_env(frappe_version: str, erpnext_version: str):
+    with open("example.env", "r+") as f:
+        content = f.read()
+        for env, var in (
+            ("FRAPPE_VERSION", frappe_version),
+            ("ERPNEXT_VERSION", erpnext_version),
+        ):
+            content = re.sub(rf"{env}=.*", f"{env}={var}", content)
+        f.seek(0)
+        f.truncate()
+        f.write(content)
+
+
+def main() -> int:
+    update_env(*get_versions())
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
diff --git a/.github/workflows/build_bench.yml b/.github/workflows/build_bench.yml
@@ -5,7 +5,7 @@ on:
     branches:
       - main
     paths:
-      - build/bench/**
+      - images/bench/**
       - docker-bake.hcl
 
   schedule:
@@ -27,7 +27,6 @@ jobs:
       - name: Build and test
         uses: docker/[email protected]
         with:
-          files: docker-bake.hcl
           targets: bench-test
 
       - name: Login
@@ -42,5 +41,4 @@ jobs:
         uses: docker/[email protected]
         with:
           targets: bench
-          files: docker-bake.hcl
           push: true
diff --git a/.github/workflows/build_develop.yml b/.github/workflows/build_develop.yml
@@ -1,83 +1,32 @@
-name: Build Develop
+name: Develop build
 
 on:
   pull_request:
     branches:
       - main
     paths:
-      - .github/workflows/build_develop.yml
-      - build/**
-      - installation/**
+      - images/nginx/**
+      - images/socketio/**
+      - images/worker/**
+      - overrides/**
       - tests/**
-      - .dockerignore
+      - compose.yaml
       - docker-bake.hcl
-      - env-example
+      - example.env
 
   schedule:
     # Every day at 12:00 pm
     - cron: 0 0 * * *
 
   workflow_dispatch:
 
-env:
-  IS_AUTHORIZED_RUN: ${{ github.repository == 'frappe/frappe_docker' && github.event_name != 'pull_request' }}
-
 jobs:
   build:
-    name: Frappe & ERPNext
-    runs-on: ubuntu-latest
-    services:
-      registry:
-        image: registry:2
-        ports:
-          - 5000:5000
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-
-      - name: Setup Buildx
-        uses: docker/setup-buildx-action@v1
-        with:
-          driver-opts: network=host
-
-      - name: Login
-        uses: docker/login-action@v1
-        if: env.IS_AUTHORIZED_RUN == 'true'
-        with:
-          username: ${{ secrets.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-
-      - name: Build Frappe
-        uses: docker/[email protected]
-        with:
-          files: docker-bake.hcl
-          targets: frappe-develop-test
-          load: true
-
-      - name: Push Frappe to local registry
-        uses: docker/[email protected]
-        with:
-          files: docker-bake.hcl
-          targets: frappe-develop-test-local
-          push: true
-
-      - name: Test Frappe
-        run: ./tests/test-frappe.sh
-
-      - name: Build ERPNext
-        uses: docker/[email protected]
-        with:
-          files: docker-bake.hcl
-          targets: erpnext-develop-test
-          load: true
-
-      - name: Test ERPNext
-        run: ./tests/test-erpnext.sh
-
-      - name: Push
-        if: env.IS_AUTHORIZED_RUN == 'true'
-        uses: docker/[email protected]
-        with:
-          files: docker-bake.hcl
-          targets: frappe-develop,erpnext-develop
-          push: true
+    uses: frappe/frappe_docker/.github/workflows/docker-build-push.yml@main
+    with:
+      repo: erpnext
+      version: develop
+      push: ${{ github.repository == 'frappe/frappe_docker' && github.event_name != 'pull_request' }}
+    secrets:
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}