Skip to content

Commit 4c3daae

Browse files
tedivmtedivm
committed
first version of celery image
0 parents  commit 4c3daae

File tree

13 files changed

+430
-0
lines changed

13 files changed

+430
-0
lines changed

‎.dockerignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Ignore everything
2+
**
3+
4+
!/assets/*

‎.github/workflows/image-build.yml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
2+
name: Celery Image Builder
3+
4+
on:
5+
push:
6+
branches:
7+
- 'main'
8+
paths-ignore:
9+
- '**/README.md'
10+
- 'templates/**'
11+
- ".github/workflows/versionator.yml"
12+
schedule:
13+
- cron: '4 0 * * WED'
14+
15+
jobs:
16+
Uvicorn-Builder:
17+
runs-on: ubuntu-latest
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
python_versions: ["3.7", "3.8", "3.9", "3.10"]
22+
package_versions: ["5.2.3"]
23+
target_base: ["full", "slim", "alpine"]
24+
steps:
25+
- name: Checkout repository
26+
uses: actions/checkout@v2
27+
28+
- name: "Create and push"
29+
uses: multi-py/action-python-image-builder@v1
30+
with:
31+
package: "celery[gevent]"
32+
package_latest_version: "5.2.3"
33+
maintainer: "Robert Hafner <[email protected]>"
34+
python_version: ${{ matrix.python_versions }}
35+
target_base: ${{ matrix.target_base }}
36+
package_version: ${{ matrix.package_versions }}
37+
registry_password: ${{ secrets.GITHUB_TOKEN }}
38+
dockerfile: "${{ github.workspace }}/dockerfile"
39+
docker_build_path: "${{ github.workspace }}/"
40+

‎.github/workflows/versionator.yml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
2+
name: Celery Version Updater
3+
4+
# Every 30 minutes check for a new version of the package.
5+
on:
6+
push:
7+
branches:
8+
- 'main'
9+
# Don't self-trigger
10+
paths-ignore:
11+
- '**/README.md'
12+
- ".github/workflows/image-build.yml"
13+
schedule:
14+
- cron: '13,43 * * * *'
15+
16+
17+
jobs:
18+
Version-Updater:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Checkout repository
22+
uses: actions/checkout@v2
23+
with:
24+
token: ${{ secrets.WORKFLOW_GITHUB_TOKEN }}
25+
26+
- name: "Update Build"
27+
uses: multi-py/action-python-versionator@main
28+
with:
29+
package: "celery"
30+
git_name: "Robert Hafner"
31+
git_email: "[email protected]"
32+
action_path: ${{ github.workspace }}/.github/workflows/image-build.yml

‎LICENSE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2022 Robert Hafner
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# TEMPLATE_PYTHON_PACKAGE
2+
3+
4+
5+
6+
## Python Versions
7+
8+
This project actively supports these Python versions:
9+
10+
* 3.10
11+
* 3.9
12+
* 3.8
13+
* 3.7
14+
* 3.6
15+
16+
17+
## Image Variants
18+
19+
Like the upstream Python containers themselves a variety of image variants are supported.
20+
21+
22+
### Full
23+
24+
The default container type, and if you're not sure what container to use start here. It has a variety of libraries and build tools installed, making it easy to extend.
25+
26+
27+
28+
### Slim
29+
30+
This container is similar to Full but with far less libraries and tools installed by default. If yo're looking for the tiniest possible image with the most stability this is your best bet.
31+
32+
33+
34+
### Alpine
35+
36+
This container is provided for those who wish to use Alpine. Alpine works a bit differently than the other image types, as it uses `musl` instead of `glibc` and many libaries are not well tested under `musl` at this time.
37+
38+
39+
40+
## Architectures
41+
42+
Every tag in this repository supports these architectures:
43+
44+
* linux/amd64
45+
* linux/arm64
46+
* linux/arm/v7
47+
48+
49+
50+
## Tags
51+
* Recommended Image: `ghcr.io/multi-py/multipy-image-template:py3.10-0.1.1`
52+
* Slim Image: `ghcr.io/multi-py/multipy-image-template:py3.10-slim-0.1.1`
53+
54+
Tags are based on the package version, python version, and the upstream container the container is based on.
55+
56+
| TEMPLATE_PYTHON_PACKAGE Version | Python Version | Full Container | Slim Container | Alpine Container |
57+
|-----------------------|----------------|----------------|----------------|------------------|
58+
| latest | 3.10 | py3.10-latest | py3.10-slim-latest | py3.10-alpine-latest |
59+
| latest | 3.9 | py3.9-latest | py3.9-slim-latest | py3.9-alpine-latest |
60+
| latest | 3.8 | py3.8-latest | py3.8-slim-latest | py3.8-alpine-latest |
61+
| latest | 3.7 | py3.7-latest | py3.7-slim-latest | py3.7-alpine-latest |
62+
| latest | 3.6 | py3.6-latest | py3.6-slim-latest | py3.6-alpine-latest |
63+
| 0.1.1 | 3.10 | py3.10-0.1.1 | py3.10-slim-0.1.1 | py3.10-alpine-0.1.1 |
64+
| 0.1.1 | 3.9 | py3.9-0.1.1 | py3.9-slim-0.1.1 | py3.9-alpine-0.1.1 |
65+
| 0.1.1 | 3.8 | py3.8-0.1.1 | py3.8-slim-0.1.1 | py3.8-alpine-0.1.1 |
66+
| 0.1.1 | 3.7 | py3.7-0.1.1 | py3.7-slim-0.1.1 | py3.7-alpine-0.1.1 |
67+
| 0.1.1 | 3.6 | py3.6-0.1.1 | py3.6-slim-0.1.1 | py3.6-alpine-0.1.1 |
68+
| 0.1.0 | 3.10 | py3.10-0.1.0 | py3.10-slim-0.1.0 | py3.10-alpine-0.1.0 |
69+
| 0.1.0 | 3.9 | py3.9-0.1.0 | py3.9-slim-0.1.0 | py3.9-alpine-0.1.0 |
70+
| 0.1.0 | 3.8 | py3.8-0.1.0 | py3.8-slim-0.1.0 | py3.8-alpine-0.1.0 |
71+
| 0.1.0 | 3.7 | py3.7-0.1.0 | py3.7-slim-0.1.0 | py3.7-alpine-0.1.0 |
72+
| 0.1.0 | 3.6 | py3.6-0.1.0 | py3.6-slim-0.1.0 | py3.6-alpine-0.1.0 |
73+
74+
75+
### Older Tags
76+
77+
Older tags are left for historic purposes but do not receive updates. A full list of tags can be found on the package's [registry page](https://github.com/multi-py/multipy-image-template/pkgs/container/multipy-image-template).
78+
79+

‎assets/entrypoint.sh

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
#! /usr/bin/env bash
2+
set -e
3+
4+
#
5+
# The follow block is based on code from tiangolo/gunicorn-uvicorn-docker
6+
# MIT License: https://github.com/tiangolo/gunicorn-uvicorn-docker/blob/master/LICENSE
7+
#
8+
9+
if [ -f /app/app/worker.py ]; then
10+
DEFAULT_MODULE_NAME=app.worker
11+
elif [ -f /app/worker.py ]; then
12+
DEFAULT_MODULE_NAME=worker
13+
fi
14+
MODULE_NAME=${MODULE_NAME:-$DEFAULT_MODULE_NAME}
15+
VARIABLE_NAME=${VARIABLE_NAME:-celery}
16+
export APP_MODULE=${APP_MODULE:-"$MODULE_NAME.$VARIABLE_NAME"}
17+
18+
19+
# If there's a prestart.sh script in the /app directory or other path specified, run it before starting
20+
PRE_START_PATH=${PRE_START_PATH:-/app/prestart.sh}
21+
echo "Checking for script in $PRE_START_PATH"
22+
if [ -f $PRE_START_PATH ] ; then
23+
echo "Running prestart script $PRE_START_PATH"
24+
. "$PRE_START_PATH"
25+
else
26+
echo "There is no prestart script at $PRE_START_PATH"
27+
fi
28+
29+
#
30+
# End of tiangolo/gunicorn-uvicorn-docker block
31+
#
32+
33+
34+
if [[ "$ENABLE_BEAT" == "true" ]]; then
35+
COMMAND="python -m celery -A $APP_MODULE beat -s /var/celery/celerybeat-schedule"
36+
else
37+
POOL=${POOL:-prefork}
38+
if [[ "$POOL" = "gevent" ]] || [[ "$POOL" = "eventlet" ]] ; then
39+
CONCURRENCY=${CONCURRENCY:-100}
40+
else
41+
CONCURRENCY=${CONCURRENCY:-2}
42+
fi
43+
COMMAND="python -m celery -A $APP_MODULE worker \
44+
--pool=$POOL \
45+
--concurrency=$CONCURRENCY \
46+
--prefetch-multiplier=${PREFETCH_MULTIPLIER:-4}"
47+
if [[ ! -z "$QUEUES" ]]; then
48+
COMMAND="$COMMAND --queues=$QUEUES"
49+
fi
50+
fi
51+
52+
COMMAND="$COMMAND --loglevel=${LOG_LEVEL:-INFO}"
53+
54+
echo $COMMAND
55+
$COMMAND

‎assets/worker.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
from celery import Celery
2+
3+
celery = Celery("ExampleApp")
4+
5+
6+
@celery.task
7+
def hello_world():
8+
print("Hello World!")
9+
10+
11+
@celery.on_after_finalize.connect
12+
def setup_periodic_tasks(sender, **kwargs):
13+
print("Enabling Test Task")
14+
sender.add_periodic_task(15.0, hello_world.s(), name="Test Task")

‎docker-compose.yaml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
version: "3"
2+
3+
services:
4+
worker-scheduler:
5+
build:
6+
context: .
7+
args:
8+
package_version: 5.2.3
9+
package: celery[gevent]
10+
environment:
11+
- 'CELERY_BROKER=pyamqp://guest@rabbitmq//'
12+
- "ENABLE_BEAT=true"
13+
depends_on:
14+
- rabbitmq
15+
16+
worker:
17+
build:
18+
context: .
19+
args:
20+
package_version: 5.2.3
21+
package: celery[gevent]
22+
environment:
23+
- 'CELERY_BROKER=pyamqp://guest@rabbitmq//'
24+
depends_on:
25+
- rabbitmq
26+
27+
rabbitmq:
28+
image: rabbitmq

‎dockerfile

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
ARG python_version=3.9
2+
ARG build_target=$python_version
3+
ARG publish_target=$python_version
4+
5+
FROM python:$build_target as Builder
6+
7+
# Add arguments to container scope
8+
ARG build_target
9+
ARG package
10+
ARG package_version
11+
12+
# Only add build tools for alpine image. The ubuntu based images have build tools already.
13+
# Only runs if `apk` is on the system.
14+
RUN if which apk ; then apk add python3-dev libffi-dev libevent-dev build-base ; fi
15+
16+
# Install package and build all dependencies.
17+
RUN pip install $package==$package_version
18+
19+
# Build our actual container now.
20+
FROM python:$publish_target
21+
22+
# Add args to container scope.
23+
ARG publish_target
24+
ARG python_version
25+
ARG package
26+
ARG maintainer=""
27+
ARG TARGETPLATFORM=""
28+
LABEL python=$python_version
29+
LABEL package=$package
30+
LABEL maintainer=$maintainer
31+
LABEL org.opencontainers.image.description="python:$publish_target $package:$package_version $TARGETPLATFORM"
32+
33+
# Used for Celery Beat.
34+
RUN mkdir /var/celery
35+
36+
# Copy all of the python files built in the Builder container into this smaller container.
37+
COPY --from=Builder /usr/local/lib/python$python_version /usr/local/lib/python$python_version
38+
39+
# Entrypoint Script
40+
COPY ./assets/entrypoint.sh /entrypoint.sh
41+
RUN chmod +x /entrypoint.sh
42+
43+
# Example application so container "works" when run directly.
44+
COPY ./assets/worker.py /app/worker.py
45+
WORKDIR /app/
46+
47+
ENV PYTHONPATH=/app
48+
49+
CMD ["/entrypoint.sh"]

‎makefile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
3+
build_test:
4+
docker build --build-arg package_version=5.2.3 --build-arg package=celery\[gevent\] -t multi-py-celery:test .
5+
6+
run_test:
7+
docker compose up --build

0 commit comments

Comments
 (0)