-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Independent container updates #1080
base: main
Are you sure you want to change the base?
Changes from all commits
81ee267
3d28ae2
197325b
f60c43f
af55d26
5c9a38d
27647cc
f6562ae
7002ab8
6aff845
db33038
5001328
22d235c
5a4ddb1
1e9e468
379c9f8
d667c28
431f0cb
aac6c63
ccae6c5
5202d79
9889710
668ee71
0724f86
537d23e
5acb302
e078e9b
60674ea
835970b
a540fc5
0f2d81d
b4818ce
35704b8
b37815a
c9c301d
df3efa8
a4fa6aa
ecb3d87
fb89f00
83418f0
3e861cc
b5bfbb5
ab51a71
43cb02b
ec4028b
4621902
cf7a3db
9bf663f
01f7b37
0c063b5
7baddd0
8381b2f
7e28319
d5d3038
33ee158
7f83505
4073a62
30ec1f1
2476ed6
4a4bf7c
bba427d
43f6d89
d93c99f
7e4cd66
22d01a4
49c4cee
356d848
3d579c8
f175739
3ea4917
760948b
c313c6d
53a7028
f00f962
3f6c134
49b54aa
a82ba28
2aeb53a
cff3ac2
c405eb9
f1dac59
7eb54c3
264f1d1
052c352
5bd5157
bbac103
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,168 @@ | ||
name: Release multi-arch container image | ||
|
||
on: | ||
workflow_dispatch: | ||
push: | ||
branches: | ||
- main | ||
- "test/**" | ||
schedule: | ||
- cron: "0 0 * * *" # Run every day at 00:00 UTC. | ||
|
||
env: | ||
REGISTRY: ghcr.io/${{ github.repository_owner }} | ||
REGISTRY_USER: ${{ github.actor }} | ||
REGISTRY_PASSWORD: ${{ github.token }} | ||
IMAGE_NAME: dangerzone/dangerzone | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
platform: | ||
- linux/amd64 | ||
- linux/arm64 | ||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: Get current date | ||
id: date | ||
run: echo "date=$(date +'%Y%m%d')" >> $GITHUB_OUTPUT | ||
|
||
- name: Prepare | ||
run: | | ||
platform=${{ matrix.platform }} | ||
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV | ||
|
||
- name: Docker meta | ||
id: meta | ||
uses: docker/metadata-action@v5 | ||
with: | ||
images: | | ||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} | ||
|
||
- name: Login to GHCR | ||
uses: docker/login-action@v3 | ||
with: | ||
registry: ghcr.io | ||
username: ${{ github.repository_owner }} | ||
password: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up QEMU | ||
uses: docker/setup-qemu-action@v3 | ||
|
||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v3 | ||
|
||
- name: Build and push by digest | ||
id: build | ||
uses: docker/build-push-action@v6 | ||
with: | ||
context: ./dangerzone/ | ||
file: Dockerfile | ||
build-args: | | ||
DEBIAN_ARCHIVE_DATE=${{ steps.date.outputs.date }} | ||
## Remove potentially incorrect Docker provenance. | ||
#provenance: false | ||
platforms: ${{ matrix.platform }} | ||
labels: ${{ steps.meta.outputs.labels }} | ||
outputs: type=image,"name=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}",push-by-digest=true,name-canonical=true,push=true | ||
|
||
- name: Export digest | ||
run: | | ||
mkdir -p ${{ runner.temp }}/digests | ||
digest="${{ steps.build.outputs.digest }}" | ||
touch "${{ runner.temp }}/digests/${digest#sha256:}" | ||
|
||
- name: Upload digest | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: digests-${{ env.PLATFORM_PAIR }} | ||
path: ${{ runner.temp }}/digests/* | ||
if-no-files-found: error | ||
retention-days: 1 | ||
|
||
merge: | ||
runs-on: ubuntu-latest | ||
needs: | ||
- build | ||
outputs: | ||
digest: ${{ steps.image.outputs.digest }} | ||
image: ${{ steps.image.outputs.image }} | ||
steps: | ||
- uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
|
||
- name: Compute image tag | ||
id: tag | ||
run: | | ||
DATE=$(date +'%Y%m%d') | ||
TAG=$(git describe --long --first-parent | tail -c +2) | ||
echo "tag=${DATE}-${TAG}" >> $GITHUB_OUTPUT | ||
|
||
- name: Download digests | ||
uses: actions/download-artifact@v4 | ||
with: | ||
path: ${{ runner.temp }}/digests | ||
pattern: digests-* | ||
merge-multiple: true | ||
|
||
- name: Login to GHCR | ||
uses: docker/login-action@v3 | ||
with: | ||
registry: ghcr.io | ||
username: ${{ github.repository_owner }} | ||
password: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v3 | ||
|
||
#- name: Docker meta | ||
# id: meta | ||
# uses: docker/metadata-action@v5 | ||
# with: | ||
# images: | | ||
# ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} | ||
# tags: | | ||
# type=ref,event=branch | ||
# type=ref,event=pr | ||
# type=semver,pattern={{version}} | ||
# type=semver,pattern={{major}}.{{minor}} | ||
|
||
- name: Create manifest list and push | ||
working-directory: ${{ runner.temp }}/digests | ||
run: | | ||
IMAGE=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.tag.outputs.tag }} | ||
DIGESTS=$(printf '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@sha256:%s ' *) | ||
docker buildx imagetools create -t ${IMAGE} ${DIGESTS} | ||
|
||
- name: Inspect image | ||
id: image | ||
run: | | ||
# NOTE: Set the image as an output because the `env` context is not | ||
# available to the inputs of a reusable workflow call. | ||
image_name="${REGISTRY}/${IMAGE_NAME}" | ||
echo "image=$image_name" >> "$GITHUB_OUTPUT" | ||
docker buildx imagetools inspect ${image_name}:${{ steps.tag.outputs.tag }} | ||
digest=$(docker buildx imagetools inspect ${image_name}:${{ steps.tag.outputs.tag }} --format "{{json .Manifest}}" | jq -r '.digest') | ||
echo "digest=$digest" >> "$GITHUB_OUTPUT" | ||
|
||
# This step calls the container workflow to generate provenance and push it to | ||
# the container registry. | ||
provenance: | ||
needs: | ||
- merge | ||
permissions: | ||
actions: read # for detecting the Github Actions environment. | ||
id-token: write # for creating OIDC tokens for signing. | ||
packages: write # for uploading attestations. | ||
uses: slsa-framework/slsa-github-generator/.github/workflows/[email protected] | ||
with: | ||
digest: ${{ needs.merge.outputs.digest }} | ||
image: ${{ needs.merge.outputs.image }} | ||
registry-username: ${{ github.actor }} | ||
secrets: | ||
registry-password: ${{ secrets.GITHUB_TOKEN }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,23 +3,25 @@ | |
import platform | ||
import shutil | ||
import subprocess | ||
from typing import List, Tuple | ||
from typing import IO, Callable, List, Optional, Tuple | ||
|
||
from . import errors | ||
from .util import get_resource_path, get_subprocess_startupinfo | ||
|
||
CONTAINER_NAME = "dangerzone.rocks/dangerzone" | ||
OLD_CONTAINER_NAME = "dangerzone.rocks/dangerzone" | ||
CONTAINER_NAME = "ghcr.io/almet/dangerzone/dangerzone" # FIXME: Change this to the correct container name | ||
RUNTIME_NAME = "podman" if platform.system() == "Linux" else "docker" | ||
|
||
log = logging.getLogger(__name__) | ||
|
||
|
||
def subprocess_run(*args, **kwargs) -> subprocess.CompletedProcess: | ||
"""subprocess.run with the correct startupinfo for Windows.""" | ||
return subprocess.run(*args, startupinfo=get_subprocess_startupinfo(), **kwargs) | ||
|
||
|
||
def get_runtime_name() -> str: | ||
if platform.system() == "Linux": | ||
runtime_name = "podman" | ||
else: | ||
# Windows, Darwin, and unknown use docker for now, dangerzone-vm eventually | ||
runtime_name = "docker" | ||
return runtime_name | ||
return RUNTIME_NAME | ||
|
||
|
||
def get_runtime_version() -> Tuple[int, int]: | ||
|
@@ -40,9 +42,8 @@ def get_runtime_version() -> Tuple[int, int]: | |
|
||
cmd = [runtime, "version", "-f", query] | ||
try: | ||
version = subprocess.run( | ||
version = subprocess_run( | ||
cmd, | ||
startupinfo=get_subprocess_startupinfo(), | ||
capture_output=True, | ||
check=True, | ||
).stdout.decode() | ||
|
@@ -112,13 +113,7 @@ def delete_image_tag(tag: str) -> None: | |
) | ||
|
||
|
||
def get_expected_tag() -> str: | ||
"""Get the tag of the Dangerzone image tarball from the image-id.txt file.""" | ||
with open(get_resource_path("image-id.txt")) as f: | ||
return f.read().strip() | ||
|
||
|
||
def load_image_tarball() -> None: | ||
def load_image_tarball_from_gzip() -> None: | ||
log.info("Installing Dangerzone container image...") | ||
p = subprocess.Popen( | ||
[get_runtime(), "load"], | ||
|
@@ -147,3 +142,90 @@ def load_image_tarball() -> None: | |
) | ||
|
||
log.info("Successfully installed container image from") | ||
|
||
|
||
def load_image_tarball_from_tar(tarball_path: str) -> None: | ||
cmd = [get_runtime(), "load", "-i", tarball_path] | ||
subprocess_run(cmd, check=True) | ||
log.info("Successfully installed container image from %s", tarball_path) | ||
|
||
|
||
def tag_image_by_digest(digest: str, tag: str) -> None: | ||
"""Tag a container image by digest. | ||
The sha256: prefix should be omitted from the digest. | ||
""" | ||
image_id = get_image_id_by_digest(digest) | ||
cmd = [get_runtime(), "tag", image_id, tag] | ||
log.debug(" ".join(cmd)) | ||
subprocess_run(cmd, check=True) | ||
|
||
|
||
def get_image_id_by_digest(digest: str) -> str: | ||
"""Get an image ID from a digest. | ||
The sha256: prefix should be omitted from the digest. | ||
""" | ||
cmd = [ | ||
get_runtime(), | ||
"images", | ||
"-f", | ||
f"digest=sha256:{digest}", | ||
"--format", | ||
"{{.Id}}", | ||
] | ||
log.debug(" ".join(cmd)) | ||
process = subprocess_run(cmd, check=True, capture_output=True) | ||
# In case we have multiple lines, we only want the first one. | ||
return process.stdout.decode().strip().split("\n")[0] | ||
|
||
|
||
def container_pull(image: str, manifest_digest: str, callback: Callable): | ||
"""Pull a container image from a registry.""" | ||
cmd = [get_runtime_name(), "pull", f"{image}@sha256:{manifest_digest}"] | ||
process = subprocess.Popen( | ||
cmd, | ||
stdout=subprocess.PIPE, | ||
stderr=subprocess.STDOUT, | ||
text=True, | ||
bufsize=1, | ||
) | ||
|
||
for line in process.stdout: # type: ignore | ||
callback(line) | ||
|
||
process.wait() | ||
if process.returncode != 0: | ||
raise errors.ContainerPullException( | ||
f"Could not pull the container image: {process.returncode}" | ||
) | ||
|
||
|
||
def get_local_image_digest(image: str) -> str: | ||
""" | ||
Returns a image hash from a local image name | ||
""" | ||
# Get the image hash from the "podman images" command. | ||
# It's not possible to use "podman inspect" here as it | ||
# returns the digest of the architecture-bound image | ||
cmd = [get_runtime_name(), "images", image, "--format", "{{.Digest}}"] | ||
log.debug(" ".join(cmd)) | ||
try: | ||
result = subprocess_run( | ||
cmd, | ||
capture_output=True, | ||
check=True, | ||
) | ||
lines = result.stdout.decode().strip().split("\n") | ||
if len(lines) != 1: | ||
raise errors.MultipleImagesFoundException( | ||
f"Expected a single line of output, got {len(lines)} lines" | ||
) | ||
image_digest = lines[0].replace("sha256:", "") | ||
if not image_digest: | ||
raise errors.ImageNotPresentException( | ||
f"The image {image} does not exist locally" | ||
) | ||
Comment on lines
+223
to
+226
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If I get what you're talking about, you're handling here the case where the image exists locally, but Podman/Docker somehow don't return a digest for it. That's a weird one indeed, I've seen it while testing things out, but never could figure out why it happens. At any rate, I'd change the message here to show that the image exists locally, but the container engine reports no digest for it. Solely to not get confused with the error message below. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We rely on I agree this isn't ideal, but I don't really see a way around it. Maybe my eyes are too close to the screen: if you see another way to do it let me know :-) |
||
return image_digest | ||
except subprocess.CalledProcessError as e: | ||
raise errors.ImageNotPresentException( | ||
f"The image {image} does not exist locally" | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While working on reproducible builds, I started to realize that our container image becomes marginally smaller if we compress it with gzip. That could be because Buildkit compresses the files in each layer, but I'm not sure yet. Anyway, what I'm getting at is that this code path is a strong candidate for ✂️ .
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neat, let's cut these hairy hairs!