Skip to content

Commit 2432a18

Browse files
KrishVora01KrishVora01
authored
KAFKA-16373: KIP-1028: Adding code to support Apache Kafka Docker Official Images (apache#16027)
This PR aims to add JVM based Docker Official Image for Apache Kafka as per the following KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka This PR adds the following functionalities: Introduces support for Apache Kafka Docker Official Images via: GitHub Workflows: - Workflow to prepare static source files for Docker images - Workflow to build and test Docker official images - Scripts to prepare source files and perform Docker image builds and tests A new directory for Docker official images, named docker/docker_official_images. This is the new directory to house all Docker Official Image assets. Co-authored-by: Vedarth Sharma <[email protected]> Reviewers: Manikumar Reddy <[email protected]>, Vedarth Sharma <[email protected]>
1 parent 0143c72 commit 2432a18

8 files changed

+444
-1
lines changed

‎.github/workflows/docker_official_image_build_and_test.yml

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: Docker Official Image Build Test
17+
18+
on:
19+
workflow_dispatch:
20+
inputs:
21+
image_type:
22+
type: choice
23+
description: Docker image type to build and test
24+
options:
25+
- "jvm"
26+
kafka_version:
27+
description: Kafka version for the docker official image. This should be >=3.7.0
28+
required: true
29+
30+
jobs:
31+
build:
32+
runs-on: ubuntu-latest
33+
steps:
34+
- uses: actions/checkout@v3
35+
- name: Set up Python 3.10
36+
uses: actions/setup-python@v3
37+
with:
38+
python-version: "3.10"
39+
- name: Install dependencies
40+
run: |
41+
python -m pip install --upgrade pip
42+
pip install -r docker/requirements.txt
43+
- name: Build image and run tests
44+
working-directory: ./docker
45+
run: |
46+
python docker_official_image_build_test.py kafka/test -tag=test -type=${{ github.event.inputs.image_type }} -v=${{ github.event.inputs.kafka_version }}
47+
- name: Run CVE scan
48+
uses: aquasecurity/trivy-action@master
49+
with:
50+
image-ref: 'kafka/test:test'
51+
format: 'table'
52+
severity: 'CRITICAL,HIGH'
53+
output: scan_report_${{ github.event.inputs.image_type }}.txt
54+
exit-code: '1'
55+
- name: Upload test report
56+
if: always()
57+
uses: actions/upload-artifact@v3
58+
with:
59+
name: report_${{ github.event.inputs.image_type }}.html
60+
path: docker/test/report_${{ github.event.inputs.image_type }}.html
61+
- name: Upload CVE scan report
62+
if: always()
63+
uses: actions/upload-artifact@v3
64+
with:
65+
name: scan_report_${{ github.event.inputs.image_type }}.txt
66+
path: scan_report_${{ github.event.inputs.image_type }}.txt

‎.github/workflows/prepare_docker_official_image_source.yml

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: Docker Prepare Docker Official Image Source
17+
18+
on:
19+
workflow_dispatch:
20+
inputs:
21+
image_type:
22+
type: choice
23+
description: Docker image type to build and test
24+
options:
25+
- "jvm"
26+
kafka_version:
27+
description: Kafka version for the docker official image. This should be >=3.7.0
28+
required: true
29+
30+
jobs:
31+
build:
32+
runs-on: ubuntu-latest
33+
steps:
34+
- uses: actions/checkout@v3
35+
- name: Set up Python 3.10
36+
uses: actions/setup-python@v3
37+
with:
38+
python-version: "3.10"
39+
- name: Install dependencies
40+
run: |
41+
python -m pip install --upgrade pip
42+
pip install -r docker/requirements.txt
43+
- name: Build Docker Official Image Artifact
44+
working-directory: ./docker
45+
run: |
46+
python prepare_docker_official_image_source.py -type=${{ github.event.inputs.image_type }} -v=${{ github.event.inputs.kafka_version }}
47+
- name: Upload Docker Official Image Artifact
48+
if: success()
49+
uses: actions/upload-artifact@v4
50+
with:
51+
name: ${{ github.event.inputs.kafka_version }}
52+
path: docker/docker_official_images/${{ github.event.inputs.kafka_version }}

‎build.gradle

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,8 @@ if (repo != null) {
233233
'**/generated/**',
234234
'clients/src/test/resources/serializedData/*',
235235
'docker/test/fixtures/secrets/*',
236-
'docker/examples/fixtures/secrets/*'
236+
'docker/examples/fixtures/secrets/*',
237+
'docker/docker_official_images/.gitkeep'
237238
])
238239
}
239240
} else {

‎docker/docker_official_image_build_test.py

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
#!/usr/bin/env python
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
"""
19+
Python script to build and test a docker image
20+
This script is used to generate a test report
21+
22+
Usage:
23+
docker_official_image_build_test.py --help
24+
Get detailed description of each option
25+
26+
Example command:-
27+
docker_official_image_build_test.py <image_name> --image-tag <image_tag> --image-type <image_type> --kafka-version <kafka_version>
28+
29+
This command will build an image with <image_name> as image name, <image_tag> as image_tag (it will be latest by default),
30+
<image_type> as image type (jvm by default), <kafka_version> for the kafka version for which the image is being built, and
31+
run tests on the image.
32+
-b can be passed as additional argument if you just want to build the image.
33+
-t can be passed if you just want to run tests on the image.
34+
"""
35+
36+
import argparse
37+
from distutils.dir_util import copy_tree
38+
import shutil
39+
from common import execute
40+
from docker_build_test import run_docker_tests
41+
import tempfile
42+
import os
43+
44+
45+
def build_docker_official_image(image, tag, kafka_version, image_type):
46+
image = f'{image}:{tag}'
47+
current_dir = os.path.dirname(os.path.realpath(__file__))
48+
temp_dir_path = tempfile.mkdtemp()
49+
copy_tree(f"{current_dir}/docker_official_images/{kafka_version}/{image_type}",
50+
f"{temp_dir_path}/{image_type}")
51+
copy_tree(f"{current_dir}/docker_official_images/{kafka_version}/jvm/resources",
52+
f"{temp_dir_path}/{image_type}/resources")
53+
command = f"docker build -f $DOCKER_FILE -t {image} $DOCKER_DIR"
54+
command = command.replace("$DOCKER_FILE", f"{temp_dir_path}/{image_type}/Dockerfile")
55+
command = command.replace("$DOCKER_DIR", f"{temp_dir_path}/{image_type}")
56+
try:
57+
execute(command.split())
58+
except:
59+
raise SystemError("Docker Image Build failed")
60+
finally:
61+
shutil.rmtree(temp_dir_path)
62+
63+
64+
if __name__ == '__main__':
65+
parser = argparse.ArgumentParser()
66+
parser.add_argument(
67+
"image", help="Image name that you want to keep for the Docker image")
68+
parser.add_argument("--image-tag", "-tag", default="latest",
69+
dest="tag", help="Image tag that you want to add to the image")
70+
parser.add_argument("--image-type", "-type", choices=[
71+
"jvm"], default="jvm", dest="image_type", help="Image type you want to build")
72+
parser.add_argument("--kafka-version", "-v", dest="kafka_version",
73+
help="Kafka version for which the source for docker official image is to be built")
74+
parser.add_argument("--build", "-b", action="store_true", dest="build_only",
75+
default=False, help="Only build the image, don't run tests")
76+
parser.add_argument("--test", "-t", action="store_true", dest="test_only",
77+
default=False, help="Only run the tests, don't build the image")
78+
args = parser.parse_args()
79+
kafka_url = f"https://downloads.apache.org/kafka/{args.kafka_version}/kafka_2.13-{args.kafka_version}.tgz"
80+
if args.build_only or not (args.build_only or args.test_only):
81+
if args.kafka_version:
82+
build_docker_official_image(args.image, args.tag, args.kafka_version, args.image_type)
83+
else:
84+
raise ValueError(
85+
"--kafka-version is required argument for jvm docker official image image")
86+
if args.test_only or not (args.build_only or args.test_only):
87+
run_docker_tests(args.image, args.tag, kafka_url, args.image_type)

‎docker/docker_official_images/.gitkeep

Whitespace-only changes.

‎docker/extract_docker_official_image_artifact.py

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env python
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
"""
19+
Python script to extract docker official images artifact and give it executable permissions
20+
This script is used to extract docker official images artifact and give it executable permissions
21+
22+
Usage:
23+
extract_docker_official_image_artifact.py --help
24+
Get detailed description of each option
25+
26+
Example command:-
27+
extract_docker_official_image_artifact.py --path_to_downloaded_artifact <artifact_path>
28+
29+
This command will build an extract the downloaded artifact, and copy the contents to the
30+
docker_official_images directory. If the extracted artifact contents already exist in the
31+
docker_official_images directory , they will be overwritten, else they will be created.
32+
33+
"""
34+
import os
35+
import argparse
36+
import zipfile
37+
import shutil
38+
from pathlib import Path
39+
40+
def set_executable_permissions(directory):
41+
for root, _, files in os.walk(directory):
42+
for file in files:
43+
path = os.path.join(root, file)
44+
os.chmod(path, os.stat(path).st_mode | 0o111)
45+
46+
47+
def extract_artifact(artifact_path):
48+
docker_official_images_dir = Path(os.path.dirname(os.path.realpath(__file__)), "docker_official_images")
49+
temp_dir = Path('temp_extracted')
50+
try:
51+
if temp_dir.exists():
52+
shutil.rmtree(temp_dir)
53+
temp_dir.mkdir()
54+
with zipfile.ZipFile(artifact_path, 'r') as zip_ref:
55+
zip_ref.extractall(temp_dir)
56+
artifact_version_dirs = list(temp_dir.iterdir())
57+
if len(artifact_version_dirs) != 1:
58+
raise Exception("Unexpected contents in the artifact. Exactly one version directory is expected.")
59+
artifact_version_dir = artifact_version_dirs[0]
60+
target_version_dir = Path(os.path.join(docker_official_images_dir, artifact_version_dir.name))
61+
target_version_dir.mkdir(parents=True, exist_ok=True)
62+
for image_type_dir in artifact_version_dir.iterdir():
63+
target_image_type_dir = Path(os.path.join(target_version_dir, image_type_dir.name))
64+
if target_image_type_dir.exists():
65+
shutil.rmtree(target_image_type_dir)
66+
shutil.copytree(image_type_dir, target_image_type_dir)
67+
set_executable_permissions(target_image_type_dir)
68+
finally:
69+
if temp_dir.exists():
70+
shutil.rmtree(temp_dir)
71+
72+
if __name__ == '__main__':
73+
parser = argparse.ArgumentParser()
74+
parser.add_argument("--path_to_downloaded_artifact", "-artifact_path", required=True,
75+
dest="artifact_path", help="Path to zipped artifacy downloaded from github actions workflow.")
76+
args = parser.parse_args()
77+
extract_artifact(args.artifact_path)

‎docker/generate_kafka_pr_template.py

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env python
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
"""
19+
Python script to prepare the PR template for the docker official image
20+
This script is used to prepare the PR template for the docker official image
21+
22+
Usage:
23+
Example command:-
24+
generate_kafka_pr_template.py --help
25+
Get detailed description of each option
26+
27+
generate_kafka_pr_template.py --image-type <image_type>
28+
29+
This command will build a PR template for <image_type> as image type (jvm by default) based docker official image,
30+
on the directories present under docker/docker_official_images.
31+
This PR template will be used to raise a PR in the Docker Official Images Repo.
32+
"""
33+
34+
import os
35+
import subprocess
36+
import sys
37+
import argparse
38+
from pathlib import Path
39+
40+
41+
# Returns the hash of the most recent commit that modified any of the specified files.
42+
def file_commit(*files):
43+
return subprocess.check_output(["git", "log", "-1", "--format=format:%H", "HEAD", "--"] + list(files)).strip().decode('utf-8')
44+
45+
46+
# Returns the latest commit hash for all files in a given directory.
47+
def dir_commit(directory):
48+
docker_required_scripts = [str(path) for path in Path(directory).rglob('*') if path.is_file()]
49+
files_to_check = [os.path.join(directory, "Dockerfile")] + docker_required_scripts
50+
return file_commit(*files_to_check)
51+
52+
53+
# Split the version string into parts and convert them to integers for version comparision
54+
def get_version_parts(version):
55+
return tuple(int(part) for part in version.name.split('.'))
56+
57+
58+
def main():
59+
parser = argparse.ArgumentParser()
60+
parser.add_argument("--image-type", "-type", choices=[
61+
"jvm"], default="jvm", dest="image_type", help="Image type you want to build")
62+
args = parser.parse_args()
63+
self = os.path.basename(__file__)
64+
current_dir = os.path.dirname(os.path.abspath(__file__))
65+
docker_official_images_dir = Path(os.path.join(current_dir, "docker_official_images"))
66+
highest_version = ""
67+
68+
header = f"""
69+
# This file is generated via https://github.com/apache/kafka/blob/{file_commit(os.path.join(current_dir, self))}/docker/generate_kafka_pr_template.py
70+
Maintainers: The Apache Kafka Project <[email protected]> (@ApacheKafka)
71+
GitRepo: https://github.com/apache/kafka.git
72+
"""
73+
print(header)
74+
versions = sorted((d for d in docker_official_images_dir.iterdir() if d.is_dir()), key=get_version_parts, reverse=True)
75+
highest_version = max(versions).name if versions else ""
76+
77+
for dir in versions:
78+
version = dir.name
79+
tags = version + (", latest" if version == highest_version else "")
80+
commit = dir_commit(dir.joinpath(args.image_type))
81+
82+
info = f"""
83+
Tags: {tags}
84+
Architectures: amd64,arm64v8
85+
GitCommit: {commit}
86+
Directory: ./docker/docker_official_images/{version}/{args.image_type}
87+
"""
88+
print(info.strip(), '\n')
89+
90+
91+
if __name__ == "__main__":
92+
main()

0 commit comments

Comments
 (0)