Skip to content

Commit 3f34713

Browse files
authored
ci: Re-factor fluentd CI workflows (fluent#1472)
* Bump Dockerfile to v1.17.1. Signed-off-by: Josh Baird <[email protected]> * Update Dockerfile. Signed-off-by: Josh Baird <[email protected]> * Try bumping bigdecimal. Signed-off-by: Josh Baird <[email protected]> * Re-factor arm64 Dockerfile. Signed-off-by: Josh Baird <[email protected]> * Include mime types. Signed-off-by: Josh Baird <[email protected]> * Do not build detect-exceptions. Signed-off-by: Josh Baird <[email protected]> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <[email protected]> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <[email protected]> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <[email protected]> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <[email protected]> * Rename some things. Signed-off-by: Josh Baird <[email protected]> * Hm. Signed-off-by: Josh Baird <[email protected]> * Add arm64. Signed-off-by: Josh Baird <[email protected]> * Try new workflow. Signed-off-by: Josh Baird <[email protected]> * Try new workflow. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Produce a manifest. Signed-off-by: Josh Baird <[email protected]> * Add support for Docker Hub. Signed-off-by: Josh Baird <[email protected]> * Add support for Docker Hub. Signed-off-by: Josh Baird <[email protected]> * Add support for Docker Hub. Signed-off-by: Josh Baird <[email protected]> * Add support for Docker Hub. Signed-off-by: Josh Baird <[email protected]> * Add support for Docker Hub. Signed-off-by: Josh Baird <[email protected]> * Prep workflow for prod. Signed-off-by: Josh Baird <[email protected]> * Remove old Dockerfiles. Signed-off-by: Josh Baird <[email protected]> * Add notes. Signed-off-by: Josh Baird <[email protected]> * Reorg some things. Signed-off-by: Josh Baird <[email protected]> * Update BUILD docs. Signed-off-by: Josh Baird <[email protected]> * Cleanup. Signed-off-by: Josh Baird <[email protected]> * Use VERSION file. Signed-off-by: Josh Baird <[email protected]> * Use VERSION file. Signed-off-by: Josh Baird <[email protected]> * Oops. Signed-off-by: Josh Baird <[email protected]> * Modify VERSION. Signed-off-by: Josh Baird <[email protected]> * Modify VERSION. Signed-off-by: Josh Baird <[email protected]> * Add ARG. Signed-off-by: Josh Baird <[email protected]> * Add ARG. Signed-off-by: Josh Baird <[email protected]> * Add ARG. Signed-off-by: Josh Baird <[email protected]> * Add ARG. Signed-off-by: Josh Baird <[email protected]> * Add ARG. Signed-off-by: Josh Baird <[email protected]> * Update README. Signed-off-by: Josh Baird <[email protected]> * Update README. Signed-off-by: Josh Baird <[email protected]> * Update README. Signed-off-by: Josh Baird <[email protected]> * Reset. Signed-off-by: Josh Baird <[email protected]> * Use fluent user. Signed-off-by: Josh Baird <[email protected]> * Tets. Signed-off-by: Josh Baird <[email protected]> * Test. Signed-off-by: Josh Baird <[email protected]> * Set amd64 fluentd path. Signed-off-by: Josh Baird <[email protected]> * One more try. Signed-off-by: Josh Baird <[email protected]> * This is it. Signed-off-by: Josh Baird <[email protected]> * Reset. Signed-off-by: Josh Baird <[email protected]> --------- Signed-off-by: Josh Baird <[email protected]>
1 parent 4f9d71b commit 3f34713

File tree

15 files changed

+235
-824
lines changed

15 files changed

+235
-824
lines changed

.github/workflows/build-fd-image.yaml

Lines changed: 0 additions & 415 deletions
This file was deleted.
Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
name: Publish Fluentd image
2+
3+
on:
4+
workflow_dispatch:
5+
6+
env:
7+
GHCR_REPO: 'ghcr.io/${{ github.repository }}/fluentd'
8+
DOCKERHUB_REPO: 'kubesphere/fluentd'
9+
10+
permissions:
11+
contents: read
12+
packages: write
13+
14+
jobs:
15+
get-version:
16+
name: Fetch fluentd version
17+
runs-on: ubuntu-latest
18+
outputs:
19+
VERSION: ${{ steps.get-version.outputs.VERSION }}
20+
steps:
21+
- name: Checkout code
22+
uses: actions/checkout@v4
23+
with:
24+
ref: ${{ github.ref }}
25+
26+
- name: Read fluentd version from VERSION file
27+
id: get-version
28+
run: |
29+
VERSION=$(cat cmd/fluent-watcher/fluentd/VERSION)
30+
echo "VERSION=${VERSION}" >> $GITHUB_OUTPUT
31+
32+
determine-tags:
33+
needs: [get-version]
34+
runs-on: ubuntu-latest
35+
name: Determine image tags
36+
outputs:
37+
IMAGE_BASE_TAG: ${{ steps.determine-tags.outputs.IMAGE_BASE_TAG }}
38+
IMAGE_MAJOR_MINOR: ${{ steps.determine-tags.outputs.IMAGE_MAJOR_MINOR }}
39+
40+
steps:
41+
- name: Determine image version tag
42+
id: determine-tags
43+
run: |
44+
VERSION=${{ needs.get-version.outputs.VERSION }}
45+
VERSION_WITHOUT_V=${VERSION#v}
46+
MAJOR_MINOR=$(echo $VERSION_WITHOUT_V | cut -d. -f1-2)
47+
48+
if skopeo inspect docker://${{ env.GHCR_REPO }}:${VERSION}; then
49+
echo "${VERSION} tag already exists, assuming we're building a patch release!"
50+
LATEST_PATCH_VERSION=$(skopeo list-tags docker://${{ env.GHCR_REPO }} | grep -E "${VERSION}-[0-9]+" | sort | uniq | tail -1 | tr -d \" | cut -d'-' -f2 | tr -d ',')
51+
NEW_PATCH_VERSION=$((LATEST_PATCH_VERSION + 1))
52+
IMAGE_BASE_TAG="${VERSION}-${NEW_PATCH_VERSION}"
53+
echo "Building patch release ${IMAGE_BASE_TAG}!"
54+
else
55+
echo "${VERSION} tag does not exist, assuming we're building a new release!"
56+
IMAGE_BASE_TAG="${VERSION}"
57+
fi
58+
59+
echo "IMAGE_BASE_TAG=$IMAGE_BASE_TAG" >> $GITHUB_OUTPUT
60+
echo "IMAGE_MAJOR_MINOR=$MAJOR_MINOR" >> $GITHUB_OUTPUT
61+
62+
build:
63+
name: Build/push image (${{ matrix.platform }})
64+
runs-on: ${{ matrix.runs-on }}
65+
needs: [get-version,determine-tags]
66+
permissions:
67+
actions: read
68+
packages: write
69+
outputs:
70+
digest_amd64: ${{ steps.output-digests.outputs.amd64 }}
71+
digest_arm64: ${{ steps.output-digests.outputs.arm64 }}
72+
tags: ${{ steps.image-metadata.outputs.tags }}
73+
strategy:
74+
fail-fast: false
75+
matrix:
76+
platform:
77+
- linux/amd64
78+
- linux/arm64
79+
include:
80+
- runs-on: ubuntu-latest
81+
- runs-on: ubuntu-24.04-arm # Builds arm64 on arm64 hosts
82+
platform: linux/arm64
83+
steps:
84+
- uses: actions/checkout@v4
85+
86+
- name: Set up Docker Buildx
87+
uses: docker/setup-buildx-action@v3
88+
with:
89+
platforms: linux/amd64,linux/arm64
90+
91+
- name: Login to GHCR
92+
uses: docker/login-action@v3
93+
with:
94+
registry: ghcr.io
95+
username: ${{ github.actor }}
96+
password: ${{ secrets.GITHUB_TOKEN }}
97+
98+
- name: Login to Docker Hub
99+
uses: docker/login-action@v3
100+
with:
101+
registry: docker.io
102+
username: ${{ secrets.REGISTRY_USER }}
103+
password: ${{ secrets.REGISTRY_PASSWORD }}
104+
105+
- name: Configure image tags
106+
id: image-metadata
107+
uses: docker/metadata-action@v5
108+
with:
109+
images: |
110+
${{ env.GHCR_REPO }}
111+
${{ env.DOCKERHUB_REPO }}
112+
flavor:
113+
latest=false
114+
tags: |
115+
type=raw,value=latest
116+
type=raw,value=${{ needs.determine-tags.outputs.IMAGE_BASE_TAG }}
117+
type=raw,value=v${{ needs.determine-tags.outputs.IMAGE_BASE_TAG }}
118+
type=raw,value=${{ needs.determine-tags.outputs.IMAGE_MAJOR_MINOR }}
119+
type=raw,value=v${{ needs.determine-tags.outputs.IMAGE_MAJOR_MINOR }}
120+
121+
- name: Build and push image
122+
id: build
123+
uses: docker/build-push-action@v6
124+
with:
125+
context: .
126+
file: cmd/fluent-watcher/fluentd/Dockerfile
127+
platforms: ${{ matrix.platform }}
128+
labels: ${{ steps.image-metadata.outputs.labels }}
129+
provenance: false
130+
build-args:
131+
FLUENTD_BASE_VERSION=${{ needs.get-version.outputs.VERSION }}
132+
outputs: type=image,"name=${{ env.GHCR_REPO }},${{ env.DOCKERHUB_REPO }}",push-by-digest=true,name-canonical=true,push=true
133+
134+
- name: Output image digests
135+
id: output-digests
136+
run: |
137+
platform="${{ matrix.platform }}"
138+
# Convert "linux/amd64" to just amd64 for the output variable name
139+
arch=${platform#linux/}
140+
echo "${arch}=${{ steps.build.outputs.digest }}" >> $GITHUB_OUTPUT
141+
142+
manifest:
143+
name: Publish image manifest
144+
runs-on: ubuntu-latest
145+
needs: [build, determine-tags]
146+
steps:
147+
- name: Login to GHCR
148+
uses: docker/login-action@v3
149+
with:
150+
registry: ghcr.io
151+
username: ${{ github.actor }}
152+
password: ${{ secrets.GITHUB_TOKEN }}
153+
154+
- name: Login to Docker Hub
155+
uses: docker/login-action@v3
156+
with:
157+
registry: docker.io
158+
username: ${{ secrets.REGISTRY_USER }}
159+
password: ${{ secrets.REGISTRY_PASSWORD }}
160+
161+
- name: Create image manifest
162+
uses: int128/docker-manifest-create-action@v2
163+
with:
164+
push: true
165+
tags: ${{ needs.build.outputs.tags }} # Includes GHCR and Docker Hub
166+
sources: |
167+
${{ env.GHCR_REPO }}@${{ needs.build.outputs.digest_amd64 }}
168+
${{ env.GHCR_REPO }}@${{ needs.build.outputs.digest_arm64 }}

RELEASE.md

Lines changed: 12 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -128,33 +128,22 @@ This repo includes a "development" chart in the [charts/](./charts/fluent-operat
128128
- Bump `version` and `appVersion` in the [charts/fluet-operator/Chart.yaml](./charts/fluent-operator/Chart.yaml) file in this repo
129129
- Manually "sync" (copy, open a PR) the local [chart](./charts/fluent-operator) to [fluent/helm-charts](https://github.com/fluent/helm-charts/tree/main/charts/fluent-operator/)
130130

131-
# Automations
131+
# Fluentd/Fluent-bit Images
132132

133-
## Bump fluent-bit version
133+
Fluent Operator uses [custom builds](./cmd/fluent-watcher/README.md) of both Fluentd and Fluent-bit. These images can (and often should be) be published out-of-band of Fluent Operator releases.
134134

135-
To bump the fluent-bit version, you can run the [pipeline](https://github.com/fluent/fluent-operator/actions/workflows/bump-fluent-bit-version.yaml), by simply providing a x.y.z version number. You can check the [fluent-bit releases](https://github.com/fluent/fluent-bit/releases) to get the version number.
135+
## Fluent-bit
136136

137-
Or you can run the following command. Note that we need to remove the leading 'v'.
137+
To publish a new fluent-bit image:
138138

139-
```bash
140-
curl --silent "https://api.github.com/repos/fluent/fluent-bit/releases/latest" |
141-
grep '"tag_name":' |
142-
sed -E 's/.*"([^"]+)".*/\1/'
143-
```
144-
145-
The pipeline will use sed to replace the outdated fluent-bit version with the wanted one. A list of files can be found below:
146-
147-
- cmd/fluent-watcher/fluentbit/VERSION
148-
- config/samples/fluentbit_v1alpha2_fluentbit.yaml
149-
- docs/best-practice/forwarding-logs-via-http/deploy/fluentbit-fluentBit.yaml
150-
- manifests/kubeedge/fluentbit-fluentbit-edge.yaml
151-
- manifests/logging-stack/fluentbit-fluentBit.yaml
152-
- manifests/quick-start/fluentbit.yaml
153-
- manifests/regex-parser/fluentbit-fluentBit.yaml
154-
- charts/fluent-operator/values.yaml
139+
* Execute the [bump-fluent-bit-version](https://github.com/fluent/fluent-operator/actions/workflows/bump-fluent-bit-version.yaml) workflow dispatch to generate a PR to update fluent-bit version references in this repo
140+
* Merge the PR generated by the [bump-fluent-bit-version](https://github.com/fluent/fluent-operator/actions/workflows/bump-fluent-bit-version.yaml) workflow
141+
* Execute the [build-fb-image](https://github.com/fluent/fluent-operator/actions/workflows/build-fb-image.yaml) workflow dispatch to build and publish the new image
155142

156-
After the pipeline finishes, we just need to review and merge the pull requests.
143+
## Fluentd
157144

158-
## Build the fluent-watcher image for fluent-bit
145+
To publish a new fluentd image:
159146

160-
This [pipeline](https://github.com/fluent/fluent-operator/actions/workflows/build-fb-image.yaml) is used to build the image. It will read the version number from `cmd/fluent-watcher/fluentbit/VERSION`, which can be update automatically via the [pipeline](#bump-fluent-bit-version).
147+
* Open a PR to update all references of the image/tag in this repo with the new image tag (TODO: automate this like the fluent-bit release process) including the `cmd/fluent-watcher/fluentd/VERSION` file
148+
* Merge the PR
149+
* Execute the [publish-fluentd-image](https://github.com/fluent/fluent-operator/actions/workflows/publish-fluentd-image.yaml) workflow dispatch to build and publish the new image

cmd/fluent-watcher/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,6 @@ We strive to never overwrite existing image tags (eg, `ghcr.io/fluent/fluent-ope
1414

1515
## Building
1616

17-
As a maintainer, to build the `ghcr.io/fluent/fluent-operator/fluent-bit` and `ghcr.io/fluent/fluent-operator/fluentd` images, you can run the "Build Fluent Bit image" or "Build Fluentd image" Github Action workflows in the "Actions" tab of this repository.
17+
As a maintainer, to build the `ghcr.io/fluent/fluent-operator/fluent-bit` and `ghcr.io/fluent/fluent-operator/fluentd` images, you can run the "Build Fluent Bit image" or "Publish Fluentd image" Github Action workflows in the "Actions" tab of this repository.
1818

1919
Always specify the upstream Fluent Bit version (eg, `3.1.2`, `3.1.3`, etc) when running this workflow. If the CI workflow detects that an image tag already exists for the version specified, it will assume that a patch release needs to be built and will automatically add a patch version to the image tag (eg, `3.1.2-1`, `3.1.2-2`, etc).

cmd/fluent-watcher/fluentd/Dockerfile

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
ARG FLUENTD_BASE_VERSION
2+
FROM golang:1.24.0 as builder
3+
4+
RUN mkdir -p /fluentd
5+
RUN mkdir -p /code
6+
COPY . /code/
7+
WORKDIR /code
8+
RUN echo $(ls -al /code)
9+
RUN CGO_ENABLED=0 go build -ldflags '-w -s' -o /fluentd/fluentd-watcher /code/cmd/fluent-watcher/fluentd/main.go
10+
11+
# See https://github.com/fluent/fluentd-docker-image/issues/425
12+
FROM fluent/fluentd:v${FLUENTD_BASE_VERSION}-debian-1.0
13+
14+
LABEL org.opencontainers.image.description "A Fluentd image for use with fluent-operator"
15+
16+
USER root
17+
18+
RUN buildDeps="make gcc g++ libc-dev" \
19+
&& apt-get update \
20+
&& apt-get install -y --no-install-recommends $buildDeps \
21+
&& gem install \
22+
fluent-plugin-s3 \
23+
fluent-plugin-grok-parser \
24+
fluent-plugin-rewrite-tag-filter \
25+
fluent-plugin-oss \
26+
fluent-plugin-dedot_filter \
27+
fluent-plugin-sumologic_output \
28+
fluent-plugin-kafka \
29+
fluent-plugin-label-router \
30+
fluent-plugin-record-modifier \
31+
fluent-plugin-multi-format-parser \
32+
fluent-plugin-aws-elasticsearch-service \
33+
fluent-plugin-opensearch \
34+
fluent-plugin-grafana-loki \
35+
fluent-plugin-cloudwatch-logs \
36+
fluent-plugin-datadog \
37+
fluent-plugin-prometheus \
38+
&& gem sources --clear-all \
39+
&& apt-get purge -y --auto-remove \
40+
-o APT::AutoRemove::RecommendsImportant=false \
41+
$buildDeps \
42+
&& rm -rf /var/lib/apt/lists/* \
43+
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
44+
45+
COPY ./cmd/fluent-watcher/fluentd/etc/fluent.conf /fluentd/etc/
46+
COPY ./cmd/fluent-watcher/fluentd/etc/app.conf /fluentd/etc/
47+
48+
COPY --from=builder /fluentd/fluentd-watcher /fluentd/bin/fluentd-watcher
49+
50+
USER fluent
51+
52+
ENTRYPOINT ["/fluentd/bin/fluentd-watcher"]

cmd/fluent-watcher/fluentd/Dockerfile.amd64

Lines changed: 0 additions & 85 deletions
This file was deleted.

0 commit comments

Comments
 (0)