Skip to content

Commit

Permalink
CICD: Improve Workflows.
Browse files Browse the repository at this point in the history
Make multiple improvements to the existing workflows:
- create a shared custom action to deal with version number tracking
  and generation;
- use this action to add the branch name to the version for pull
  request builds;
- create a shared workflow for all debian-ish builds to avoid re-use
  by copy / paste;
- remove potential security risks by eliminating the use of
  pre-evaluated expressions (`${{ ... }}`) inside scripts;
- update outdated GitHub action versions;
- improve the consistency by renaming scripts acording to have a `.sh`
  extension;
- improve naming of generated artefacts for pull requests to include
  the correct version.

@dirkh: Unfortunately this is potentially going to break builds when it is
merged, as there is no good way to 'test' a merge build short of
merging.
We'll just have to deal with the fallout of it in a follow-up pull
request.

Signed-off-by: Michael Keller <[email protected]>
  • Loading branch information
mikeller committed May 12, 2024
1 parent e20ec92 commit 6fc8310
Show file tree
Hide file tree
Showing 32 changed files with 399 additions and 536 deletions.
56 changes: 56 additions & 0 deletions .github/actions/manage-version/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Manage the Subsurface CICD versioning

inputs:
no-increment:
description: Only get the current version, do not increment it even for pushevents
default: false
nightly-builds-secret:
description: The secret to access the nightly builds repository
default: ''

outputs:
version:
description: The long form version number
value: ${{ steps.version_number.outputs.version }}
buildnr:
description: The build number
value: ${{ steps.version_number.outputs.buildnr }}

runs:
using: composite
steps:
- name: atomically create or retrieve the build number and assemble release notes for a push (i.e. merging of a pull request)
if: github.event_name == 'push' && ! inputs.no-increment
env:
NIGHTLY_BUILDS_SECRET: ${{ inputs.nightly-builds-secret }}
shell: bash
run: |
if [ -z "$NIGHTLY_BUILDS_SECRET" ]; then
echo "Need to supply the secret for the nightly-builds repository to increment the version number, aborting."
exit 1
fi
scripts/get-atomic-buildnr.sh $GITHUB_SHA $NIGHTLY_BUILDS_SECRET "CICD-release"
- name: retrieve the current version number in all other cases
if: github.event_name != 'push' || inputs.no-increment
env:
PULL_REQUEST_BRANCH: ${{ github.event.pull_request.head.ref }}
shell: bash
run: |
echo "-pull-request-$PULL_REQUEST_BRANCH" > latest-subsurface-buildnumber-extension
- name: store version number for the build
id: version_number
env:
PULL_REQUEST_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
shell: bash
run: |
git config --global --add safe.directory $GITHUB_WORKSPACE
# For a pull request we need the information from the pull request branch
# and not from the merge branch on the pull request
git checkout $PULL_REQUEST_HEAD_SHA
version=$(scripts/get-version.sh)
echo "version=$version" >> $GITHUB_OUTPUT
buildnr=$(scripts/get-version.sh 1)
echo "buildnr=$buildnr" >> $GITHUB_OUTPUT
git checkout $GITHUB_SHA
8 changes: 4 additions & 4 deletions .github/workflows/android-dockerimage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,17 @@ jobs:
VERSION: ${{ '5.15.2' }} # the version numbers here is based on the Qt version, the third digit is the rev of the docker image

steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v4

- name: Build the name for the docker image
id: build_name
run: |
v=${{ env.VERSION }}
b=${{ github.ref }} # -BRANCH suffix, unless the branch is master
v=$VERSION
b=$GITHUB_REF # -BRANCH suffix, unless the branch is master
b=${b/refs\/heads\//}
b=${b,,} # the name needs to be all lower case
if [ $b = "master" ] ; then b="" ; else b="-$b" ; fi
echo "NAME=subsurface/android-build${b}:${v}" >> $GITHUB_OUTPUT
echo "NAME=$GITHUB_REPOSITORY_OWNER/android-build${b}:${v}" >> $GITHUB_OUTPUT
- name: Build and Publish Linux Docker image to Dockerhub
uses: elgohr/Publish-Docker-Github-Action@v5
Expand Down
59 changes: 31 additions & 28 deletions .github/workflows/android.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Android

on:
push:
paths-ignore:
Expand All @@ -11,45 +12,44 @@ on:
branches:
- master


jobs:
buildAndroid:
runs-on: ubuntu-latest
build:
env:
BUILD_ROOT: ${{ github.workspace }}/..
KEYSTORE_FILE: ${{ github.workspace }}/../subsurface.keystore
runs-on: ubuntu-latest
container:
image: docker://subsurface/android-build:5.15.2

steps:
- name: checkout sources
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

- name: atomically create or retrieve the build number and assemble release notes
- name: set the version information
id: version_number
if: github.event_name == 'push'
run: |
bash scripts/get-atomic-buildnr.sh ${{ github.sha }} ${{ secrets.NIGHTLY_BUILDS }} "CICD-release"
version=$(cat release-version)
echo "version=$version" >> $GITHUB_OUTPUT
- name: store dummy version and build number for non-push build runs
if: github.event_name != 'push'
run: |
echo "100" > latest-subsurface-buildnumber
echo "CICD-pull-request" > latest-subsurface-buildnumber-extension
uses: ./.github/actions/manage-version
with:
nightly-builds-secret: ${{ secrets.NIGHTLY_BUILDS }}

- name: set up the keystore
if: github.event_name == 'push'
env:
ANDROID_KEYSTORE_BASE64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
run: |
echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 -d > $KEYSTORE_FILE
echo "$ANDROID_KEYSTORE_BASE64" | base64 -d > $KEYSTORE_FILE
- name: run build
id: build
env:
ANDROID_KEYSTORE_PASSWORD: pass:${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
KEYSTORE_ALIAS: ${{ secrets.ANDROID_KEYSTORE_ALIAS }}
BUILDNR: ${{ steps.version_number.outputs.buildnr }}
run: |
# this is rather awkward, but it allows us to use the preinstalled
# Android and Qt versions with relative paths
cd $BUILD_ROOT
cd ..
ln -s /android/5.15.* .
ln -s /android/build-tools .
ln -s /android/cmdline-tools .
Expand All @@ -62,12 +62,20 @@ jobs:
git config --global --add safe.directory $GITHUB_WORKSPACE
git config --global --add safe.directory $GITHUB_WORKSPACE/libdivecomputer
# get the build number via curl so this works both for a pull request as well as a push
BUILDNR=$(curl -q https://raw.githubusercontent.com/subsurface/nightly-builds/main/latest-subsurface-buildnumber)
export OUTPUT_DIR="$GITHUB_WORKSPACE"
export KEYSTORE_FILE="$KEYSTORE_FILE"
export KEYSTORE_PASSWORD="pass:${{ secrets.ANDROID_KEYSTORE_PASSWORD }}"
export KEYSTORE_ALIAS="${{ secrets.ANDROID_KEYSTORE_ALIAS }}"
bash -x ./subsurface/packaging/android/qmake-build.sh -buildnr ${BUILDNR}
bash -x ./subsurface/packaging/android/qmake-build.sh -buildnr $BUILDNR
- name: delete the keystore
if: github.event_name == 'push'
run: |
rm $KEYSTORE_FILE
- name: publish pull request artifacts
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v4
with:
name: Subsurface-Android-${{ steps.version_number.outputs.version }}
path: Subsurface-mobile-*.apk

# only publish a 'release' on push events (those include merging a PR)
- name: upload binaries
Expand All @@ -81,8 +89,3 @@ jobs:
fail_on_unmatched_files: true
files: |
Subsurface-mobile-${{ steps.version_number.outputs.version }}.apk
- name: delete the keystore
if: github.event_name == 'push'
run: |
rm $KEYSTORE_FILE
17 changes: 7 additions & 10 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,14 @@ jobs:
matrix:
# Override automatic language detection by changing the below list
# Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
language: ['cpp', 'javascript']
language: ['c-cpp', 'javascript-typescript']

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
# We must fetch at least the immediate parents so that if this is
# a pull request then we can checkout the head.
fetch-depth: 2
fetch-depth: 0
submodules: recursive

- name: get container ready for build
run: |
Expand All @@ -51,7 +50,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
Expand All @@ -60,13 +59,11 @@ jobs:
# queries: ./path/to/local/query, your-org/your-repo/queries@main

- name: Build
env:
SUBSURFACE_REPO_PATH: ${{ github.workspace }}
run: |
cd ..
git config --global --add safe.directory ${SUBSURFACE_REPO_PATH}
git config --global --add safe.directory ${SUBSURFACE_REPO_PATH}/libdivecomputer
git config --global --add safe.directory $GITHUB_WORKSPACE
git config --global --add safe.directory $GITHUB_WORKSPACE/libdivecomputer
bash -e -x subsurface/scripts/build.sh -desktop -build-with-webkit
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
uses: github/codeql-action/analyze@v3
20 changes: 14 additions & 6 deletions .github/workflows/coverity-scan.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Coverity Scan Linux Qt 5.9

on:
schedule:
- cron: '0 18 * * *' # Daily at 18:00 UTC
Expand All @@ -11,7 +12,10 @@ jobs:

steps:
- name: checkout sources
uses: actions/checkout@v1
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

- name: add build dependencies
run: |
Expand All @@ -30,11 +34,15 @@ jobs:
qtquickcontrols2-5-dev libbluetooth-dev libmtp-dev
- name: configure environment
env:
SUBSURFACE_REPO_PATH: ${{ github.workspace }}
run: |
git config --global --add safe.directory ${SUBSURFACE_REPO_PATH}
git config --global --add safe.directory ${SUBSURFACE_REPO_PATH}/libdivecomputer
git config --global --add safe.directory $GITHUB_WORKSPACE
git config --global --add safe.directory $GITHUB_WORKSPACE/libdivecomputer
- name: get the version information
id: version_number
uses: ./.github/actions/manage-version
with:
no-increment: true

- name: run coverity scan
uses: vapier/coverity-scan-action@v1
Expand All @@ -44,5 +52,5 @@ jobs:
email: [email protected]
command: subsurface/scripts/build.sh -desktop -build-with-webkit
working-directory: ${{ github.workspace }}/..
version: $(/scripts/get-version)
version: ${{ steps.version_number.outputs.version }}
description: Automatic scan on github actions
3 changes: 3 additions & 0 deletions .github/workflows/documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ jobs:
- name: Checkout Sources
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

- name: Process the Documentation
id: process_documentation
Expand Down
20 changes: 11 additions & 9 deletions .github/workflows/fedora-copr-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,16 @@ jobs:
setup-build:
name: Submit build to Fedora COPR
# this seems backwards, but we want to run under Fedora, but Github doesn' support that
container: fedora:latest
runs-on: ubuntu-latest
container:
image: fedora:latest

steps:
- name: Check out sources
uses: actions/checkout@v1
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

- name: Setup build dependencies in the Fedora container
run: |
Expand All @@ -28,13 +32,11 @@ jobs:
git config --global --add safe.directory /__w/subsurface/subsurface
git config --global --add safe.directory /__w/subsurface/subsurface/libdivecomputer
- name: atomically create or retrieve the build number
- name: set the version information
id: version_number
if: github.event_name == 'push'
run: |
bash scripts/get-atomic-buildnr.sh ${{ github.sha }} ${{ secrets.NIGHTLY_BUILDS }} "CICD-release"
version=$(cat release-version)
echo "version=$version" >> $GITHUB_OUTPUT
uses: ./.github/actions/manage-version
with:
nightly-builds-secret: ${{ secrets.NIGHTLY_BUILDS }}

- name: Setup API token for copr-cli
env:
Expand All @@ -53,5 +55,5 @@ jobs:
- name: run the copr build script
run: |
cd ..
bash -x subsurface/packaging/copr/make-package.sh ${{ github.ref_name }}
bash -x subsurface/packaging/copr/make-package.sh $GITHUB_REF_NAME
22 changes: 13 additions & 9 deletions .github/workflows/ios.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: iOS

on:
push:
paths-ignore:
Expand All @@ -19,7 +20,10 @@ jobs:
run: sudo xcode-select -s "/Applications/Xcode_11.7.app"

- name: checkout sources
uses: actions/checkout@v1
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive

- name: setup Homebrew
run: brew install autoconf automake libtool pkg-config
Expand All @@ -31,24 +35,24 @@ jobs:
ref: main
path: qt-ios

- name: store dummy version and build number for test build
- name: set the version information
id: version_number
run: |
echo "100" > latest-subsurface-buildnumber
echo "CICD-test-build" > latest-subsurface-buildnumber-extension
version=$(scripts/get-version)
echo "version=$version" >> $GITHUB_OUTPUT
uses: ./.github/actions/manage-version
with:
nightly-builds-secret: ${{ secrets.NIGHTLY_BUILDS }}

- name: build Subsurface-mobile for iOS
env:
VERSION: ${{ steps.version_number.outputs.version }}
run: |
cd ${{ github.workspace }}/..
cd ..
git config --global --add safe.directory $GITHUB_WORKSPACE
git config --global --add safe.directory $GITHUB_WORKSPACE/libdivecomputer
export IOS_QT=$GITHUB_WORKSPACE/qt-ios
echo "build for simulator"
bash -x $GITHUB_WORKSPACE/packaging/ios/build.sh -simulator
# We need this in order to be able to access the file and publish it
mv build-Subsurface-mobile-Qt_5_14_1_for_iOS-Release/Release-iphonesimulator/Subsurface-mobile.app ${{ github.workspace }}/Subsurface-mobile-${{ steps.version_number.outputs.version }}.app
mv build-Subsurface-mobile-Qt_5_14_1_for_iOS-Release/Release-iphonesimulator/Subsurface-mobile.app $GITHUB_WORKSPACE/Subsurface-mobile-$VERSION.app
- name: publish artifacts
uses: actions/upload-artifact@v4
Expand Down
Loading

0 comments on commit 6fc8310

Please sign in to comment.