CI: Implement automatic dynamic versioning for pre-releases (#469)

Author: aaronsteers

.github/workflows/pypi_publish.yml

@@ -5,7 +5,7 @@
 # Note: We may want to rename this file at some point. However, if we rename the workflow file name,
 # we have to also update the Trusted Publisher settings on PyPI.
 
-name: Packaging and Publishing
+name: CDK Publish
 
 on:
   push:
@@ -14,7 +14,12 @@ on:
   workflow_dispatch:
     inputs:
       version:
-        description: "Note that this workflow is intended for prereleases. For public-facing stable releases, please use the GitHub Releases workflow instead: https://github.com/airbytehq/airbyte-python-cdk/blob/main/docs/RELEASES.md. If running this workflow from main or from a dev branch, please enter the desired version number here, for instance 1.2.3dev0 or 1.2.3rc1."
+        description: >
+          Note that this workflow is intended for prereleases. For public-facing stable releases,
+          please use the GitHub Releases workflow instead:
+          https://github.com/airbytehq/airbyte-python-cdk/blob/main/docs/RELEASES.md.
+          For prereleases, please leave the version blank to use the detected version. Alternatively,
+          you can override the dynamic versioning for special use cases.
         required: false
       publish_to_pypi:
         description: "Publish to PyPI. If true, the workflow will publish to PyPI."
@@ -37,17 +42,30 @@ jobs:
     name: Build Python Package
     runs-on: ubuntu-24.04
     steps:
-      - name: Detect Release Tag Version
+
+      - name: Checkout CDK Repo
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Detect Prerelease Version using Dunamai
+        uses: mtkennerly/dunamai-action@v1
+        with:
+          args: --style pep440
+          env-var: DETECTED_VERSION
+
+      - name: Detect Release Tag Version from git ref ('${{ github.ref_name }}')
         if: startsWith(github.ref, 'refs/tags/v')
         run: |
+          echo "Overriding Dunamai detected version: '${{ env.DETECTED_VERSION || 'none' }}'"
+          # Extract the version from the git ref
           DETECTED_VERSION=${{ github.ref_name }}
-          echo "Version ref set to '${DETECTED_VERSION}'"
           # Remove the 'v' prefix if it exists
           DETECTED_VERSION="${DETECTED_VERSION#v}"
-          echo "Setting version to '$DETECTED_VERSION'"
+          echo "Setting detected version to '$DETECTED_VERSION'"
           echo "DETECTED_VERSION=${DETECTED_VERSION}" >> $GITHUB_ENV
 
-      - name: Validate and set VERSION from git ref ('${{ github.ref_name }}') and input (${{ github.event.inputs.version || 'none' }})
+      - name: Validate and set VERSION (detected='${{ env.DETECTED_VERSION }}', input='${{ github.event.inputs.version || 'none' }}')
         id: set_version
         run: |
           INPUT_VERSION=${{ github.event.inputs.version }}
@@ -84,10 +102,6 @@ jobs:
             echo "IS_PRERELEASE=true" >> $GITHUB_OUTPUT
           fi
 
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-
       - uses: hynek/build-and-inspect-python-package@v2
         env:
           # Pass in the evaluated version from the previous step

docs/RELEASES.md

@@ -21,11 +21,13 @@ _Note:_
 
 This process is slightly different from the above, since we don't necessarily want public release notes to be published for internal testing releases. The same underlying workflow will be run, but we'll kick it off directly:
 
-1. Navigate to the "Packaging and Publishing" workflow in GitHub Actions.
-2. Type the version number - including a valid pre-release suffix. Examples: `1.2.3dev0`, `1.2.3rc1`, `1.2.3b0`, etc.
-3. Select `main` or your dev branch from the "Use workflow from" dropdown.
-4. Select your options and click "Run workflow".
-5. Monitor the workflow to ensure the process has succeeded.
+1. Navigate to the "Publish CDK" workflow in GitHub Actions.
+2. Select your dev branch (or `main`) from the "Use workflow from" dropdown.
+3. Leave the version number blank, allowing the CI workflow to pick a version number
+   using [Dunamai](https://dunamai.readthedocs.io).
+4. Select from the other options and click "Run workflow".
+5. Monitor the workflow to ensure the process has succeeded. You will see the
+   version number in the GitHub Actions job output and in GitHub Environments view.
 
 ## Understanding and Debugging Builder and SDM Releases
 
@@ -59,15 +61,15 @@ To manually test changes against a dev image of SDM before committing to a relea
 
 Once the publish pipeline has completed, choose a connector to test. Set the base_image in the connector's metadata to your pre-release version in Dockerhub (make sure to update the SHA as well).
 Next, build the pre-release image locally using `airbyte-ci connectors —name=<source> build`.
-You can now run connector interfaces against the built image using the pattern
`docker run airbyte/<source-name>:dev <spec/check/discover/read>`.
+You can now run connector interfaces against the built image using the pattern`docker run airbyte/<source-name>:dev <spec/check/discover/read>`.
 The connector's README should include a list of these commands, which can be copy/pasted and run from the connector's directory for quick testing against a local config.
 You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.
 
 #### Pretesting Low-Code Python connectors
 
 Once the publish pipeline has completed, set the version of `airbyte-cdk` in the connector's pyproject.toml file to the pre-release version in PyPI.
-Update the lockfile and run connector interfaces via poetry:
`poetry run source-<name> spec/check/discover/read`.
-You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.


+Update the lockfile and run connector interfaces via poetry:`poetry run source-<name> spec/check/discover/read`.
+You can also run `airbyte-ci connectors —name=<source> test` to run the CI test suite against the dev image.
 
 #### Pretesting in Cloud
 

pyproject.toml

@@ -27,6 +27,7 @@ version = "0.0.0"  # Version will be calculated dynamically.
 
 [tool.poetry-dynamic-versioning]
 enable = true
+style = "pep440"  # Ensures compatibility with PyPI
 
 [tool.poetry.dependencies]
 python = ">=3.10,<3.13"