Generate release notes automatically from pull requests (#3174)

This alleviates two issues:
1. Developers routinely hit conflicts on their pull requests, requiring
merging. The original code had attempted to alleviate this by having
multiple different items that could be replaced by category, but it was
insufficient.
2. In part due to the above, presumably, a lot of release notes in the
past were sparse, by generating it automatically from the PR, this will
be much harder.

I considered just using GitHub's `generate-notes` functionality, but
that is rather limited. This change doesn't really expand on using that
yet, but puts us in a better place to do so.

This also updates the release workflow to use the new script, and
updates the release notes so that they have the new format for every
version since `4.0.559.0`. This also means the release notes now include
the patch releases of `4.0.559.*`. I also did some manual additions to
the release notes.

---------

Co-authored-by: Alec Grieser <[email protected]>

Author: ScottDugas

.github/release.yml

@@ -23,28 +23,25 @@ jobs:
         with:
           ref: ${{ inputs.branch }}
           ssh-key: ${{ secrets.DEPLOY_KEY }}
+          fetch-tags: true
+          # fetch all the history to make sure that we have the last release
+          # I tried fetching part of the history, but I just couldn't get it to work, and fetching all still takes like 5s
+          fetch-depth: 0
       - name: Setup Base Environment
         uses: ./actions/setup-base-env
       - name: Setup FDB
         uses: ./actions/setup-fdb
-
-      # Push a version bump back to main. There are failure scenarios that can result
-      # in published artifacts but an erroneous build, so it's safer to bump the version
-      # at the beginning
       - name: Configure Git
         run: |
           git config --global user.name 'FoundationDB CI'
           git config --global user.email '[email protected]'
-      - name: Increment Version
-        run: python build/versionutils.py gradle.properties -u PATCH --increment --commit
-      - name: Push Version Update
-        run: git push
 
       - name: Build and publish release
         uses: ./actions/release-build-publish
         with:
           gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
           gpg_passphrase: ${{ secrets.GPG_PASSPHRASE }}
+          update_type: PATCH
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           MAVEN_USER: ${{ secrets.MAVEN_USER }}

.github/workflows/patch_release.yml

@@ -16,29 +16,26 @@ jobs:
         uses: actions/[email protected]
         with:
           ssh-key: ${{ secrets.DEPLOY_KEY }}
+          fetch-tags: true
+          # fetch all the history to make sure that we have the last release
+          # I tried fetching part of the history, but I just couldn't get it to work, and fetching all still takes like 5s
+          fetch-depth: 0
       - name: Setup Base Environment
         id: setup-base
         uses: ./actions/setup-base-env
       - name: Setup FDB
         uses: ./actions/setup-fdb
-
-      # Push a version bump back to main. There are failure scenarios that can result
-      # in published artifacts but an erroneous build, so it's safer to bump the version
-      # at the beginning
       - name: Configure git
         run: |
           git config --global user.name 'FoundationDB CI'
           git config --global user.email '[email protected]'
-      - name: Increment version
-        run: python build/versionutils.py gradle.properties --increment --commit
-      - name: Push version increment
-        run: git push
 
       - name: Build and publish
         uses: ./actions/release-build-publish
         with:
           gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
           gpg_passphrase: ${{ secrets.GPG_PASSPHRASE }}
+          update_type: BUILD
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           MAVEN_USER: ${{ secrets.MAVEN_USER }}

.github/workflows/release.yml

@@ -71,7 +71,7 @@ importantly, it keeps the community open and communicative.
 
 ### Opening a Pull Request
 
-When opening a pull request (PR), please ensure that the title of the pull request
+When opening a pull request (PR), please ensure that the description of the pull request
 or the commit message indicates the issue that it is addressing (see 
 [Closing issues with keywords](https://help.github.com/articles/closing-issues-using-keywords/)).
 For example:
@@ -82,12 +82,28 @@ This will automatically create an association between the PR and the issue that
 it is addressing and, upon merging of the PR into the main code line, will 
 automatically mark the issue as resolved.
 
-If your pull request results in a user-visible change to the Record Layer, you should
-also update the [release notes](docs/sphinx/source/ReleaseNotes.md). For most changes, it
-is sufficient to fill in one of the bullets in the "next release" section of that
-document. You should include a short description of the change as well as filling in
-the issue number. The "next release" section is commented out, so the change won't
-be visible in our documentation until the next time a release is cut.
+PRs should have titles that clearly indicate what the change is accomplishing
+as we use these to generate release notes. You can glance at
+[release notes](docs/sphinx/source/ReleaseNotes.md) for inspiration.
+
+They should also have one of the following labels:
+- `breaking change`: For any breaking change
+- `enhancement`: For any new feature or enhancement
+- `bug fix`: For bug fixes
+- `performance`: For performance improvements
+- `dependencies`: For updates to dependency versions
+- `build improvement`: For updates to our build system that shouldn't have user visible impacts
+- `testing improvement`: For new test coverage, or improvements to testing infrastructure
+- `documentation`: For improvements to our documentation
+
+(Note: `build improvement`/`testing improvement`/`documentation` are combined in one
+grouping in the release notes, that is collapsed).
+
+[release-notes-config.json](build/release-notes-config.json) describes how labels are
+converted into categories in the release notes.
+If a PR has multiple labels, it will appear in the first grouping as listed above /
+in [release-notes-config.json](build/release-notes-config.json) (i.e., if it is has
+`enhancement` and `dependencies`, it will only appear under `enhancement`).
 
 ### Reporting issues
 

CONTRIBUTING.md

@@ -7,15 +7,58 @@ inputs:
   gpg_passphrase:
     description: 'GPG passphrase for artifact signing'
     required: true
+  update_type:
+    description: 'One of MAJOR, MINOR, BUILD, PATCH'
+    required: true
 
 runs:
   using: "composite"
   steps:
-    - name: Get version
-      id: get_version
+    - name: Get old version
+      id: get_old_version
+      shell: bash
+      run: |
+        echo "version=$(python build/versionutils.py gradle.properties)" >> "$GITHUB_OUTPUT"
+    # Push a version bump back to main. There are failure scenarios that can result
+    # in published artifacts but an erroneous build, so it's safer to bump the version
+    # at the beginning
+    - name: Increment version
+      shell: bash
+      run: python build/versionutils.py gradle.properties --increment --commit -u ${{ inputs.update_type }}
+    - name: Get new version
+      id: get_new_version
       shell: bash
       run: |
         echo "version=$(python build/versionutils.py gradle.properties)" >> "$GITHUB_OUTPUT"
+    # We also want to push the tag, because that will be used for the next release's release notes
+    - name: Create tag
+      shell: bash
+      run: git tag -m "Release ${{ steps.get_new_version.outputs.version }}" -f "${{ steps.get_new_version.outputs.version }}"
+
+    # We want to do this before anything else, because if the later steps fail, we want to make sure that the full
+    # change log includes all changes, even if they reference a release that was never actually published.
+    - name: Update release notes
+      shell: bash
+      run: |
+        python ./build/create_release_notes.py \
+        --config ./build/release-notes-config.json \
+        --release-notes-md docs/sphinx/source/ReleaseNotes.md \
+        --skip-commit $(git log -n 1 --format=%H HEAD) \
+        --repository ${{ github.repository }} \
+        --commit \
+        ${{ steps.get_old_version.outputs.version }} ${{ steps.get_new_version.outputs.version }}
+    # We move the tag to after the release notes are updated so that later steps (i.e. sphinx) will pick up the udpated
+    # release notes
+    - name: Move tag to HEAD
+      shell: bash
+      run: git tag -m "Release ${{ steps.get_new_version.outputs.version }}" -f "${{ steps.get_new_version.outputs.version }}"
+
+    # push the changes to gradle.properties, the release notes, and the tag as one operation, so if it fails,
+    # it will be as if the release never did anything
+    - name: Push Version Update
+      shell: bash
+      run: git push origin HEAD "${{ steps.get_new_version.outputs.version }}"
+
     - name: Run Gradle Test
       uses: ./actions/gradle-test
       with:
@@ -29,24 +72,16 @@ runs:
         ORG_GRADLE_PROJECT_signingPassword: ${{ inputs.gpg_passphrase }}
 
     # Post release: Update various files which reference version
-    - name: Update release notes
-      shell: bash
-      run: ARTIFACT_VERSION="${{ steps.get_version.outputs.version }}" ./build/update_release_notes.bash
+    # Updating the yaml files has to be done after the tests complete, or it will mark tests as failing that aren't
+    # supported by the previous version.
     - name: Update YAML test file versions
       uses: ./actions/run-gradle
       with:
         gradle_command: updateYamsql -PreleaseBuild=true
     - name: Commit YAML updates
       shell: bash
-      run: python ./build/commit_yamsql_updates.py "${{ steps.get_version.outputs.version }}"
+      run: python ./build/commit_yamsql_updates.py "${{ steps.get_new_version.outputs.version }}"
 
-    # Create and push the tag
-    - name: Create tag
-      shell: bash
-      run: git tag -m "Release ${{ steps.get_version.outputs.version }}" -f "${{ steps.get_version.outputs.version }}"
-    - name: Push tag
-      shell: bash
-      run: git push origin "${{ steps.get_version.outputs.version }}"
     - name: Push Updates
       id: push_updates
       shell: bash
@@ -62,12 +97,12 @@ runs:
       with:
         branch: release-build
         branch-suffix: timestamp
-        title: "Updates for ${{ steps.get_version.outputs.version }} release"
+        title: "Updates for ${{ steps.get_new_version.outputs.version }} release"
         sign-commits: true
         body: |
-          Updates from release for version ${{ steps.get_version.outputs.version }}. Conflicts during the build prevented automatic updating. Please resolve conflicts by checking out the current branch, merging, and then deleting this branch.
+          Updates from release for version ${{ steps.get_new_version.outputs.version }}. Conflicts during the build prevented automatic updating. Please resolve conflicts by checking out the current branch, merging, and then deleting this branch.
 
     # Creating the PR can change the current branch. Explicitly check out the tag here for downstream builds
     - name: Revert to tag
       shell: bash
-      run: git checkout "${{ steps.get_version.outputs.version }}"
+      run: git checkout "${{ steps.get_new_version.outputs.version }}"