Skip to content

Salt Maintenance Workflow: Update changelog and backport tracking #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master-new-maintaining-workflow-for-salt
Choose a base branch
from
Open

Salt Maintenance Workflow: Update changelog and backport tracking #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1392d9b
use git notes
agraul Apr 8, 2025
5dc8da5
Update section about RPM changelogs
agraul Apr 16, 2025
f080d71
update salt extensions section
agraul May 21, 2025
9fb4c25
WIP: alternative git workflow for Salt packaging
agraul May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 185 additions & 0 deletions accepted/0000-git-salt-packaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
- Feature Name: Git-based Salt Packaging
- Start Date: 2025-05-21

# Summary
[summary]: #summary

Package Salt on src.opensuse.org, with one branch per code stream.

# Motivation
[motivation]: #motivation

We want to change our packaging process for Salt, starting with Salt 3008. This is the
first release we package after moving a lot of Salt modules to Salt Extensions. This
approach should meet the following goals:

1. After merging a pull request at [openSUSE/salt](https://github.com/openSUSE/salt), a new package (RPM) is built automatically
2. Building Salt and Salt Extensions can be controlled from a single location (sources may be located elsewhere)
3. We're aligned with the new openSUSE Tumbleweed / SLE 16 workflow packaging
4. We're aligned with upstream's Salt Extension workflows for maintaining, documenting and publish

Package sources for Tumbleweed and SLE 16 will be tracked in git repositories.
[OBS](https://openbuilservice.org) uses the sources from central git forges located at
https://src.opensuse.org and https://src.suse.de respectively.


## Requirements

We maintain Salt RPMs in different code streams from the same sources. The code streams'
changelogs differ and release timings can be

- Why are we doing this?
- What use cases does it support?
- What is the expected outcome?

Describe the problem you are trying to solve, and its constraints, without coupling them too closely to the solution you have in mind. If this RFC is not accepted, the motivation can be used to develop alternative solutions.

# Detailed design
[design]: #detailed-design

## `openSUSE/Salt` on Github

Build metadata (salt.spec, salt.changes, \_multibuild, …) is moved to a subdirectory in
`openSUSE/salt` on GitHub. This allow us to include packaging updates at the time we
create pull requests, e.g. we can include an appropriate changelog together with the
changes.

Files moved to Salt repository:
- pkg/suse/README.SUSE
- pkg/suse/html.tar.bz2 ???
- pkg/suse/salt-tmpfiles.d
- pkg/suse/transactional_update.conf
- pkg/suse/update-documentation.sh
- pkg/suse/rpmchangelogs
- pkg/suse/_multibuild
- pkg/suse/salt.spec
- pkg/suse/changelogs/factory.changes
- pkg/suse/changelogs/sles15sp2.changes
- pkg/suse/changelogs/sles15sp3.changes
- pkg/suse/changelogs/sles15sp4.changes
- pkg/suse/changelogs/sles15sp5.changes
- pkg/suse/changelogs/sles15sp6.changes
- pkg/suse/changelogs/sles15sp7.changes

### RPM Changelogs

New changelog entries should be part of pull requests. It easy for the code author to
write the user-facing changelog entry while she has all the required context available.

Our changelogs differ between code streams. Most differences are due to different grouping
and entry dates, since we generally keep the package contents in sync.

To help adding new changelog entries in pull requests and update them during rebases, we
add a new Python script `rpmchangelogs`. This script wraps `osc vc` to modify all
`*.changes` files at once. It can `add`, `modify`, and `remove` the latest changelog entry
in all changelogs when the entries are the same.

We use a Github status check to prevent accidental pull requests merges without changelog entries.

## "Package-Git" repository on src.{suse.de,opensuse.org}

Package git: One repository with different branches (see below). The repository contains
openSUSE/salt as a Git submodule.

Branches:
- `factory` (devel for Tumbleweed)
- `products` (why?)
- `testing` (devel for Manager / Uyuni)
- `sles15sp5` (code stream, src.suse.de)
- `sles15sp6` (code stream, src.suse.de)
- `sles15sp7` (code stream, src.suse.de)
- `next` (why?)

### Packaging sources in salt repo

``` text
.gitattributes # created with obs-git-init
.gitignore # created with obs-git-init
.gitmodules # contains Git submodule status
salt # Git submodule
README.SUSE # extracted from `salt` Git submodule
_multibuild # extracted from `salt` Git submodule
html.tar.bz2 # extracted from `salt` Git submodule
salt-tmpfiles.d # extracted from `salt` Git submodule
salt.spec # extracted from `salt` Git submodule
salt.changes # extracted from `salt` Git submodule,for the given branch
transactional_update.conf # extracted from `salt` Git submodule
update-documentation.sh # extracted from `salt` Git submodule
```

## "Project-Git" repository on src.{suse.de,opensuse.org}

We use a single "Project-Git" repository, again with one branch per code stream. Packages
in this project are included as Git submodules, checked out at the corresponding branch.

Per convention, the project git repository is located in the "salt" organisation and
called `_ObsPrj`. An example organisation with the same layout (except that it uses a
singular `master` branch in `_ObsPrj`) is [lua](https://src.opensuse.org/lua)

### Packages

``` text
salt
salt-ext-zypper
salt-ext-transactional_update
salt-ext-rebootmgr
...
```

## Build Service project on build.{suse.de,opensuse.org}

Build Service projects configure build repositories via it's `meta`. The rest (including
`prjconf`) is maintained in the "Project-Git".

## Update End-to-end Workflow

When we merge a PR to a release/ branch in openSUSE/salt, a jenkins job updates the Package-Git repository.

``` text
1. update git submodule
2. extract files (salt.spec, \_multibuild, …)
3. rename <codestream.changes> to salt.changes
4. commit
5. push
```

This is implemented with a Makefile, the Jenkins job just calls `make` with
required variables set. The same Makefile also defines targets to similarly update Salt
Extensions.

[`workflow-direct`](https://src.opensuse.org/adamm/autogits) keeps the Project-Git
up-to-date with changes to the Package-Git repositories.

## Salt Extensions

Salt Extensions are packaged individually. Each salt extension is a typical
Python RPM, built with the standard `python-rpm-macros`, tracked in
"Package-Git" repositories. These package git repositories are included next to
Salt in the "Project-Git".

Packaging sources are different from Salt since we do not control the upstream
repositories. Specfile and changelog are not stored in the extension source repos, instead
we keep them directly in the Package-Git. Since these are new packages, we don't have
diverging changelogs and can use a single branch.

# Drawbacks
[drawbacks]: #drawbacks

Why should we **not** do this?

* obscure corner cases
* will it impact performance?
* what other parts of the product will be affected?
* will the solution be hard to maintain in the future?

# Alternatives
[alternatives]: #alternatives

- What other designs/options have been considered?
- What is the impact of not doing this?

# Unresolved questions
[unresolved]: #unresolved-questions

- What are the unknowns?
- What can happen if Murphy's law holds true?
78 changes: 43 additions & 35 deletions accepted/0000-salt-new-maintaining-workflow.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ pkg/suse/html.tar.bz2
pkg/suse/salt-tmpfiles.d
pkg/suse/transactional_update.conf
pkg/suse/update-documentation.sh
pkg/suse/mkchlog.sh
pkg/suse/rpmchangelogs
pkg/suse/_multibuild
pkg/suse/salt.spec
pkg/suse/changelogs/factory/salt.changes
Expand All @@ -77,41 +77,19 @@ This is the place now where all those files will be maintained.

#### Tracking upstream and downstream patches

This RFC proposes the usage of "obs_scm" service as the mechanism to pull the sources for the Salt package to build. This service will automatically produce a sources tarball according to a configured GitHub branch.
This RFC proposes the usage of "obs_scm" service as the mechanism to pull the sources for the Salt package to build. This service generates a sources tarball based on the content of a configured GitHub repository.

For the regular Salt maintenance, this means it won't be needed anymore to manually produce patch files to add them to the spec file, as the tarball now contains the updated sources (with the exception of EMBARGOED bugs, where patches are still needed as we cannot push any fix to public GitHub repositories).
Except for bugs under embargo, we won't use patches in the spec file. The generated tarball includes the latest changes we merged into our `openSUSE/release/<current release>` branch.
Bugs under embargo use a different workflow with manually created patches since we cannot push fixes for such bugs to our public Github repository.

To avoid losing the useful labeling of "PATCH-FIX_UPSTREAM" and "PATCH-FIX_OPENSUSE" (with a direct link to the origin PR on the spec file for each new patch we introduced into our Salt package), we will keep adding this information to the spec file on every new PR but this time without adding the patch itself, only the comment.
We will therefore not track bug fixes in the spec file anymore. The origin of a bug fix, i.e. whether it is backport from upstream or not, will be stored in a Git note attached to the commit of each bug fix. Git notes can be added after bug fixes are merged without modifying the Git history.

#### Salt RPM changelogs

As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory.
As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory. New changelog are part of pull requests to `openSUSE/salt`.
Our packaging artifacts will contain a `rpmchangelogs` Python script to easily `add`, `modify` and `remove` changelog entries for all changelogs at once.

Our packaging artifacts will contain a `mkchlog.sh`, which is a helper script to generate a changelog entry to all maintained changelog in one shot. Something like this:

```bash
echo "Generating changelog entry for Salt package"
if ! osc vc _temp.changes;
then
exit 1;
fi

echo "Update changelog files"
echo >> _temp.changes

for i in $(ls changelogs/*/salt.changes); do
echo "$(cat _temp.changes $i)" > $i
git add $i
done

rm _temp.changes
```

When creating a PR to `openSUSE/salt` the user must also include the corresponding changelog entry for all maintained changelog files.

Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR.

NOTE: I think it is better to decouple commit messages (focus on developers) from changelog entries (focus on users/customers), so I prefer to not use commit messages from "openSUSE/salt" to autogenerate the changelog entries but rather to manually write a meaningful changelog message to be included in your PR as part of your changes. Similarly to what we do in other Uyuni repositories.
We will use a Github status check to prevent us from merging pull requests without changelogs, like we already do in `uyuni-project/uyuni`

### OBS project structure

Expand Down Expand Up @@ -253,16 +231,46 @@ Feel free to open new PRs against `openSUSE/devel/master` to see this in action.

### Salt Extensions

#### Builtin extensions
The sources for the builtin Salt Extensions will be located together with the main Salt codebase at the `openSUSE/salt` GitHub repository. No new packages or subpackages will be created for these extensions as they will be part of the main `python3*-salt` package.
Salt extensions are collections of Salt modules that are maintained outside of the Salt repository.
They can be maintained by the upstream Salt Core team, by our openSUSE Salt maintainers or anyone
else. By convention, most Salt extensions are maintained in repositories in the
[salt-extensions](https://github.com/salt-extensions) Github organization.

#### Extensions Maintained by openSUSE

- transactional\_update
- zypperpkg
- rebootmgr
- btrfs
- openscap
- libvirt_events
- virt
- snapper
- suse\_ip?


Each of these extensions are maintained in a separate Git repository, created with
[`salt-extension-migrate`](https://github.com/salt-extensions/salt-extension-migrate). This tool
keeps the Git log from the main Salt repository and is the standard tool to move Salt modules to new extensions.

A) We maintain these extensions in the [salt-extensions](https://github.com/salt-extensions) Github
organization. That makes it easy to discover our Salt extensions and makes it easier for anyone to
contribute to our Salt extensions, just like any other existing Salt extension. The standard tooling
helps us with the configuration for publishing the documentation and the PyPI upload.

B) We maintain these extensions in the [openSUSE](https://github.com/openSUSE) Github organization,
to make the ownership and responsibility clear. We manually configure PyPI upload and docs publishing.

#### RPM Packages for Extensions

If a fix is needed for any of the builtin extensions, workflow would be the same as for a code fix in the main Salt package.
Salt Extensions are like any other Python package. `py2pack` can be used to generate 80% of the spec
file for a Salt Extension. That makes it easy to add new Salt Extensions or drop old ones, e.g. when
they are moved back to the Salt core repository.

#### Packaged Salt Extensions

For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository where we will maintain these extensions.
For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository for each of these extensions.

This "openSUSE/salt-extensions" repository will contain:
- a common salt-extension spec file that will generate all RPM packages
- The sources for each Salt Extension we package
- The changelog files for each maintained codestream
Expand Down