Skip to content

Commit 0386de0

Browse files
abhijatabhijat
authored
tools: Add scripts and supporting files to generate os packge repo (#5923)
* tools: Add scripts and supporting files to generate os packge repo * tests: Add automated tests to install packages for fedora,ubuntu Signed-off-by: Abhijat Malviya <[email protected]>
1 parent f28888e commit 0386de0

File tree

14 files changed

+427
-4
lines changed

14 files changed

+427
-4
lines changed

‎.github/workflows/generate-osrepo-site.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: generate-site
2+
on:
3+
workflow_dispatch:
4+
5+
jobs:
6+
gen-site:
7+
runs-on: ubuntu-latest
8+
env:
9+
SiteRoot: _site
10+
11+
name: Generate index and site assets
12+
steps:
13+
- name: Checkout Repository
14+
uses: actions/checkout@v4
15+
16+
- name: Install packaging tools
17+
# RPM tools are available on ubuntu
18+
run: sudo apt install -y rpm gpg createrepo-c dpkg-dev reprepro
19+
20+
- name: Setup requirements
21+
working-directory: tools/packaging/osrepos
22+
run: pip install -r requirements.txt
23+
24+
- name: Download packages
25+
working-directory: tools/packaging/osrepos
26+
run: python scripts/fetch-releases.py $SiteRoot
27+
28+
- name: Import GPG key
29+
id: gpg-import
30+
uses: crazy-max/ghaction-import-gpg@v6
31+
with:
32+
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
33+
34+
- name: Sign RPMs
35+
shell: sh
36+
working-directory: tools/packaging/osrepos
37+
run: sh scripts/sign-rpms.sh ${{ steps.gpg-import.outputs.fingerprint }}
38+
39+
- name: Create YUM repository
40+
# Creates metadata for YUM/DNF repository, the files were copied in the download step
41+
shell: sh
42+
working-directory: tools/packaging/osrepos
43+
run: createrepo_c -v $SiteRoot/rpm
44+
45+
- name: Sign YUM repository
46+
shell: sh
47+
working-directory: tools/packaging/osrepos
48+
run: gpg --armor --detach-sign $SiteRoot/rpm/repodata/repomd.xml
49+
50+
- name: Create APT repository
51+
# The configuration for apt repo is in tools/packaging/osrepos/reprepro-config,
52+
# which ensures the same GPG key used elsewhere in this action is used to sign
53+
# the repository
54+
shell: sh
55+
working-directory: tools/packaging/osrepos
56+
run: sh -x scripts/generate-apt-repo.sh
57+
58+
- name: Prepare assets
59+
working-directory: tools/packaging/osrepos
60+
run: |
61+
cp -aRv dragonfly.repo pgp-key.public dragonfly.sources $SiteRoot/
62+
rm -rf $SiteRoot/deb/conf
63+
64+
- name: Generate Directory Listings
65+
working-directory: tools/packaging/osrepos
66+
run: python scripts/generate-index.py $SiteRoot
67+
68+
- name: Authenticate
69+
uses: 'google-github-actions/auth@v3'
70+
with:
71+
project_id: 'dragonflydb'
72+
credentials_json: ${{ secrets.GCP_BUCKET_CREDENTIALS }}
73+
74+
- name: GCloud setup
75+
uses: 'google-github-actions/setup-gcloud@v3'
76+
77+
- name: Deploy site
78+
working-directory: tools/packaging/osrepos
79+
run: |
80+
gcloud storage rm ${{ secrets.GCP_PACKAGES_BUCKET }}/**
81+
gcloud storage rsync $SiteRoot ${{ secrets.GCP_PACKAGES_BUCKET }} --recursive --delete-unmatched-destination-objects

‎tests/dragonfly/conftest.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -433,10 +433,11 @@ def pytest_runtest_makereport(item, call):
433433
if report.when == "teardown":
434434
call_outcome = getattr(item, "call_outcome", None)
435435
log_dir = item.funcargs.get("df_log_dir")
436-
if report.failed:
437-
copy_failed_logs(log_dir, report)
438-
if call_outcome and call_outcome.failed:
439-
copy_failed_logs(log_dir, call_outcome)
436+
if log_dir:
437+
if report.failed:
438+
copy_failed_logs(log_dir, report)
439+
if call_outcome and call_outcome.failed:
440+
copy_failed_logs(log_dir, call_outcome)
440441

441442

442443
@pytest.fixture(scope="function")

‎tests/dragonfly/package_install_test.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
from testcontainers.core.container import DockerContainer
2+
3+
4+
def check(container: DockerContainer, cmd: str):
5+
result = container.exec(cmd)
6+
assert result.exit_code == 0, f"command {cmd} failed with result {result}"
7+
8+
9+
async def test_install_package_on_fedora():
10+
with DockerContainer(image="fedora:latest", tty=True) as fedora:
11+
check(
12+
fedora,
13+
"dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo",
14+
)
15+
check(fedora, "dnf -y install dragonfly")
16+
check(fedora, "dragonfly --version")
17+
18+
19+
async def test_install_package_on_ubuntu():
20+
with DockerContainer(image="ubuntu:latest", tty=True) as ubuntu:
21+
check(ubuntu, "apt update")
22+
check(ubuntu, "apt install -y curl")
23+
check(
24+
ubuntu,
25+
"curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public",
26+
)
27+
check(
28+
ubuntu,
29+
"curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources",
30+
)
31+
check(ubuntu, "apt update")
32+
check(ubuntu, "apt install -y dragonfly")
33+
check(ubuntu, "dragonfly --version")

‎tools/packaging/osrepos/README.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Package repositories for rpm and debian packages
2+
3+
This directory contains scripts and definitions for setting up YUM and apt repositories for Linux users to install
4+
dragonfly packages.
5+
6+
The repositories are served as static websites. The generate-site workflow is used to set up and deploy the sites using
7+
scripts and definitions included here.
8+
9+
The workflow does the following tasks:
10+
11+
* Download the latest 5 releases from dragonfly releases page, specifically deb and rpm assets
12+
* for deb files, only the latest package is downloaded and present (see note below)
13+
* Set up a directory structure separating deb and rpm files into version specific paths
14+
* Sign the packages (see note on GPG)
15+
* Deploy the assets prepared, along with the public GPG key and repo definitions for apt and rpm tooling
16+
17+
## Using the YUM repository
18+
19+
Add the repository using:
20+
21+
```shell
22+
sudo dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo
23+
```
24+
25+
Then install dragonfly as usual, or a specific version:
26+
27+
```shell
28+
sudo dnf -y install dragonfly-0:v1.33.1-1.fc30.x86_64
29+
```
30+
31+
## Using the APT repository
32+
33+
First download the public GPG key to an appropriate location:
34+
35+
```shell
36+
sudo curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public
37+
```
38+
39+
Then add the sources file:
40+
41+
```shell
42+
sudo curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources
43+
```
44+
45+
Finally install dragonfly using apt
46+
47+
```shell
48+
sudo apt update && sudo apt install dragonfly
49+
```
50+
51+
#### Versions in APT repository
52+
53+
Unlike the yum repo, the apt repo only has the latest version. The reason for this is the tool, `reprepro` supplied by
54+
debian to build repositories only supports multiple
55+
versions in version 5.4 onwards, and the github runner using ubuntu-latest does not have this version.
56+
57+
Another option would be to use the components feature of apt repositories in the sources file we ask users to install,
58+
but then the versions would need
59+
to be hardcoded in the sources file and the user would have
60+
to update the file with each new release which makes for a bad user experience. As of now users wanting older packages
61+
should download them directly.
62+
63+
### Signing packages
64+
65+
The packages are signed using the GPG key imported from the secret GPG_PRIVATE_KEY in this repository.
66+
67+
The corresponding public key is served with site assets, so the apt/yum/dnf based tooling can consume the public key to
68+
verify package integrity.
69+
70+
### TODO
71+
72+
- [X] debian packages signing (not required? release file is signed)
73+
- [X] debian repo metadata setup
74+
- [ ] tests asserting that packages are installable?

‎tools/packaging/osrepos/dragonfly.repo‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[dragonfly]
2+
name=Dragonfly Packages
3+
baseurl=https://packages.dragonflydb.io/rpm/
4+
enabled=1
5+
gpgcheck=1
6+
gpgkey=https://packages.dragonflydb.io/pgp-key.public

‎tools/packaging/osrepos/dragonfly.sources‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Types: deb
2+
URIs: https://packages.dragonflydb.io/deb
3+
Suites: noble
4+
Components: main
5+
Signed-By: /usr/share/keyrings/dragonfly-keyring.public

‎tools/packaging/osrepos/pgp-key.public‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
-----BEGIN PGP PUBLIC KEY BLOCK-----
2+
3+
mQINBGjkpygBEADuvzXdOXChr/e4Uh2UBne60NPjmuhpjmArfMfqySeRezJ1Nuvd
4+
AvKNuYRyCw+zsh0Zc/sSANpIdAeKPqrfZJgfEIJI0f8WVjfqsCKi+yWB7Bx0GjQ9
5+
y/xoFLKkT7p0P/F4yRlb8kQq2KVP9UvcZBETJY96TpQIJM4N3XoG+8DsELW5HYF2
6+
6sbhgmaNUsxm9oH5UqHcBc7TTgUp10GmZFR4dTeB1IffD/eLMVDMQ8ygzmVxkJPQ
7+
zEKfpFFzseTVyreQlZ5U4GDR8FiB0mY4gZxbCywNqZRycyMM7v4EHuUO0fOgRHdl
8+
5dseF+H1aEG/00JRo6zjiIbgMga0x9wYmVWvTU4wLnGoomukEMCkEQxlil1QjUlK
9+
XI0EltU03DuGki5uhYc9dSS1h74ku2xWePaMsvmxrTphRo1WQBDutzVXSIZ6NBc3
10+
BN+VBHcumVvif9aRrsfsj2CXhnOB61AW+VWk3fk0evW9cceXZDA0NgGdyeTfS7EI
11+
pioaWtmE3Uv3AfHTlNbMytxG7d7k7oAT2xV6z2IygyQZ5LI1tvSJJ+I5kZHKeruj
12+
k2bFp6H9FGi+g4kA+z9QWgkt+0UXYbjKZAs5Es1uGrRk6o1rAyVTKBKz62F0YQbK
13+
j8Q49Z6iSobaKeQG8naCVkALSM49i4Zpw3x1jUpd7k8/KhpJObq3rewqIQARAQAB
14+
tCREcmFnb25mbHkgPHBhY2thZ2luZ0BkcmFnb25mbHlkYi5pbz6JAlIEEwEKADwW
15+
IQRgvYPC7oTdikxvMGcSMEAYvD0qugUCaOSnKAMbLwQFCwkIBwICIgIGFQoJCAsC
16+
BBYCAwECHgcCF4AACgkQEjBAGLw9KrpGbw//VH2zUjaoSh7SnKGdDOA7A95o2EET
17+
ZvChxImyb6xNKfUoMajPnKcJFg514aPFKLuJl4qJmikxdqBF/bYkznCQSJcLQhsT
18+
pvkqanUh/XwBqbJye1QjBq1o0qXLgeY/Ciz2nqupwLQdzvGHO6+2Yk04T89pnZEo
19+
CDSoZKkacu8TpalStqzqDlumryXZzdZ35hAu9OT0fVc2wtcMiY3pznLG1iawNk8I
20+
bzme0ezGA/fk7xEptEbGlb1OtUV5+iG/SFEVvic8GTNf1yLQNCVK3QzD1ciL3MzR
21+
OTH8a04ov2bMxjl8bIefKE/dFBeCSKbvkfTSMAEgqUAuRp7gvoO7uHO05A5AHU2i
22+
y4agskGkgQR9u1yqUXyYIM9kkpuUqqAkwRqg1pw55LG686Xe35QYH4zbpgvr45/Q
23+
JRPFjCbLzR1ZcNyrecHgrq2M9WNlk6dtdWBSJuc7L0M8KJqfrPxQmMpMm/KR43Ey
24+
um0FCgb2J+ceO2W4GrE/DHHoNTt2iio2gMcmRXM7XTmVupsigbYk7AqGncLIQ60B
25+
94jtv16ggXIeA5sPqmyssARXtweTM+EzLLs4K79be4K5j/yyg3CxxvZcq5CZNwoi
26+
fbQgGVNb4SS+nv2r1mVe9XNSonmVVrAqSIFpptH5ahqgaRDUnmy0Lzk7qiHv02OW
27+
PjbSiwQGHDHwq98=
28+
=SOT5
29+
-----END PGP PUBLIC KEY BLOCK-----

‎tools/packaging/osrepos/reprepro-config/distributions‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Codename: noble
2+
Suite: stable
3+
Architectures: amd64 arm64
4+
Components: main
5+
Origin: Dragonfly
6+
Label: Dragonfly
7+
Description: Dragonfly APT repository
8+
SignWith: 60BD83C2EE84DD8A4C6F306712304018BC3D2ABA

‎tools/packaging/osrepos/reprepro-config/options‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
verbose

‎tools/packaging/osrepos/requirements.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
certifi>=2025.10.5
2+
charset-normalizer>=3.4.3
3+
idna>=3.10
4+
requests>=2.32.5
5+
urllib3>=2.5.0

0 commit comments

Comments
 (0)