Skip to content

Release/v4 #209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
a-klos merged 12 commits into from
Jan 22, 2026
Merged

Release/v4 #209

a-klos merged 12 commits into from
Jan 22, 2026

Conversation

@a-klos
Copy link
Member

@a-klos a-klos commented Jan 21, 2026 •
edited
Loading

This pull request introduces major improvements to CI/CD automation, release management, and environment configuration. It adds new GitHub Actions workflows for preparing releases, creating releases, building Docker images, and bumping Helm chart versions. The .dockerignore file is also refactored for better build performance, and the .env.template is updated to simplify authentication configuration. Several enhancements are made to the lint and test workflow for improved reliability and clarity.

CI/CD Workflow Additions and Improvements

  • Added .github/workflows/prepare-release.yml to automate preparing release PRs, including semantic versioning and dependency bumping for internal libraries.
  • Added .github/workflows/create-release.yml to automate tagging and publishing GitHub releases when PRs with the refresh-locks label are merged into main.
  • Added .github/workflows/build-images.yml to build and push Docker images for all major services on release, including digest artifact handling for traceability.
  • Added .github/workflows/bump-chart-version.yml to allow manual Helm chart version bumps via workflow dispatch, opening a PR for the change.

Environment and Build Configuration

  • Refactored .dockerignore to use glob patterns for Node/Nx artifacts, frontend build outputs, and Python environments, reducing build context size and improving maintainability.
  • Updated .env.template to split frontend and backend authentication variables for clarity and removed redundant frontend auth variables. [1] [2]

Lint and Test Workflow Enhancements

  • Improved conditional logic in .github/workflows/lint-and-test.yml to skip jobs for release-related PRs and updated dorny/paths-filter to v3 for change detection. [1] [2]
  • Changed Docker build commands in lint/test jobs to use dedicated dev Dockerfiles and improved image naming for clarity and reliability. [1] [2]

a-klos and others added 11 commits December 16, 2025 12:04
@a-klos
chore(deps): Major version updates of the frontends. (#137)
922b80d
@a-klos
chore: merge main
a1d3dc6
@a-klos
Update dependencies and configurations across multiple services
43877f8 
- Added torch and torchvision dependencies to document-extractor with CPU support.
- Updated .gitignore in frontend to include npm cache.
- Bumped mcp version from 1.22.0 to 1.23.0 in mcp-server's poetry.lock.
- Updated langchain-core version from 1.0.7 to 1.2.6 and added uuid-utils dependency in rag-backend's poetry.lock.
- Adjusted markers for colorama package in rag-backend's poetry.lock.
@a-klos @renovate-bot
refactor!: helmchart simple (#174)
2295388 
This pull request refactors secret management and environment variable
handling across the local development and deployment workflow, focusing
on improving flexibility and security. The changes standardize secret
references, introduce dev-only helpers for Langfuse initialization, and
update documentation and templates to reflect the new structure. The
most important changes are grouped below:

**Secret Management Refactor**

* Secrets for basic authentication, Langfuse, S3, STACKIT, and RAGAS are
now split into individual username/password or API key fields with
support for referencing Kubernetes secrets via `secretKeyRef` in
`values.yaml` and Helm templates. This enables easier integration with
external secret managers and more granular secret configuration.
(`.env.template`, `Tiltfile`, `infrastructure/README.md`, `README.md`,
[[1]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566R242-R255)
[[2]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566L248-L253)
[[3]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566L263-R312)
[[4]](diffhunk://#diff-749e06f64632f62a0c0dfbf4c4f3850e27e94ac109aa121fabd5c29469ae88deL13-R15)
[[5]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L472-R486)
[[6]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L535-R533)
[[7]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L211-R237)
F9cc2a25L27R29)
* Helm templates now use new helper functions to resolve secret names,
preferring `secretKeyRef.name` when set, and falling back to default
names. This affects backend/admin deployments and ingress
authentication.
(`infrastructure/rag/templates/_admin_backend_and_extractor_helpers.tpl`,
`infrastructure/rag/templates/_backend_helpers.tpl`,
[[1]](diffhunk://#diff-3ab40efdb049da16ac327c9fbaf8ec1d25f26efbeded4e0c2cfd7f50b976d3ceR18-R53)
[[2]](diffhunk://#diff-148b9880aef46deecdbc54ae7d004627e0aa1bf612fe139eb607861495ae9301R22-R65)
[[3]](diffhunk://#diff-148b9880aef46deecdbc54ae7d004627e0aa1bf612fe139eb607861495ae9301L112-R156)
[[4]](diffhunk://#diff-9f487482fa54d28d71fff497724bbc6741cced8a1e35c9b8829d1c6bd01dca0aL137-R145)
[[5]](diffhunk://#diff-2b6f7f2ec4938055207faa53acf7a300e0ec235db31d1cfb6896703b97292348L125-R133)

**Langfuse Dev-Only Helper**

* Added a dev-only workflow for Langfuse initialization secrets using
Kustomize and Tilt: developers can fill out a `.env.langfuse` file,
which is used to generate a stable Kubernetes secret for local
development. Documentation and Tiltfile logic were updated to support
this workflow.
(`infrastructure/kustomize/langfuse/.env.langfuse.template`,
`infrastructure/kustomize/langfuse/kustomization.yaml`, `Tiltfile`,
`README.md`, `infrastructure/README.md`,
[[1]](diffhunk://#diff-b2c7f0f443df5a8f90f932d8c1c326c35195f9e2758322412862ac844c83c257R1-R8)
[[2]](diffhunk://#diff-bd3d71bbf87e1ca957e44fdaf2c202225d57d281b5f6065f0d67eacc5e582485R1-R12)
[[3]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425R323-R330)
[[4]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R246-R251)
[[5]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566R185-R190)

**Documentation Updates**

* Updated `README.md` and `infrastructure/README.md` to reflect the new
environment variable and secret structure, clarify dev/prod secret
management, and provide step-by-step instructions for local setup and
deployment. (`README.md`, `infrastructure/README.md`,
[[1]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L211-R237)
[[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R246-R251)
[[3]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566R185-R190)
[[4]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566R242-R255)
[[5]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566L248-L253)
[[6]](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566L263-R312)

**Configuration and Template Changes**

* Updated Helm values and templates to match the new secret structure,
including support for referencing secrets and setting inline values for
all major providers (basic auth, Langfuse, S3, STACKIT, RAGAS).
(`infrastructure/README.md`,
`infrastructure/rag/templates/_admin_backend_and_extractor_helpers.tpl`,
`infrastructure/rag/templates/_backend_helpers.tpl`,
[[1]](diffhunk://#diff-9f487482fa54d28d71fff497724bbc6741cced8a1e35c9b8829d1c6bd01dca0aL137-R145)
[[2]](diffhunk://#diff-2b6f7f2ec4938055207faa53acf7a300e0ec235db31d1cfb6896703b97292348L125-R133)

**Minor Improvements**

* Added new environment variables for evaluation settings and improved
example values for clarity in documentation.
(`infrastructure/README.md`,
[infrastructure/README.mdL313-R353](diffhunk://#diff-d5b352058053e1306240bcd8ab265b3234d611212cd196b2001fbb9131048566L313-R353))

Let me know if you need more details about any specific change or how
these updates affect your workflow!

---------

Co-authored-by: Renovate Bot <renovate@whitesourcesoftware.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@a-klos @NewDev16
refactor: split Dockerfiles into prod and dev files (#95)
00a03ae 
This pull request introduces significant improvements to the development
and build workflows for both the backend services and Python libraries.
The main changes include a clearer separation between development and
production Docker builds, enhanced configuration for local development
via Tilt, and improved dependency management and testing setup for
Python libraries. These updates streamline live development, testing,
and linting processes, making them more reliable and easier to maintain.

**Build and workflow improvements:**

* Added explicit support for development mode in the `Tiltfile`,
allowing dynamic switching between `Dockerfile.dev` (for live code
updates and fast iteration) and production `Dockerfile` (for optimized
builds). Also updated Docker ignore lists for more targeted builds.
[[1]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425R10-R64)
[[2]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L154-R208)
[[3]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L196-R261)
[[4]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L229-R315)
[[5]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L274-R380)
[[6]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L321-R422)
[[7]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L335-R443)
* Updated GitHub Actions workflows to use `Dockerfile.dev` for service
builds during linting and testing, removed unnecessary build arguments,
and set Python version to 3.13 for consistency.
[[1]](diffhunk://#diff-9a979a1e38ba79e2c75e54c4bf21fe1a2a1b935e1736666565f992e634dadd0fL11-R11)
[[2]](diffhunk://#diff-9a979a1e38ba79e2c75e54c4bf21fe1a2a1b935e1736666565f992e634dadd0fL84-R84)
[[3]](diffhunk://#diff-9a979a1e38ba79e2c75e54c4bf21fe1a2a1b935e1736666565f992e634dadd0fL107-R116)
[[4]](diffhunk://#diff-ea3710a52b3ef06a333289cb6d30d80091cde61d7f198ca449e463a6a52e4d39L93-L113)

**Python library dependency and testing enhancements:**

* Refactored Poetry dependency groups in all Python libraries to
separate `test`, `lint`, and `dev` dependencies, added `setuptools
<80.9` for compatibility, and updated library versions. Also improved
`pytest` configuration for better logging and path setup.
[[1]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL7-R7)
[[2]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL68-R73)
[[3]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL93-R99)
[[4]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbR115-R118)
[[5]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L7-R7)
[[6]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L73-L80)
[[7]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323R103-R111)
[[8]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L130-R140)
[[9]](diffhunk://#diff-ddd05b1c3207b7c5f110a23524e106448461d9847d93c840c68412b9778ec3e6R59-R66)
[[10]](diffhunk://#diff-ddd05b1c3207b7c5f110a23524e106448461d9847d93c840c68412b9778ec3e6L80-R92)
[[11]](diffhunk://#diff-9c4162cc1c16dd4c7ec5e95e79df285e8c0882a1db7ff2892c746a0537d26c36L7-R7)

**Docker build improvements for Python libraries:**

* Updated `libs/Dockerfile` to use Poetry 2.1.3, improved conditional
installation logic for dev/test/lint dependencies, and ensured correct
installation of shared dependencies for interdependent libraries.

**Frontend workflow fixes:**

* Ensured `npm install` runs before frontend linting and testing in
Tilt, and expanded dependency tracking for these local resources.
[[1]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L560-R669)
[[2]](diffhunk://#diff-c2ee8653e1d6b85f0aadf87cd438a9250806c052877248442be4d434cbc52425L570-R679)

**Dependency version updates:**

* Updated several library versions and dependency specifications,
including switching `fasttext` to a stable release and updating internal
library versions to 2.2.1.
[[1]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL7-R7)
[[2]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L7-R7)
[[3]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L73-L80)
[[4]](diffhunk://#diff-9c4162cc1c16dd4c7ec5e95e79df285e8c0882a1db7ff2892c746a0537d26c36L7-R7)

Let me know if you want to walk through any of these changes in detail
or discuss how they impact your local development workflow!

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Andreas Klos <aklos@outlook.de>
@a-klos @NewDev16
refactor: update GitHub workflows for release management and image bu…
25f8c42 
…ild (#97)

This pull request restructures and improves the release automation for
the repository by splitting the previous monolithic workflow into
several specialized GitHub Actions workflows. These new workflows
automate versioning, library publishing, Docker image building, Helm
chart packaging, and dependency lockfile refreshing. Additionally, the
workflows now use label-based gating to control releases and dependency
updates, and they improve version extraction and artifact handling. The
previous workflow file `.github/workflows/semantic-release.yml` has been
removed and replaced with more maintainable, modular workflows.

**Release and Versioning Automation**

* Added `.github/workflows/prepare-release.yml` to automate semantic
version calculation and bump internal library versions, opening a PR
with the new versions and gating future steps with the `prepare-release`
label.
* Added `.github/workflows/create-release.yml` to create a Git tag and
GitHub Release when a PR with the `refresh-locks` label is merged to
`main`, extracting the version from the PR title.

**Library Publishing and Dependency Management**

* Added `.github/workflows/publish-libs-on-merge.yml` to publish Python
libraries to TestPyPI after a `prepare-release` PR is merged, update
service dependency pins, refresh lockfiles, and open a PR with updated
lockfiles using the `refresh-locks` label.

**Docker Image and Helm Chart Automation**

* Added `.github/workflows/build-images.yml` to build and publish Docker
images for all services when a release is published, capturing image
digests as artifacts.
* Added `.github/workflows/publish-chart.yml` to package and publish the
Helm chart after images are built, bumping chart versions and opening a
PR for chart version updates using the `chart-bump` label.

**Workflow Gating and Cleanup**

* Updated `.github/workflows/lint-and-test.yml` to skip jobs if any of
the release-related labels (`prepare-release`, `refresh-locks`,
`chart-bump`) are present, preventing unnecessary CI runs during release
automation.
* Removed the legacy `.github/workflows/semantic-release.yml` workflow,
which previously handled all release steps in one file, in favor of the
new modular approach.

These changes collectively make the release process more robust,
modular, and maintainable, while ensuring that versioning, publishing,
and dependency updates are tightly controlled and automated.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Andreas Klos <aklos@outlook.de>
@a-klos
feat: add continue_on_failure option to SitemapExtractor and admin fr…
cfba3b2 
…ontend (#200)

This pull request adds support for a new `continue_on_failure` option to
sitemap-based document uploads, allowing users to specify whether the
crawl should skip pages that fail to load rather than aborting the
entire process. The feature is integrated across the backend, frontend,
and documentation, and includes robust handling and testing for various
input types.

**Backend changes:**

* Added parsing and normalization for the `continue_on_failure`
parameter in `_parse_sitemap_loader_parameters`, supporting boolean and
string representations.
* Defaulted `continue_on_failure` to `True` in `aextract_content` if not
provided, ensuring failure resilience by default.

**Frontend changes:**

* Added a checkbox for "Continue on failure" to the sitemap upload UI
(`DocumentUploadContainer.vue`), defaulting to checked
[[1]](diffhunk://#diff-a6fc8bcaabdced0bd0b5b642bd5a4aa9cb124a5bbebd0762e76f9dcb0df884c1R242-R250)
[[2]](diffhunk://#diff-a6fc8bcaabdced0bd0b5b642bd5a4aa9cb124a5bbebd0762e76f9dcb0df884c1R36).
* Passed the `continueOnFailure` value from the UI through the API
payload
[[1]](diffhunk://#diff-a6fc8bcaabdced0bd0b5b642bd5a4aa9cb124a5bbebd0762e76f9dcb0df884c1R106)
[[2]](diffhunk://#diff-0f7547155cd6592b947aae6327e72dbe57073ae43aba24e82ad7ef78fee08153R103-R106).
* Updated the `SitemapConfig` interface to include `continueOnFailure`.

**Documentation and localization:**

* Documented the new parameter in `README.md` for sitemap sources.
* Added English and German translations for the new UI option and its
tooltip
[[1]](diffhunk://#diff-430c5bb0cfd37251a3388659a69ca7cff0726cd2cc40d592b79b55c9f644050dR26-R27)
[[2]](diffhunk://#diff-e485c1eda5b61acd7bba3807afc19b489ad515ba3a6feddd627596986245c334R24-R25).

**Testing:**

* Updated and expanded tests to verify correct handling and passing of
the `continue_on_failure` parameter, including type normalization and
defaulting behavior
[[1]](diffhunk://#diff-abbbfdf7e054264c0cba6be8ad96b9da67fac3205fe69c3f247807a6d6cc70deL272-R275)
[[2]](diffhunk://#diff-abbbfdf7e054264c0cba6be8ad96b9da67fac3205fe69c3f247807a6d6cc70deL431-R435)
[[3]](diffhunk://#diff-abbbfdf7e054264c0cba6be8ad96b9da67fac3205fe69c3f247807a6d6cc70deL444-R447)
[[4]](diffhunk://#diff-abbbfdf7e054264c0cba6be8ad96b9da67fac3205fe69c3f247807a6d6cc70deL465-R468).
@a-klos
chore: update poetry.lock files for multiple services
d3b9b61 
- Updated Poetry version from 2.2.1 to 2.3.1 in document-extractor, mcp-server, and rag-backend.
- Changed version format for extractor-api-lib and rag-core-api/rag-core-lib from "v3.4.0" to "3.4.0".
- Updated jsonschema-specifications version format in dependencies.
- Added new musllinux wheels for greenlet across various Python versions.
- Adjusted cffi dependency version format in extras for consistency.
@a-klos
refactor: update dependencies and markers in poetry.lock and pyprojec…
2b97fcc 
…t.toml
@a-klos
refactor: update favicon files to SVG format and correct title casing…
811db08 
… in HTML
@a-klos
chore: update pyproject.toml to set package-mode to false
ad651e3
@a-klos a-klos merged commit 88162b9 into main Jan 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stackit-sven stackit-sven stackit-sven approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@a-klos @stackit-sven