Feature/improve kernel structure

[Edwardvaneechoud]

This pull request introduces support for multiple Python kernel image flavors (base and ML) and enables per-kernel image selection, as well as tracking the exact Python package versions installed in each kernel. It also updates the Docker build and deployment workflows to build, tag, and publish these new kernel images separately from the main application images. The documentation and integration tests are updated accordingly to reflect these changes.

Kernel image flavors and per-kernel selection:

• Added two new Docker images: flowfile-kernel-base (base) and flowfile-kernel-ml (ML flavor with sklearn, xgboost, etc.), with independent versioning from the main application images. Users can now select which kernel image to use per kernel, or specify a custom image. The database schema is updated to support image_flavour and custom_image fields for kernels, with a migration to backfill existing kernels. [1] [2] [3] [4] [5]

• Docker Compose and integration test workflows are updated to support building, tagging, and using both flowfile-kernel-base and flowfile-kernel-ml images locally. [1] [2] [3]

Kernel package version tracking:

• Added a new resolved_packages JSON column to the kernels table and model, to store the exact versions of user-requested Python packages after image build. This allows the UI to display the actual installed versions. [1] [2]

CI/CD and Docker workflow improvements:

• The Docker build and publish GitHub workflow (docker-publish.yml) is refactored to:
  • Build and tag both kernel and application images for both amd64 and arm64 platforms.
  • Extract and use separate version numbers for application and kernel images.
  • Add build matrix entries for the new kernel images, and update tagging logic to use the correct version for each image.
  • Improve the summary output to clearly list all images and their respective versions. [1] [2] [3] [4] [5] [6] [7] [8]

Documentation updates:

• Expanded deployment documentation to explain the new kernel image flavors, how to select them, and how to configure per-kernel image selection and extra package installation. [1] [2] [3]

Database schema changes:

• Added Alembic migrations for image_flavour, custom_image, and resolved_packages fields in the kernels table. [1] [2]

These changes make the kernel runtime more flexible and transparent, and improve the reliability and clarity of Docker builds and deployments.

[netlify]

✅ Deploy Preview for flowfile-wasm canceled.

Name	Link
🔨 Latest commit	fb667b1
🔍 Latest deploy log	https://app.netlify.com/projects/flowfile-wasm/deploys/69f8bbda20946a00081bb42c