feat: Multi-cluster Kubernetes Task Runner#19433
Conversation
GWphua
force-pushed
the
multiple-k8s-clusters
branch
from
e3f0e73 to
f4504f6
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if (ownsExecutor && configManager != null) { | ||
| configManager.addListener( | ||
| KubernetesTaskRunnerDynamicConfig.CONFIG_KEY, | ||
| StringUtils.format(OBSERVER_KEY, Thread.currentThread().getId()), |
There was a problem hiding this comment.
The updated Thread.threadId is the replacement, introduced in Java 19. I didn't get to see any PR that drops jdk17 support yet, (correct me if im wrong), so we should still use this deprecated method.
| if (ownsExecutor && configManager != null) { | ||
| configManager.addListener( | ||
| KubernetesTaskRunnerDynamicConfig.CONFIG_KEY, | ||
| StringUtils.format(OBSERVER_KEY, Thread.currentThread().getId()), |
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.
This is an automated review by Codex GPT-5
|
Yet to provide documentation on the feature, I will need some time to make it. |
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
Reviewed 17 of 17 changed files.
This is an automated review by Codex GPT-5.5
| new DynamicConfigPodTemplateSelector(properties, effectiveConfig) | ||
| ); | ||
| } else { | ||
| return new SingleContainerTaskAdapter( |
There was a problem hiding this comment.
[P1] Default pod adapter fails for remote clusters
The multik8s factory still defaults to SingleContainerTaskAdapter, and also allows MultiContainerTaskAdapter. Both adapters build jobs by reading the Overlord pod via the configured Kubernetes client and task namespace. In this factory, that client points at the selected target cluster and namespace, where the single Druid Overlord usually does not exist. With the documented multik8s sample, which does not set a custom template adapter, the first task submitted to a remote cluster will fail while trying to copy a non-existent Overlord pod. Either require customTemplateAdapter for multik8s or change these adapters to source the pod spec from the local Overlord cluster instead of the target task cluster.
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.
Reviewed 21 of 21 changed files.
This is an automated review by Codex GPT-5.5
3 participants
Description
The existing Kubernetes task runner schedules all tasks into a single Kubernetes cluster. This PR adds a
multik8stask runner mode that can schedule tasks across multiple Kubernetes clusters while keeping a single Overlord-level view of task capacity, task state, task logs, and task reports.This is useful for Druid deployments that need to spread indexing work across multiple Kubernetes clusters, drain or disable specific clusters, or route tasks with a simple cluster selection strategy.
Added multi-cluster Kubernetes task runner
Added
MultipleKubernetesTaskRunner, which wraps multiple per-clusterKubernetesTaskRunnerinstances and exposes them through the existingTaskRunnerandTaskLogStreamerAPIs.The runner supports:
k8s_clustertask context tag injection for pod-template selectionAdded multi-cluster runner configuration
Added
MultipleKubernetesTaskRunnerConfig, configured under the existingdruid.indexer.runnerprefix, with per-cluster settings for cluster name, kubeconfig path, task namespace, Overlord identifier, and disabled state.The multi-cluster runner reuses the existing Kubernetes runner configuration for common task settings such as capacity, labels, annotations, sidecar support, task cleanup, task timeout, and shared informer behavior.
Built per-cluster Kubernetes runners
Added
MultipleKubernetesTaskRunnerFactory, which creates oneKubernetesTaskRunnerper configured cluster and registers the newmultik8srunner type inKubernetesOverlordModule.Each per-cluster runner gets its own Kubernetes client and task adapter while sharing a single task tracking executor. The factory also honors
useK8sSharedInformersby usingCachingKubernetesPeonClientwhen shared informers are enabled.Shared task runner capacity across clusters
Added
AutoscalableThreadPoolExecutorand updatedKubernetesTaskRunnerso multiple Kubernetes task runners can share one executor. This keeps the configured runner capacity as a global limit across all configured clusters instead of multiplying capacity by the number of clusters.Release note
Added experimental support for running Kubernetes indexing tasks across multiple Kubernetes clusters. Set
druid.indexer.runner.type=multik8sand configuredruid.indexer.runner.clustersto schedule tasks across multiple Kubernetes clusters from a single Overlord.Key changed/added classes in this PR
MultipleKubernetesTaskRunnerMultipleKubernetesTaskRunnerConfigMultipleKubernetesTaskRunnerFactoryMultipleKubernetesTaskRunnerDelegateAutoscalableThreadPoolExecutorKubernetesTaskRunnerKubernetesOverlordModuleThis PR has: