Support configuring defaults for pipeline component settings

[dseapy]

Summary

Currently AbstractPodTemplate and ContainerTemplate need to be specified within each pipeline component (daemon, job, vertex). It would be useful to be able to set defaults for the controller to use.

Use Cases

It would be nice to simplify the responsibility of the data-scientist creating the pipeline in a Dev environment, while also simplifying the responsibility of the devops team with the (eventual) promotion of the pipeline to a Prod environment. Allowing defaults to be set for the controller would have the following benefits:

• Streamline data-scientists writing/running successful pipelines in clusters that require specific environment-specific configuration to run well.
• Streamline devops team implementing automation to promote Pipeline resources to different environments with different constraints (i.e. fewer per-pipeline daemon replicas in Dev to save $).
• Yaml with less duplication, making it easier to work with when working with Pipelines in general.

An example of this, is that in a Prod environment one might want settings that a data-scientist making the Pipeline might not care much about in Dev. i.e. maybe not exactly this, but something like this for daemon component:

  templates:
    daemon:
      replicas: 2
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
      containerTemplate:
        resources:
          requests:
            cpu: 100m
          limits:
            memory: 200Mi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - daemon
                  - key: numaflow.numaproj.io/pipeline-name
                    operator: In
                    values:
                      - example-pipeline
              topologyKey: kubernetes.io/hostname

[dseapy]

I'm happy to work on this. There are alternative ways to accomplish this outside of numaflow (i.e. write a DefaultedPipeline CRD with controller that creates Pipeline resources), so it's not a blocker but would be nice for numaflow to support.

If this use-case does make sense for numaflow to support, let me know if you all have ideas of best approach to take for this. I can also just get started on implementing what seems best and then get feedback on a PR & go from there.

[whynowy]

I don't prefer to introduce extra CRDs for that, what about adding to the controller configmap?

apiVersion: v1
kind: ConfigMap
metadata:
  name: numaflow-controller-config
data:
  controller-config.yaml: |+
    isbsvc:
      redis:
        xxx
  pipeline-templates.yaml: |+
    daemon:
      xxx
    job:
      xxx
    vertex:
      podTemplate:
        xxxx
      containerTemplate:
        xxxx
      initContainerTemplate:
        xxx

Users still can give customizations in the pipeline spec, that will override (merge to) the global config.