DLQ for Sink

[vigith]

Problem

DLQ is already available out-of-the-box in Numaflow (using conditional forwarding) except for the Sink vertex. In the Sink vertex, the user must "manually" write to a DLQ if the Sink is down; otherwise, the backpressure caused will back-propagate and stop reading from the source. This is a significant concern when users have multiple Sinks where they expect writes to happen even if there is a failing Sink (else a failing Sink can cause the entire pipeline to stall).

Approaches

Multiple Container for fallback

e.g.,

    - name: out
      sink:
        # A simple log printing sink
        log: {}
        fallback:
          - foo
    - name: kafka-out
      sink:
        udsink:
          container:
            image: my-sink:latest
        fallback:
          kafka:
            brokers:
              - my-broker1:19700
              - my-broker2:19700
            topic: my-topic

Conditional Forwarding at Sink level

DLQ is unavailable only in the Sink because it is a terminal state, and no forwarding happens after the Sink. We could extend the conditional-forwarding and let a Sink forward to another Sink.

Use Cases

• Write to a different Sink if the current Sink is failing
  • e.g., Write to Kafka if the Database Sink is under maintenance.
• DLQ Sink can be connected back to Source and call a DLQ source, and the same messages can be fed in (solved problem via multi-source)

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Core

Preview Give feedback

1. chore: Refactor Sinks to Avoid Implementing Forwarder Interface #1662
   +0
2. feat: controller change for fallback sink #1664
   +0
3. chore: kustomization cofigurations support fallback sink #1667
   +0
4. add fallback-sink to sinker #1668
   area/sink +1
   
5. e2e tests for fallback #1682
   testing +1
6. add examples
7. doc: Fallback Sink #1691
   +0

SDK

Preview Give feedback

1. feat: Fallback Sink numaflow-go#120
   +0
2. feat: Fallback Sink numaflow-java#116
   +0
3. SDK changes for Fallback Sink numaflow-python#156
   enhancement +1
   

[whynowy]

@yhl25 - be aware we need to have a different sock file and server-info file for the fallback container.

[whynowy]

@yhl25 - be aware we need to have a different sock file and server-info file for the fallback container.

We can populate a builtin ENV for all the UD containers to indicate the container type.