error-handling.md

Error Handling

Processor and sink agents can have failure behavior configured in their pipeline.yaml file:

errors:
    on-failure: skip|fail|dead-letter
    retries: 1

The default behavior is on-failure: fail with 0 retries.

You can configure error behavior at the top of the pipeline file, and then that behavior will apply to all agents in the pipeline. Both agents in this pipeline will skip errors:

errors:
    on-failure: "skip"
pipeline:
  - name: "convert-to-structure"
    type: "document-to-json"
    input: "questions-topic"
    configuration:
      text-field: "question"
  - name: "compute-embeddings"
    type: "compute-ai-embeddings"
    configuration:
      model: "text-embedding-ada-002"
      embeddings-field: "value.question_embeddings"
      text: "{{ value.question }}"

Source agents do not have configurable error handling.

Configuration

On-failure	Behavior
skip	Skip the record
fail	Fail the processing. The executor pod is restarted. The message and all other messages that were queued or being processed locally will be processed again.
dead-letter	The failed message is put into a dead-letter queue topic. The name of the topic is computed from the “input-topic” of the agent (even for implicit topics) as `topicname + “-deadletter”` and the schema is the same.