plugin eats 400 and 500 errors after max-retry

[sandeepbhojwani]

i have a post_retry_max=1 and i use a buffer with exponential backoff and retry. When there's a 400 error the plugin just eats the record and so my buffer retry doesn't even kick-in. Also for 500, the plugin will retry for the post_retry_max times and after that just eat the record.

I was expecting the plugin to raise an error after 400 or 500 with post_retry_max exceeded. This will let the buffer plugin do the outer retry. Also metrics will capture the errors.

Here's the config i used

<match **>
    @type splunk-http-eventcollector
    @log_level debug
    # test_mode true
    server "XXXX"
    protocol https
    verify false
    token XXXX
    host logging
    index test
    check_index false
    sourcetype _json
    all_items true
    batch_size_limit 1041076 # 1MB
    post_retry_max 1
    post_retry_interval 5

    <buffer>
      @type memory
      chunk_limit_size 1MB
      queued_chunks_limit_size 64
      total_limit_size 128MB

      flush_mode interval
      flush_interval 2s
      flush_thread_count 2

      retry_type exponential_backoff
      retry_wait 10
      retry_exponential_backoff_base 2
      retry_max_interval 1280
      retry_timeout 2h
      retry_randomize true
    </buffer>
  </match>