Logstash-output-s3 can't save logs on many bucket.

[vankhoa011]

Hi all,

I make a config for logstash ,
I want if the logs match logtype==request and env==prd , it will save to Bucket 1
If the logstash logtype==request and env==stg , i will save to Bucket2.
However, all logs are save on Bucket 1.

In the beginning , I think there is something wrong with my configure or logstash condition didn't work properly.

So I tried to remove configure which save logs on Bucket2 to check the conditional logstash.
However, it worked correctly.

That's why I think the logstash output-s3 don't allow to save logs on many buckets.

output {

  if [logtype] == "request" and [env] == "prd" {
    s3 {
        access_key_id => "XXX"
        secret_access_key => "XXX"
        bucket => "XXX1"
        endpoint_region => "us-east-1"
        time_file => 1
    }
  }
  if [logtype] == "request" and [env] == "stg" {
    s3 {
      access_key_id => "XXX"
      secret_access_key => "XXX"
      bucket => "XXX2"
      endpoint_region => "us-east-1"
      time_file => 1
    }
  }
}

[ph]

I haven't tested this personaly but can you try adding a different tags option on each output?
Like this:

output {

  if [logtype] == "request" and [env] == "prd" {
    s3 {
        access_key_id => "XXX"
        secret_access_key => "XXX"
        bucket => "XXX1"
        endpoint_region => "us-east-1"
        time_file => 1
        tags => ["bucket1"]
    }
  }
  if [logtype] == "request" and [env] == "stg" {
    s3 {
      access_key_id => "XXX"
      secret_access_key => "XXX"
      bucket => "XXX2"
      endpoint_region => "us-east-1"
      time_file => 1
      tags => ["bucket2"]
    }
  }
}

[codewaffle]

I've also ran into this bug, took quite awhile to track down - in retrospect, the "S3: Cannot delete the temporary file since it doesn't exist on disk" messages in the log were pointing to this issue - if you don't set tags, the files start overwriting each other...

[guyboertje]

@codewaffle - regarding the OP: did you have two s3 outputs 9with different buckets) in if blocks the same as the original post? And was it solved by adding tags?

[codewaffle]

It was the same bucket, different prefixes - I'm a little fuzzy on the rest of the details after so much time.

Here's the config I'm using now - iirc all I did to go from 'not working' to 'working' was add the tags:

output {
    if [type] == "archive" {
        if [fields][logsink_id] == "some-logsink-xxx" {
            s3 {
                access_key_id => "******"
                secret_access_key => ""******"
                region => "us-east-1"
                bucket => "the-bucket-name"
                prefix => "prefix-1/"
                size_file => 5242880
                time_file => 30
                tags => ["prefix-1-tag"]
            }
        }
        else {
            s3 {
                access_key_id => "******"
                secret_access_key => "******"
                region => "us-east-1"
                bucket => "the-bucket-name"
                prefix => "prefix-2/"
                size_file => 5242880
                time_file => 30
                codec => "json"
                tags => ["prefix-2-tag"]
            }
        }
    }
    else
    {
        gelf { ... }
    }
}

[MrPrimate]

I can confirm the problem, and that adding tags resolves the issue.

I had multiple s3 outputs, and without the tags, the files overwrite each other.

My symptoms manifested as each bucket getting the same log file. This error was in my logstash logs:

:message=>"S3: Cannot delete the temporary file since it doesn't exist on disk", :filename=>"ls.s3.eu...

[ph]

With the support of dynamic prefix and uuid that should fix that situation see #102

[suveerrao]

I am trying to send my kubernetes logs to s3 using logstash(v5.4.3) and logstash-output-s3 (5.0.7). logstash is creating logstash-programmatic-access-test-object but not the actual log files. Below are my input and output configurations. Am I taking a wrong input plugin ? please do suggest.

input {
  file {
    # kubelet created symlinks to docker container logs
    path => "/var/log/containers/*.log"
    tags => ["kubernetes", "docker"]
    sincedb_path => "/var/log/containers/sincedb-kubernetes"
  }
}

output {
  if "kubernetes" and "docker" in [tags] {
     s3{
      access_key_id => "XXXXXXXXXX"
      secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXX"
      region => "us-east-1"
      bucket => "nonapp-logs"
      size_file => 2048
      time_file => 5
      prefix => "non-app-logs-%{+YYYY.MM.dd}"
      canned_acl => "private"
      server_side_encryption => true
    }
  }
}