MultipartUploadError:Net::ReadTimeout error

[danhermann]

This error occurs in version 4.1.7 of the S3 output with version 2.11.236 of the AWS SDK as shipped with Logstash 6.7.1. It appears to work correctly with S3 output 4.1.7 and AWS SDK 2.11.177 as shipped with Logstash 6.5.4.

[bitva77]

I'm seeing this as well.

Just upgraded to 6.8.3. I'm still seeing the timeout but it seems to retry successfully now.

Will update as I get more info...

No, the output times out and does not recover or retry successfully. 4.1.9 of the plugin

[2019-10-07T17:50:40,512][ERROR][logstash.outputs.s3 ] Uploading failed, retrying. {:exception=>Aws::S3::MultipartUploadError, :message=>"multipart upload failed: Net::ReadTimeout; Net::ReadTimeout; Net::ReadTimeout; Net::ReadTimeout", :path=>"/tmp/MinioTemp/e89bfafe-1554-4af1-8f71-999f3193d42d/2019-10/ls.s3.173d01e4-93f8-47f8-963f-4675d8978ff1.2019-10-07T17.08.part0.txt", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.319/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:81:inabort_upload'",
`

[bitva77]

I've taken logstash-output-s3-4.1.7 from the 6.5.4 Logstash image and have copied the contents of it into the logstash-output-s3-4.1.9 folder in the 6.8.3 Logstash image and am running that now.

RUN rm -rf /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9 ADD logstash-output-s3-4.1.7 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9/

So far it's running fine and has worked through the queue we had. Will update this thread with more information.

Not sure how the AWS SDK thing works so I'll try to roll that one back next if it starts failing again.

[bitva77]

I reverted AWS SDK to 2.11.177 but still have the issue.

I noticed in the error that concurrent-ruby-1.1.5 is used. I've now reverted that to 1.0.5 and will report back.

Currently running fine....

UPDATE

Still same problem. Looks like it happens on bigger file uploads (16MB or so)

[bitva77]

Been running Logstash 6.5.4 without issue for a few hours now. Definitely something wrong in 6.7 and above. Have not tried 7.x yet.

[inqueue]

Some users are blocked from upgrading due to this issue. Is there a resolution in sight?

[inqueue]

Hi, any news?

[kvmuralidhar]

Upload is failing with logstash version 7.0.0 and s3 output plugin version 4.1.8/4.1.10 as well. The file size is 200MB (no issues with logstash version 6.5.4 and s3 plugin version 4.1.7)

This is preventing us from upgrading to the newer version of logstash. I hope there is a resolution in sight soon.

Thanks
Murali

[kvmuralidhar]

Tried logstash version 7.3.2 this morning and the issue exists.

[kvmuralidhar]

I tried by setting the size of the file to 15MB (instead of 200MB) and it's failing to upload a small file as well. Here is the trace info from the logs.

Uploading failed, retrying. {:exception=>Aws::S3::MultipartUploadError, :message=>"multipart upload failed: Net::ReadTimeout; Net::ReadTimeout; Net::ReadTimeout; Net::ReadTimeout", :path=>"/u/elasticStack/logstash-working-dir/ls-szng1/tmp/482ec854-eb7b-4bd2-ba8c-92ca83f08f5e/ls.s3.59a7b8a1-3906-4331-9e16-ee7afe4bc2bc.2020-02-05T20.42.part18.txt.gz", :backtrace=>["/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.330/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:81:in abort_upload'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.330/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:70:in upload_parts'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.330/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:44:in upload'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.330/lib/aws-sdk-resources/services/s3/file_uploader.rb:32:in upload'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.330/lib/aws-sdk-resources/services/s3/object.rb:252:in upload_file'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9/lib/logstash/outputs/s3/uploader.rb:38:in upload'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9/lib/logstash/outputs/s3/uploader.rb:29:in block in upload_async'", "/u/elasticStack/ls_szng1/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/java_executor_service.rb:85:in run'"]}

thanks
Murali

[bitva77]

Confirmed same issue on 7.5.1 as well.

[kares]

anyone tried setting up a minimal reproducer using a sample file and LS with s3 output?

[kvmuralidhar]

Hi Kares,

I was able to reproduce the issue with a simple logstash configuration as shown below. The s3 bucket we are using is internal to our environment and not provided by aws. Attaching herewith the logstash configuration (minus the endpoint).

Logstash version 7.3.2 and logstash-output-s3 plugin version 4.1.10

sample-s3.conf.txt
Thanks
Murali

[kares]

Thanks MV, gave it a spin against AWS but not luck reproducing, using:

   s3 {
        #access_key_id => "someaccessid"
        #bucket => "lstest"
        encoding => "gzip"
        #secret_access_key => "${DL_SCRTACS_KEY}"
        #region => "internal-region"
        size_file => 20971520
        # upload_multipart_threshold => 52428800
        # upload_workers_count => 6
        temporary_directory => "${LS_WORKING_DIR}"
        #endpoint => "http://internal-region.storage.cloud.domain.com"
        #validate_credentials_on_root_bucket => false

	#kares:
        aws_credentials_file => "aws_credentials.yml"
        bucket => "kares-test1"

        codec => json_lines
        additional_settings => {
          "force_path_style" => true
        }
   }

... was able to upload 20M gzips just fine:

[muralivenkataraman]

Hi Kares,

We are using CEPH object storage instead of s3 provided by AWS. No issues uploading a 200Mb file to CEPH object storage with logstash version 6.5.4 or 6.6.1

Thanks
Murali