Allow non-AWS endpoints

This is useful for local Ceph S3 deployments.  Fixes #10.  Fixes #65.

Author: Andrew Gaul

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 4.0.6
+  - Support for non-AWS endpoints
+
 ## 4.0.5
   - Delete the file on disk after they are succesfully uploaded to S3 #122 #120
   - Added logging when an exception occur in the Uploader's `on_complete` callback

CONTRIBUTORS

@@ -11,6 +11,7 @@ Contributors:
 * Nick Ethier (nickethier)
 * Pier-Hugues Pellerin (ph)
 * Richard Pijnenburg (electrical)
+* Andrew Gaul (andrewgaul)
 
 Note: If you've sent us patches, bug reports, or otherwise contributed to
 Logstash, and you aren't on the list above and want to be, please let us know

lib/logstash/outputs/s3.rb

@@ -13,6 +13,7 @@
 require "pathname"
 require "aws-sdk"
 require "logstash/outputs/s3/patch"
+require "uri"
 
 Aws.eager_autoload!
 
@@ -64,9 +65,11 @@
 # This is an example of logstash config:
 # [source,ruby]
 # output {
-#    s3{
+#    s3 {
 #      access_key_id => "crazy_key"             (required)
 #      secret_access_key => "monkey_access_key" (required)
+#      endpoint => "http://127.0.0.1:8080"      (optional, used for non-AWS endpoints, default = "")
+#      force_path_style => false                (optional, used for non-AWS endpoints, default = false)
 #      region => "eu-west-1"                    (optional, default = "us-east-1")
 #      bucket => "your_bucket"                  (required)
 #      size_file => 2048                        (optional) - Bytes
@@ -106,6 +109,13 @@ class LogStash::Outputs::S3 < LogStash::Outputs::Base
   # S3 bucket
   config :bucket, :validate => :string, :required => true
 
+  # Specify a custom endpoint for use with non-AWS S3 implementations, e.g.,
+  # Ceph.  Provide a URL in the format http://127.0.0.1:8080/
+  config :endpoint, :validate => :string
+
+  # When false, specify the bucket in the subdomain.  When true, specify the bucket in the path.
+  config :force_path_style, :validate => :bool, :default => false
+
   # Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file.
   # If you have tags then it will generate a specific size file for every tags
   ##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
@@ -270,6 +280,7 @@ def full_options
     options = Hash.new
     options[:s3_signature_version] = @signature_version if @signature_version
     options.merge(aws_options_hash)
+           .merge(endpoint_options)
   end
 
   def normalize_key(prefix_key)
@@ -286,6 +297,18 @@ def upload_options
     }
   end
 
+  def endpoint_options
+    if @endpoint
+      uri = URI(@endpoint)
+      {
+        :endpoint => @endpoint,
+        :force_path_style => @force_path_style,
+      }
+    else
+      {}
+    end
+  end
+
   private
   # We start a task in the background for check for stale files and make sure we rotate them to S3 if needed.
   def start_periodic_check