Fluent OSS output plugin buffers event logs in local files and uploads them to OSS periodically in background threads.
This plugin splits events by using the timestamp of event logs. For example, a log '2019-04-09 message Hello' is reached, and then another log '2019-04-10 message World' is reached in this order, the former is stored in "20190409.gz" file, and latter in "20190410.gz" file.
Fluent OSS input plugin reads data from OSS periodically.
This plugin uses MNS on the same region of the OSS bucket. We must setup MNS and OSS event notification before using this plugin.
This document shows how to setup MNS and OSS event notification.
This plugin will poll events from MNS queue and extract object keys from these events, and then will read those objects from OSS. More info at https://github.com/aliyun/fluent-plugin-oss
| Variable Name | Type | Required | Default | Description |
|---|---|---|---|---|
| endpoint | string | Yes | - | OSS endpoint to connect to' |
| bucket | string | Yes | - | Your bucket name |
| access_key_id | *secret.Secret | Yes | - | Your access key id Secret |
| aaccess_key_secret | *secret.Secret | Yes | - | Your access secret key Secret |
| path | string | No | fluent/logs | Path prefix of the files on OSS |
| upload_crc_enable | bool | No | true | Upload crc enabled |
| download_crc_enable | bool | No | true | Download crc enabled |
| open_timeout | int | No | 10 | Timeout for open connections |
| read_timeout | int | No | 120 | Timeout for read response |
| oss_sdk_log_dir | string | No | /var/log/td-agent | OSS SDK log directory |
| key_format | string | No | %{path}/%{time_slice}%{index}%{thread_id}.%{file_extension} | The format of OSS object keys |
| store_as | string | No | gzip | Archive format on OSS: gzip, json, text, lzo, lzma2 |
| auto_create_bucket | bool | No | false | desc 'Create OSS bucket if it does not exists |
| overwrite | bool | No | false | Overwrite already existing path |
| check_bucket | bool | No | true | Check bucket if exists or not |
| check_object | bool | No | true | Check object before creation |
| hex_random_length | int | No | 4 | The length of %{hex_random} placeholder(4-16) |
| index_format | string | No | %d | sprintf format for %{index} |
| warn_for_delay | string | No | - | Given a threshold to treat events as delay, output warning logs if delayed events were put into OSS |
| format | *Format | No | - | Format |
| buffer | *Buffer | No | - | Buffer |