Add docs to cover vector axiom sink changes and migration (#114)

Signed-off-by: Darach Ennis <[email protected]>
Co-authored-by: Mano Toth <[email protected]>
Co-authored-by: Mano Toth <[email protected]>

Author: darach

send-data/vector.mdx

@@ -16,9 +16,11 @@ Vector is a lightweight and ultra-fast tool for building observability pipelines
 
 Follow the [quickstart guide in the Vector documentation](https://vector.dev/docs/setup/quickstart/) to install Vector, and to configure sources and sinks.
 
-## timestamp field
+<Warning>
+If you use Vector version v0.41.1 or earlier, use the `@timestamp` field instead of `_time` to specify the timestamp of the events. For more information, see [Timestamp in legacy Vector versions](#timestamp-in-legacy-vector-versions).
 
-Logs and metrics sent via vector should use `@timestamp` as the timestamp field like: `{"@timestamp":"2022-04-14T21:30:30.658Z..."}`, not `_time`. Axiom accepts many date strings and timestamps without knowing the format in advance, including Unix Epoch, RFC3339, and ISO 8601.
+If you upgrade from Vector version v0.41.1 or earlier to a newer version, update your configuration. For more information, see [Upgrade from legacy Vector version](#upgrade-from-legacy-vector-version).
+</Warning>
 
 ## Configuration
 
@@ -65,6 +67,7 @@ type = "remap"
 inputs = ["VECTOR_SOURCE_ID"]
 source = '''
   . = del(.FIELD_TO_REMOVE)
+'''
 
 [sinks.SINK_ID]
 type = "axiom"
@@ -79,7 +82,7 @@ Replace `FIELD_TO_REMOVE` with the field you want to remove.
 Any changes to Vector’s `file` method can make the code example above outdated. If this happens, please refer to the [official Vector documentation on the `file` method](https://vector.dev/docs/reference/configuration/sources/file/), and we kindly ask you to inform us of the issue using the feedback tool at the bottom of this page.
 </Note>
 
-## Send Kubernetes logs to Axiom 
+## Send Kubernetes logs to Axiom
 
 Send Kubernetes logs to Axiom using the Kubernetes source.
 
@@ -117,7 +120,7 @@ TOKEN is used to ingest or query data to your dataset. API token can be generate
 
 [See creating an API token for more](/reference/tokens)
 
-## Send Docker logs to Axiom 
+## Send Docker logs to Axiom
 
 To send Docker logs using the Axiom sink, you need to create a configuration file, for example, `vector.toml`, with the following content:
 
@@ -145,7 +148,7 @@ vector --config /path/to/vector.toml
 
 Vector collects logs from Docker and forward them to Axiom using the Axiom sink. You can view and analyze your logs in your dataset.
 
-## Send AWS S3 logs to Axiom 
+## Send AWS S3 logs to Axiom
 
 To send AWS S3 logs using the Axiom sink, create a configuration file, for example, `vector.toml`, with the following content:
 
@@ -155,7 +158,6 @@ type = "aws_s3"
 bucket = "my-bucket"  # replace with your bucket name
 region = "us-west-2"  # replace with the AWS region of your bucket
 
-  
 [sinks.axiom]
 type = "axiom"
 inputs = ["my_s3_source"]
@@ -165,7 +167,7 @@ token = "your_api_token"  # replace with your Axiom API token
 
 Finally, run Vector with the configuration file using `vector --config ./vector.toml`. This starts Vector and begins reading logs from the specified S3 bucket and sending them to the specified Axiom dataset.
 
-## Send Kafka logs to Axiom 
+## Send Kafka logs to Axiom
 
 To send Kafka logs using the Axiom sink, you need to create a configuration file, for example, `vector.toml`, with the following code:
 
@@ -186,7 +188,7 @@ token = "your_api_token"  # replace with your Axiom API token
 
 Finally, you can start Vector with your configuration file: `vector --config /path/to/your/vector.toml`
 
-## Send NGINX metrics to Axiom 
+## Send NGINX metrics to Axiom
 
 To send NGINX metrics using Vector to the Axiom sink, first enable NGINX to emit metrics, then use Vector to capture and forward those metrics. Here is a step-by-step guide:
 
@@ -230,7 +232,7 @@ token = "your_api_token"  # replace with your Axiom API token
 
 Finally, you can start Vector with your configuration file: `vector --config /path/to/your/vector.toml`
 
-## Send Syslog logs to Axiom 
+## Send Syslog logs to Axiom
 
 To send Syslog logs using the Axiom sink, you need to create a configuration file, for example, `vector.toml`, with the following code:
 
@@ -244,11 +246,11 @@ mode="tcp"
 [sinks.axiom]
 type="axiom"
 inputs = [ "my_source_id" ] # required
-dataset="your_dataset_name" # replace with the name of your Axiom dataset 
+dataset="your_dataset_name" # replace with the name of your Axiom dataset
 token="your_api_token" # replace with your Axiom API token
 ```
 
-## Send Prometheus metrics to Axiom 
+## Send Prometheus metrics to Axiom
 
 To send Prometheus scrape metrics using the Axiom sink, you need to create a configuration file, for example, `vector.toml`, with the following code:
 
@@ -266,4 +268,69 @@ dataset = "your_prometheus_dataset"  # replace with the name of your Axiom datas
 token = "your_api_token"  # replace with your Axiom API token
 ```
 
-Check out the [advanced configuration on Batch, Buffer configuration, and Encoding on Vector Documentation](https://vector.dev/docs/reference/configuration/sinks/axiom/)
+Check out the [advanced configuration on Batch, Buffer configuration, and Encoding on Vector Documentation](https://vector.dev/docs/reference/configuration/sinks/axiom/)
+
+## Timestamp in legacy Vector versions
+
+If you use Vector version v0.41.1 or earlier, use the `@timestamp` field instead of `_time` to specify the timestamp in the event data you send to Axiom. For example: `{"@timestamp":"2022-04-14T21:30:30.658Z..."}`. For more information, see [Requirements of the timestamp field](/reference/field-restrictions#requirements-of-the-timestamp-field). In the case of Vector version v0.41.1 or earlier, the requirements explained on the page apply to the `@timestamp` field, not to `_time`.
+
+If you use Vector version v0.42.0 or newer, use the `_time` field as usual for other collectors.
+
+### Upgrade from legacy Vector version
+
+If you upgrade from Vector version v0.41.1 or earlier to a newer version, change all references from the `timestamp` field to the `_time` field and remap the logic.
+
+Example `vrl` file:
+
+```vrl example.vrl
+# Set time explicitly rather than allowing Axiom to default to the current time
+. = set!(value: ., path: ["_time"],  data: .timestamp)
+
+# Remove the original value as it is effectively a duplicate
+del(.timestamp)
+```
+
+Example Vector configuration file:
+
+```toml
+# ...
+
+[transforms.migrate]
+type = "remap"
+inputs = [ "k8s"]
+file= 'example.vrl' # See above
+
+[sinks.debug]
+type = "axiom"
+inputs = [ "migrate" ]
+dataset = "my-axiom-dataset" # No change
+token = "your-token" # No change
+
+[sinks.debug.encoding]
+codec = "json"
+```
+
+### Set compression algorithm
+
+Upgrading to Vector version v0.42.0 or newer automatically enables the `zstd` compression algorithm by default.
+
+To set another compression algorithm, use the example below:
+
+```toml
+# ...
+
+[transforms.migrate]
+type = "remap"
+inputs = [ "k8s"]
+file= 'example.vrl' # See above
+
+[sinks.debug]
+type = "axiom"
+compression = "gzip" # Set the compression algorithm
+inputs = [ "migrate" ]
+dataset = "my-axiom-dataset" # No change
+token = "your-token" # No change
+
+[sinks.debug.encoding]
+codec = "json"
+```