Skip to content

development: guide: final style #2013

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions development/developer-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ For cryptographic support, Fluent Bit uses the system installed version of OpenS

When you write Fluent Bit code, you'll use the Fluent Bit versions of the standard C functions for working with memory:

* [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `malloc`, allocates memory.
* [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `calloc`, allocates memory and initializes it to zero.
* [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `realloc`.
* [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `free`, releases allocated memory.
- [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `malloc`, allocates memory.
- [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `calloc`, allocates memory and initializes it to zero.
- [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `realloc`.
- [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `free`, releases allocated memory.

{% hint style="info" %}
Many types have specialized create and destroy functions, like [`flb_sds_create()` and `flb_sds_destroy()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h).
Expand All @@ -33,9 +33,9 @@ In general, you should use SDS strings in any string processing code. SDS string

Fluent Bit has its own network connection library. The key types and functions are defined in the following header files:

* [`flb_upstream.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h)
* [`flb_http_client.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h)
* [`flb_io.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h)
- [`flb_upstream.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h)
- [`flb_http_client.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h)
- [`flb_io.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h)

The following code demonstrates an HTTP request in Fluent Bit:

Expand Down Expand Up @@ -318,7 +318,8 @@ The [stdout plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out
Fluent Bit provides a standalone environment for development. Developers who use different operating systems or distributions can develop on a basic, common stack. The development environment provides the required libraries and tools for you.

Development environments are provided for:
- [Devcontainer](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#devcontainer)

- [Development container](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#devcontainer)
- [Vagrant](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#vagrant).

## Testing
Expand Down
14 changes: 7 additions & 7 deletions development/library_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ Fluent Bit is written in C and can be used from any C or C++ application.

Fluent Bit runs as a service, which means that the exposed API provides interfaces to create and manage contexts, specify inputs and outputs, set configuration parameters, and set routing paths for events or records. A typical usage of this library involves:

* Creating library instance and contexts and setting their properties.
* Enabling input plugins and setting their properties.
* Enabling output plugins and setting their properties.
* Starting the library runtime.
* Optionally ingesting records manually.
* Stopping the library runtime.
* Destroying library instances and contexts.
- Creating library instance and contexts and setting their properties.
- Enabling input plugins and setting their properties.
- Enabling output plugins and setting their properties.
- Starting the library runtime.
- Optionally ingesting records manually.
- Stopping the library runtime.
- Destroying library instances and contexts.

## Data types

Expand Down
15 changes: 7 additions & 8 deletions development/msgpack-format.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,26 @@
# Msgpack format
# MessagePack format

Fluent Bit **always** handles every Event message as a structured message using a binary serialization data format called [MessagePack](https://msgpack.org/).
Fluent Bit always handles every Event message as a structured message using a binary serialization data format called [MessagePack](https://msgpack.org/).

## Fluent Bit usage

MessagePack is a standard and well-defined format, refer to the official documentation for full details.
This section provides an overview of the specific types used by Fluent Bit within the format to help anyone consuming it.

* The data structure used by Fluent Bit is a 2-length [`fixarray`](https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family) of the timestamp and the data.
* The timestamp comes from [`flb_time_append_to_msgpack`])(https://github.com/fluent/fluent-bit/blob/2138cee8f4878733956d42d82f6dcf95f0aa9339/src/flb_time.c#L197), so it's either a `uint64`, a `float64`, or a [`fixext`](https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family) where the 4 MSBs are the seconds (big-endian `uint32`) and 4 LSBs are nanoseconds.
* The data itself is just a [`msgpack` map](https://github.com/msgpack/msgpack/blob/master/spec.md#map-format-family) with the keys as strings.
- The data structure used by Fluent Bit is a 2-length [`fixarray`](https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family) of the timestamp and the data.
- The timestamp comes from [`flb_time_append_to_msgpack`](https://github.com/fluent/fluent-bit/blob/2138cee8f4878733956d42d82f6dcf95f0aa9339/src/flb_time.c#L197), so it's either a `uint64`, a `float64`, or a [`fixext`](https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family) where the 4 MSBs are the seconds (big-endian `uint32`) and 4 LSBs are nanoseconds.
- The data itself is a [`msgpack` map](https://github.com/msgpack/msgpack/blob/master/spec.md#map-format-family) with the keys as strings.

## Example

Set up Fluent Bit to send in `msgpack` format to a specific port.

```bash
docker run --rm -it --network=host fluent/fluent-bit /fluent-bit/bin/fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=msgpack -v

```

We could send this to stdout but as it is a serialized format you would end up with strange output.
As an example we use the [Python msgpack library](https://msgpack.org/#languages) to deal with it:
You could send this to stdout but as it's a serialized format you would end up with strange output.
As an example, use the [Python `msgpack` library](https://msgpack.org/#languages) to handle it:

```python
#Python3
Expand Down
40 changes: 24 additions & 16 deletions development/wasm-filter-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ There are no additional requirements to execute Wasm plugins.

#### Build `flb-wamrc` (optional)

`flb-wamrc` is a `flb`-prefixed AOT (ahead of time) compiler that's provided from [`wasm-micro-runtime`](https://github.com/bytecodealliance/wasm-micro-runtime).
`flb-wamrc` is a `flb`-prefixed Ahead of Time (AOT) compiler that's provided from [`wasm-micro-runtime`](https://github.com/bytecodealliance/wasm-micro-runtime).

For `flb-wamrc` support, you must install the LLVM infrastructure and some additional libraries (`libmlir`, `libPolly`, `libedit`, and `libpfm`). For example:

Expand All @@ -22,13 +22,13 @@ For `flb-wamrc` support, you must install the LLVM infrastructure and some addit

### For Wasm programs

Currently, Fluent Bit supports the following Wasm tool chains:
Fluent Bit supports the following Wasm tool chains:

* Rust on `wasm32-unknown-unknown`
* rustc 1.62.1 (e092d0b6b 2022-07-16) or later
* [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi`
* v0.24.0 or later
* [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later.
- Rust on `wasm32-unknown-unknown`
- rustc 1.62.1 (`e092d0b6b` 2022-07-16) or later
- [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi`
- v0.24.0 or later
- [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later.

## Get started

Expand All @@ -44,8 +44,13 @@ To support AOT-compiled Wasm execution as filter plugins, build Fluent Bit with

Once compiled, you can see new plugins that handle `wasm`, for example:

```shell
bin/fluent-bit -h
```

Which returns results similar to:

```text
$ bin/fluent-bit -h
Usage: fluent-bit [OPTION]
Inputs
# ... other input plugin stuffs
Expand All @@ -60,9 +65,9 @@ Filters

The Fluent Bit Wasm filter assumes C ABI, also known as `wasm32-unknown-unknown` on Rust target and `wasm32-wasi` on TinyGo target.

### To Install Additional Components
### Install additional components

TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-unknown-unknown` target, you must install `wasm32-unknown-unknown` by using [rustup](https://rustup.rs/). Then, install the target components as follows:
TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-unknown-unknown` target, you must install `wasm32-unknown-unknown` by using [`rustup`](https://rustup.rs/). Then, install the target components as follows:

```text
rustup target add wasm32-unknown-unknown
Expand Down Expand Up @@ -116,7 +121,7 @@ pipeline:
function_name: super_awesome_filter
# Note: run Fluent Bit from the 'wasm_path' location.
accessible_paths: /path/to/fluent-bit

outputs:
- name: stdout
match: '*'
Expand Down Expand Up @@ -152,7 +157,8 @@ For example, one of the sample [Rust Wasm filters](https://github.com/fluent/flu
[0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}]
[0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}]
```
Another example of a Rust Wasm filter is the [flb_filter_iis](https://github.com/kenriortega/flb_filter_iis) filter.

Another example of a Rust Wasm filter is the [`flb_filter_iis`](https://github.com/kenriortega/flb_filter_iis) filter.

This filter takes the [Internet Information Services (IIS)](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis) [w3c logs](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis#select-w3c-fields-to-log) (with some custom modifications) and transforms the raw string into a standard Fluent Bit JSON structured record.

Expand All @@ -172,7 +178,7 @@ pipeline:
wasm_path: /plugins/flb_filter_iis_wasm.wasm
function_name: flb_filter_log_iis_w3c_custom
accessible_paths: .

outputs:
- name: stdout
match: 'iis.*'
Expand Down Expand Up @@ -213,6 +219,7 @@ The output after the filter logic will be:
```

This filter approach offers several advantages inherent to programming languages. For example:

- It can be extended by adding type conversion to fields, such as `sc_bytes`, `cs_bytes`, and `time_taken`. You can use this to validate data results.
- It allows for the use of conditions to apply more descriptive filters. For example, you can get only all logs that contain status codes higher than `4xx` or `5xx`.
- It can be used to define allow lists and deny lists using a data structure array or a file to store predefined IP addresses.
Expand Down Expand Up @@ -254,6 +261,7 @@ AOT compiling should generate CPU architecture-dependent objects. If you want to

### Further examples

* [C filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c)
* [Rust Filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust)
* [TinyGo filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go)
- [C filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c)
- [Rust Filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust)
- [TinyGo
filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go)
23 changes: 12 additions & 11 deletions development/wasm-input-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ There are no additional requirements to execute Wasm plugins.

### For Wasm programs

Fluent Bit supports the following Wasm toolchains:
Fluent Bit supports the following Wasm tool chains:

* Rust on `wasm32-unknown-unknown`
* rustc 1.62.1 (e092d0b6b 2022-07-16) or later
* [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi`
* v0.24.0 or later
* [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later
- Rust on `wasm32-unknown-unknown`
- rustc 1.62.1 (`e092d0b6b` 2022-07-16) or later
- [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi`
- v0.24.0 or later
- [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later

## Get started

Expand Down Expand Up @@ -50,7 +50,7 @@ Wasm input in Fluent Bit assumes WASI ABI, also known as `wasm32-wasi` on Rust t

### Install additional components

TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-wasi` target, you must install `wasm32-wasi` by using [rustup](https://rustup.rs/). Then, install the target components as:
TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-wasi` target, you must install `wasm32-wasi` by using [`rustup`](https://rustup.rs/). Then, install the target components as:

```text
rustup target add wasm32-wasi
Expand All @@ -74,10 +74,10 @@ pipeline:
tag: exec.wasi.local
wasi_path: /path/to/wasi_built_json.wasm
# For security reasons, WASM/WASI program cannot access its outer world
# without accessible permissions. Uncomment below 'accessible_paths' and
# without accessible permissions. Uncomment below 'accessible_paths' and
# run Fluent Bit from the 'wasi_path' location:
# accessible_paths /path/to/fluent-bit

outputs:
- name: stdout
match: '*'
Expand All @@ -92,7 +92,7 @@ pipeline:
Tag exec.wasi.local
WASI_Path /path/to/wasi_built_json.wasm
# For security reasons, WASM/WASI program cannot access its outer world
# without accessible permissions. Uncomment below 'accessible_paths' and
# without accessible permissions. Uncomment below 'accessible_paths' and
# run Fluent Bit from the 'wasi_path' location:
# accessible_paths /path/to/fluent-bit

Expand All @@ -104,4 +104,5 @@ pipeline:
{% endtab %}
{% endtabs %}

For an example that handles structured logs, see the [Rust `serde-json` example](https://github.com/fluent/fluent-bit/tree/master/examples/wasi_serde_json).
For an example that handles structured logs, see the [Rust `serde-json`
example](https://github.com/fluent/fluent-bit/tree/master/examples/wasi_serde_json).
3 changes: 3 additions & 0 deletions vale-styles/FluentBit/Headings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ exceptions:
- GCP
- GitLab
- GitHub
- Go
- Google
- Google Cloud
- Google Cloud BigQuery
Expand Down Expand Up @@ -82,6 +83,7 @@ exceptions:
- LTSV
- macOS
- Marketplace
- MessagePack
- Microsoft Fabric
- MongoDB
- NATS
Expand Down Expand Up @@ -119,5 +121,6 @@ exceptions:
- Visual
- Vivo Exporter
- VS
- Wasm
- WebSocket
- Windows