|
1 | 1 | # DOCKER CONTAINER LOG COLLECTOR |
2 | | -Zebrium's docker container log collector collects container logs and sends logs to Zebrium for automated Incident detection. |
3 | | -Our github repository is located [here](https://github.com/zebrium/ze-docker-log-collector). |
4 | 2 |
|
5 | | -# ze-docker-log-collector |
| 3 | +Zebrium's docker container log collector collects container logs and sends logs to Zebrium for automated Incident detection. This is achieved by using the [Fluentd logging driver for Docker](https://docs.docker.com/config/containers/logging/fluentd/) and the Zebrium Fluentd [output plugin.](https://github.com/zebrium/fluentd-output-zebrium) |
6 | 4 |
|
7 | 5 | ## Getting Started |
8 | | -### Docker |
9 | | -Use the following command to create a docker log collector container: |
10 | | -``` |
11 | | -sudo docker run -d --name="zdocker-log-collector" --restart=always \ |
12 | | - -v=/var/run/docker.sock:/var/run/docker.sock \ |
13 | | - -e ZE_LOG_COLLECTOR_URL="<ZE_LOG_COLLECTOR_URL>" \ |
14 | | - -e ZE_LOG_COLLECTOR_TOKEN="<ZE_LOG_COLLECTOR_TOKEN>" \ |
15 | | - -e ZE_HOSTNAME="<HOSTNAME>" \ |
16 | | - -e ZE_DEPLOYMENT_NAME="YOUR_DEPLOYMENT_NAME_HERE" \ |
17 | | - zebrium/docker-log-collector:latest |
18 | | -``` |
19 | 6 |
|
20 | | -### Docker Compose |
21 | | -Use the following configuration file to deploy via docker-compose command: |
22 | | -``` |
23 | | -version: '3.5' |
24 | | -
|
25 | | -services: |
26 | | - zdocker-log-collector: |
27 | | - image: zebrium/docker-log-collector:latest |
28 | | - restart: always |
29 | | - volumes: |
30 | | - - /var/run/docker.sock:/var/run/docker.sock |
31 | | - environment: |
32 | | - ZE_LOG_COLLECTOR_URL: "<ZE_LOG_COLLECTOR_URL>" |
33 | | - ZE_LOG_COLLECTOR_TOKEN: "<ZE_LOG_COLLECTOR_TOKEN>" |
34 | | - ZE_DEPLOYMENT_NAME: "<YOUR_DEPLOYMENT_NAME_HERE>" |
35 | | - ZE_HOSTNAME: "<HOSTNAME>" |
36 | | -``` |
37 | | -### AWS Elastic Container Service (ECS) |
| 7 | +When sending your logs from your docker daemon to Zebrium, there are two configuration options for where your log collector can be installed in configured. The collector can be installed within the docker daemon context that you are sending all the logs from, or it could be installed on an external host, and have the logs routed to it by each docker daemon. |
38 | 8 |
|
39 | | -Add the following service to ECS on EC2 cluster configuration. |
| 9 | +### Deploying the Collector |
| 10 | + |
| 11 | +Regardless on the installation method, you will start the collector using the following command, substituting the token and URL in for the values found in your Zebrium Integration and Collectors page. Additional ENVS listed [below](#environment-variables) can be specified to the collector to further extend the functionality. |
| 12 | + |
| 13 | +```bash |
| 14 | +docker run -p 24224:24224 -e ZE_LOG_COLLECTOR_URL=<URL> -e ZE_LOG_COLLECTOR_TOKEN=<TOKEN> --restart always zebrium/docker-log-collector:latest |
40 | 15 | ``` |
41 | | -services: |
42 | | - zdocker-log-collector: |
43 | | - image: zebrium/docker-log-collector:latest |
44 | | - restart: always |
45 | | - volumes: |
46 | | - - /var/run/docker.sock:/var/run/docker.sock |
47 | | - environment: |
48 | | - ZE_LOG_COLLECTOR_URL: "<ZE_LOG_COLLECTOR_URL>" |
49 | | - ZE_LOG_COLLECTOR_TOKEN: "<ZE_LOG_COLLECTOR_TOKEN>" |
50 | | - ZE_DEPLOYMENT_NAME: "<YOUR_DEPLOYMENT_NAME_HERE>" |
| 16 | + |
| 17 | +### Configuring the Docker daemon |
| 18 | + |
| 19 | +Once our collector has been deployed and configured, we need to modify the docker daemon configuration to start sending logs to the collector. For a complete list of configuration options, please see the [official docker documentation](https://docs.docker.com/config/containers/logging/fluentd/). The docker daemon is located in `/etc/docker/daemon.json` on Linux host and `C:\ProgramData\docker\config\daemon.json` on windows host. For more about the docker daemon.json, see the [official documentation](https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file) |
| 20 | + |
| 21 | +Add the following configuration to your daemon.json file, substituting `<fluentd-address>` for the address of your log collector. If your log collector is deployed in the same docker daemon, then use `127.0.0.1:24224` as your address. |
| 22 | + |
| 23 | +```bash |
| 24 | +{ |
| 25 | +"log-driver": "fluentd", |
| 26 | + "log-opts": { |
| 27 | + "fluentd-address": "<fluentd-address>", |
| 28 | + "fluentd-async": "true" |
| 29 | + } |
| 30 | +} |
51 | 31 | ``` |
52 | | -To collect container logs from all nodes in an ECS cluster, zdocker-log-collector service must be configured to run as an ECS daemon task. Please follow the steps below to configure the daemon task: |
53 | | -1. Log in to the AWS console and navigate to the ECS Clusters section. Click into your cluster you run the Agent on. |
54 | | -2. Choose Service tab, click on the Create button. |
55 | | -3. For launch type, select EC2, for service type, select DAEMON, type a service name, and click on Next step. |
56 | | -4. For Load balance type option, select None, and click on Next step. On next page, select Next step without configuring Auto Scaling. |
57 | | -5. Review and click on Create Service. |
58 | | - |
59 | | -Please note ECS tasks must be configured to use 'json-file' Log Driver for Zebrium log collector to receive container logs. If there is special log configuration on ECS instances, for example, using UserData section on instance to set log configuration, those configurations may need to be modified or deleted. |
60 | | - |
61 | | -## Environment Variables |
62 | | -The following environment variables are supported by the collector: |
63 | | -<table> |
64 | | - <tr> |
65 | | - <th>Environment Variables</th> |
66 | | - <th>Description</th> |
67 | | - <th>Default value</th> |
68 | | - <th>Note</th> |
69 | | - </tr> |
70 | | - <tr> |
71 | | - <td>ZE_LOG_COLLECTOR_URL</td> |
72 | | - <td>Zebrium log host server URL</td> |
73 | | - <td>None. Must be set by user</td> |
74 | | - <td>Provided by Zebrium once your account has been created.</td> |
75 | | - </tr> |
76 | | - <tr> |
77 | | - <td>ZE_LOG_COLLECTOR_TOKEN</td> |
78 | | - <td>Authentication token</td> |
79 | | - <td>None. Must be set by user</td> |
80 | | - <td>Provided by Zebrium once your account has been created.</td> |
81 | | - </tr> |
82 | | - <tr> |
83 | | - <td>ZE_HOSTNAME</td> |
84 | | - <td>Hostname of docker host</td> |
85 | | - <td>Empty. Optional</td> |
86 | | - <td>If ZE_HOSTNAME is not set, container hostname is used as source host for logs.</td> |
87 | | - </tr> |
88 | | - <tr> |
89 | | - <td>ZE_MAX_INGEST_SIZE</td> |
90 | | - <td>Maximum size of post request for Zebrium log server</td> |
91 | | - <td>1048576 bytes. Optional</td> |
92 | | - <td>Unit is in bytes</td> |
93 | | - </tr> |
94 | | - <tr> |
95 | | - <td>ZE_FLUSH_TIMEOUT</td> |
96 | | - <td>Interval between sending batches of log data to Zebrium log server.</td> |
97 | | - <td>30 seconds. Optional</td> |
98 | | - <td>Unit is in seconds. Please note Zebrium output plugin sends data immediately to log server when accumulated data reaches ZE_MAX_INGEST_SIZE bytes.</td> |
99 | | - </tr> |
100 | | - <tr> |
101 | | - <td>ZE_FILTER_NAME</td> |
102 | | - <td>Collect logs for containers whose names match filter name pattern. These can include wildcards, for example, <i>my_container1*</i></td> |
103 | | - <td>Empty. Optional</td> |
104 | | - <td></td> |
105 | | - </tr> |
106 | | - <tr> |
107 | | - <td>ZE_FILTER_LABELS</td> |
108 | | - <td>Collect logs for containers whose labels match the labels as defined in ZE_FILTER_LABELS. The format is: <i>label1:label1_value,label2:label2_value</i> These can include wildcards, for example, <i>my_label:xyz*</i></td> |
109 | | - <td>Empty. Optional</td> |
110 | | - <td></td> |
111 | | - </tr> |
112 | | - |
113 | | -</table> |
114 | | - |
115 | | - |
116 | | -## Testing your installation |
117 | | -Once the docker log collector software has been deployed in your environment, your container logs and incident detection will be available in the Zebrium UI. |
118 | | - |
119 | | -## Contributors |
120 | | -* Brady Zuo (Zebrium) |
| 32 | + |
| 33 | +Once the daemon file is updated, restart the docker daemon for the new changes to take effect. After this, your should be able to view the logs of the log collector and verify that it is receiving and forwarding logs to Zebrium. |
| 34 | + |
| 35 | +### Environment Variables |
| 36 | + |
| 37 | +Below is a list of environment variables that are available for configuration of the Fluentd container. |
| 38 | + |
| 39 | +| Environment Variables | Default | Description | Required | |
| 40 | +|-------------------|-------------------|-------------------| ---| |
| 41 | +| ZE_LOG_COLLECTOR_URL | "" | Zebrium URL Endpoint for log ingestion| yes| |
| 42 | +| ZE_LOG_COLLECTOR_TOKEN | "" | Zebrium ZAPI token for log ingestion| yes| |
| 43 | +| ZE_DEPLOYMENT_NAME | "default" | Zebrium Service Group Name. Read more [here](https://docs.sciencelogic.com/zebrium/latest/Content/Web_Zebrium/Key_Concepts.html#service-groups)| no| |
| 44 | +| FLUSH_INTERVAL | "60s" | Buffer Flush Interval| no| |
| 45 | +| ZE_LOG_LEVEL | "info" | Sets the log level for the output plugin | no | |
| 46 | +| VERIFY_SSL | "true" | Enables or disables SSL verification on endpoint| no| |
| 47 | + |
| 48 | +## Additional Resources |
| 49 | + |
| 50 | +* [Github repository](https://github.com/zebrium/ze-docker-log-collector) |
| 51 | +* [Fluentd Documentation](https://www.fluentd.org/guides/recipes/docker-logging) |
| 52 | +* [Docker Plugin Documentation](https://docs.docker.com/config/containers/logging/fluentd/) |
0 commit comments