Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ assignees: ''

---

> [!CAUTION]
> Do not paste credentials into this issue. This includes Kubernetes `Secret` manifests,
> base64 or decoded secret values, private values files, tokens, private keys, and MongoDB
> URIs. The output of `helm get all`, `helm template`, and `kubectl get secret` can contain
> credentials. Redact them before you paste. This issue is public.
>
> To report a security vulnerability, contact the maintainers privately. Do not open a
> public issue.

## Summary
_Give a brief description of the problem._

Expand All @@ -26,8 +35,9 @@ _What is affected by this issue, and why does it matter?_
- Cloud provider / platform:

### Pre-flight checks
- [ ] `helm lint ./graylog` passes
- [ ] `helm lint ./charts/graylog` passes
- [ ] Checked existing issues for duplicates
- [ ] I removed every credential, token, and secret value from the text, logs, and screenshots in this issue

### Steps to reproduce the issue:
1. ...
Expand Down
49 changes: 45 additions & 4 deletions charts/graylog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,20 @@ and let `cert-manager` do the rest!

# Installation
## Pre Installation
> [!WARNING]
> If you are managing the deployment of this chart through GitOps (ArgoCD, Flux etc), you must manually generate the Graylog Secrets resource. Otherwise auto generated credentials will rotate after initial deployment. Refer to the [Graylog Secrets](../../docs/graylog-secrets.md) doc for requirements of that secret.

1. If Argo CD or Flux manages this release, create the Graylog `Secret` before you install the
chart. Then set `global.existingSecretName` to the name of that Secret. Make sure that the
Secret contains every required key in the [Graylog Secrets](../../docs/graylog-secrets.md)
guide. The optional keys in that guide are only necessary for the features that use them.
2. An external Secret requires an external MongoDB. Set `mongodb.communityResource.enabled=false`
and put the connection string in `GRAYLOG_MONGODB_URI`. The chart refuses to render an external
Secret together with the bundled MongoDB. See
[examples/values-existing-secret-external-mongodb.yaml](../../examples/values-existing-secret-external-mongodb.yaml)
for a complete example.

> [!CAUTION]
> Chart-generated credentials can change after the first deployment when a GitOps controller
> renders the chart. An externally managed Secret prevents this change.

## Installing on Kubernetes

Expand Down Expand Up @@ -363,10 +375,29 @@ It is best-effort. It cannot stop inputs, so on a node still receiving traffic t
journal never reaches zero and the hook gives up. It also cannot tell an upgrade
from a scale-in, so it slows both. Enable it if you routinely scale in.

> [!CAUTION]
> Do not enable the drain and reduce the replica count in one change. The preStop hook is
> part of the pod spec, so an existing pod does not get the hook until Kubernetes replaces
> it. Kubernetes deletes the highest-ordinal pod with its old spec, and that pod drains
> nothing.

Enable the drain before you scale in:

1. Set `graylog.lifecycle.preStopDrain.enabled=true` and keep the replica count unchanged.
2. Wait for every pod to carry the new spec. With the default
`graylog.updateStrategy.type: RollingUpdate`, run
`kubectl rollout status sts/graylog -n graylog`. With `OnDelete`, Kubernetes does not replace
the pods for you: delete each one, highest ordinal first, and wait for its replacement to
become ready before you delete the next.
3. Make sure that the highest-ordinal pod carries the hook. Run
`kubectl get pod graylog-<n> -n graylog -o jsonpath='{.spec.containers[0].lifecycle.preStop}'`.
An empty result means that the pod still runs the old spec. Do not scale in yet.
4. Reduce the replica count by one.

| Key | Description | Default |
|---|---|---|
| `enabled` | Hold termination while the journal drains. | `false` |
| `endpointPropagationDelaySeconds` | Sleep before the first sample, waiting for endpoint removal to propagate. Raise it if a load balancer targets pod IPs directly. | `15` |
| `endpointPropagationDelaySeconds` | Sleep before the first sample, waiting for endpoint removal to propagate. If a load balancer targets pod IPs directly, increase this value and `graylog.terminationGracePeriodSeconds` by the same number of seconds. | `15` |
| `shutdownReserveSeconds` | Held back out of the grace period for Graylog's own shutdown. | `45` |
| `pollIntervalSeconds` | Seconds between journal-depth samples. | `2` |
| `stallPolls` | Give up after this many polls with no new low. | `10` |
Expand Down Expand Up @@ -964,8 +995,18 @@ See [the included guide](../../docs/mongodb-backup-restore.md) if you need to ta
Graylog's MongoDB database and restore it with `mongorestore`.

# Uninstall

> [!WARNING]
> A scale-in to zero replicas stops every Graylog pod. The StatefulSet does not set
> `podManagementPolicy`, so Kubernetes deletes the pods one at a time, from the highest ordinal
> down. The automatic drain is off by default, so unprocessed messages stay in the journal on
> volumes that Helm does not delete.
>
> If the release still receives traffic, stop the inputs first. Then drain each pod. See
> [Scaling in safely](../../docs/graylog-message-handling.md#scaling-in-safely).

```sh
# optional: scale Graylog down to zero
# optional: scale Graylog down to zero, after the journals are drained
kubectl scale sts graylog -n graylog --replicas 0 && kubectl wait --for=delete pod graylog-0 -n graylog

# remove chart
Expand Down
20 changes: 11 additions & 9 deletions docs/graylog-message-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ ordinals first — StatefulSets terminate from the highest ordinal down.
zero. Endpoint removal alone does not stop long-lived TCP connections, UDP senders,
or internally generated inputs.

> [!WARNING]
> Do not send `DELETE` to `/api/system/inputs/<id>`. That endpoint deletes the input
> definition. To stop a running input, send `DELETE` to `/api/cluster/inputstates/<id>`.
> This call leaves the input configured, so a `PUT` on the same path starts it again.
> `/api/system/inputstates/<id>` stops the input on one node only.
> `/api/cluster/inputstates/<id>` covers every node, which is what quiesces the journal.

```sh
# list inputs
curl -su "admin:$PASS" http://localhost:9000/api/system/inputs | jq '.inputs[] | {id, title}'
Expand All @@ -102,13 +109,6 @@ curl -su "admin:$PASS" -H 'X-Requested-By: cli' \

Or **System > Inputs** in the UI. Requires admin auth.

> [!WARNING]
> Stop inputs via `inputstates`, never `DELETE /api/system/inputs/<id>` — that one is
> "Terminate input on this node" and removes the input definition. `inputstates` stops
> a running input and leaves it configured, so `PUT` on the same path starts it again.
> `/api/system/inputstates/<id>` is the node-scoped variant; `/api/cluster/inputstates/<id>`
> covers every node, which is what quiesces the journal.

Every state-changing Graylog API call needs an `X-Requested-By` header (any value) or
it fails with `CSRF protection header is missing`. That applies to the `lbstatus`
override below too.
Expand Down Expand Up @@ -219,8 +219,10 @@ Defaults: `300 - 15 - 45 = 240s`. The chart fails to render if that is not posit
**settle** is a plain sleep before the first sample. Endpoint removal is eventually
consistent, and measuring during that window reads ingest that is about to stop —
which skews the drain rate and can trip the feasibility abort on a pod that would
have drained fine. Raise it if a load balancer targets pod IPs directly (ALB/NLB IP
mode deregisters slower than kube-proxy), and raise the grace period with it.
have drained fine. If a load balancer targets pod IPs directly, increase
`graylog.lifecycle.preStopDrain.endpointPropagationDelaySeconds`. Then increase
`graylog.terminationGracePeriodSeconds` by the same number of seconds. ALB and NLB in
IP mode deregister slower than kube-proxy.

### What it cannot do

Expand Down
Loading