Problem
Currently we check at the TCP level whether the NiFi port is open for readiness/startup.
That does not mean however that a node is connected to the cluster.
Value / why now
NiFi 2.x has a dedicated and separate management server which exposes an unauthenticated /health and a /health/cluster endpoint.
We can now replace the TCP check with one that's actually meaningful and allows for rolling restarts.
This has caused support tickets in the past.
Scope
Replace the probes for NiFi 2.x:
- startupProbe: gate on bootstrap via the management server
/health (200 once the app server has started). Keep the generous failure budget.
- readinessProbe: gate on cluster membership via
/health/cluster (200 only when CONNECTED/CONNECTING, else 503).
Both target the local management server (default 127.0.0.1:52020, plain HTTP) via an exec probe, since it binds localhost only.
Proposed probes:
.startup_probe(Probe {
period_seconds: Some(10),
failure_threshold: Some(120),
exec: Some(ExecAction { command: Some(vec![
"curl".into(), "-fsS".into(), "-o".into(), "/dev/null".into(),
"http://127.0.0.1:52020/health".into(),
])}),
..
})
// This is new
.readiness_probe(Probe {
period_seconds: Some(10),
failure_threshold: Some(3),
exec: Some(ExecAction { command: Some(vec![
"curl".into(), "-fsS".into(), "-o".into(), "/dev/null".into(),
"http://127.0.0.1:52020/health/cluster".into(),
])}),
..
})Done when
- On NiFi 2.x, a node reports Ready only once
/health/cluster returns 200, and startup is gated on /health.
- A node that is up but not yet cluster-connected is observably not Ready (check during a rolling restart).
Appetite
Expected effort: 1-2 days. If it takes longer, stop and flag it.
Open questions
- Check how the
/health/cluster endpoint behaves with a single-node cluster
- Confirm
/health only becomes reachable after the flow is loaded.
- Pin the management address explicitly (set
org.apache.nifi.management.server.address via bootstrap)
rather than rely on the 52020 default, for robustness across versions?
Problem
Currently we check at the TCP level whether the NiFi port is open for readiness/startup.
That does not mean however that a node is connected to the cluster.
Value / why now
NiFi 2.x has a dedicated and separate management server which exposes an unauthenticated
/healthand a/health/clusterendpoint.We can now replace the TCP check with one that's actually meaningful and allows for rolling restarts.
This has caused support tickets in the past.
Scope
Replace the probes for NiFi 2.x:
/health(200 once the app server has started). Keep the generous failure budget./health/cluster(200 only when CONNECTED/CONNECTING, else 503).Both target the local management server (default
127.0.0.1:52020, plain HTTP) via an exec probe, since it binds localhost only.Proposed probes:
Done when
/health/clusterreturns 200, and startup is gated on/health.Appetite
Expected effort: 1-2 days. If it takes longer, stop and flag it.
Open questions
/health/clusterendpoint behaves with a single-node cluster/healthonly becomes reachable after the flow is loaded.org.apache.nifi.management.server.addressvia bootstrap)rather than rely on the 52020 default, for robustness across versions?