This issue is related to org.apache.storm.zookeeper.ClientZookeeper
Sniffing inbound ZooKeeper traffic during normal cluster-state polling shows a paired sequence for each read path.
Actually, when Storm reads cluster state from ZooKeeper, it first asks "does this node exist?" and only then asks "give me its data", two requests to the zk for a single logical read.
The initial check is unnecessary: the read call already returns the data when the node exists and reports a "no node" result when it doesn't. As a result, every value Storm reads (assignments, supervisor info, heartbeats, errors) sends zk roughly twice the requests it needs.
This is not a tuple data-plan performance issue; these are coordination reads. It is an avoidable request volume against the zk.
I propose to fix these methods with the pattern attached:
- getData
- getDataWithVersion:
- getVersion:
current
GIVEN path, wantWatch:
if EXISTS path (requesting a watch if wantWatch): // request #1
result <- READ path (requesting a watch if wantWatch) // request #2
return result.value
else:
return EMPTY // exists-watch (if requested) is already set
proposed
GIVEN path, wantWatch:
result <- READ path (requesting a watch if wantWatch) // request #1
if result is present:
return result.value // watch already set if requested
else: // node is absent (e.g. `NoNodeException`)
if wantWatch:
WATCH path // request #2, only here
return EMPTY
on any other error:
FAIL
This issue is related to
org.apache.storm.zookeeper.ClientZookeeperSniffing inbound ZooKeeper traffic during normal cluster-state polling shows a paired sequence for each read path.
Actually, when Storm reads cluster state from ZooKeeper, it first asks "does this node exist?" and only then asks "give me its data", two requests to the zk for a single logical read.
The initial check is unnecessary: the read call already returns the data when the node exists and reports a "no node" result when it doesn't. As a result, every value Storm reads (assignments, supervisor info, heartbeats, errors) sends zk roughly twice the requests it needs.
This is not a tuple data-plan performance issue; these are coordination reads. It is an avoidable request volume against the zk.
I propose to fix these methods with the pattern attached:
current
proposed