Skip to content

Use Node-IPs for external reachability and localities #641

@adwk67

Description

@adwk67
Member

Scope

This issue involves the following topics:

  • ensuring that HBase is reachable externally via listener endpoints (see parent issue)
  • ensure that these endpoints (or a load balancer that uses them) is consistently used in the UI screens
  • ensure the HBase advertises the nodes it is running on so that e.g. an accurate locality can be derived (see feat: HBase resolvable endpoints docker-images#1159)

Image

UI endpoint resolution - previous notes

UI endpoint resolution

The endpoints exposed in the HBase UI (mainly the master) are not resolvable for end-users. This means that, while it is possible to use the listener classes to write discovery information to ConfigMaps (so that users can easily reach master/regionserver/rest-server UIs), links referenced in these webpages - most obviously the list of region servers - are dead.

Context

  • Hbase uses the following internally for hostname and port resolution. If these settings are not in hbase-site.xml then a default mechanism is used.
    • hbase.{ master | regionserver| rest }.{ hostname | port }
  • the default mechanism involves a DNS lookup (see here and here), which also requires that reverse DNS lookups not be disabled. See here.
  • the endpoint information written to Zookeeper is what is used for listing the regionservers, and is also used for internal binding
    • this causes problems as HBase does not have access to the host network
  • tweaking internal details for the sake of user-facing endpoints runs the risk of breaking things, so a safer solution would be to patch HBase to use listener information only for the purpose of providing externally-reachable endpoints

Proposed solution

Node advertisement

  • the aim is to use the proto struct extensions when calculating localities etc. Not yet researched.

Activity

self-assigned this
on Apr 15, 2025
adwk67

adwk67 commented on Apr 23, 2025

@adwk67
MemberAuthor

Notes

HBase master- (HMaster extends HRegionServer) and region-servers (HRegionServer) do not have a direct connection: server data is persisted/shared in ZooKeeper and also in HBase meta tables. The data in ZooKeeper uses protobuf entities which serialize from / deserialize to ServerName objects, which are used in a variety of places in the code to share server data (servlet rendering, RPC calls, ZooKeeper persistence etc. etc.).

Listener data (hostname and port) can be written to custom config settings in hbase-site.xml on start-up, once the listener data has been written to the pods. These settings can then be retrieved for each server. However, the host/port are used for internal RPC calls as well as UI data, and node IPs (in the case of external-stable/external-unstable listener classes) are not reachable without giving access to the host network.

Extending proto entities

Extending ServerName (and the associated protobuf class) to add two new fields - externalHostname and externalPort - has the advantage of keeping that data with other server meta data: when the active master changes (or when region servers are added) the existing mechanisms for tracking these changes are unaffected. However, this is fairly invasive as ServerName is the object used in multiple places in the code. If we want to use these fields in a just-in-time manner, then MasterStatusServlet needs to be patched to replace hostname/port with the "external" values. This is not that simple, though, as the whole ServerName object is used as the key for internal caching, and the link is thus broken between the server name and the metrics that should also be rendered in the UI.

Changes to UI rendering

Alternatively, we can leave the MasterStatusServlet code as-is (i.e. do not attempt to replace host with external host etc.) and make the changes in the .jamon and .jsp files, where the host/port are ready out and displayed directly.

Patch Status

There is a WIP-patch available in this branch (associated changes for hbase-operator changes are here).

adwk67

adwk67 commented on Apr 24, 2025

@adwk67
MemberAuthor

I've asked about this on the dev mailing list: https://lists.apache.org/thread/rx7box0f7b02pg0xnyhlh0sj5b7zp6td

adwk67

adwk67 commented on Apr 29, 2025

@adwk67
MemberAuthor

Moving this into Development:Track with the following comments:

  1. the WIP patch works, apart from the parts of the UI that return information from RegionReplicaInfo: it is unclear where the ServerName (which currently is missing the externalHostname and externalPort fields in the patch) is being written for these objects and fixing this will probably require a further change to proto entities (i.e getting ever more invasive)
  2. if listener endpoints are used within HBase .jsp code, this will make it difficult to get things to work with a loadbalancer or Ingress, as those will be the URLs that should be displayed in the UI
  3. this could be achieved by writing the load balancer endpoint to hbase-site.xml at start-up via the entrypoint script
  4. there are various things hard coded in the patch, which the user should be able to "turn off" - doable, but not done
  5. most pertinently: if it possible to use Openshift Routes for endpoint resolution (which it is), it may be possible to do a similar thing with non-Openshift clusters

Points 2. and 5. are the ones which make me doubt that this (patching HBase) is the correct/best approach.

moved this to Development: Track in Stackable Engineeringon Apr 29, 2025
changed the title [-]Make HBase UI endpoints resolvable[/-] [+]Make HBase externally reachable[/+] on Apr 30, 2025
changed the title [-]Make HBase externally reachable[/-] [+]Use Node-IPs for external reachability and localities[/+] on Apr 30, 2025
moved this from Development: Track to Development: In Progress in Stackable Engineeringon May 19, 2025
moved this from In Progress to Done in Stackable End-to-End Coordinationon Jul 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Development

Participants

@adwk67

Issue actions

    Use Node-IPs for external reachability and localities · Issue #641 · stackabletech/hbase-operator