Kube State Metrics CustomResourceState configurations for Gateway API resources.
The contents of this repository are intended to be used with kube-state-metrics. To use the CustomResourceState, see the configuration information for how to configure an existing or separate kube-state-metrics instance with a custom resource configuration.
The CustomResourceState is available at ./config/default/custom-resource-state.yaml
For easier consumption via kustomize, a ./kustomization.yaml
is available that generates a ConfigMap named custom-resource-state with the
CustomResourceState data in a key called custom-resource-state.yaml.
An example of how to use this with kube-promethues in shown in ./config/examples/kube-prometheus The kustomization config does the following:
- mounts the ConfigMap as a volume in the kube-state-metrics Deployment
- adds the
--custom-resource-state-config-filearg to the container, referencing the file in the ConfigMap - patches the
kube-state-metricsClusterRole with permissions forcustomresourcedefinitionsand the various Gateway API resources in thegateway.networking.k8s.ioapiGroup - changes the kube-state-metrics image to a version that supports CustomResourceState and has various issues fixed.
- includes example Grafana dashboards
- includes example Prometheus Alert rules
A set of Grafana dashboards are available in ./config/examples/dashboards and on grafana.com You can import them directly into Grafana and modify as needed. The dashboards are divided by resources (GatewayClasses, Gateways, HTTPRoutes, GRPCRoutes, TLSRoutes, TCPRoutes and UDPRoutes), with variables for filtering, and links to drill down from a GatewayClass to a Gateway to all the routes.
https://grafana.com/grafana/dashboards/19432
https://grafana.com/grafana/dashboards/19433
https://grafana.com/grafana/dashboards/19434
https://grafana.com/grafana/dashboards/19570
https://grafana.com/grafana/dashboards/19572
https://grafana.com/grafana/dashboards/19571
https://grafana.com/grafana/dashboards/19573
A set of example Alerts are available in ./config/examples/rules. You can create the PrometheusRule resource directly or modify it as needed.
All metrics have GVK labels to allow more specific matching and filtering. For example:
{
customresource_group="gateway.networking.k8s.io"
customresource_kind="Gateway",
customresource_version="v1beta1"
}
In addition, all metrics have a name and namespace label, where applicable:
{
namespace="<NAMESPACE>",
name="<NAME>"
}
All metrics are prefixed with gatewayapi_.
For example, gatewayapi_gateway_status.
Information about a Gateway, Gauge
gatewayapi_gateway_info{namespace="<NAMESPACE>",name="<GATEWAY>",gatewayclass="<GATEWAYCLASS_NAME>"} 1
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_gateway_labels{namespace="<NAMESPACE>",name="<GATEWAY>",GATEWAY_LABEL_NAME="<GATEWAY_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_gateway_created{namespace="<NAMESPACE>",name="<GATEWAY>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_gateway_deleted{namespace="<NAMESPACE>",name="<GATEWAY>"} 1690879977
Per Listener information, Gauge
gatewayapi_gateway_listener_info{namespace="<NAMESPACE>",name="<GATEWAY>",listener_name="<LISTENER_NAME>",port="<PORT>",protocol="<PROTOCOL>",hostname="<HOSTNAME>","tls_mode"="<Passthrough|Terminate>"} 1
Status Conditions of Gateway, Gauge, 1 or 0 (1 means this condition type currently applies to this gateway)
gatewayapi_gateway_status{namespace="<NAMESPACE>",name="<GATEWAY>",type="<Accepted|Scheduled|Ready|Other>"} 1
Number of attached routes for an individual listener, Gauge
gatewayapi_gateway_status_listener_attached_routes{namespace="<NAMESPACE>",name="<GATEWAY>",listener_name="<LISTENER_NAME>"} 5
Address types and values, Gauge
gatewayapi_gateway_status_address_info{namespace="<NAMESPACE>",name="<GATEWAY>",type="<IPAddress|Hostname|DOMAIN_PREFIXED_STRING_IDENTIFIER>",value="<ADDRESS>"} 1
Information about a Gateway, Gauge
gatewayapi_gatewayclass_info{name="<GATEWAYCLASS_NAME>"} 1
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_gatewayclass_labels{name="<GATEWAYCLASS_NAME>",GATEWAYCLASS_LABEL_NAME="<GATEWAYCLASS_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_gatewayclass_created{name="<GATEWAYCLASS_NAME>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_gatewayclass_deleted{name="<GATEWAYCLASS_NAME>"} 1690879977
Status Conditions of GatewayClass, Gauge, 1 or 0 (1 means this condition type currently applies to this gateway)
gatewayapi_gatewayclass_status{name="<GATEWAYCLASS_NAME>",type="<Accepted>"} 1
gatewayapi_gatewayclass_status_supported_features{name="<GATEWAYCLASS_NAME>",features="<FeatureX>"}
NOTE There is no _info label for HTTPRoute as no additional information is available to show outside of what's already in below metrics.
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_httproute_labels{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>",HTTPROUTE_LABEL_NAME="<HTTPROUTE_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_httproute_created{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_httproute_deleted{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>"} 1690879977
Hostnames to match against the HTTP Host header, Gauge
gatewayapi_httproute_hostname_info{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>",hostname="<HOSTNAME>"}
Parent References that the route wants to be attached to, Gauge
gatewayapi_httproute_parent_info{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Parent References that the route is attached to based on the status, Gauge
gatewayapi_httproute_status_parent_info{name="<HTTPROUTE_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_grpcroute_labels{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>",grpcroute_LABEL_NAME="<grpcroute_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_grpcroute_created{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_grpcroute_deleted{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Hostnames to match against the HTTP Host header, Gauge
gatewayapi_grpcroute_hostname_info{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>",hostname="<HOSTNAME>"}
Parent References that the route wants to be attached to, Gauge
gatewayapi_grpcroute_parent_info{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Parent References that the route is attached to based on the status, Gauge
gatewayapi_grpcroute_status_parent_info{name="<GRPCRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_tcproute_labels{name="<TCPRoute_NAME>",namespace="<NAMESPACE>",tcproute_LABEL_NAME="<tcproute_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_tcproute_created{name="<TCPRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_tcproute_deleted{name="<TCPRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Parent References that the route wants to be attached to, Gauge
gatewayapi_tcproute_parent_info{name="<TCPRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Parent References that the route is attached to based on the status, Gauge
gatewayapi_tcproute_status_parent_info{name="<TCPRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_tlsroute_labels{name="<TLSRoute_NAME>",namespace="<NAMESPACE>",tlsroute_LABEL_NAME="<tlsroute_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_tlsroute_created{name="<TLSRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_tlsroute_deleted{name="<TLSRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Hostnames to match against the HTTP Host header, Gauge
gatewayapi_tlsroute_hostname_info{name="<TLSRoute_NAME>",namespace="<NAMESPACE>",hostname="<HOSTNAME>"}
Parent References that the route wants to be attached to, Gauge
gatewayapi_tlsroute_parent_info{name="<TLSRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Parent References that the route is attached to based on the status, Gauge
gatewayapi_tlsroute_status_parent_info{name="<TLSRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Kubernetes labels converted to Prometheus labels, Gauge
gatewayapi_udproute_labels{name="<UDPRoute_NAME>",namespace="<NAMESPACE>",udproute_LABEL_NAME="<udproute_LABEL_VALUE>"} 1
Unix creation timestamp in seconds, Gauge
gatewayapi_udproute_created{name="<UDPRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Unix deletion timestamp in seconds, Gauge
gatewayapi_udproute_deleted{name="<UDPRoute_NAME>",namespace="<NAMESPACE>"} 1690879977
Parent References that the route wants to be attached to, Gauge
gatewayapi_udproute_parent_info{name="<UDPRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Parent References that the route is attached to based on the status, Gauge
gatewayapi_udproute_status_parent_info{name="<UDPRoute_NAME>",namespace="<NAMESPACE>",parent_group="<PARENT_GROUP>",parent_kind="<PARENT_KIND>",parent_name="<PARENT_NAME>",parent_namespace="<PARENT_NAMESPACE>"}
Dashboards are written in jsonnet, and use the grafonnet library.
Resulting dashboard json files are checked in.
To generate dashboards, run make generate-dashboards.
Local development can be done using a combination of automatic jsonnet execution and using the grafana-operator to automatically update dashboards in Grafana. This allows for a relatviely fast development loop where you can change a dashboard jsonnet file, save it, then see the changes automically in a browser.
To set up the local development environment, run the following:
./hack/local_dev.shGrafonnet is a powerful tool, but it may not cover all scenarios in its documentation. If you encounter issues or roadblocks, don’t be afraid to experiment with different approaches. Learning through trial and error can often lead to better understanding and innovative solutions. Remember, every challenge is an opportunity to learn.
Grafana’s user interface can be an invaluable reference when working with transformations and overrides in Grafonnet. If you’re unsure about how to implement a specific feature in Grafonnet, try creating it in the Grafana UI first. Then, export the dashboard as JSON. This exported JSON can serve as a reference for how to implement the same feature in Grafonnet.
Consider creating abstract representations of panels that are used repeatedly across your dashboards. This not only helps avoid repetition but also ensures consistency in the layout and design of your panels.
Abstraction can also make your code more readable and maintainable, especially when dealing with complex input patterns. When deciding whether to create a new abstraction or reuse an existing one, keep the “Don’t Repeat Yourself” (DRY) principle in mind. If you find yourself writing similar code for multiple panels, it might be time to consider abstraction.
By following these guidelines, you can navigate Grafonnet development more effectively and efficiently. Remember, the goal is not just to create functional dashboards, but also to write clean, maintainable code that can be understood and modified by others.
Contributions are welcome in the form of bugs, feature requests & pull requests. Before submitting a new issue, please use the search function to see if there is already a similar report or pull request. Roadmap items will be represented as Issues.