Skip to content

OCPBUGS-55399: cluster-node-tuning-operator Pod of the hosted cluster is not listening on 60000 port which makes the target to be down #1346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion pkg/metrics/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ import (
"os"
"time"

ntoconfig "github.com/openshift/cluster-node-tuning-operator/pkg/config"

"k8s.io/klog/v2"

"github.com/prometheus/client_golang/prometheus/promhttp"
Expand Down Expand Up @@ -47,7 +49,11 @@ func init() {
// it does so only if the CA bundle changed from the current CA bundle on record.
func DumpCA(caBundle string) {
if caBundle != server.caBundle {
server.caBundleCh <- caBundle
select {
case server.caBundleCh <- caBundle:
default:
klog.Infof("Metrics server CA channel not ready, skipping CA bundle update")
}
Comment on lines +52 to +56
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is welcome and seems correct, but why is it needed in the context of this PR to change the metrics server?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just after retesting this without the change and it seems to work without it. When trying out fixes for this bug, I was of the belief that the NTO was blocked on this line as it would be trying to receive a response that would never come due to the switch to HTTP but it seems that it works just fine without it. Either way, I'm willing to either keep this block in this PR or open a new one with just this little piece of code. Nice catch 🙌

Copy link
Author

@georgelipceanu georgelipceanu Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to explore options before to resort to just disable tls and expose the port to everyone. I'm concerned about information leaking. How can we prevent it?

Regarding the switch to HTTP, from what I've found and experimented with, I found that the NTO needed RBAC permissions to get the ConfigMap from the hosted cluster's kube-system namespace while the Control-Plane-Operator in the hosted cluster, responsible for deploying the operator, did not have permission to create RBAC resources in the hosted cluster's kube-system namespace.

I personally wasn't able to find a way to keep HTTPS with the current configuration but I would be willing to take a deeper dive into it HTTP won't be secure enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR, IIRC, we had two issues with the metrics server reported to us by the Security team.

I'd be very reluctant to merge this without meeting both requirements.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, in that case I'll keep looking into keeping HTTPS with the metrics server for HyperShift. However if the new DumpCA() code looks good then I would be willing to repurpose this PR for this code only. Is this something that the team would be interested in?

}
}

Expand Down Expand Up @@ -126,6 +132,27 @@ func (Server) Start(ctx context.Context) error {
// and restarted with the current files. Every non-nil return from this function is fatal
// and will restart the whole operator.
func RunServer(port int, ctx context.Context) error {
if ntoconfig.InHyperShift() {
klog.Info("starting metrics server.")
handler := promhttp.HandlerFor(registry, promhttp.HandlerOpts{})
router := http.NewServeMux()
router.Handle("/metrics", handler)
srv := &http.Server{
Addr: fmt.Sprintf(":%d", port),
Handler: router,
}
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
klog.Errorf("error from metrics server: %v", err)
}
}()
<-ctx.Done()
klog.Info("stopping insecure metrics server")

shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return srv.Shutdown(shutdownCtx)
}
// Set up and start the file watcher.
watcher, err := fsnotify.NewWatcher()
if watcher == nil || err != nil {
Expand Down