Include note about upgrade steps for target allocator for operator v1beta1

[jaronoff97]

opentelemetry.io/content/en/blog/2024/prom-and-otel/index.md

Line 299 in 81c8e86

### Configuration

Hello! We had someone reach out on this issue mentioning that the docs are using the old version of the target allocator and collector CRDs (v1alpha1) and there is a notable breaking change between those versions for the target allocator. We have a document for the upgrade guide and it would be great to edit this blog post to mention that if you are using v1beta1 be sure to follow the upgrade guide. I'm not sure if these back-compatible notices are something we've done before, but it seems like users are being misled a bit right now.

[lenalebt]

I'm that one person having trouble setting everything up :-).

I followed https://opentelemetry.io/blog/2024/prom-and-otel/ and basically ignored the rest of the docs and upgrade guides, whereever they may be. I am specifically living in scaleway k8s right now (https://www.scaleway.com/en/kubernetes-kapsule/), where they have a managed prometheus for me, but (for reasons beyond my understanding) do not export standard metrics from kube-state-metrics nor node-exporter automatically (and they are not running by default in their clusters). Me wanting to install them and leverage otel-collector to ingest that into that managed prometheus is basically the outline of my task.

My main troubles were roughly:

• I was unsure whether I should expect a specific "targetallocator" pod or not. For some reason, that one did not get started, but I did not have any errors in logs. Mentioning that a pod should come up would help. I found the defined service, and no endpoints, which made me quite sure a pod should exist.
• The targetallocator pod, after I had it running, did not want to start because I only installed PodMonitor and ServiceMonitor CRDs (as the blog post mentioned). Through the existence of TargetAllocator missing scrape configs from ServiceMonitor/PodMonitor when using v1beta1 spec vs v1alpha1 opentelemetry-operator#3528 I learned that support for the other CRDs has been added, so I added them (and patched the ClusterRole mentioned in the post).

Still on it bringing it to life, will post an update if I had more troubles that could get mentioned in some docs...

[lenalebt]

Apparently it's really important to add this:

  podMonitorSelector: {}
  serviceMonitorSelector: {}

to the config of the targetallocator, as it won't pick up the service monitors otherwise. Did default values change? I dunno. This is the complete configuration part I applied:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: opentelemetry
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-targetallocator-sa
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      serviceMonitorSelector: {}
  config:
  # ...

Hope that helps :-). It is working now.

[svrnm]

@jaronoff97 @lenalebt thanks for reporting this! We treat blog posts as pages that decay over time, and they are true at the point of writing. If the blog post serves as a point of reference we should consider moving the relevant part of the content to docs.

[lenalebt]

Totally unterstand this! Still, it is a quite recent post, so I somehow expected it to still be "quite right". But right, docs would probably be better 😊. And in the end, the blog post still was really helpful.

[svrnm]

What we can do is add a note to the blog post (as initially request by @jaronoff97) that this is the case and people should go to that new resource.

At the same time it would be great if the documentation for that would also be available through https://opentelemetry.io/docs/kubernetes/operator/

[rtkgupta]

@lenalebt You are right, I am hitting the same roadblocks, the blog is outdated. Did you find any other place which helped you figure the configuration out? I am looking at, which seems outdated as well - https://github.com/open-telemetry/opentelemetry-operator/tree/282c169f0499694dd0b780cf225bc8680c05071d?tab=readme-ov-file#target-allocator.
It would be awesome if you could have a blog post or GitHub repo outlining how you got it to work!

[lenalebt]

Hey @rtkgupta sorry, I am really bad at remembering stuff. I can only reconstruct from what I have written in here, but the one thing I remember was that it took me quite a while until I understood what actually was going wrong.

I'll post a few parts of my current config and hope that it helps you...

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: opentelemetry
  namespace: default
spec:

...

  targetAllocator:
    allocationStrategy: consistent-hashing
    enabled: true
    filterStrategy: relabel-config
    observability:
      metrics: {}
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      scrapeInterval: 30s
      serviceMonitorSelector: {}
    replicas: 1
    resources: {}
    serviceAccount: opentelemetry-targetallocator-sa

...

  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
          - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            job_name: kubernetes-nodes-cadvisor
            kubernetes_sd_configs:
            - role: node
            relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$$1/proxy/metrics/cadvisor
              source_labels:
              - __meta_kubernetes_node_name
              target_label: __metrics_path__
            scheme: https
            scrape_interval: 30s
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
          - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            job_name: kubernetes-kubelet
            kubernetes_sd_configs:
            - role: node
            relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$$1/proxy/metrics
              source_labels:
              - __meta_kubernetes_node_name
              target_label: __metrics_path__
            scheme: https
            scrape_interval: 120s
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
        target_allocator:
          collector_id: ${POD_NAME}
          endpoint: http://opentelemetry-targetallocator
          interval: 60s

The scrape config was to get cadvisor and kubelet to work, probably not necessary to get the targetallocator itself running.

Hope this helps!

[rtkgupta]

Definitely helps, thank you @lenalebt!