Are there any way to monitor/alert failed "mutations" of a pod by opentelemetry-operator

[veziak]

Component(s)

auto-instrumentation

Describe the issue you're reporting

I would like to be able to monitor/alert situation when pods are not "mutated" by opentelemetry-operator.
For example if opentelemetry-operator is down, annotated pods are not going to be "mutated" and will continue to run without instrumentation.
What are the options to handle this situation

[iblancasa]

I think you will need to do something custom... maybe you can query those pods with the annotation but without init containers?

[veziak]

That's what I was thinking about, but it will require a new application bundled with opetelemetry-operator helm chart, that queries k8s api and expose custom metrics that could be collected by grafana/prometheus. Quite a bit of work, so I wanted to check that I do not miss a simpler solution.

[iblancasa]

That's what I was thinking about, but it will require a new application bundled with opetelemetry-operator helm chart, that queries k8s api and expose custom metrics that could be collected by grafana/prometheus. Quite a bit of work, so I wanted to check that I do not miss a simpler solution.

But if the operator is down... the new application very likely will not be there neither.

[paolostancato]

This promql has worked for me:

count by (pod) (kube_pod_container_info{container="my-main-container"}) unless (count by (pod) (kube_pod_container_info{container="otc-container"}))

It'll list all the pods that have my-main-container but not the otc-container.