Add proposal for embedded exporters in OTel Collector

[ArthurSens]

No description provided.

[quentinmit]

It sounds like this proposed architecture would require that Prometheus exporters be implemented in Go and explicitly linked into the OTel Collector binary. We previously had solved this in the Collector with the prometheusexecreceiver which just allows directly spawning a Prometheus exporter (written in any language) as a child process, but it was removed from the Collector because of concerns around the security of launching config-specified binaries.

I think it's a significant roadblock to require that exporters be written in Go, per-exporter code changes (to adopt the new ExporterLifecycleManager interface), AND require a custom build of the Collector that includes all the desired exporters. I'm not sure how to resolve those issues without running the exporters as a separate process, though.

[ringerc]

Go's lack of a decent interface for runtime loading severely limits the options here. It's possible to combine https://pkg.go.dev/plugin with CGO and dlopen() but gets clumsy and complicated fast, especially if plugins aren't built with exactly the same go runtime version. Lots of mess adapting interfaces too.

Rust based exporters are starting to appear and are likely to grow in popularity over time so more strongly tying exporter=go might not be ideal.

[quentinmit]

Go's lack of a decent interface for runtime loading severely limits the options here. It's possible to combine https://pkg.go.dev/plugin with CGO and dlopen() but gets clumsy and complicated fast, especially if plugins aren't built with exactly the same go runtime version. Lots of mess adapting interfaces too.

AIUI, the Collector objects to loading code from a config-controlled path, whether that's as a plugin or dlopen or fork+exec. I don't actually agree with that stance, so I'm happy to help try to convince OTel to support something akin to the prometheusexecreceiver again.

I think there could be a really elegant child process solution here, where you run HTTP and/or gRPC over the subprocess's stdin to ship Prometheus and/or OTLP metrics back from the exporter to the Collector.

Rust based exporters are starting to appear and are likely to grow in popularity over time so more strongly tying exporter=go might not be ideal.

I'm not super familiar with the whole exporter ecosystem, but my impression was that there's already a decent roster of Python exporters as well.

[ArthurSens]

It sounds like this proposed architecture would require that Prometheus exporters be implemented in Go and explicitly linked into the OTel Collector binary. We previously had solved this in the Collector with the prometheusexecreceiver which just allows directly spawning a Prometheus exporter (written in any language) as a child process, but it was removed from the Collector because of concerns around the security of launching config-specified binaries.

Interesting! This is the first time I hear about this component. I can totally understand why it was deprecated, and if I understand it correctly, it just executed binaries, right? You still needed to run the usual prometheusreceiver and configure it to scrape the executed binaries, is that correct?

With the approach I'm suggesting here, I understand the limitations of only supporting exporters written in Go, but do you think the overall experience is better or worse than the previous attempt? I can see value in not spawning new subprocesses, and not requiring a scrape configuration also simplifies things in my point of view :)

I think it's a significant roadblock to require that exporters be written in Go, per-exporter code changes (to adopt the new ExporterLifecycleManager interface), AND require a custom build of the Collector that includes all the desired exporters. I'm not sure how to resolve those issues without running the exporters as a separate process, though.

I agree it's a roadblock if our goal is to enable ALL exporters independently of language. I was aiming for a slightly less ambitious project, to give a way forward to very widely used exporters like node_exporter, kube-state-metrics, blackbox_exporter, cadvisor, mysql_exporter, etc etc.

With that said, I'd be super happy to listen to ideas that support even more languages and incorporate them into the proposal. My preference would continue something that we can embed as part of the Collector Pipeline, and not as separate processes that still require extra scrape configuration.

I think there could be a really elegant child process solution here, where you run HTTP and/or gRPC over the subprocess's stdin to ship Prometheus and/or OTLP metrics back from the exporter to the Collector.

That sounds cool! Would you be willing to write down your ideas in a Google Doc? Or even a counter-proposal to this one?

[bwplotka]

Just as a datapoint - majority of Prometheus exporters are written in Go.

[roidelapluie]

1. A while back, we worked on moving some exporters to packages instead of basically a main.go. Why can't OTel Collector use the existing Collector interfaces? I do not see that mentioned.

2. In practice, I wonder if the "single binary approach" will be possible at all -- after 2 years, there will be 10 versions of all the dependencies fighting each other in go.mod.

3. Was a plugin system thought for in the OTLP community? Something similar to what Terraform does. It would solve many issues - but remove the "1 binary" goal - wich is really a difficult one to maintain with such a large scope.

[quentinmit]

It sounds like this proposed architecture would require that Prometheus exporters be implemented in Go and explicitly linked into the OTel Collector binary. We previously had solved this in the Collector with the prometheusexecreceiver which just allows directly spawning a Prometheus exporter (written in any language) as a child process, but it was removed from the Collector because of concerns around the security of launching config-specified binaries.

Interesting! This is the first time I hear about this component. I can totally understand why it was deprecated, and if I understand it correctly, it just executed binaries, right? You still needed to run the usual prometheusreceiver and configure it to scrape the executed binaries, is that correct?

I don't understand why it was deprecated. :) AIUI the concern was "if the user can configure a command to run, then we will run the command, and that command could do anything". Of course that's true (and if you give configuration access to someone you don't give shell access to, they could use that to escalate to a shell), but the Collector config also lets you read and write arbitrary files, as well as open arbitrary network connections, all of which can easily be escalated to running a command. You need to treat the Collector config as sensitive either way.

No, the prometheusexecreceiver both ran the exporter process and automatically configured and ran scraping for it.

With the approach I'm suggesting here, I understand the limitations of only supporting exporters written in Go, but do you think the overall experience is better or worse than the previous attempt? I can see value in not spawning new subprocesses, and not requiring a scrape configuration also simplifies things in my point of view :)

In isolation for a single exporter, it's better, but I think as an ecosystem of exporters, it's worse because you can only use a limited subset of exporters (only the ones that are written in Go and have adopted the new interface).

I think it's a significant roadblock to require that exporters be written in Go, per-exporter code changes (to adopt the new ExporterLifecycleManager interface), AND require a custom build of the Collector that includes all the desired exporters. I'm not sure how to resolve those issues without running the exporters as a separate process, though.

I agree it's a roadblock if our goal is to enable ALL exporters independently of language. I was aiming for a slightly less ambitious project, to give a way forward to very widely used exporters like node_exporter, kube-state-metrics, blackbox_exporter, cadvisor, mysql_exporter, etc etc.

With that said, I'd be super happy to listen to ideas that support even more languages and incorporate them into the proposal. My preference would continue something that we can embed as part of the Collector Pipeline, and not as separate processes that still require extra scrape configuration.

I think there could be a really elegant child process solution here, where you run HTTP and/or gRPC over the subprocess's stdin to ship Prometheus and/or OTLP metrics back from the exporter to the Collector.

That sounds cool! Would you be willing to write down your ideas in a Google Doc? Or even a counter-proposal to this one?

I can try to put something together, but I'm not sure I'm familiar enough with the high-level Prometheus direction to know which style would be preferred (HTTP or gRPC? OTLP, text format, or proto format metrics? etc.)

2. In practice, I wonder if the "single binary approach" will be possible at all -- after 2 years, there will be 10 versions of all the dependencies fighting each other in go.mod.

Great question. I agree that this is potentially a significant blocker for OTel Collector adoption - today, most people use kitchen-sink binaries that contain ~every public component, but if the ecosystem grows as designed, that's not going to be viable going forward.

3. Was a plugin system thought for in the OTLP community? Something similar to what Terraform does. It would solve many issues - but remove the "1 binary" goal - wich is really a difficult one to maintain with such a large scope.

I think what I suggest above (running a child process with HTTP or gRPC) is very similar to how Terraform plugins get used. I'm not aware of any proposals along those lines in the OTel world right now.

[ArthurSens]

I've discussed this offline with @roidelapluie, and he raised some good points that the exporter-toolkit may not be the best place to build this interface, if we decide to do so. The exporter toolkit was designed to facilitate HTTP interactions, and creating a scraper and adapter seems far from the original design. I'll update the proposal to mention that we'll create a stand-alone Go library for this work.

I also see the dependency hell problem, still figuring that out 🤗

[ArthurSens]

No, the prometheusexecreceiver both ran the exporter process and automatically configured and ran scraping for it.

That's fantastic. That allows integration with exporters written in any language, and the Go dependencies problem wouldn't exist. However, specifically for Go, this approach feels like doing extra work to have a client_model.dto encoded into Prometheus exposition format to then be converted back to client_model.dto and finally to OTel's format.

@quentinmit, another problem I see with prometheusexecreceiver is how to package the whole thing. If you depend on separate binaries, the collector container needs to include all those binaries with it. Therefore, a plugin system over gRPC seems like a better approach to solving the problem. If OTel Collector had something like "Component Marketplace", that would be ideal.

We're dreaming a little bit here, though. I don't want to let "Perfect" get in the way of "Progress". I'd be happy to collaborate with anyone who wants to push this plugin system forward, but it feels too far away at the moment.

I can try to put something together, but I'm not sure I'm familiar enough with the high-level Prometheus direction to know which style would be preferred (HTTP or gRPC? OTLP, text format, or proto format metrics? etc.)

We don't expect you to write a perfect proposal from the start; you can suggest your preferences, and the community will give you feedback on their preferences as well. Just like how you're doing here :)