Spotlight's 100% tracer sampler interferes with health checks

[rassie]

How do you use Sentry?

Self-hosted/on-premise

Version

2.35.0

Steps to Reproduce

I'm writing this in full awareness of XKCD 1172...

In my environment, we are developing our services using a local Kubernetes cluster. The deployment configuration is as close to production deployment as it can be, so that we employ Kubernetes health checks on our services. We are using logging filters and Sentry samplers to filter out requests going to /health from logs and traces.

Now we want to expand our Sentry experience with Spotlight, Apparently, in #4207 a change has been introduced to sample at 100% for everything with Spotlight in development. Maybe our configuration is unadjusted, but in our case, this leads to /health traces overwhelming the Spotlight overlay and also the logs if set to DEBUG.

1. Add a primitive endpoint to your application
2. Add a tracing sampler to your Sentry SDK configuration filtering that endpoint
3. Enable debug on your Sentry SDK
4. Add Spotlight to the configuration
5. Call the endpoint multiple times and observe both logs and overlay getting immer fuller

Expected Result

I expect Spotlight to work exactly like Sentry would, i.e. respecting sampler configuration. It's perfectly possible my expectation is wrong, I'm ready to add code to my services to make it work like I expect it to. From what I can see, it's currently not possible. For example, setting a DSN results in SDK trying to parse it or trying to connect to it, so that a placeholder DSN won't work and I don't want to use a real DSN in development. Leaving the DSN out results in the described behaviour.

Actual Result

Logs and overlay overflow with useless data.

[linear]

PY-1813 Spotlight's 100% tracer sampler interferes with health checks

[sentrivana]

Hey @rassie, thanks for raising. @BYK Can you take a look? 🙏🏻