Significant performance impact

[slafs]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.34.1

Steps to Reproduce

Given that #2116 is already closed,
I've decided to open a new issue on the same topic.

We're seeing a significant performance impact of using sentry's SDK in our FastAPI and Django projects, both running behind Uvicorn.

Not sure if this is expected, but the performance hit when only calling sentry_sdk.init() is a bit terrifying TBH.

Please see https://github.com/slafs/sentry_perf for details.

Expected Result

I'd expect far less impact on the performance, given that the SDK doesn't do anything yet (no error capture, no transaction capture and no event sending).

Actual Result

In some cases the performance is around 2x or even 4x worse (e.g. from 20k req/s to 5k req/s)

[linear]

PY-1797 Significant performance impact

[szokeasaurusrex]

Hi @slafs, thank you for raising this issue and for providing a detailed reproduction. We will investigate

[sentrivana]

Just an update -- we've already identified a couple of potential smaller performance improvements, but we're still looking into where most of the overhead is coming from. Thanks again for providing a full fledged setup to repro, it's immensely helpful.

[sentrivana]

Couple more updates:

• The lifespan benchmark is better just by the virtue of not instrumenting everything we normally would. We usually patch all middlewares to capture middleware spans, and if the SDK is initialized in the lifespan, this doesn't happen (because it's too late and the middlewares have already been created) -- so it's not equivalent to an init on the top level. It's more akin to initing the FastAPI/Starlette integrations with middleware_spans=False.

  So the way we advertise this in the FastAPI/Django docs is correct (init should be top level, or in settings.py). We can def improve the general "you need to init() in an async function" warning message to make it clear that this is only the case for stuff like smaller scripts, etc.

• Even if tracing is not enabled explicitly and the SDK is not sending spans to Sentry, the SDK is still tracing in the background in order to do stuff like continue incoming traces and connect related telemetry together.

The more I look at this the more it seems to me like there is no single thing that we can optimize and everything will be much faster. I've identified a couple of pain points but removing them only makes things marginally better. Anyways, still continuing to look into this, and also discussing other, potentially more invasive changes with the team.

[sentrivana]

So after looking into this for a long time I'm coming to the conclusion that we're seeing is basically the overhead of doing something in an endpoint vs doing nothing. I've profiled this left and right with different profilers and fixed up stuff that is popping up as most CPU intensive but none of it brings a sizeable benefit when compared to a bare-bones app simply because with Sentry, there's now something being done in the endpoints.

If we look at the absolute numbers, with Sentry the app is still serving requests sub-1ms:

Time per request:       0.265 [ms] (mean, across all concurrent requests)

That being said, we're doing the following:

• Merging a PR which removes some performance papercuts
• Updating docs with regards to where to init() in an async app
• Talking internally about removing some overhead due to scope forking (longer-term change which is also not likely to make performance skyrocket)
• Adding an async transport in 3.0 which should remove the worst offender when it comes to blocking in the SDK
• Investigating where more async improvements would make sense

What can be done on the user side to alleviate overhead:

• Setting middleware_spans=False. We're creating a span each time a middleware is run and it adds up.

In any case though, we can also look into your app and try to figure out if there's anything that can be done there.