In-progress check-ins not being sent intermittently with Celery integration

[gaprl]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.29.1

Steps to Reproduce

We've recently shipped an improvement to the Crons' monitor details page, and noticed that many in-progress check-ins are never being sent across all of our monitors. To reproduce, just set up our automatic detection integration with Celery and go to the monitor details page.

Expected Result

All check-ins are sending the in-progress check-ins, and not only the completion check-in.

Actual Result

In-progress check-ins are being missed:

Here's some of the impacted monitors:

• [Internal 🔗 ] monitors-detect-broken-monitor-envs
• [Internal 🔗 ] fetch-release-registry-data

[antonpirker]

Hey @gaprl !

Do I get this right: we get ok check-ins for check-ins that we never sent a in_progress check-in?

Can we check on the server if the in_progress check-ins have been received but have been discarded for some reason (like because they are in a wrong format?)

[evanpurkhiser]

I think the best way to track this down would probably to start by having some mode where the sentry SDK logs when it sends in_progress and when it sends oks, do we have anything like that yet?

[antonpirker]

if you set debug=True you will see that there has been an envelope of type monitor sent (which is a check-in) but you can not see if it is ok/ in_progress…
But I guess this would be a nice debug output to have.

[antonpirker]

Hey @gaprl I now checked the client outcomes in the admin interface and it seems that every once in a while the server is dropping check-ins (here on May 16th there where 16 check-ins dropped):

Can we somehow find out why those where dropped and what check-ins those where?

Another thing that jumped into my eye is that the monitors-detect-broken-monitor-envs are all very short lived tasks (sometimes only 100ms long). We had a problem before where check ins where processed in the wrong order. Could this also be happening here due to the in_progress and the ok checking landing in different queues and thus the ok is processed before the in_progress?

[mdefeche]

If that can help, I have a cron that has constantly the same issue (#84358). Happens only in prod environment (same architecture (django + celery + django-celery-beat on AWS + ECS + Fargate), tens of other crons work fine.

DSN: https://dca869850db34a8a8a4c9fef41bf5840@o1156924.ingest.sentry.io/4503942203310080

[evanpurkhiser]

I've done some research for this specific issue. When check-ins come in out-of-order we're not correctly adding in the date_in_progress. We need to add this to correctly account for the fact that we did receive a check-in. Right now we just discard the in-progress since we know we already completed the check.

You can see here the check-ins are being processed out of order

Here's one including exact processing time (the second timestamp)

Unfortunately @mdefeche this appears like it may be a different issue.

Going to move this issue back to the sentry repo and address it there.

[mdefeche]

@evanpurkhiser , how can I investigate further the issue on our side ? Can I log calls to crons' API ?