Uptime Monitoring: Share your experience and expectations

[therealarkin]

Hello Everyone!

I'm Eran from the Sentry product team. We're always looking for opportunities to improve and extend our services to help you manage your applications more effectively. One such area that has come to our attention is Uptime Monitoring!

As we contemplate a service in this direction, we'd love to understand how you currently handle uptime monitoring, what challenges you face, and how an integrated Sentry service might add value.

Here are some starter questions to spur the conversation:

1. How do you currently monitor your websites or web applications' uptime? Which tools or services do you use for this purpose, if any?
2. How effective are these tools? What aspects do you like, and where do you feel they fall short?
3. How often do you encounter website downtime? What are the usual causes (e.g., server issues, code bugs, etc.)?
4. What features would you like to see in an uptime monitoring service from Sentry?

Your input is invaluable to us as we explore this new venture. All thoughts, experiences, and ideas are welcome! Even if you have never used an uptime monitoring service, we're curious to hear what you'd expect from one.

Thanks for taking the time to contribute. We're looking forward to hearing your thoughts and ideas!

[defp]

I like https://gatus.io/ feature rich, easy customization

[therealarkin]

@defp - thanks! Which features have you particularly found helpful and why?

[cleptric]

1. https://betteruptime.com/
2. Super effective. They offer a variety of notification channels, status pages, and a deep level of customization of your uptime checks.
3. Mostly borked deploys.
4. Multi-region support (check uptime from various pops around the globe), a simple UI, an API to automate uptime check creation and SSL certificate monitoring.

[therealarkin]

Thanks @cleptric! Which customization you found the most useful and why?
Also, do you mean broken deploys?

[cleptric]

Yeah, borked is slang for not working properly 😬

I like all the different assertions a monitor can have when checking for uptime.

For the checks, there are a lot of parameters someone can tweak, like the check interval, adding auth and payloads, following redirects or not, when to check for expired SSL certs and much more.

[aldy505]

+1 for SSL certificate monitoring, most free uptime monitoring software don't have a good emphasis on that one.

[PaitoAnderson]

1. https://uptimerobot.com/
2. No issues in the 6+ years using it, quite basic but does the job.
3. ~4 a year, usually odd web server crash or cloud outage.
4. Support to call multiple people from a known phone number so we can set this as an emergency contact in our phone.

[estahn]

We started using Grafana Synthetic Monitoring quite recently.

https://grafana.com/blog/2022/03/10/best-practices-for-alerting-on-synthetic-monitoring-metrics-in-grafana-cloud/

[therealarkin]

Thank you for sharing! What do you like about it? What do you dislike?

[aldy505]

1. https://github.com/louislam/uptime-kuma
2. I like it because it's free. Can monitor lots of types (HTTP, ICMP, DNS, gRPC request, MQTT, MariaDB, Postgres, Push (like the reverse of webhook), and even Steam game server). Has TLS certificate expiry notification and status
3. Mostly because of network failures (ISP uptime SLA problems), and sometimes because of it's in the middle of a release pipeline.
4. Everything that Uptime Kuma has + a separate UI that can run on a separate network (like literally I can spawn a VM for the dashboard on another cloud provider). See TLS certificate status reporting for Push monitor type louislam/uptime-kuma#3385 for detail on how I use this.

[dwd]

I have used:

• SmokePing (pretty good basic visualisation, but no alerting)
• New Relic (amazingly good, but gets pricey very quickly if you take full advantage)
• Google Cloud's own (fair, but limited, and obviously can't tell you if the entire data centre is underwater/melted/etc).
• A bunch of other stuff (like, I once used a cronjob that drove a shell script that drove ping and SNPP, and if you know what the latter even is then we should get together and cry into a beer together).

Things I like:

• New Relic's integration was really nice - so if a response was slower than expected, we could dive into the APM of the tested API and see why.
• New Relic did all the TLS cert validation, too, but would also warn if the expiry was less than (say) a week away.
• NR's full synthetics stuff is impressive - it'll run a whole headless browser for you if you want, as well as a JS script of Stuff To Do.
• SmokePing's visualisation are great for seeing beyond "Is this thing up?" - the display of distribution on the y axis is really nice. Ping me, and I'll give you a link to my personal one.
• NR's "where would you like this to be run from" is pretty nice.
• SmokePing can do ICMP and DNS as well (actually better than) other probes.

You might notice I don't say anything much positive about Google's - it gives value, for sure, and it does the basic job. But beyond dumping an alert out if we've a long running database migration on deploy, it's not doing much more.

So my future plan is this: https://github.com/dwd/sentrynet - which is (or will be, it's only doing very basic stuff now) an attempt to make a synthetics source we can deploy mega-cheap in a bunch of locations (hosting a small docker image is like $5 a month these days), and give us the shiny integration with Sentry (Issues and Performance) that my tiny heart craves.

So... your questions:

1. Yeah, Google Cloud.
2. I mean, it detects downtime, but absolutely nothing beyond that.
3. We get a ~false positive if a migration takes too long. Other than that, our longest downtime was because Google's data centre melted.
4. Everything! Mostly, I think I can Sentrynet to get the right data in, but the visualisations aren't as shiny as I'd like. But downtime is (or should be) rare - I want to be able to find brown-outs in service more effectively, and that means more flexible alerting and reporting than most of these tools offer. New Relic is an exception here, but I think Sentrynet + Sentry Anomalies + some better visualiation could be a winner.

[ipeterov]

What I find really useful is an ability to create Selenium-like tests, and not just checking that a URL returns 200 to some response. Uptime.com has this feature.

It's really useful for downtimes caused by interaction of frontend & backend (for example, backend using an incorrect CDN url, resulting in a blank page in production). Those are really hard to cover by regular tests, so having a tool notify you immediately after you broke something, instead of waiting for users to wake you up is really nice.

The thing I disliked about the uptime.com implementation is that it runs at most every 5 minutes, which makes for a not-very-accurate uptime calculation, and also delays my average time to reaction by 2.5 minutes, which is a pretty big deal. It would be nice to see a once-a-minute option.

[edgariscoding]

Definitely interested in this and it seems like a perfect feature for Sentry.

[darioseidl]

We've been using UptimeRobot and HetrixTools, free or paid, depending on the company. Both work quite well.

Features that I like/need:

• checking for TLS certificate expiration
• checking pages with basic auth
• monitoring other services/ports, not just HTTP(S)
• blacklist monitoring
• generating a (public) status page for all monitors
• sending notifications to mail/Slack/Telegram channels (so that you don't need yet another app on your phone)

[JoseAlban]

Using freshping.io ATM

Pros

• Very basic, it tests pre-set status endpoints, including CDN and servers
• It summarises uptime and sends email notifications with percentage
• public status page for all monitors, with optional custom domain

Cons

• Does not wait for canvas to be fully rendered, and our app is canvas-heavy
• Does not have any advanced feature specific checks after the single page app is loaded, so we can be up but unavailable feature wise

[orenaksakal]

• How do you currently monitor your websites or web applications' uptime? Which tools or services do you use for this purpose, if any?
  Self hosted uptimekuma

• How effective are these tools? What aspects do you like, and where do you feel they fall short?
  I like it but would love to have uptime/error reporting and performance metrics in one place like glitchtip

• How often do you encounter website downtime? What are the usual causes (e.g., server issues, code bugs, etc.)?
  Not very often

• What features would you like to see in an uptime monitoring service from Sentry?
  Ping, expected response body, 1m, 5m etc incremental checks, SSL checks and notifications, monitoring grouping e.g. project-api, project-site, response timeout, simple status pages

[adetch]

@therealarkin any news on an offering in this space? If nothing in sentry, it would be great to just have an integration w/ https://uptime.betterstack.com/ to highlight periods with downtime

[d3thomas]

@therealarkin What is the status on this? Currently looking for a monitoring solution for my FastAPI's. All looks good and works great on the implementation. The Uptime Monitoring is not yet there I see, but is a requirement for a client of mine. Would love to have that in Sentry! Can you tell us a bit more on if this is still on the roadmap, if yes when can we expect it?

[gaprl]

Hey there @d3thomas, we're happy to announce we've recently released an early access version of uptime! We'd love for you to check it out and provide any feedback at [email protected].

[d3thomas]

@gaprl Great thank you, I will check it out!

[edgariscoding]

Also interested in this

[tdtm]

We're currently using BetterUptime so fwiw my key points are similar to others comparing with BetterUptime.

A key thing I would add, and that Sentry is in a perfect position to bring, would be to have more levels on public status pages (or internal dashboards for that matter). So rather than up/down, we'd also have "degraded" if errors are above a certain reference level.

Even better yet, having this grouped by certain rules instead of just being based on a certain page returning a certain status code and/or string. I would imagine the main rule would be route segments for most users but I suppose that can be a bit arbitrary.

For example, Sentry would know that I have a high error rate at example.com/dashboard but everything else is OK; so it could be "Up - Dashboard degraded". A high error rate at example.com/members/:memberId/edit could be "Up - Member management degraded". But then very high error rates thorough the whole app could just be classified as "down".

This would be much smarter than the oversimplified up/down that most services currently allow.

[tdtm]

Regarding the degraded aspect, I love the idea of responding to the uptime monitor with a standardized response like proposed in https://datatracker.ietf.org/doc/html/draft-inadarei-api-health-check so the uptime monitor can flag as....

Downtime if:

• no response (eg network issue)
• error response (5xx)
• monitor reports "fail"

Or degraded if:

• response time is X% above baseline
• getting intermittent errors (say the threshold is 5 failures in a row, and we did not get 5 failures in a row but we got 50% failures in the last 5 minutes)
• monitor reports "warn" (to use the RFC language)
• Sentry's error tracker observes significantly more errors than normal (e.g. >10% error rate)

Or just up/operating normally otherwise.

We'd also prefer putting this behind some sort of simple authentication to avoid disclosing unnecessary details much to a potential attacker.

[tdtm]

For a multi-service setup it would also depend on what "area" of the application is encountering problems.

The authentication being down means the whole product is down, as it's critical. Nobody can do anything is auth isn't working.

If the only thing failing is file uploads, this means the product experience is degraded, but I woudn't report that as downtime.

[therealarkin]

This is 🔥 congrats to the team: https://sentry.io/changelog/uptime-monitoring-early-adopters-release. Looks like @gaprl will take it from here 😃

[vpmedia]

Hi!
I would like an option in the Sentry UI to completely opt-out from this feature.
I have no auto uptime alert configured and robots.txt deny all is in place, still I'm getting monitoring calls to my servers.

Thanks!

[vishal2-wiai]

Is dedicated Uptime Monitoring available in the open source self-hosted version as well? I am unable to find a way to enable it.

[aldy505]

Yes it does! See here getsentry/self-hosted#3304

[fnagel]

I would need response time monitoring. Having stats over the response time is a good metric if something is wrong.

[yashshanker]

How can I configure conditions/notifications for Uptime Alerts ?

The screenshots in the docs show Set Conditions section, which is not there under Uptime Alert.

[noqcks]

Just trying uptime monitoring for the first time today and have some thoughts.

1. What status codes will cause uptime monitoring to fail? There's nothing that indicates what status codes will trigger it. I would hope and assume that anything other than 2xx fails.

2. What is ownership? I have no idea where my alerts will go if an uptime monitor fails. The text describing ownership also doesnt tell me how alerts are routed.

3. How do I test it? Is there a way to trigger an alert so that I can see what the failure message looks like in Slack? Would love that.

[venkigvs2002]

got any reply??

[philprime]

We started using Sentry Uptime Monitoring for an instance of HashiCorp Vault polling the /sys/health endpoint, which returns different status codes but also details in the JSON response body (Documentation).

To summarize the issue:

• Vault allows storing secrets and can be sealed from access until administrator interaction.
• If the vault is sealed, the endpoint will return a 503 HTTP Status
• If the deployment crashes, the load balancer in my cluster also returns 503
• Therefore status code is not sufficient to identify the problem

What I need:

• Check fields in the response JSON and use it as uptime conditions, i.e. '.sealed' IS 'false'

{
  "clock_skew_ms": 0,
  "cluster_id": "995f749a-9bc3-fbe0-a185-ccba6999fc73",
  "cluster_name": "vault-cluster-e65c0563",
  "echo_duration_ms": 0,
  "enterprise": false,
  "initialized": true,
  "performance_standby": false,
  "replication_dr_mode": "disabled",
  "replication_performance_mode": "disabled",
  "sealed": false,
  "server_time_utc": 1709559327,
  "standby": false,
  "version": "1.16.0-rc2"
}

• JQ could be a reference for key-path-based access

[tdtm]

Agreed, this would also be useful for RFC 2629 health checks, as well as any similar custom schema, so a generic jq-style feature would be very versatile

[christianrr]

Indeed, checking the response for some specific content is super important for uptime monitors IMO.
JSON-support would be great, however, checking the string content of the response with contains/not-contains would be sufficient for most cases (maybe this would be an easier first step)

[brianguertin]

It's nice to see uptime monitoring in Sentry. But it would be nice if there was a public "status" page? Like https://status.sentry.io/

[edgariscoding]

What would be amazing!