Prom Garmin Scraper

Introduction

This tool is a Python-based scraping application that collects daily health and activity data from my Garmin account, exposes it as Prometheus metrics, and provides endpoints for backfilling historical data, which is especially useful if I lose prometheus data. The metrics are designed for visualization and alerting in Grafana and Prometheus, helping me monitor my fitness, health, heart rate, stress, activity.

The inspiration came from when I was recovering from covid and was trying to pin point when my long covid had improved - by using my oxygen intake data on my watch. Garmin provide this data on their app and on weekly reports, but I wanted an in-depth approach to it.

Setup

Prerequisites

• Python 3.11+
• Docker (optional but recommended for local testing)
• Garmin account credentials
• Prometheus and Grafana for metrics collection and visualization

Environment Variables

Set the following environment variables for the app to function:

• GARMIN_USER: Your Garmin account username
• GARMIN_PASS: Your Garmin account password
• GARTH_FOLDER: Path to store authentication tokens - generated by the app and then used for 90 days.
• SLACK_CHANNEL: (Optional) Slack channel for sync alerts
• SLACK_USER_ID: (Optional) Slack user ID for notifications
• SLACK_BOT_TOKEN: (Optional) Slack bot token for sending messages

Installation

1. Right now this is a public repo, so you can clone it and build it and run it locally.
2. However, you won't be able to access the docker images that my GHA workflow builds.
3. At a later stage I might push the image to a public docker repo, but until then you'll need to find your own route to deploying it.

Architecture and Design

• This project follows a modular, pythonic style - splitting them into easy maintable components
• The core of the app is written in Python, using Flask to expose the required endpoints
• It relies on a manual process to sync my watch, but there are alerts to remind me to do it.
• Metrics are exposed on /metrics using the prometheus-client library - these are then scraped by prometheus using Pod Annotations
• It includes Helm charts for easy deployment to my Raspberry Pi K3s cluster, including alerts
• The backfill endpoint is affectively my way of restoring data from a backup for my garmin data. The endpoint currently only generates the backfill timeseries chunks - it does not put them into the prometheus filesystem.

Functionality

• /metrics: Exposes Prometheus metrics for scraping.
• /daily: Triggers a daily data scrape from Garmin and updates metrics. I'm triggering this as a raw cronjob on my Raspberr Pi. At a later date I may change the endpoint to run on a schedule.
• /backfill?days=N: Backfills historical data for the past N days and generates TSDB-compatible files. You'll then need to pass that to a folder where prometheus can ingest it to. Beware, the metric names can lead to different metrics appearing for the same name.

Metrics include heart rate, stress, activity, body battery, sleep, and derived metrics for richer analysis.

Endpoints

• GET /metrics: Prometheus metrics endpoint.
• GET /daily: Fetches and exposes the latest daily Garmin data.
• GET /backfill?days=N: Backfills and processes N days of historical data.

Dashboards

I've exported my dashboard as json, mostly for version control. A direct copy may not work as it's based on your datasource, but it's worth looking at for some examples. There's nothing too complicated like rates or deltas, it's mostly just standard queries on timeseries and pie chart panels.

Alerts

As part of the helm chart I've got a PrometheusRule set. This generates rules that my local alertmanager can alert on.

Example alerts you can configure in Prometheus:

• RestingHeartRateSpike: Detects spikes in 7-day resting heart rate (possible sickness).
• SedentaryBehavior: Alerts if you’ve been sedentary for over 12 hours.
• ElevatedStressLevels: Alerts if your average stress level is high over several days.
• LowBodyBatteryRecovery: Detects poor overnight recovery.
• InsufficientActivity: Alerts if you’ve been active less than 1 hour.
• PoorSleepQuality: Alerts if SPO2 during sleep is low.
• HighStressToActivityRatio: Alerts if stress duration is high relative to activity.
• OvertrainingWarning: Detects drops in heart rate variability.
• GarminDataStale: Alerts if your watch hasn’t synced in over 24 hours.

See helm/templates/garmin-alerts.yaml for some rules you can copy.

Planned Features

• Fix the login so it works on ARM64. Right now I'm relying on a manual copy process.
• Better refinement of my alerts
• The backfill endpoint will automatically backfill right into prometheus, taking out the manual steps

Conclusion and Learnings

This tool takes my garmin data and allows me to visualise it on Grafana, and alert on it where required. I learned a few new things from it that I never learned in industry

• How the prometheus backfill functionality works
• How request headers differ between Raspberry Pi (ARM64) vs my machine. The login returns a 401 on my Pi. The initial login won't work on a Raspberry Pi, so I need to run it locally first and then copy the oauth credentials up! It's a bit maddening and I suspect there's some difference in user-agent/other headers that I havne't had time to look into.