Add Datadog GitHub health metrics sync

Cargo.toml

@@ -25,7 +25,7 @@ reqwest = { version = "0.12", features = ["json", "blocking"] }
 
 # Serialization
 serde = { version = "1", features = ["derive"] }
-serde_json = "1"
+serde_json = { version = "1", features = ["preserve_order"] }
 
 # Database
 rusqlite = { version = "0.32", features = ["bundled"] }

README.md

@@ -21,8 +21,8 @@ src/             # Rust source (single binary: github-tools)
 docs/            # GitHub Pages — interactive HTML dashboards (ECharts)
 data/            # Committed snapshots: JSON inputs
   {owner}_{repo}/issues/  # Issues/PRs JSON split by year (2024.json, 2025.json, ...)
+  {owner}_{repo}/discussions/  # Discussions JSON split by year
 out/             # Gitignored — all generated and local-only files
-  historical/    # Raw JSON fetched from GitHub API
   db/            # SQLite databases
   summaries/     # Generated CSVs
   purge/         # Purge audit logs (local only)
@@ -41,6 +41,8 @@ Commands read credentials from the environment. Keep them in a single `secrets.e
 
 ```dotenv
 GITHUB_TOKEN=...
+DD_API_KEY=...       # Datadog metric submission only
+DD_SITE=datadoghq.com
 DOCKER_USERNAME=...   # purge commands only
 DOCKER_PASSWORD=...   # purge commands only
 ```
@@ -67,6 +69,8 @@ Fetch:
   fetch-labels       Fetch all labels for a repository
 
 Pipeline:
+  sync-metrics       Fetch GitHub data locally, rebuild DB, and submit Datadog metrics
+  push-metrics       Submit a Datadog snapshot from an existing local DB
   generate-all       Build DB + summaries + charts for all repos (workflow)
   build-db           Load issues JSON into SQLite database
   generate-summaries Generate CSV summaries from SQLite database
@@ -99,7 +103,12 @@ for repo in vectordotdev/vector vectordotdev/vrl quickwit-oss/quickwit quickwit-
 done
 ```
 
-Writes to `out/historical/`. The fetched JSON must be split by year and promoted to `data/` to commit as a snapshot. Issues/PRs are stored in `data/{owner}_{repo}/issues/{year}.json`.
+Fetches merge directly into the committed snapshot layout:
+
+- Issues/PRs: `data/{owner}_{repo}/issues/{year}.json`
+- Discussions: `data/{owner}_{repo}/discussions/{year}.json`
+
+Use `--since 30d` (or another ISO/relative value) for an incremental refresh. Fresh records replace matching record numbers, while older snapshot data is retained.
 
 ## 2. Generate DB, summaries, and charts
 
@@ -142,3 +151,100 @@ Outputs:
 - `out/summaries/{owner}_{repo}_automated_review_stats.json` — stats snapshot picked up by `generate-all`
 
 Re-run `generate-all` after collecting stats to update the dashboard with the AI review chart.
+
+## 5. Datadog project-health metrics
+
+`sync-metrics` is the end-to-end path intended for scheduled automation:
+
+```shell
+op run --env-file secrets.env -- github-tools sync-metrics \
+  --repo vectordotdev/vector \
+  --lookback 7d \
+  --activity-window 30d
+```
+
+The command:
+
+1. Fetches a complete GitHub snapshot into the runner-local `data/{owner}_{repo}/` directory.
+2. Rebuilds `out/db/{owner}_{repo}.db` from that local snapshot.
+3. Reconstructs one snapshot at each UTC midnight in `--lookback`, adds the current snapshot, and sends them to Datadog. `--activity-window` independently controls the rolling period summarized by the closed-item metrics.
+
+The fetched JSON and SQLite database are temporary automation inputs. The command never stages, commits, or pushes them.
+
+Use `--dry-run` to perform the fetch and database rebuild while printing, but not submitting, the resulting metrics. To preview metrics entirely offline after a database has been built:
+
+```shell
+github-tools push-metrics \
+  --repo vectordotdev/vector \
+  --history 30d \
+  --activity-window 30d \
+  --dry-run
+```
+
+Use `--output-json` when a Datadog workflow or agent owns the Datadog connection. It performs the same calculation without submitting and emits one final JSON envelope. Each object in `batches` is a size-safe request body that can be sent directly to `POST /api/v2/series`:
+
+```shell
+github-tools sync-metrics \
+  --repo quickwit-oss/quickwit \
+  --lookback 7d \
+  --activity-window 30d \
+  --output-json
+```
+
+```json
+{"format":"datadog-series-batches-v1","series_count":1,"point_count":1,"batches":[{"series":[{"metric":"github.health.v2.issues","type":3,"points":[{"timestamp":1788278400,"value":42}],"tags":["repo:quickwit-oss/quickwit"]}]}]}
+```
+
+The default prefix is `github.health.v2`; override it with `--prefix`. Increment the prefix version before changing the metric dimensions so a clean backfill cannot mix incompatible tag schemas. The metrics are:
+
+| Metric | Meaning | Important tags |
+|---|---|---|
+| `github.health.v2.issues` | Open issue backlog at each snapshot | `repo`, `issue_type`, `age`, labels |
+| `github.health.v2.prs` | Open non-draft PR backlog at each snapshot | `repo`, `age`, labels |
+| `github.health.v2.discussions` | Open discussions at each snapshot | `repo`, `category`, `answered`, `age` |
+| `github.health.v2.issues.closed` | Issues closed in the rolling window ending at each snapshot | `repo`, `window`, `issue_type`, labels |
+| `github.health.v2.prs.closed` | PRs closed in the rolling window ending at each snapshot | `repo`, `window`, labels |
+| `github.health.v2.discussions.closed` | Discussions closed in the rolling window ending at each snapshot | `repo`, `window`, `category`, `answered` |
+
+All are gauges. Rolling-window totals remain gauges because each point is a complete window snapshot, not a count accumulated during the reporting interval.
+
+### Historical backfill and weekly updates
+
+Historical Metrics Ingestion must be enabled before submitting the backfill. For a new prefix:
+
+1. Submit a one-day seed using `--lookback 1d --activity-window 30d` so the metric names exist.
+2. In Datadog **Metrics Summary**, choose **Configure Metrics → Enable historical metrics** and select the `github.health.v2` namespace.
+3. Submit the one-time backfill with `--lookback 450d --activity-window 30d`.
+4. Schedule subsequent runs at a fixed UTC time with `--lookback 7d --activity-window 30d`.
+
+Closed metrics are sparse: a metric with no matching closures is not emitted. If a closed metric is absent after the seed, use a wider activity window to create its name before enabling Historical Metrics Ingestion.
+
+The 450-day limit fits within Datadog's default 15-month metric retention. The seven-day weekly lookback intentionally avoids rewriting the preceding week's UTC-midnight snapshots. A failed or skipped weekly run can leave a gap and requires an explicit catch-up run.
+
+Datadog treats points older than one hour as historical and bills them as indexed custom metrics. Older points also have higher ingestion latency; see [Historical Metrics Ingestion](https://docs.datadoghq.com/metrics/custom_metrics/historical_metrics/).
+
+After enabling it, an existing database can be backfilled without fetching GitHub again:
+
+```shell
+github-tools push-metrics \
+  --repo quickwit-oss/quickwit \
+  --history 450d \
+  --activity-window 30d
+```
+
+Dashboard totals must sum the dimensional series. For example:
+
+```text
+sum:github.health.v2.issues{repo:quickwit-oss/quickwit}
+sum:github.health.v2.prs{repo:quickwit-oss/quickwit}
+sum:github.health.v2.discussions{repo:quickwit-oss/quickwit}
+sum:github.health.v2.issues.closed{repo:quickwit-oss/quickwit,window:30d}
+sum:github.health.v2.prs.closed{repo:quickwit-oss/quickwit,window:30d}
+sum:github.health.v2.discussions.closed{repo:quickwit-oss/quickwit,window:30d}
+```
+
+Do not use `avg:` for repository totals: each metric is split into multiple tag combinations for its breakdown dimensions.
+
+An external scheduler invokes `sync-metrics` once per repository. Its GitHub token needs read access to the source repository. Direct submission needs `DD_API_KEY`; alternatively, `--output-json` lets a Datadog workflow submit each generated batch through a managed HTTP connection. Non-US1 accounts should also set `DD_SITE` (for example, `datadoghq.eu`) for direct submission.
+
+Tags and series are emitted in deterministic order, and an identical metric name, timestamp, and tag combination is safe to resubmit because Datadog retains the most recently submitted value. Current GitHub snapshots do not contain the full timelines for labels, issue types, PR draft transitions, discussion categories/answers, or close/reopen cycles. Historical breakdowns using those mutable fields are therefore current-state approximations. Use a new prefix version for a clean corrective backfill if the schema changes; exact lifecycle reconstruction would require fetching GitHub timeline events.

src/commands/mod.rs

@@ -9,5 +9,6 @@ pub mod fetch_issues;
 pub mod fetch_labels;
 pub mod generate_summaries;
 pub mod purge;
+pub mod push_metrics;
 pub mod remove_legacy_label;
 pub mod workflows;