feat(backend): sync git commit metadata into commits tables (#2079) by felipebergamin · Pull Request #2118 · kernelci/dashboard

felipebergamin · 2026-09-15T21:06:29Z

Summary

Store git commit metadata in Postgres instead of reading it from KCIDB submissions. Author, committer, subject, message, and ordered parents live on commits, commit_identity, commit_message, and commit_parents. The same hash can appear on many checkouts, so that data does not live on checkouts.

This PR includes the work previously reviewed in #2113:

Schema ([#2079] Add commits and commit_parents tables #2089): commits, commit_identity, commit_message, commit_parents.
One-shot library ([#2079] Fetch commit metadata via SHA-only partial clone on tmpfs #2090): parse_commit on an existing repo, and a throwaway fetch on GIT_SCRATCH_DIR. No database writes. The mirror ingest reuses that parser.

Two daily cron jobs fill those tables from git:

sync_commit_mirror (04:00) fetches the ancestor closure of checkout branches, from the tree-names.yaml allowlist, into one bare repo at GIT_MIRROR_DIR.
sync_commit_ingest (10:00) parses new objects from that mirror and inserts them. --fill-gaps covers checkout hashes the mirror still does not have.

When the mirror job finishes, it writes GIT_MIRROR_DIR/mirror-size-bytes. With PROMETHEUS_METRICS_ENABLED=true, the metrics process exposes:

git_mirror_size_bytes — apparent size of the mirror, in bytes
git_mirror_size_mtime_seconds — Unix mtime of that file, so time() - git_mirror_size_mtime_seconds is how long ago the size was recorded

GitMirrorSizeHigh fires when the size stays above 10 GB for 30 minutes. Steady state is about 6–8 GB. The series stay absent until the first completed fetch. The alert shows in Prometheus and does not page.

Details: docs/sync-commits.md.

Closes #2090

Test plan

poetry run pytest kernelCI_app/tests/unitTests/helpers/commitSync_test.py kernelCI_app/tests/unitTests/helpers/gitCommit_test.py kernelCI_app/tests/unitTests/utils/git_mirror_size_test.py
gitCommit_test.py covers fixture parse (author, committer, subject, ordered parents), parse against an existing repo with no fetch, and typed errors for fetch failure, oversized or treeful packs, and a bad URL
poetry run python manage.py sync_commit_mirror --mirror-dir /tmp/git-mirror against a small allowlist
Confirm /tmp/git-mirror/mirror-size-bytes is a single integer and matches the directory size
With PROMETHEUS_METRICS_ENABLED=true, open http://localhost:8001/metrics/ and check git_mirror_size_bytes and git_mirror_size_mtime_seconds
poetry run python manage.py sync_commit_ingest --mirror-dir /tmp/git-mirror --dry-run parses objects and writes nothing
Ingest without --dry-run inserts commits and parent edges, and a second run skips hashes already stored

* Add `Commits` model: unique `git_commit_hash`, nullable author/committer/subject/message/`fetched_from_url` * Add `CommitParents` model: FKs to `commits.id`, `ord` with 0 = first parent, unique `(commit_id, ord)` * Add migration `0021_commits_and_commit_parents` (`commits`, `commit_parents` tables and indexes) * Join from `checkouts.git_commit_hash` without new checkout columns Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Dump commits and commit_parents in full so git ancestry survives restore. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

…ci#2089) Keep commits rows narrow: unique git name/email in commit_identity, subject plus UTF-8 body in commit_message as bytea. update_db dumps and restores the new tables and still accepts a flat legacy commits CSV.

alanpeixinho · 2026-09-23T22:05:50Z

@@ -0,0 +1,59 @@
+---


maybe we shoudl keep those skill related files only on the dedicated PR.

alanpeixinho · 2026-09-24T19:59:55Z

+        parser.add_argument(
+            "--dry-run",
+            action="store_true",
+            help="Parse as requested, write nothing to the database or tips file.",


the dry run still updates with fetch (so it is not totally dry), even if we are not writing to database, might be something worth pointing.

alanpeixinho · 2026-09-24T20:07:26Z

+        for metadata in metadatas
+    ]
+    Commits.objects.bulk_create(
+        rows, ignore_conflicts=True, batch_size=UPSERT_BATCH_SIZE


we are ignoring on conflicts here, we should opt to either update on conflict or change the upsert name.

Add a SHA-only helper for commit author/parents so later sync can reuse parsing without duplicating git format handling. Signed-off-by: Felipe Bergamin <felipebergamin@profusion.mobi>

- Move fetch timeout and max pack size to Django settings (GIT_FETCH_TIMEOUT_SECONDS, GIT_FETCH_MAX_PACK_BYTES) - Document URL preference tiers and why the git CLI is used - Reuse author/committer constants in tests

…nelci#2079) Fetch allowlisted trees into GIT_MIRROR_DIR and ingest the tip delta into commits / commit_parents (not checkout hashes). Probe filter support, keep kernel.org packs, split mirror vs ingest crons, and exclude stored tips with ^sha on rev-list stdin. Signed-off-by: Felipe Bergamin <felipebergamin@profusion.mobi>

Commit objects are immutable, so existing hashes stay as stored. Dry-run help now says the mirror fetch still updates the repo on disk, while ingest does not fetch.

Author and committer now live on commit_identity, and subject and message on commit_message.

Nested identity and message stores pushed commit_store over the complexity limit.

Record the mirror's on-disk size when sync_commit_mirror finishes so Prometheus can alert above 10 GB. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Felipe Bergamin <felipebergamin@profusion.mobi>

alanpeixinho · 2026-10-07T15:35:05Z

+
+    if not dry_run:
+        write_stored_tips(repo_dir, list_tips(repo_dir))
+        out("tips recorded; next run only ingests what is new")


I am unsure here ... but what happens if we fail on ingestion? could we potentially move the tip and loose some hashes from being ingested again later?

alanpeixinho · 2026-10-07T15:41:53Z

+            self.stdout.write("Commits migration completed")
+
+    def restore_commits(self) -> None:
+        with TextIOWrapper(self.snapshot_archive.extractfile("commits.csv")) as file:


I see we are checking for file existing before opening for the other csv, should we also add it here?

alanpeixinho · 2026-10-07T15:55:51Z

+        ).decode()
+    except (CommitMetadataError, ValueError) as exc:
+        logger.warning("verify-pack %s failed: %s", idx, exc)
+        return set()


if we have an error, we would return an empty set. Wouldnt it cause the pack_reject_reason to return None (no reason)?

alanpeixinho · 2026-10-07T17:23:33Z

+        finally:
+            self._record_mirror_size(mirror_dir)
+
+    def _record_mirror_size(self, mirror_dir: Path) -> None:


I dont know how much this is realy worth it, but maybe adding an option to run gc aggressive could help gain some space (but it is kinda expensive, so I dont know if this helps much).
This is really some idea being thrown, so, feel free to ignore if it could bring more trouble than it is worth it.

Did some test on mainline, and got from 3.56 GiB to 2.20 GiB. But it also took more than 30min :/

alanpeixinho · 2026-10-07T17:42:08Z

+          severity: warning
+        annotations:
+          summary: "Git commit mirror is larger than 10 GB"
+          description: "GIT_MIRROR_DIR is {{ $value | humanize1024 }}B after the last sync_commit_mirror run. Steady state is about 6–8 GB."


this is small, but we are using humanize1024 (base 1024), while our check is using base 1000, maybe we should use 10 * 1024 * 1024 * 1024

Pass --shallow-since only after ingest has caught up, and make sync_commit_gc delete unreachable objects immediately. A failed ingest no longer advances synced-tips. Co-authored-by: Cursor <cursoragent@cursor.com>

A per-tip cat-file asked the promisor remote for commits gc had already deleted, so the mirror fetch stalled before it started. Co-authored-by: Cursor <cursoragent@cursor.com>

alanpeixinho

Nice job

alanpeixinho · 2026-10-14T14:40:50Z

+    # After that 400, git sits on the dead connection instead of exiting. Give curl
+    # a stall deadline, and keep auto-gc from repacking a multi-GB mirror mid-run.
+    _git(repo, "config", "http.lowSpeedLimit", "1000")
+    _git(repo, "config", "http.lowSpeedTime", "60")


I was having some trouble fetching the repo, so I had to increase the lowSpeedTime threshold, might be safe to have higher values.

felipebergamin force-pushed the feat/2079/sync-commit-metadata branch from 2185f8e to 93ed169 Compare September 18, 2026 19:04

felipebergamin marked this pull request as ready for review September 21, 2026 19:28

alanpeixinho added 3 commits September 23, 2026 09:58

feat(backend): snapshot and restore commits in update_db

4ef6f7f

Dump commits and commit_parents in full so git ancestry survives restore. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

felipebergamin force-pushed the feat/2079/sync-commit-metadata branch from 93ed169 to 85dcd65 Compare September 23, 2026 20:19

alanpeixinho reviewed Sep 24, 2026

View reviewed changes

felipebergamin mentioned this pull request Sep 25, 2026

feat(backend): parse and fetch single-commit metadata (#2090) #2113

Closed

4 tasks

felipebergamin force-pushed the feat/2079/sync-commit-metadata branch from 85dcd65 to 16397d9 Compare September 25, 2026 20:00

felipebergamin added 6 commits September 25, 2026 17:36

feat(backend): parse and fetch single-commit metadata (kernelci#2090)

8f8b96d

Add a SHA-only helper for commit author/parents so later sync can reuse parsing without duplicating git format handling. Signed-off-by: Felipe Bergamin <felipebergamin@profusion.mobi>

fix(backend): address review on commit metadata fetch

a4fb0ae

- Move fetch timeout and max pack size to Django settings (GIT_FETCH_TIMEOUT_SECONDS, GIT_FETCH_MAX_PACK_BYTES) - Document URL preference tiers and why the git CLI is used - Reuse author/committer constants in tests

fix(backend): stop calling ignore-on-conflict an upsert

e4c426a

Commit objects are immutable, so existing hashes stay as stored. Dry-run help now says the mirror fetch still updates the repo on disk, while ingest does not fetch.

fix(backend): sort commit sync test imports

6fe41d3

fix(backend): write synced commits onto the split schema

07d64ab

Author and committer now live on commit_identity, and subject and message on commit_message.

felipebergamin force-pushed the feat/2079/sync-commit-metadata branch from 80974f5 to 07d64ab Compare September 25, 2026 21:01

felipebergamin and others added 2 commits September 25, 2026 18:10

fix(backend): move commit sync fakes out of the fixture

992b1fa

Nested identity and message stores pushed commit_store over the complexity limit.

feat(backend): expose git mirror size after each fetch

bfa9a45

Record the mirror's on-disk size when sync_commit_mirror finishes so Prometheus can alert above 10 GB. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Felipe Bergamin <felipebergamin@profusion.mobi>

felipebergamin force-pushed the feat/2079/sync-commit-metadata branch from ef2be10 to bfa9a45 Compare September 29, 2026 20:00

felipebergamin requested a review from alanpeixinho September 29, 2026 20:07

alanpeixinho reviewed Oct 7, 2026

View reviewed changes

felipebergamin and others added 2 commits October 8, 2026 19:38

feat(backend): prune the git mirror without losing uningested commits

93a9758

Pass --shallow-since only after ingest has caught up, and make sync_commit_gc delete unreachable objects immediately. A failed ingest no longer advances synced-tips. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(backend): stop the fetch from re-downloading pruned tips

c204b6c

A per-tip cat-file asked the promisor remote for commits gc had already deleted, so the mirror fetch stalled before it started. Co-authored-by: Cursor <cursoragent@cursor.com>

alanpeixinho approved these changes Oct 14, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(backend): sync git commit metadata into commits tables (#2079) - #2118

feat(backend): sync git commit metadata into commits tables (#2079)#2118
felipebergamin wants to merge 13 commits into
kernelci:mainfrom
profusion:feat/2079/sync-commit-metadata

felipebergamin commented Sep 15, 2026 •

edited

Loading

Uh oh!

alanpeixinho Sep 23, 2026

Uh oh!

alanpeixinho Sep 24, 2026

Uh oh!

alanpeixinho Sep 24, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho Oct 7, 2026

Uh oh!

alanpeixinho left a comment

Uh oh!

alanpeixinho Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

felipebergamin commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alanpeixinho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

felipebergamin commented Sep 15, 2026 •

edited

Loading