Skip to content

fix(gitlab): Heal webhooks on integration reinstall#119340

Draft
billyvg wants to merge 2 commits into
masterfrom
billyvong/fix/gitlab-webhook-heal-on-reinstall
Draft

fix(gitlab): Heal webhooks on integration reinstall#119340
billyvg wants to merge 2 commits into
masterfrom
billyvong/fix/gitlab-webhook-heal-on-reinstall

Conversation

@billyvg

@billyvg billyvg commented Jul 9, 2026

Copy link
Copy Markdown
Member

Stacked on #119333 (webhook install-path logging). Review that first; this PR is the heal-only diff on top.

Summary

Fixes GitLab webhooks not being (re)created after uninstall → reinstall of the integration.

GitlabRepositoryProvider.on_create_repository no longer blindly trusts a stored webhook_id and returns early. It addresses the hook directly by id and lets GitLab be the source of truth:

Stored webhook_id Hook on GitLab Action
set exists update_project_webhook — refresh token + events in place (heals a rotated secret)
set gone (404) create a fresh hook, store the new id
unset create a fresh hook

Root cause

On uninstall, disassociate_organization_integration clears Repository.integration_id but leaves config["webhook_id"] intact and never deletes the GitLab hook. On reinstall, the base provider merges that stale webhook_id forward, and the old guard returned early — so no working webhook existed afterward. Reinstalling with a new OAuth application also rotates the webhook secret (sha1(hostname + client_id)), so the surviving hook's token stopped matching and inbound events were rejected (gitlab.webhook.invalid-token-secret). Updating the hook in place re-pushes the current token; a missing hook is recreated.

Why address by id (not list-and-match)

We only ever persist a webhook_id for a hook we created, and GitLab never reuses hook ids. So the stored id resolves to exactly one of two states — our hook, or 404. Addressing it directly:

  • Can't clobber another party's hook — we act on our own id, never enumerate and guess. A GitLab project can carry multiple hooks (other orgs, other tooling); we never touch them.
  • No pagination surface — no first-page get_project_webhooks scan that could miss the hook and create a duplicate.
  • Fewer calls — one PUT on the common heal path (previously GET-list + PUT); PUT + POST only when the hook is genuinely gone.

Rate-limit safety (re: the reverted #117801)

#117801 was reverted because it flooded GitLab: it hooked post_install and fanned out update_project_webhook across every org × every repo sharing the integration, unconditionally, on every (re)install.

This fix is structurally different:

  • Conditional: the extra GitLab call (a single update_project_webhook, plus a create only on 404) runs only when a repo already has a stored webhook_id. Fresh repos cost nothing extra (one POST, as before).
  • No steady-state cost: the periodic scm_repo_sync_beat sweep only feeds new/relinked repos into on_create_repository; in steady state nothing fires — the healing is a one-time post-reinstall reconciliation.
  • Bounded & per-tenant: the bulk sync path chunks work at SYNC_BATCH_SIZE = 100 per create_repos_batch task, scoped to a single org.
  • Self-throttling: a GitLab 429 → ApiRateLimitedError (not in the session auto-retry list) aborts the batch; create_repos_batch retries after delay=120s. No hammering.

Known follow-up: every reactivation with a stored webhook_id still issues a PUT even when nothing rotated. A stacked PR adds a local secret-fingerprint short-circuit so steady-state reactivations make zero GitLab calls — kept separate to keep this diff focused on correctness.

Observability (from #119333)

Install path logs on the sentry.integrations.gitlab logger: on_create_repository (every invocation), webhook_refreshed (hook updated in place), webhook_stale_recreated (stale id 404'd → recreated), webhook_created (fresh hook). Each on_create_repository ends in exactly one terminal event, logged after the GitLab call succeeds.

Notes

  • Scope stays within the GitLab provider; no change to the shared disassociate_organization_integration.
  • A non-404 failure while updating the hook surfaces as an error and does not fall through to create a duplicate hook (covered by tests).

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 9, 2026
@billyvg billyvg force-pushed the billyvong/ref/gitlab-webhook-creation-logging branch from e96bb0f to 11a8033 Compare July 9, 2026 20:24
@billyvg billyvg force-pushed the billyvong/fix/gitlab-webhook-heal-on-reinstall branch from 9768d24 to cd7f2ab Compare July 9, 2026 20:24
@sentry

sentry Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Sentry Snapshot Testing

Name Added Removed Changed Renamed Unchanged Skipped Status
sentry-frontend
sentry-frontend
0 0 0 0 451 0 ✅ Unchanged

⚙️ sentry-frontend Snapshot Settings

@billyvg billyvg force-pushed the billyvong/ref/gitlab-webhook-creation-logging branch from 11a8033 to 761c010 Compare July 9, 2026 20:34
@billyvg billyvg force-pushed the billyvong/fix/gitlab-webhook-heal-on-reinstall branch 2 times, most recently from f18fe26 to c95ae46 Compare July 10, 2026 17:42
@billyvg billyvg force-pushed the billyvong/ref/gitlab-webhook-creation-logging branch from 761c010 to c39d665 Compare July 10, 2026 18:31
@billyvg billyvg force-pushed the billyvong/fix/gitlab-webhook-heal-on-reinstall branch from c95ae46 to 16058ad Compare July 10, 2026 18:31
Base automatically changed from billyvong/ref/gitlab-webhook-creation-logging to master July 10, 2026 20:32
billyvg and others added 2 commits July 10, 2026 16:32
Add structured logging to GitlabRepositoryProvider.on_create_repository so
the webhook install path is observable in Sentry Logs (no Datadog metrics).

Three logs, all on the sentry.integrations.gitlab logger:
- gitlab.repository.on_create_repository on every invocation, with
  has_existing_webhook, to gauge how often the path runs.
- gitlab.repository.webhook_created on success, so actual GitLab hook
  creation calls can be counted (i.e. request volume to GitLab).
- gitlab.repository.webhook_creation_skipped when a stale webhook_id is
  already present and creation is skipped.

No change to webhook creation behavior. This baseline lets us confirm that
reinstalls silently skip webhook creation (stale webhook_id survives
uninstall) and verify any future fix without flooding GitLab with requests.

Co-Authored-By: Claude <noreply@anthropic.com>
When a GitLab integration is uninstalled and reinstalled, the stored
webhook_id survives in Repository.config (disassociate_organization_integration
clears integration_id but leaves the config intact and never deletes the
GitLab hook). The old guard saw a truthy webhook_id and returned without
touching GitLab, so no working webhook existed after reinstall. If the
reinstall used a new OAuth app, the webhook secret rotated and the surviving
hook's token no longer matched, so incoming events were rejected.

Verify-then-refresh/create in on_create_repository instead of trusting a
stored webhook_id: list the project's hooks and, if the stored hook is still
present, refresh its token + events (healing the rotated-secret case); if it
is gone, create a fresh hook and store the new id.

Co-Authored-By: Claude <noreply@anthropic.com>
@billyvg billyvg force-pushed the billyvong/fix/gitlab-webhook-heal-on-reinstall branch from 16058ad to 6cfc743 Compare July 10, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant