Task/db backed OIDC

[muzamilkm]

Summary

Moves native DevLake OIDC provider configuration from deployment environment variables into the existing MySQL-backed access-management feature.

Customer administrators can now validate, save and activate an OIDC provider from Config UI without changing deployment configuration or restarting DevLake. The client secret is encrypted before persistence and the database becomes the authoritative provider source after activation.

The same validated provider is also synchronized to Grafana Generic OAuth through Grafana's supported SSO Settings API, so customers configure their IdP once in DevLake while DevLake and Grafana continue to own their own users and sessions.

Included

• Adds database-backed OIDC provider configuration, candidate staging and source activation state.
• Encrypts persisted OIDC client secrets with AES-256-GCM using deployment-owned key material. Client secrets remain write-only and are never returned through the API.
• Adds customer-admin APIs and an Authentication section under User Management for:
  • validating provider discovery
  • saving and updating a staged provider
  • activating database-backed configuration
  • retrying Grafana synchronization
  • enabling, disabling and retiring provider configuration where allowed
• Keeps the existing native DevLake OIDC authorization-code, PKCE, nonce/state, verified-email, session and access-directory flows rather than introducing a second authentication path.
• Adds a database-backed provider runtime that replaces the environment provider only after explicit activation and does not fall back to stale environment credentials afterward.
• Adds restricted outbound OIDC discovery, JWKS and token-exchange networking with HTTPS enforcement, private-address rejection, redirect limits and bounded timeouts.
• Preserves configured secrets for non-secret provider edits without exposing the stored credential to handlers or the browser. Changing the OAuth client identity requires an explicit replacement secret.
• Adds provider revision and Grafana synchronization state so partial cross-service updates are visible and retryable rather than silently treated as successful.
• Synchronizes Generic OAuth to Grafana through PUT /api/v1/sso-settings/generic_oauth using a dedicated deployment-owned Grafana server-admin machine identity.
• Keeps Grafana users, roles, sessions and database ownership independent from DevLake. No Grafana database tables or Grafana source code are modified.
• Adds safe compensation/recovery handling when Grafana accepts an activation but the DevLake database transition cannot complete.
• Adds audit events for provider lifecycle and Grafana synchronization without logging credentials, tokens or upstream secret material.
• Adds local/deployment configuration for backend-only OIDC encryption and Grafana management credentials.

Authentication and Grafana Boundary

DevLake remains the customer-facing source of truth for OIDC configuration.

The customer registers one confidential OIDC client with both application callbacks:

https://<devlake-host>/api/auth/callback
https://<grafana-host>/login/generic_oauth

DevLake performs discovery and uses the resulting provider configuration for its existing native OIDC flow. The same validated configuration is mapped into Grafana Generic OAuth through Grafana's SSO Settings API.

Grafana continues to own its own user database and session. Existing pre-created Grafana users are still matched through the existing Generic OAuth flow and clicking Dashboards continues to reuse the browser's IdP session for a seamless second login.

Grafana OSS does not allow an ordinary service-account token to obtain the instance-level settings:read / settings:write permissions required by the SSO Settings API. The integration therefore uses a dedicated local Grafana server-admin machine account supplied as a backend-only deployment secret. This account is separate from the customer's human Grafana administrator.

Scope and Upstream Coupling

Most of the feature is contained within the existing fork-owned access/authentication boundaries:

• backend/server/api/access
• backend/server/api/auth
• config-ui/src/routes/access

Upstream-facing changes are limited to the required integration points:

• OIDC helper support for database-provided runtime configuration and restricted HTTP transport
• auth runtime/provider source selection
• API route registration
• one additive migration registration
• deployment configuration

The implementation does not modify Grafana source, Grafana database internals, unrelated plugins or shared Config UI infrastructure.

Verification

The implementation has been validated locally across the full provider lifecycle, including:

• database migration and provider/candidate persistence
• AES-256-GCM encryption, tamper rejection and key rotation
• issuer and scope validation
• private-address and cloud-metadata SSRF rejection
• staged provider revisions without disrupting the active environment provider
• customer-admin RBAC and CSRF enforcement
• secret redaction in API responses and audit records
• environment-to-database provider activation
• live Google OIDC sign-in after database activation
• Grafana Generic OAuth synchronization through the SSO Settings API
• live Grafana authentication reload without container restart
• Grafana synchronization retry
• active-provider disable/retire guards
• REST API-key authentication remaining independent from interactive OIDC
• public callback URL handling without leaking internal Docker hostnames

Focused checks include:

go test ./server/api/access -count=1
go test ./helpers/oidchelper -count=1
go vet ./server/api/access ./helpers/oidchelper
go build ./server/api/auth ./server/api/access ./helpers/oidchelper ./server/api
Config UI production build
route-local Config UI utility tests
gofmt
git diff --check

The local end-to-end test run covered 24 lifecycle/security scenarios and completed successfully, including live DevLake Google login and live Grafana Generic OAuth synchronization.

Future Work

• Multiple active OIDC providers and identity linking. The current access directory stores one (issuer, subject) identity directly on an access user, so this PR deliberately supports one active database provider at a time.
• Supporting Google, Entra or other IdPs concurrently for the same person requires a separate identity model, such as auth_access_identities, plus explicit identity linking, session revocation across identities and migration of existing users.
• This is intentionally deferred because the current PR already covers the complete database configuration, encrypted-secret, runtime cutover, Grafana synchronization and customer-admin workflow. Adding multi-provider identity semantics here would substantially expand an already security-sensitive change.
• Local email/password authentication remains a separate follow-up and can later use the same identity model if required.

Staging and Production Follow-up

Application support does not provision the deployment-owned secrets or Grafana machine identity itself.

Before rollout, deployment automation/DevOps must provide:

• the OIDC credential-encryption key material to the Lake backend only
• canonical public DevLake and Grafana URLs
• a dedicated Grafana devlake-system server-admin machine user and random password
• the Grafana management credential to the Lake backend only

The Grafana machine identity can be provisioned using Grafana's documented server-admin APIs and is independent from the customer-facing Grafana administrator.

Staging should first migrate the existing Google provider from environment configuration into the database, verify DevLake and Grafana login after activation and then remove the old provider credentials from the runtime environment.

[ubaidkhalidarbisoft]

Reviewed the whole change. The architecture is sound — the two-phase candidate/activation model, the write-only secret contract, AES-GCM with per-provider AAD, and the restricted egress transport are all the right shapes for this problem, and the test coverage on the crypto and normalization helpers is genuinely good.

That said, I found several things I'd want resolved before this ships, plus a number of smaller ones. Inline comments have the detail; summarizing the ones I'd treat as blockers:

1. EnableOIDCProvider silently no-ops while logging every user out. oidcProviderFromCandidate doesn't copy ID, so when a candidate revision is staged the provider passed into setOIDCProviderEnabled has ID == 0. The UPDATE ... WHERE id = 0 matches zero rows, but RevokeProviderSessions still revokes every session for that provider and the handler returns 200 with enabled: true. UpdateColumns doesn't surface rows-affected, so nothing catches it.

2. Retiring a provider permanently burns its key and issuer. RetireOIDCProvider sets retired_at but the unique indexes on provider_key and issuer_url cover retired rows too. Re-creating a provider with the same key/issuer afterwards hits a constraint violation and surfaces as a 500 with no path forward short of manual SQL.

3. Saving a draft provider mutates live Grafana. In SaveOIDCProvider, the pre-activation path calls syncGrafana(..., enabled: false, ...), which PUTs the new client into Grafana's generic_oauth settings and disables it. If the customer already had Grafana Generic OAuth configured (via ini/env or a previous provider), that config is overwritten and OAuth login to Grafana breaks — before the admin has activated anything, and with no restore path if they never do.

4. A transient DB error during RefreshOIDCProvider locks everyone out until restart. The error branch calls replaceProviderState(databaseOIDCUnavailableConfig(...)), which installs an empty provider map. Fail-closed is the right instinct, but there's no retry and no self-heal, so a momentary DB blip during an enable/activate turns into an indefinite login outage.

Smaller items (SSRF edge cases, swallowed Grafana client construction error, unreachable backend routes, a UI refresh gap that hides the retry button) are inline.

---

Edited after review discussion. I originally listed a fifth blocker — gorm:"type:blob" being invalid Postgres DDL. @muzamilkm pushed back that this deployment is MySQL, and that's substantially right: the fork already uses MySQL-only type:longtext in the notion/hubspot/salesforce models, so this is an established pattern rather than an oversight. I've downgraded that comment to non-blocking; the suggestion there is now just a zero-cost hedge (dropping the tag changes nothing on MySQL). Leaving the correction visible rather than silently editing it away.

One process note: the PR description says the local run "covered 24 lifecycle/security scenarios and completed successfully." Blockers 1, 2, and 3 above are all lifecycle scenarios, so it'd be worth reconciling what those 24 actually exercised — I suspect the candidate-revision paths weren't among them. Related: see my separate comment on CI coverage, which is why none of this was caught automatically.