Skip to content

Inject OBO access token in apps run-local#5795

Open
ZaSkittles wants to merge 3 commits into
databricks:mainfrom
ZaSkittles:zack-loebel/apps-run-local-obo-token
Open

Inject OBO access token in apps run-local#5795
ZaSkittles wants to merge 3 commits into
databricks:mainfrom
ZaSkittles:zack-loebel/apps-run-local-obo-token

Conversation

@ZaSkittles

@ZaSkittles ZaSkittles commented Jul 2, 2026

Copy link
Copy Markdown

What did you change, and why?

databricks apps run-local starts a local proxy that fronts the app and
injects the identity headers the deployed Apps OAuth2 proxy would
(X-Forwarded-Email, X-Forwarded-User, ...) — but not
X-Forwarded-Access-Token. That token is the on-behalf-of (OBO) credential
the deployed platform forwards so an app can call Databricks APIs as the
calling user. Locally it is simply absent, so every OBO code path sees the
header as missing and fails, and OBO-gated surfaces can't be exercised in a
browser without deploying.

This injects X-Forwarded-Access-Token too, minted from the CLI's own
credentials (the same profile already used for CurrentUser.Me), matching
the deployed proxy.

  • libs/appproxy: add InjectHeaderFunc(key, fn) for a header whose value is
    resolved per request. Used for the token because it expires: the SDK token
    source returns a cached token without blocking in the steady state and
    refreshes on demand, so a browser session that outlives a single token keeps
    working. A resolution error fails the request with 502 rather than
    forwarding it without the header. Both the HTTP and WebSocket paths now go
    through a shared applyInjectedHeaders (on the WebSocket path the headers
    are applied before the connection is hijacked, so a resolution error's 502
    still reaches the client).
  • cmd/apps/run_local.go: inject the token from w.Config.GetTokenSource(),
    mirroring the existing idiom in cmd/apps/logs.go. Only OAuth-based auth can
    mint a token this way; for PAT/basic auth GetTokenSource() has nothing to
    exchange (a PAT is not a JWT assertion). Rather than 502 every request, we
    probe the source once at setup and, when it can't mint a token, log a warning
    and forward without the header — restoring the pre-OBO behavior for those
    users. OBO paths still can't be exercised without OAuth, and the warning says
    so and points at databricks auth login.
  • libs/apps/runlocal: export the header name as a constant.

How do you know it works?

Unit tests cover the new InjectHeaderFunc (per-request resolution, the
502-on-error path for HTTP and WebSocket, and the PAT fallback that omits the
header instead of 502ing). Manually verified end-to-end against a Lakehouse
App: a request straight to the app returns 401 (OBO fails, as before), while
the same request through the run-local proxy returns 200 with OBO-backed pages
rendering as the calling user.

@ZaSkittles ZaSkittles force-pushed the zack-loebel/apps-run-local-obo-token branch from f7263cb to 1c6609e Compare July 2, 2026 03:49
@ZaSkittles ZaSkittles marked this pull request as ready for review July 2, 2026 03:56
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/cmd/apps/ - approved by @MarioCadenas

Files: cmd/apps/run_local.go, cmd/apps/run_local_test.go

/libs/apps/ - approved by @MarioCadenas

Files: libs/apps/runlocal/headers.go

General files (require maintainer)

Files: NEXT_CHANGELOG.md, libs/appproxy/appproxy.go, libs/appproxy/appproxy_test.go
Based on git history:

  • @denik -- recent work in ./, cmd/apps/, libs/appproxy/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose) can approve all areas.
See OWNERS for ownership rules.

@theof-db theof-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to test this e2e locally?

@ZaSkittles

Copy link
Copy Markdown
Author

Were you able to test this e2e locally?

yes! I was able to run it against my app locally and see the token forwarded to my app. I wrote a note about this in the how it works section

@fjakobs fjakobs requested a review from MarioCadenas July 6, 2026 09:22

@MarioCadenas MarioCadenas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: PAT / basic-auth users get a 502 on every request — run-local becomes unusable for them

setupProxy unconditionally registers the token header from w.Config.GetTokenSource(), but that only returns a working source for OAuth-based auth. For PAT (token = dapi… / DATABRICKS_TOKEN) and basic auth, GetTokenSource() returns an errorTokenSource because the credentials provider isn't an OAuthCredentialsProvider (SDK config.Config.GetTokenSource; PAT uses credentials.FromCredentials). So the per-request tokenSource.Token(ctx) errors on every request, applyInjectedHeaders propagates it, and handleHTTP returns 502 and never forwards to the app.

This is a regression — PAT users could use run-local before this PR (static headers only). Reproduced end-to-end against this branch: a request that should return 200 instead returns

502 Bad Gateway
Error resolving injected header: OAuth Token not supported for current auth type pat

Suggested fix: when the auth type has no OAuth token, either (a) skip the token injection and still forward the request, or (b) fail fast at setupProxy with one actionable error (e.g. "run-local OBO token injection requires OAuth auth; you're using PAT") — rather than a silent per-request 502 with an opaque gateway error.

Also worth a test that drives a request through setupProxy with a non-OAuth config to lock this in — the current TestSetupProxyPortInUse uses a PAT config but never issues a request, so it misses this.

Comment thread libs/appproxy/appproxy.go Outdated
defer middlewareConn.Close()

// Inject headers on the upgrade request too, matching the HTTP path.
if err := p.applyInjectedHeaders(r); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebSocket 502 is written after Hijack(), so the client never sees it. applyInjectedHeaders(r) runs after hj.Hijack() + defer middlewareConn.Close(); once the connection is hijacked, http.Error(w, …) no longer reaches the client (w is detached), so on a token-resolution failure the client just gets a closed connection with no 502. Move this block to before hj.Hijack() so the error branch's http.Error still lands. (Not a leak — r.Write is correctly skipped.)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — moved applyInjectedHeaders (and its http.Error 502 branch) to before hj.Hijack(), so on a resolution failure the error still reaches the client instead of a bare closed connection. Added TestProxyInjectHeaderFuncErrorWebSocket to lock in that the 502 lands on the upgrade path.

Zack Loebel and others added 2 commits July 6, 2026 16:51
The deployed Apps platform fronts each slot with an OAuth2 proxy that
injects the caller's on-behalf-of (OBO) token as X-Forwarded-Access-Token.
`databricks apps run-local` starts a local proxy that injects the identity
headers (X-Forwarded-Email, etc.) but not the token, so OBO code paths
can't be exercised locally — the app sees the header as missing and every
OBO route fails.

Mint a token from the CLI's own credentials (the same profile already used
for CurrentUser.Me) and inject it as X-Forwarded-Access-Token. The value is
resolved per request via a new appproxy InjectHeaderFunc so a browser
session outliving a single token still gets a refreshed one; the SDK token
source returns a cached token without blocking in the steady state. A
resolution failure fails the request with 502 rather than forwarding it
without the header.

Co-authored-by: Isaac
- run-local: probe the token source once; on non-OAuth auth (PAT/basic)
  log a warning and forward without X-Forwarded-Access-Token instead of
  502ing every request. Restores pre-OBO behavior for those users.
- appproxy: move header injection before Hijack on the WebSocket path so
  a resolution error's 502 still reaches the client.
- tests: drive a request through setupProxy with a PAT config (no 502,
  header omitted); assert the WebSocket 502-before-Hijack path.

Co-authored-by: Isaac
@ZaSkittles ZaSkittles force-pushed the zack-loebel/apps-run-local-obo-token branch from 1c6609e to d8b6636 Compare July 6, 2026 18:15
@ZaSkittles

Copy link
Copy Markdown
Author

@MarioCadenas good catch, and thanks for reproducing it end-to-end — you're right that this was a regression for PAT/basic-auth users.

The root cause is that a PAT can't be turned into an OAuth token here. GetTokenSource() only yields OAuth tokens (it returns an errorTokenSource for PAT/basic auth), and the only exchange primitive in the SDK — ApiClient.GetOAuthTokenPOST /oidc/v1/token with the jwt-bearer grant — requires a JWT assertion as input. A PAT is an opaque dapi… string, not a JWT, so there's nothing to exchange; that endpoint re-scopes an OAuth token you already hold (it's how serving mints DataPlane OBO tokens), it can't bootstrap OAuth from a PAT. Short of forcing an interactive auth login, there's no way to mint one.

So I switched to a best-effort approach: probe the token source once at setupProxy; if it can't mint a token (PAT/basic auth), log a warning and forward without the header rather than 502 every request. That restores the pre-OBO behavior for those users — run-local works, they just don't get OBO locally, with a warning explaining why and pointing at databricks auth login.

Also added the test you asked for: TestSetupProxyPATOmitsTokenHeader drives a real request through setupProxy with a PAT config and asserts a 200 with the header omitted (not a 502).

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 5795
  • Commit SHA: 642be1bd283e07f06b3c8b3661b2fad8cbccc5d8

Checks will be approved automatically on success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants