debug: permissions.read token is not exported to Azure CLI in Agent sandbox

[krubenok]

Diagnostic Summary

• Pipeline: Internal Azure DevOps PR review pipeline (redacted)
• Definition ID: Redacted
• Build ID: Redacted
• Run URL: Redacted internal ADO URL
• Result: Failed
• Failing stage/job/step: Stage 3 → SafeOutputs → Execute safe outputs (Stage 3)
• First failed timeline record: Execute safe outputs (Stage 3) (Bash task, exit code 1)
• Suspected root cause: With permissions.read configured, ado-aw v0.45.1 tells the agent that az devops is authenticated through $AZURE_DEVOPS_EXT_PAT, but the generated Agent step does not export that variable. An agent path that uses direct Azure CLI therefore fails authentication and emits report-incomplete, which intentionally fails Stage 3.
• Confidence: High

Evidence

Relevant log excerpts

The read token was successfully acquired before the Agent job:

Starting: Acquire ADO token (SC_READ_TOKEN)
...
Finishing: Acquire ADO token (SC_READ_TOKEN)

MCPG received the token and successfully launched the Azure DevOps MCP server:

[LAUNCHER] Env passthrough: ADO_MCP_AUTH_TOKEN=... (from MCPG process)
Routes: /mcp/<server> where <server> is one of: [azure-devops safeoutputs]

The generated prompt told the agent:

Azure DevOps management — az devops, az pipelines, az repos, az boards.
These are authenticated automatically from $AZURE_DEVOPS_EXT_PAT when the
pipeline declares permissions: read:. List/inspect operations Just Work.

The agent used direct az devops (in this run after loading a review skill) and reported:

az devops invoke ... no authenticated PAT/AAD identity was available

Stage 3 then processed the resulting report-incomplete record and failed by design:

Found 1 safe output(s) to execute
[1/1] report-incomplete - ✗ - Agent reported task incomplete: ...
Total: 1 | Success: 0 | Warnings: 0 | Failed: 1
Bash exited with code '1'.

Generated YAML mismatch

The token is only passed to MCPG:

-e ADO_MCP_AUTH_TOKEN="$SC_READ_TOKEN"
...
env:
  SC_READ_TOKEN: $(SC_READ_TOKEN)

The generated Run copilot (AWF network isolated) step has no SC_READ_TOKEN or AZURE_DEVOPS_EXT_PAT entry:

env:
  GITHUB_TOKEN: $(GITHUB_APP_TOKEN)
  GITHUB_READ_ONLY: 1
  COPILOT_OTEL_ENABLED: 'true'
  COPILOT_OTEL_EXPORTER_TYPE: file
  COPILOT_OTEL_FILE_EXPORTER_PATH: /tmp/awf-tools/staging/otel.jsonl

This matches the v0.45.1 compiler test in src/compile/common.rs, which explicitly asserts that the engine environment does not contain AZURE_DEVOPS_EXT_PAT because the token is handled by MCPG. That design conflicts with src/compile/extensions/azure_cli.rs, whose documentation and generated prompt state that the variable is set inside AWF for direct Azure CLI use.

Timeline observations

• Agent succeeded.
• Detection succeeded.
• SafeOutputs failed while executing the single report-incomplete output.
• The Conclusion job ran because it uses and(always(), not(canceled())).
• No 401/403 occurred in Stage 3; docs/safe-output-permissions.md does not apply to this failure.

Changes since last successful build

Unknown. The failure is reproducible from the generated v0.45.1 YAML shape and does not depend on compilation drift.

Environment

• Agent source file: Internal path redacted
• Compiled pipeline YAML: Internal path redacted
• Compilation in sync: Yes (ado-aw check returned OK)
• ado-aw version: 0.45.1
• AWF version: 0.27.9
• MCPG version: 0.4.1
• Agent pool: Microsoft-hosted Ubuntu pool
• OS/image: ubuntu-22.04
• Engine/model: Copilot (model redacted)
• Relevant MCP servers: Azure DevOps MCP, SafeOutputs

Analysis

• Stage classification: Cross-stage cause surfaced in Stage 3 (SafeOutputs)
• Why this stage failed: The Agent could not authenticate a direct az devops read because the advertised $AZURE_DEVOPS_EXT_PAT was absent. It correctly emitted report-incomplete; Stage 3 treats that output as failed execution and exits nonzero.

The Azure DevOps MCP route itself was healthy and authenticated. The problem is the separate direct-CLI contract introduced by the always-on Azure CLI extension: the prompt advertises authentication that the generated Agent environment does not provide.

Root Cause

• Root cause: Compiler/runtime contract mismatch in v0.45.1: permissions.read is mapped to ADO_MCP_AUTH_TOKEN for MCPG only, while the Azure CLI extension promises AZURE_DEVOPS_EXT_PAT inside AWF without generating the required Agent-step environment mapping.
• Category: Compiler bug
• Ruled-out causes: stale compiled YAML; failure to mint the read token; MCPG startup/routing failure; Stage 3 ADO 401/403; service-connection authorization failure
• Related recent changes: Azure CLI always-on extension and its generated advisory

Issue

• Title: debug: permissions.read token is not exported to Azure CLI in Agent sandbox
• Label: bug

[jamesadevine]

Thanks - aware of this - I have a fix, but want to add a permission bitmask check before restoring this feature.

[jamesadevine]

Also sorry for the delay all up, I have been working on tests for the past few weeks to up the signal given to the agents. The tests uncovered this issue!

[krubenok]

Should we consider allowing the ADO MCP version being defined in the front matter so that we can have an escape hatch for this in future?

[jamesadevine]

Yes! Indeed - there's a lot of hardening to do in this area.

Minimal fix is to allow greater azure-devops mcp configuration. including version (or sha) pinning if possible.

I am not going to fix azure cli support yet. I am looking at options to better fix this, but in essence, I cannot trust users to configure service connections correctly. There are also no simple checks to ensure a service connection/token is read only. Any detection has to be comprehensive, and even a comprehensive detection can miss capabilities. For this to work safely and deliver the promises of aw, work is required.

[jamesadevine]

DevOps MCP is the viable alternative route for now, at least this permits constrained tool access via frontmatter config...

[jamesadevine]

#1824 and release 0.49.0 contain ado-proxy which sits in between any call to azure devops. It blocks any operation except GETs (reads) and can be scoped using the permissions.read front matter. As of this version az and azure devops mcp support is fully reenabled.