Problem
Unit analyzer tests do not prove the complete artifact download -> extraction -> root resolution -> analysis -> rendering path. A real proxy smoke exposed PipelineArtifact zips containing a repeated top-level artifact-name directory (agent_outputs_<id>/agent_outputs_<id>/...), which caused audit, OTel, firewall, MCP, proxy, and graph analyzers to look one level too high while unit tests stayed green.
Scope
- Add deterministic integration fixtures representing actual ADO PipelineArtifact zip layouts, including both flat and repeated artifact-name roots.
- Exercise
ado-aw audit console and --json, ado-aw trace, MCP-author audit_build, and MCP-author trace_failure through their public entry points.
- Cover agent, detection, and safe-output artifact families individually and together, including
--artifacts selection and --no-cache refresh.
- Include realistic sanitized logs for ado-proxy decisions/lifecycle, AWF firewall, MCPG, OTel/aw_info, detection verdict, safe-output proposals/execution, missing-tool/data/noop reports, and ADO timeline jobs.
- Add a small live candidate-runner contract test that audits a completed smoke child after artifact publication, without relying on mutable hand-created build definitions.
Required assertions
- Flat and repeated-root archives produce identical AuditData.
- Every expected analyzer section is populated: ado_proxy_analysis, firewall_analysis, policy_analysis, mcp_server_health/tool usage/failures, metrics/engine config, detection, safe outputs, jobs, and pipeline graph when source is available.
- Console, JSON, trace, audit_build, and trace_failure agree on shared facts.
- Artifact filters never analyze excluded families.
- Cached and
--no-cache runs are equivalent after normalization.
- Unknown/malformed schemas produce bounded warnings while other artifact evidence remains available.
- No raw URLs/query values/header values/bodies/tokens or malformed source lines leak into rendered or structured output.
- Missing/corrupt files degrade independently instead of suppressing unrelated analyzers.
- Downloaded-file paths and artifact-root selection are deterministic across numeric build suffixes.
- Findings/recommendations match expected denial, lifecycle, MCP, firewall, detection, and safe-output scenarios without changing build success semantics.
Acceptance
- Integration suite runs offline and deterministically in normal Rust CI.
- Live runner assertion is isolated to the existing candidate smoke framework and emits an actionable failure with build/artifact identifiers.
- Regression test fails against the pre-fix repeated-root behavior.
Reference the discovery build 630125 and parent orchestrator 630120.
Problem
Unit analyzer tests do not prove the complete artifact download -> extraction -> root resolution -> analysis -> rendering path. A real proxy smoke exposed PipelineArtifact zips containing a repeated top-level artifact-name directory (
agent_outputs_<id>/agent_outputs_<id>/...), which caused audit, OTel, firewall, MCP, proxy, and graph analyzers to look one level too high while unit tests stayed green.Scope
ado-aw auditconsole and--json,ado-aw trace, MCP-authoraudit_build, and MCP-authortrace_failurethrough their public entry points.--artifactsselection and--no-cacherefresh.Required assertions
--no-cacheruns are equivalent after normalization.Acceptance
Reference the discovery build
630125and parent orchestrator630120.