Skip to content

test(audit): add end-to-end integration suites for audit and debug commands #1819

Description

@jamesadevine

Problem

Unit analyzer tests do not prove the complete artifact download -> extraction -> root resolution -> analysis -> rendering path. A real proxy smoke exposed PipelineArtifact zips containing a repeated top-level artifact-name directory (agent_outputs_<id>/agent_outputs_<id>/...), which caused audit, OTel, firewall, MCP, proxy, and graph analyzers to look one level too high while unit tests stayed green.

Scope

  • Add deterministic integration fixtures representing actual ADO PipelineArtifact zip layouts, including both flat and repeated artifact-name roots.
  • Exercise ado-aw audit console and --json, ado-aw trace, MCP-author audit_build, and MCP-author trace_failure through their public entry points.
  • Cover agent, detection, and safe-output artifact families individually and together, including --artifacts selection and --no-cache refresh.
  • Include realistic sanitized logs for ado-proxy decisions/lifecycle, AWF firewall, MCPG, OTel/aw_info, detection verdict, safe-output proposals/execution, missing-tool/data/noop reports, and ADO timeline jobs.
  • Add a small live candidate-runner contract test that audits a completed smoke child after artifact publication, without relying on mutable hand-created build definitions.

Required assertions

  • Flat and repeated-root archives produce identical AuditData.
  • Every expected analyzer section is populated: ado_proxy_analysis, firewall_analysis, policy_analysis, mcp_server_health/tool usage/failures, metrics/engine config, detection, safe outputs, jobs, and pipeline graph when source is available.
  • Console, JSON, trace, audit_build, and trace_failure agree on shared facts.
  • Artifact filters never analyze excluded families.
  • Cached and --no-cache runs are equivalent after normalization.
  • Unknown/malformed schemas produce bounded warnings while other artifact evidence remains available.
  • No raw URLs/query values/header values/bodies/tokens or malformed source lines leak into rendered or structured output.
  • Missing/corrupt files degrade independently instead of suppressing unrelated analyzers.
  • Downloaded-file paths and artifact-root selection are deterministic across numeric build suffixes.
  • Findings/recommendations match expected denial, lifecycle, MCP, firewall, detection, and safe-output scenarios without changing build success semantics.

Acceptance

  • Integration suite runs offline and deterministically in normal Rust CI.
  • Live runner assertion is isolated to the existing candidate smoke framework and emits an actionable failure with build/artifact identifiers.
  • Regression test fails against the pre-fix repeated-root behavior.

Reference the discovery build 630125 and parent orchestrator 630120.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions