Enable audit logging for external API calls

[peteroden]

Problem

The agent makes security-sensitive external API calls to GitLab, Jira, and Copilot/LLM providers, as well as performs critical operations like git push, MR comment posting, and approval gate decisions. Currently, there is structured logging via structlog but no dedicated audit trail for security-sensitive operations.

Without audit logging, we cannot:

• Detect unauthorized API calls or suspicious patterns
• Investigate security incidents (e.g., token compromise, webhook replay)
• Demonstrate compliance with security logging standards (OWASP A09, SOC 2)
• Track which actor triggered which external action
• Correlate audit events back to originating webhooks or poll cycles

Business Context

Why now? This is item #7 from the Recommended Hardening list in docs/wiki/security-model.md. As the agent operates in production environments with access to GitLab repositories, Jira projects, and LLM APIs via sensitive tokens (GITLAB_TOKEN, JIRA_API_TOKEN, GITHUB_TOKEN), we need visibility into security-sensitive operations.

Impact of not solving:

• No forensic trail if tokens are compromised
• Cannot detect malicious webhook replay attacks
• Cannot prove compliance during security audits
• Limited ability to investigate customer-reported issues involving agent actions

OWASP Mapping: This addresses A09:2021 – Security Logging and Monitoring Failures.

Success Metrics

1. Coverage: 100% of external API calls (GitLab, Jira, Copilot SDK) emit audit log entries
2. Traceability: Every audit entry includes correlation ID linking back to originating webhook/poll event
3. Searchability: Audit logs can be queried by actor, action, target, timestamp
4. Performance: Audit logging adds <5ms overhead per operation
5. Zero data loss: Audit entries are never dropped (even if external sink is unavailable)

User Stories

US-1: Security Engineer investigates suspicious activity

As a security engineer
I want to query audit logs for all GitLab API calls made by the agent in the last 24 hours
So that I can investigate whether a token was compromised and identify affected resources

Acceptance Criteria:

• Audit log entries include: timestamp (ISO 8601 with timezone), actor (webhook event ID or poll cycle ID), action (e.g., gitlab.api.post_mr_comment), target (e.g., project_id=123, mr_iid=45), outcome (success/failure), correlation_id
• Logs are queryable by time range, action type, target resource
• Failed operations are logged with error classification (e.g., auth_failure, rate_limit, network_error)

---

US-2: Compliance auditor verifies authentication events

As a compliance auditor
I want to review all webhook authentication successes and failures
So that I can verify that only valid HMAC-authenticated requests are processed

Acceptance Criteria:

• Audit log entry for every webhook POST to /webhook endpoint
• Entry includes: HMAC validation result (valid or invalid), source IP (if available from headers), project_id (if valid), rejection reason (if invalid)
• Invalid HMAC attempts are logged with full context (timestamp, IP, project_id if parseable)

---

US-3: Operator traces coding command execution

As a platform operator
I want to trace all /copilot command executions from MR comment to git push
So that I can debug failed coding workflows and verify approval gate enforcement

Acceptance Criteria:

• Audit trail for: MR comment received (/copilot command), coding task started (K8s Job or local executor), patch generated, patch validated, git push executed, approval gate decision (approved/rejected)
• All entries share the same correlation_id
• Approval gate decisions include: approval_required (true/false), approver (username if manual approval), decision_reason

---

US-4: DevOps configures audit log destination

As a DevOps engineer
I want to configure where audit logs are sent (stdout, file, external SIEM)
So that I can integrate audit logs with our existing security monitoring infrastructure

Acceptance Criteria:

• Config option: AUDIT_LOG_DESTINATION (env var) with values: stdout (default), file, syslog, http
• If file: AUDIT_LOG_FILE_PATH specifies target file (supports log rotation)
• If http: AUDIT_LOG_HTTP_ENDPOINT specifies target URL (JSON POST)
• Audit logs use structured JSON format with consistent schema
• Fallback: if external sink fails, log to stdout + emit error metric

---

Acceptance Criteria (Overall)

AC-1: External API Call Coverage

• All GitLab API calls logged:
  • gitlab.api.get_mr_details
  • gitlab.api.get_mr_diff
  • gitlab.api.post_mr_note (review comments)
  • gitlab.api.post_commit_discussion
  • gitlab.api.get_project_info
  • gitlab.api.clone_repo (git clone via URL)
  • gitlab.api.push_branch (git push)
• All Jira API calls logged:
  • jira.api.get_issue
  • jira.api.transition_issue
  • jira.api.add_comment
  • jira.api.search_issues
• All Copilot/LLM calls logged:
  • copilot.sdk.review_request (review mode)
  • copilot.sdk.coding_request (coding mode)
  • copilot.sdk.response_received

AC-2: Authentication Events

• Webhook HMAC validation success logged with project_id, event_type
• Webhook HMAC validation failure logged with rejection_reason, source_ip (if available)

AC-3: Sensitive Operations

• Git push operations logged with: repo, branch, commit_sha, correlation_id
• MR comment posts logged with: project_id, mr_iid, comment_type (review/note), correlation_id
• /copilot command executions logged with: command, project_id, mr_iid, initiator (comment author), correlation_id
• Approval gate decisions logged with: decision (approve/reject), reason, approver, correlation_id

AC-4: Log Entry Schema

Each audit log entry includes:

{
  "timestamp": "2025-01-15T14:32:15.123Z",
  "event_type": "audit",
  "action": "gitlab.api.post_mr_note",
  "actor": {
    "type": "webhook_event" | "poll_cycle" | "admin_action",
    "id": "<correlation_id>"
  },
  "target": {
    "resource_type": "merge_request",
    "project_id": 123,
    "mr_iid": 45
  },
  "outcome": "success" | "failure",
  "error": "<error_classification if failure>",
  "correlation_id": "<UUID>",
  "service": "gitlab-copilot-agent"
}

AC-5: Configuration

• AUDIT_LOG_ENABLED env var (default: true)
• AUDIT_LOG_DESTINATION env var (values: stdout, file, http)
• AUDIT_LOG_FILE_PATH env var (if destination=file)
• AUDIT_LOG_HTTP_ENDPOINT env var (if destination=http)
• Validation: service fails to start if destination=file and path is unwritable

AC-6: Correlation IDs

• Each webhook event generates a unique correlation_id (UUID) at ingress (webhook.py)
• Poll cycles generate correlation_id per poll iteration
• correlation_id propagated through: task execution, K8s Job pods (via env var), Redis results, git operations, API calls
• All audit log entries for a single webhook include the same correlation_id
• correlation_id included in OpenTelemetry trace context

AC-7: Performance & Reliability

• Audit logging is asynchronous (does not block API calls)
• If external sink is unavailable, logs written to stdout as fallback
• Audit log buffer size configurable (default: 1000 entries)
• Metric: audit_log_entries_total counter (labels: action, outcome)
• Metric: audit_log_sink_errors_total counter (labels: destination)

---

Scope

In Scope

• Audit log entries for all external API calls (GitLab, Jira, Copilot SDK)
• Audit log entries for webhook HMAC validation (success/failure)
• Audit log entries for sensitive operations (git push, MR comments, /copilot commands, approval gates)
• Structured JSON schema for audit log entries
• Correlation ID generation and propagation
• Configuration for audit log destination (stdout, file, http)
• Async audit logging (non-blocking)
• Fallback to stdout if external sink fails
• Prometheus metrics for audit log volume and errors

Out of Scope

• Full audit log aggregation/indexing system (use external SIEM like Splunk, Datadog)
• Audit log retention policies (handled by external log sink)
• Real-time alerting on suspicious patterns (handled by SIEM)
• Audit log encryption at rest (handled by external sink)
• Audit log signing/tamper-proofing (future enhancement)
• Backward compatibility with existing log format (this is a new structured audit stream)

---

Dependencies

• Existing structured logging infrastructure (structlog)
• OpenTelemetry trace context propagation (telemetry.py)
• Redis state storage (for correlating Job pod results with correlation_id)
• Kubernetes Job executor (for propagating correlation_id to pods)

---

Risks

1. Performance overhead: Audit logging adds latency to API calls
   • Mitigation: Async logging with buffering; benchmark shows <5ms overhead acceptable
2. Log volume explosion: High-traffic agents generate GB/day of audit logs
   • Mitigation: Sampling config (e.g., only log errors + 10% of successes) as future enhancement
3. Correlation ID propagation gaps: Some code paths may not propagate correlation_id
   • Mitigation: Code review checklist; add correlation_id to all Task models and executor methods
4. External sink failures: If HTTP endpoint is down, logs could be lost
   • Mitigation: Fallback to stdout; retry queue with bounded buffer

---

References

• Security Model: docs/wiki/security-model.md (item feat(jira): Jira poller and config #7 in Recommended Hardening)
• OWASP A09: Security Logging and Monitoring Failures
• Structured Logging: Current implementation uses structlog (telemetry.py)
• OpenTelemetry: Trace context provides correlation_id foundation

---

Implementation Notes

Suggested approach (for Developer Agent):

1. Create audit_logger.py module with AuditLogger class
2. Add correlation_id field to webhook/poll event models
3. Wrap all GitLab/Jira/Copilot API calls with audit log decorators
4. Add audit log entries in webhook.py for HMAC validation
5. Propagate correlation_id through Task models and executor methods
6. Add config validation in config.py
7. Add Prometheus metrics in metrics.py
8. Add integration tests verifying audit log entries for key operations

Estimated effort: Medium (3-5 PRs, stack structure recommended)
Priority: High (security hardening item)