Skip to content

test(executor-e2e): add create-github-issue and set-github-issue-type scenarios #1798

Description

@jamesadevine

Task: add create-github-issue + set-github-issue-type scenarios to executor-e2e

Add deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in githubnext/ado-aw. Both currently ship with zero runtime
coverage
— their only proof is a wiremock unit test.

Background

create-github-issue and set-github-issue-type were added by PR #1670
(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and removed the smoke-failure-reporter smoke case, which
was the only thing exercising create-github-issue at runtime. Net effect:
coverage went from "thin" to "none".

executor-e2e excludes it by design today. tests/executor-e2e/README.md
says: "Excluded (out of scope or GitHub-only): the GitHub-only
create-issue."
That exclusion predates the tools being first-class public
safe outputs, and is what you are reversing.

Confusable, do not be misled: scripts/ado-script/src/executor-e2e/github-issue.ts
is the harness's own failure reporter (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.

What to build

Add scripts/ado-script/src/executor-e2e/scenarios/github-issue.ts exporting
githubIssueScenarios, and register it in scenarios/index.ts (append to
allScenarios — the order is deterministic and existing entries must not move).

Cover, at minimum:

  1. create-github-issue — files an issue, asserts it exists on GitHub with
    the expected title (including the configured title-prefix), body, and
    config-injected labels.
  2. set-github-issue-type — sets the issue type on an existing issue.
  3. The same-run temporary-ID handoff — this is the highest-value case and
    the one with the weakest current proof. create-github-issue accepts an
    optional temporary_id, and set-github-issue-type accepts an
    issue_number that may be either a real number or that temporary id. A
    single executor run must resolve the handoff. A wrong REST shape or a
    scoping failure here is invisible today.

Consider also (use judgement, do not gold-plate):

  • allowed-labels rejection — config is default-deny for labels, unlike
    set-github-issue-type.allowed which is default-allow. Use
    expectedFailure for the rejection path.
  • Clearing an issue type: issue_type: "" is the documented clear operation.

The contract you must follow

Read scripts/ado-script/src/executor-e2e/scenario.ts first — it is the
authoritative contract and is well commented. Key points:

  • A scenario is { tool, config, setup, ndjson, assert, cleanup } with an
    optional id (set id when several scenarios share one tool),
    files, env, and expectedFailure.
  • ndjson() returns the executor entry without the name field — the
    runner injects name: <tool>.
  • No LLM is involved. You craft the NDJSON directly, run the real ado-aw execute binary, then assert via REST.
  • ScenarioContext gives you buildId, prefix(tool)
    (ado-aw-det-<buildId>-<tool>), workDir, log, and rest.
  • Throw SkipError from setup() when a precondition is missing, so an
    incomplete environment records as skipped, not failed.
  • Cleanup gotcha: if setup() throws a non-SkipError, cleanup() is
    NOT called — tear down anything you partially created before rethrowing.
    See scenarios/pr.ts setupPr for the established pattern.
  • assert() may populate state that cleanup() needs (e.g. the created issue
    number read off the executor result) — do that before any fallible check,
    so a later assertion failure still leaves cleanup able to tear down.

scenarios/wiki.ts is the cleanest model: env-var override, REST discovery,
SkipError, deterministic naming, real cleanup.

Two things that make this genuinely different from every existing scenario

1. It asserts against GitHub, not ADO. ScenarioContext.rest is AdoRest
and will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in
executor-e2e/github-issue.ts (findOpenIssueByTitle, createGitHubIssue,
diagnoseGitHubAuthFailure) rather than writing a fourth GitHub client — lift
them into a shared module if that is cleaner, but do not duplicate them.

2. GitHub issues cannot be deleted. Every other scenario tears its objects
down completely. You cannot. cleanup() must close the issue (and prefer a
[e2e]-style title prefix plus the ado-aw-det-<buildId>-… marker so leaked
issues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.

Environment

  • Definition 2550 (executor e2e) already has EXECUTOR_E2E_GITHUB_TOKEN
    provisioned
    — no new secret provisioning is required. Confirm the token's
    scope actually permits issue-type mutation on the target repo; issue types
    are an org-level construct and may need more than plain Issues write. If it
    does not, SkipError is the correct response, not a hard failure.
  • Existing env vars follow the E2E_* convention (E2E_ISSUE_REPO,
    E2E_WIKI_NAME, E2E_QUEUE_PIPELINE_ID, …). Follow it, and treat an
    unexpanded ADO macro (literal $(NAME)) as unset — github-issue.ts already
    has that guard, reuse it.
  • Issue types are a closed set defined by the repository/org owner. Discover
    the available types via REST rather than hardcoding a name, and SkipError
    when none exist — a hardcoded type will rot.

Validation

cd scripts/ado-script
npm run typecheck
npx vitest run src/executor-e2e
npm run build:executor-e2e

Add unit coverage alongside the existing src/executor-e2e/__tests__/. Prove
the temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.

Then update tests/executor-e2e/README.md: remove create-issue from the
"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.

Out of scope

  • Do not add these tools to the smoke suite (tests/smoke/cases.json) — the
    point of executor-e2e is deterministic Stage 3 coverage with no agent.
  • Do not add a GitHub PAT to the smoke agentic lane. Issue test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator #1796 covers
    orchestrator-side failure reporting separately and puts the token on the
    orchestrator, not the lane.
  • Do not change the Rust implementation of either tool. If you find a genuine
    bug, report it rather than fixing it inline.

Definition of done

  • Scenarios registered and passing against the real project
  • Temporary-ID handoff proven, mutation-checked
  • Cleanup closes every issue it creates, limitation documented
  • Missing preconditions skip rather than fail
  • README exclusion line corrected
  • npm run typecheck, npx vitest run src/executor-e2e,
    npm run build:executor-e2e all green

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions