You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Task: add create-github-issue + set-github-issue-type scenarios to executor-e2e
Add deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in githubnext/ado-aw. Both currently ship with zero runtime
coverage — their only proof is a wiremock unit test.
Background
create-github-issue and set-github-issue-type were added by PR #1670
(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and removed the smoke-failure-reporter smoke case, which
was the only thing exercising create-github-issue at runtime. Net effect:
coverage went from "thin" to "none".
executor-e2e excludes it by design today. tests/executor-e2e/README.md
says: "Excluded (out of scope or GitHub-only): the GitHub-only create-issue." That exclusion predates the tools being first-class public
safe outputs, and is what you are reversing.
Confusable, do not be misled:scripts/ado-script/src/executor-e2e/github-issue.ts
is the harness's own failure reporter (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.
What to build
Add scripts/ado-script/src/executor-e2e/scenarios/github-issue.ts exporting githubIssueScenarios, and register it in scenarios/index.ts (append to allScenarios — the order is deterministic and existing entries must not move).
Cover, at minimum:
create-github-issue — files an issue, asserts it exists on GitHub with
the expected title (including the configured title-prefix), body, and
config-injected labels.
set-github-issue-type — sets the issue type on an existing issue.
The same-run temporary-ID handoff — this is the highest-value case and
the one with the weakest current proof. create-github-issue accepts an
optional temporary_id, and set-github-issue-type accepts an issue_number that may be either a real number or that temporary id. A
single executor run must resolve the handoff. A wrong REST shape or a
scoping failure here is invisible today.
Consider also (use judgement, do not gold-plate):
allowed-labels rejection — config is default-deny for labels, unlike set-github-issue-type.allowed which is default-allow. Use expectedFailure for the rejection path.
Clearing an issue type: issue_type: "" is the documented clear operation.
The contract you must follow
Read scripts/ado-script/src/executor-e2e/scenario.ts first — it is the
authoritative contract and is well commented. Key points:
A scenario is { tool, config, setup, ndjson, assert, cleanup } with an
optional id (set id when several scenarios share one tool), files, env, and expectedFailure.
ndjson() returns the executor entry without the name field — the
runner injects name: <tool>.
No LLM is involved. You craft the NDJSON directly, run the real ado-aw execute binary, then assert via REST.
ScenarioContext gives you buildId, prefix(tool)
(ado-aw-det-<buildId>-<tool>), workDir, log, and rest.
Throw SkipError from setup() when a precondition is missing, so an
incomplete environment records as skipped, not failed.
Cleanup gotcha: if setup() throws a non-SkipError, cleanup() is
NOT called — tear down anything you partially created before rethrowing.
See scenarios/pr.tssetupPr for the established pattern.
assert() may populate state that cleanup() needs (e.g. the created issue
number read off the executor result) — do that before any fallible check,
so a later assertion failure still leaves cleanup able to tear down.
scenarios/wiki.ts is the cleanest model: env-var override, REST discovery, SkipError, deterministic naming, real cleanup.
Two things that make this genuinely different from every existing scenario
1. It asserts against GitHub, not ADO.ScenarioContext.rest is AdoRest
and will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in executor-e2e/github-issue.ts (findOpenIssueByTitle, createGitHubIssue, diagnoseGitHubAuthFailure) rather than writing a fourth GitHub client — lift
them into a shared module if that is cleaner, but do not duplicate them.
2. GitHub issues cannot be deleted. Every other scenario tears its objects
down completely. You cannot. cleanup() must close the issue (and prefer a [e2e]-style title prefix plus the ado-aw-det-<buildId>-… marker so leaked
issues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.
Environment
Definition 2550 (executor e2e) already has EXECUTOR_E2E_GITHUB_TOKEN
provisioned — no new secret provisioning is required. Confirm the token's
scope actually permits issue-type mutation on the target repo; issue types
are an org-level construct and may need more than plain Issues write. If it
does not, SkipError is the correct response, not a hard failure.
Existing env vars follow the E2E_* convention (E2E_ISSUE_REPO, E2E_WIKI_NAME, E2E_QUEUE_PIPELINE_ID, …). Follow it, and treat an
unexpanded ADO macro (literal $(NAME)) as unset — github-issue.ts already
has that guard, reuse it.
Issue types are a closed set defined by the repository/org owner. Discover
the available types via REST rather than hardcoding a name, and SkipError
when none exist — a hardcoded type will rot.
Validation
cd scripts/ado-script
npm run typecheck
npx vitest run src/executor-e2e
npm run build:executor-e2e
Add unit coverage alongside the existing src/executor-e2e/__tests__/. Prove
the temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.
Then update tests/executor-e2e/README.md: remove create-issue from the
"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.
Out of scope
Do not add these tools to the smoke suite (tests/smoke/cases.json) — the
point of executor-e2e is deterministic Stage 3 coverage with no agent.
Task: add
create-github-issue+set-github-issue-typescenarios to executor-e2eAdd deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in
githubnext/ado-aw. Both currently ship with zero runtimecoverage — their only proof is a wiremock unit test.
Background
create-github-issueandset-github-issue-typewere added by PR #1670(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and removed the
smoke-failure-reportersmoke case, whichwas the only thing exercising
create-github-issueat runtime. Net effect:coverage went from "thin" to "none".
executor-e2eexcludes it by design today.tests/executor-e2e/README.mdsays: "Excluded (out of scope or GitHub-only): the GitHub-only
create-issue." That exclusion predates the tools being first-class publicsafe outputs, and is what you are reversing.
Confusable, do not be misled:
scripts/ado-script/src/executor-e2e/github-issue.tsis the harness's own failure reporter (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.
What to build
Add
scripts/ado-script/src/executor-e2e/scenarios/github-issue.tsexportinggithubIssueScenarios, and register it inscenarios/index.ts(append toallScenarios— the order is deterministic and existing entries must not move).Cover, at minimum:
create-github-issue— files an issue, asserts it exists on GitHub withthe expected title (including the configured
title-prefix), body, andconfig-injected labels.
set-github-issue-type— sets the issue type on an existing issue.the one with the weakest current proof.
create-github-issueaccepts anoptional
temporary_id, andset-github-issue-typeaccepts anissue_numberthat may be either a real number or that temporary id. Asingle executor run must resolve the handoff. A wrong REST shape or a
scoping failure here is invisible today.
Consider also (use judgement, do not gold-plate):
allowed-labelsrejection — config is default-deny for labels, unlikeset-github-issue-type.allowedwhich is default-allow. UseexpectedFailurefor the rejection path.issue_type: ""is the documented clear operation.The contract you must follow
Read
scripts/ado-script/src/executor-e2e/scenario.tsfirst — it is theauthoritative contract and is well commented. Key points:
{ tool, config, setup, ndjson, assert, cleanup }with anoptional
id(setidwhen several scenarios share onetool),files,env, andexpectedFailure.ndjson()returns the executor entry without thenamefield — therunner injects
name: <tool>.ado-aw executebinary, then assert via REST.ScenarioContextgives youbuildId,prefix(tool)(
ado-aw-det-<buildId>-<tool>),workDir,log, andrest.SkipErrorfromsetup()when a precondition is missing, so anincomplete environment records as skipped, not failed.
setup()throws a non-SkipError,cleanup()isNOT called — tear down anything you partially created before rethrowing.
See
scenarios/pr.tssetupPrfor the established pattern.assert()may populate state thatcleanup()needs (e.g. the created issuenumber read off the executor result) — do that before any fallible check,
so a later assertion failure still leaves cleanup able to tear down.
scenarios/wiki.tsis the cleanest model: env-var override, REST discovery,SkipError, deterministic naming, real cleanup.Two things that make this genuinely different from every existing scenario
1. It asserts against GitHub, not ADO.
ScenarioContext.restisAdoRestand will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in
executor-e2e/github-issue.ts(findOpenIssueByTitle,createGitHubIssue,diagnoseGitHubAuthFailure) rather than writing a fourth GitHub client — liftthem into a shared module if that is cleaner, but do not duplicate them.
2. GitHub issues cannot be deleted. Every other scenario tears its objects
down completely. You cannot.
cleanup()must close the issue (and prefer a[e2e]-style title prefix plus theado-aw-det-<buildId>-…marker so leakedissues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.
Environment
2550(executor e2e) already hasEXECUTOR_E2E_GITHUB_TOKENprovisioned — no new secret provisioning is required. Confirm the token's
scope actually permits issue-type mutation on the target repo; issue types
are an org-level construct and may need more than plain Issues write. If it
does not,
SkipErroris the correct response, not a hard failure.E2E_*convention (E2E_ISSUE_REPO,E2E_WIKI_NAME,E2E_QUEUE_PIPELINE_ID, …). Follow it, and treat anunexpanded ADO macro (literal
$(NAME)) as unset —github-issue.tsalreadyhas that guard, reuse it.
the available types via REST rather than hardcoding a name, and
SkipErrorwhen none exist — a hardcoded type will rot.
Validation
cd scripts/ado-script npm run typecheck npx vitest run src/executor-e2e npm run build:executor-e2eAdd unit coverage alongside the existing
src/executor-e2e/__tests__/. Provethe temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.
Then update
tests/executor-e2e/README.md: removecreate-issuefrom the"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.
Out of scope
tests/smoke/cases.json) — thepoint of executor-e2e is deterministic Stage 3 coverage with no agent.
agenticlane. Issue test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator #1796 coversorchestrator-side failure reporting separately and puts the token on the
orchestrator, not the lane.
bug, report it rather than fixing it inline.
Definition of done
npm run typecheck,npx vitest run src/executor-e2e,npm run build:executor-e2eall green