Skip to content

Merge main evaluation lifecycle into the landing-page release - #278

Merged
Asher Johnson Hogan (asherjh) merged 18 commits into
release/planner-landing-pagefrom
users/asherj/sync-main-landing-page-release
Sep 14, 2026
Merged

Merge main evaluation lifecycle into the landing-page release#278
Asher Johnson Hogan (asherjh) merged 18 commits into
release/planner-landing-pagefrom
users/asherj/sync-main-landing-page-release

Conversation

@asherjh

@asherjh Asher Johnson Hogan (asherjh) commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Description

Synchronizes the landing-page release with main's evaluation lifecycle. The release combines evaluation generation, review, promotion, execution, and result export with MCP-powered landing-page configuration and the schema-v2 widget telemetry receiver.

  • Integrates the evaluation lifecycle scripts, skills, setup extraction, and command routing from main.
  • Keeps lifecycle-script CI alongside the landing-page, MCP foundation, FlightCheck, and installer jobs, with release branch and PR event filters intact.
  • Preserves the telemetry receiver's authenticated tenant identity, host context, and timezone-aware ISO source timestamps.
  • Retains Maker Profile 0.4.25, newer than main's 0.4.24, with the matching bundled VSIX and landing-page Quick Action.

Integration

Targets release/planner-landing-page. Merge this PR using Create a merge commit to retain main's ancestry before completing PR 268 — Add MCP-powered ESS landing-page configuration.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Documentation update

Testing

  • Combined lifecycle-script, MCP, FlightCheck, setup, and telemetry suites: 1,878 passed, 1 platform-specific skip. The existing lifecycle CI exclusions for installer tests were retained.
  • Maker Profile: 41 tests passed; installer smoke/parser harness: 43 passed.
  • Python source lint passed. CI YAML retains the exact release job definitions plus main's lifecycle job.
  • VSIX manifest, package version, extension source, and changelog match the packaged 0.4.25 artifact; only that current VSIX is bundled.
  • Independent review found no merge-caused functional blockers. All files carrying the schema-v2 telemetry fix are unchanged from the release input.

Checklist

  • My code follows the existing style
  • I have added/updated tests where applicable
  • I have updated documentation as needed

nehacopilot and others added 18 commits September 4, 2026 06:05
@nehacopilot
feat(evaluations): add grounded generation and quality exports
50f7757 
Add dispatcher-based evaluation creation, catalogue-grounded generation, deterministic CSV export, and folder-scoped quality validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
feat(evaluations): add review and promotion lifecycle
b5ee5c6 
Persist review markers through setup and scoped push, reconcile local and deployed state, and safely promote workspace sets before cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
feat(evaluations): run test sets and analyze results
efee0c2 
Add deployed-set discovery, connected Copilot Studio profile selection, asynchronous run execution, history retrieval, and evidence-based result analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): explain review-blocked test sets
883fe5d 
Return explicit reasons for pending reviews and locally completed reviews that have not been pushed, while keeping blocked sets unavailable for execution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
feat(evaluations): add maker and reviewer choice prompts
d6bb693 
Require structured test-set selection, offer makers self-edit or SME feedback paths, and clarify that reviewer recommendations return official edits and execution ownership to the maker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): make reviewer workflow feedback-only
ba82bdd 
Limit reviewer actions to inspection and written recommendations while keeping all source edits, validation, push, and execution with the maker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
feat(evaluations): refresh review sets automatically
8f17612 
Refresh the configured agent and rebuild its baseline before listing review work, using the existing checkpointed refresh flow so reviewers do not need a separate pull step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): preserve unmapped test case ids
79f5bbb 
Keep the original test case identifier as the display fallback when local component metadata is unavailable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): address review and runtime gaps
3666ea4 
Preserve explicit review-block messages, require synchronized review completion, validate runtime configuration, clean collision-safe exports, and keep Python 3.11 compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
docs(evaluations): register run lifecycle
aed5b53 
Redirect the run command from the monorepo root and document generation, review, push, execution, history, and results behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): use signed-in profile connections
2e98915 
Keep PPAPI and Power Apps connection discovery on the same account, reject ambiguous profiles, and include required agent tool bindings in evaluation run requests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
chore(evaluations): add MinimalBot run POC
17530b0 
Add an opt-in TEST-ring utility that clones an evaluation payload with fresh component IDs, resolves the signed-in profile and tool bindings, starts PPAPI, and reports request identifiers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
fix(evaluations): prefer latest matching profile
b57793a 
When the signed-in account has multiple connected profiles, select its most recently updated profile instead of treating the selection as ambiguous.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nehacopilot
revert(evaluations): remove directly pushed MinimalBot POC
12ec360 
Move the MinimalBot evaluation POC back behind the dedicated setup pull request while retaining the signed-in profile and tool connection fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
apurvabanka_microsoft
requested changes
bfed56e
apurvabanka_microsoft
requested changes
d5dea9f
@apurvabanka
Merge pull request #267 from microsoft/nbethapudi/evaluation-lifecycle
3236a51 
: Evaluation lifecycle (grounded generation → review → run → analyze)
@asherjh
release: merge main evaluation lifecycle into landing-page release
b5bdc80 
Keep the release current with main while preserving the complete landing-page experience and the schema-v2 telemetry receiver, including source timestamps and authenticated tenant context.

- merge main evaluation lifecycle and shared setup integration
- retain both lifecycle and landing-page/MCP CI jobs with release event filters
- preserve the telemetry fix and the versioned Maker Profile 0.4.25 package

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4e43a4e1-8e81-4a9e-aa02-966b029b12d3
@asherjh
Asher Johnson Hogan (asherjh) merged commit b114992 into release/planner-landing-page Sep 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dcowen91 Drew Owen (dcowen91) dcowen91 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@asherjh @dcowen91 @nehacopilot @apurvabanka