Skip to content

feat(planner): grounded ESS rollout planner skill, roles attestation, and WeveNova/WorkIQ MCP integration - #258

Open
Harsheet jain (jainharsheet77) wants to merge 85 commits into
mainfrom
feature/planner-skill
Open

Harsheet jain (jainharsheet77) wants to merge 85 commits into
mainfrom
feature/planner-skill

Conversation

@jainharsheet77

Copy link
Copy Markdown
Contributor

Description

Adds the grounded ESS rollout planner to the ESS Maker Skills kit, along with the supporting roles-attestation skill and the WeveNova/WorkIQ MCP integration that backs them.

Highlights:

  • /planner skill — a grounded, interview-driven ESS rollout planner (Step 1): scenario-first interview, Learn-supported native-connector grounding, editable plan Markdown round-trip, dependency-ordered tasks, and eager eval preview.
  • Shared planner sync — persists the plan to the shared planner service automatically once built and resolves the single active plan via project.activePlanId.
  • /roles skill — role attestation, a shared two-tier person resolver (user-consentable Graph scope with a WorkIQ people-resolution fallback), and task assignment that reuses person resolution.
  • AgentConfig MCP — mirrors the full WeveNova attestable-role registry (Entra + Power Platform), pins the WorkIQ MCP to @microsoft/workiq@1.0.0, and surfaces clearer planner error detail.
  • Onboarding/setup branches by persona so makers route into planning, and non-setup personas are nudged to /setup into the plan's environment.
  • Tests — extensive new coverage across tests/planner, tests/roles, tests/mcp/*, and tests/scripts (~4.9k lines of tests).

Scope: solutions/ess-maker-skills/ (skills + workspace), tests/, dev-specs/, and one .github/ change. No samples/ content is touched.

Related issue

Refs #253

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor / cleanup

Testing

  • New unit/contract test suites added under tests/planner/, tests/roles/, tests/mcp/planner/, tests/mcp/agentconfig/, and tests/scripts/ (plan model, capture, sync, research, roles, person resolution, MCP client/server contracts, widget protocol).
  • Review feedback from the internal planner PR (feat(agentconfig-mcp): WeveNova planner MCP for planner-skill integration #253) was addressed across both the skill side and the MCP side.

Checklist

  • My code follows the existing style
  • I have added/updated tests where applicable
  • I have updated documentation as needed

Static validation (samples/ only)

This PR does not touch anything under samples/ (changes are under solutions/, tests/, dev-specs/, and .github/), so sample static validation is not applicable.

Validation
- YAML parse: N-A
- AdaptiveDialog kind: N-A
- XML parse: N-A
- Filename convention (new): N-A
- Folder convention (new, incl. README.md): N-A
- Diff scope (samples/ only): N-A
- Secrets / internal URLs: N-A

harsheetjain and others added 30 commits August 11, 2026 17:53
…Step 1)

Implements Step 1 of the ADK plan-generation dev spec: a local-first,
structured Plan the /planner skill authors for an ESS rollout.

- scripts/planner/: Plan model (atomic IO, validation, summary render, Flow-2
  grouped-by-role discovery), TOC-first Microsoft Learn research selection,
  observe-mode output capture (/setup -> environmentId from config.json), an
  absent-safe roles-source seam, and a CLI the skill drives.
- src/skills/planner/ + planner.prompt.md: the grounded interview ->
  Learn-grounded roles/tasks -> Flow-1 person assignment -> capture playbook,
  wired into copilot-instructions routing and the menu. Planner is the one
  experience allowed before setup (planning decides a greenfield deployment).
- tests/planner/: 49 tests (pure logic + local IO; no network/cassettes).
- dev-specs/adk-plan-generation/: the design doc.

The local Plan is shaped like the WeveNova Plan/Task entities so a future sync
is a field copy. WeveNova, tenant inventory, and the roles source are all
optional, absent-safe seams; the Plan is authoritative on disk without them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Capture and expose scenario-to-scenario ordering (the PM spec's "HR knowledge
before HR ticketing") without a new typed collection, consistent with the
one-Context-bag model:

- A scenario in scope is a Context entry (group "scenario"); a dependency edge
  is a Context entry (group "scenarioDependsOn", key "A -> B", scalar value =
  kind requires|recommends, description = rationale/PM-spec citation).
- A grounded PM-spec seed (knowledge -> ticketing) lets the planner advise the
  sponsor; unmet_scenario_dependencies() surfaces prerequisites not in scope.
- Exposed via the interview (check-deps) and rendered in summary with a
  met/MISSING status; ordering is then enforced by the task produces/consumes
  DAG.
- New CLI: add-scenario, add-scenario-dependency, check-deps. 15 new tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ct onboarding framing

- Routing: greenfield "set up ESS for the first time / where do I start / how do
  I get started" now routes to /planner (not /setup); the planner emits "run
  /setup" as the first task. Added the gate exception + trigger phrases.
- Onboarding framing: /setup (onboarding) connects the kit to an ALREADY-deployed
  ESS agent and records its details - it does not create the environment or
  install ESS (those are portal/admin prerequisites on a new tenant). Reworded
  model.md, capture.md, planner.prompt.md, capture.py docstring, and the dev-spec.
- Back-propagation documented: the details /setup records in .local/config.json
  (environmentId, dataverseEndpoint, agent slug/schema/folder) flow to later
  tasks via config.json (every skill reads it) and via the pinned
  primaryEnvironment artifact (tasks that consume it) - e.g. topic create.

Docs/framing only; no logic change. 64 tests still pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ut, stronger skill steps

Addresses a real run that stopped after a single "run setup" task and skipped
the system/scenario questions:

- CLI: `task-brief` shows an assignee how to do a task, their role, the resolved
  values it consumes (e.g. the env id setup produced - the back-propagation), and
  the keys to capture. `pin-output` commits what an assignee created (Workday
  connection, Entra app, eval suite) onto the plan - the generic ask-mode
  counterpart to capture-setup.
- Model: Plan.resolved_consumes() + Plan.task_brief().
- Skill instructions made prescriptive so the agent runs the WHOLE flow: the
  interview must capture which systems + scenarios (mandatory) before Phase 3;
  Phase 3 emits the full grounded task set (setup + one connect per system +
  authoring per scenario + evals + publish), not just setup; Phase 5 briefs each
  assignee with the env id and commits what they create back onto the plan.
- 5 new tests (69 total).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, check-deps, research extraction, summary timestamp

Ground the planner on the PM spec model (scenarios come from the maker + Microsoft Learn; a business-scenario catalogue is an optional implementation choice) and fix four greenfield smoke-test bugs.

Grounding:
- Remove the invented scenario_catalogue.json + catalogue.py (a business-scenario list the PM spec does not define).
- Add planner_facts.json + facts.py holding non-Learn facts ONLY: scenario dependencies (each with an explicit source) + a recognition lexicon. Not a scenario catalogue.
- known_scenario_dependencies() now reads the facts file; no false "PM spec" citation. The knowledge->ticketing edge is sourced "ess-design-guidance" and flagged confirm-citation (verified: absent from pm-spec.md and the ADO spec repo).
- Drop scenarios/suggest-scenarios CLI (prompt->fixed-list mapping).

Bug fixes:
- bug1: add-system CLI + Plan.set_system write scoped keys (system.<area>) so multiple target systems no longer collide on one reused key; interview.md asks per-area.
- bug2: check-deps + Plan.scenario_dependency_status() surface MET dependencies, not only unmet.
- bug3: research.extract_signals/strip_html/fetch_page_text + `research --extract` pull role/output candidates off fetched Learn pages.
- bug4: plan carries updatedAt (bumped on save); summary renders Generated + Updated.

Tests: 82 pass, 1 live skipped. Adds tests/planner/test_facts.py; updates scenario/research/cli/plan_model tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…connectors

The interview asked "which backend systems should ESS connect to" with improvised
examples (incl. ADP, which has NO native ESS connector). Ground the systems
capture in Phase-1 Learn research instead of improvising:

- Native ESS integrations = Workday, ServiceNow HRSD/ITSM, SAP SuccessFactors
  (each has an ESS Learn page in the TOC). Derive the set from research; don't
  name systems from memory.
- SharePoint / M365 content is a knowledge source, not a data-system connector.
- A system with no native connector (ADP, Jira, Dynamics 365, custom HTTP API)
  routes to /create (custom Power Automate flow), NOT a connect task.

model.md Phase 3 now emits a create task (not a fabricated connect task) for
non-native systems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
If a plan already exists, /planner asked for the objective again because the
plan-existence check was buried at the bottom of SKILL.md (after the phases +
"the interview must capture the objective" emphasis).

- Promote plan lookup to the FIRST step ("First - resume or start"): if
  workspace/plan/plan.json exists, show its latest state (summary) and the tasks
  the person can pick up, role-gated (Flow 2), and do NOT re-run the interview or
  re-ask the objective. Start over only on explicit confirmation.
- A task is shown only if the person holds the role it needs; role resolution is
  best-effort until the roles source / MCP exists (future work).
- Frame the phases as "building a new / extending a plan"; reinforce the resume
  gate in planner.prompt.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ironment

/setup should be deliberate and plan-driven, not auto-fired by the gate. Drive it
from plan state:

- The Power Platform admin's setup task decides/creates the environment and pins
  primaryEnvironment. Every OTHER persona must connect their own kit to that same
  environment before their task's skill works.
- Plan.kit_setup_nudge(): for a non-setup kit-skill task, once primaryEnvironment
  is pinned, returns the env id/url to connect to; None for the setup task itself,
  non-kit tasks, or when no env is pinned yet (then the setup task is the
  prerequisite, not a nudge).
- task_brief surfaces it; the task-brief CLI prints "First connect your kit: run
  /setup and choose environment <envId>".
- Skill docs (mytasks/capture/model/SKILL) explain the plan-driven nudge.

Tests: 83 pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The gate's welcome told every user to "Type /setup", which is a VS Code prompt-file
command that does not exist in the Copilot CLI. Reword to name both surfaces: type
/setup in VS Code, or just say "set up ESS" in the CLI/any chat. The routing
exceptions already treat "explicitly asked to run setup" the same as /setup, so
natural-language invocation works.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ship the kit's CLI entry points inside the repo so anyone who clones it gets them
automatically (no per-user ~/.copilot copy or sync needed). The Copilot CLI
auto-discovers project skills from .github/skills/<name>/SKILL.md.

- solutions/ess-maker-skills/.github/skills/setup/SKILL.md
- solutions/ess-maker-skills/.github/skills/planner/SKILL.md

Names match the VS Code prompt-file commands (/setup, /planner) so the skill name
is consistent across Copilot in VS Code and the CLI. Each is a thin launcher that
honors the kit's copilot-instructions then follows the real
src/skills/{onboarding,planner}/SKILL.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tasks are assigned a Learn-grounded role at creation (the role that should be able
to pick the task). Make that grounding explicit and auditable:

- new_task / add-task gain --role-source: the Microsoft Learn URL that grounds the
  task's role (optional task.roleSource field). task_brief surfaces it as
  "Role grounded in: <url>".
- research.md: `research --extract` surfaces role candidates per page; carry the
  (role, source URL) pair into task creation.
- model.md: every task gets a Learn-sourced --role + --role-source at creation;
  the person is assigned later (Phase 4 / future external roles API).
- assign.md + dev-spec: document the future external roles API — RoleSource
  .list_holders resolves role->person, and a task can be assigned a user together
  with the role. The role stays Learn-grounded; the API only resolves people.

Tests: 85 pass (adds role-source coverage).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d it

When /setup finishes it now nudges the maker to record it on the rollout plan
instead of waiting for someone to notice.

- onboarding step3-flightcheck.md gains a plan-aware handoff (3.4): at the end of
  setup (readiness check skipped or done), if workspace/plan/plan.json exists,
  offer to mark the setup task complete and pin the environment, running
  `capture-setup --complete`. Plan-conditional, so standalone /setup users are
  unaffected.
- Plan.setup_task_id() finds the plan's /setup task (the onboarding-skill task,
  not a portal "provision" task). `capture-setup --task` is now optional and
  auto-detects it, so the handoff is turnkey.
- capture.md documents the auto-detect and the automatic offer.

Tests: 87 pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…description

The WeveNova Task entity has no `action` field, so emitting one would be rejected
on persist. Remove it entirely — a Task is described by title + description (the
description states the "how": which command to run, or a portal/manual step).

- plan_model: drop the action_* builders, ACTION_KINDS/ONBOARDING_SKILLS, the
  `action` param/field on new_task, and _validate_action. Setup detection is keyed
  on the grounded produces/consumes signal (the task that produces
  primaryEnvironment), not action. Summary tasks table drops the Action column;
  task_brief returns `description`.
- cli: remove _build_action and the --skill/--action-kind/--ref flags; add-task is
  title + description (+ role / role-source / produces / consumes).
- skill docs (model/capture/mytasks/research/SKILL): title + description, no action.
- dev-spec: remove task.action from the schema, JSON examples, and §9/§10/§12/
  appendix; state the WeveNova Task entity has no action field.
- tests rewritten to description-based tasks; 87 pass, no action references remain.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tep-2 entities

Audited every field against the Step-2 "Plan Enrichment & Persistence" dev spec
(the WeveNova AgentConfigurationPlan/Task entities), removing invented fields and
correcting spec-backed ones.

Removed (not in the WeveNova entity):
- task.roleSource - the role's Learn grounding lives in the research context
  (Step-2 §7.6 prerequisites[].sourceUrl), never as a task field.
- plan.notes - §7.1 defines only Context + Outputs; free-form notes are Context
  entries, not a top-level field.
- plan.generatedAt / plan.updatedAt - the entity timestamps are server-owned
  (CreatedAt/UpdatedAt, §7.5); the local pre-sync file no longer sets them (summary
  drops the Generated/Updated line).

Kept / corrected (confirmed present in Step-2):
- Principal.Role {roleId, directoryRef?} and Principal.User {oid, directoryRef?}
  (§7.4) - RESTORED the optional directoryRef I had wrongly removed.
- task.produces / task.consumes (IList<string> keys, §7.2); setup detection via
  produces primaryEnvironment.

Result: plan = {schemaVersion, planId, projectId, status, context, tasks, outputs};
task = {id, title, description, assignedTo, state, produces, consumes}. Docs
(model/research/dev-spec) updated. 86 tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…o context)

A real run went system-first ("Which back-end system?" -> Workday) and then
reduced scenarios to that one system's capabilities (profile/time-off/pay),
never asking which scenario TYPES the maker wanted (HR knowledge, HR ticketing,
IT ticketing). Reorder the interview so it builds the scenario context first:

- Q2 now captures scenarios / jobs-to-be-done FIRST, prompting with the grounded
  HR knowledge / HR ticketing / IT ticketing / data-actions framing (examples,
  not a fixed catalogue).
- Q3 asks the system PER scenario, only after scenarios are captured.
- Explicit rule: picking a system must not narrow the scenario set; a maker on
  Workday may still want HR knowledge and IT ticketing. "Required before Phase 3"
  reordered: objective -> scenarios -> system-per-scenario.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…alogue

Add the authoritative ESS scenario catalogue (scenario list, priority order, and
dependency edges) as vendored data, and wire the interview to it so a sponsor's
goal is captured against real categories instead of ad-hoc examples.

- scripts/planner/scenario_catalogue.md: vendored decision-layer snapshot - the
  category map (6 OOB categories / 43 scenarios + extensible E1-E7/Facilities),
  the default priority order + tiers, and the dependency edges. Per-scenario
  detail (fields, setup, connectors, roles) stays fetched from Microsoft Learn at
  render time. No internal repo path/source is recorded in the file.
- interview.md: Phase 2 reads the catalogue; Q2 maps the sponsor's goal to the
  catalogue categories (HR Knowledge, HR/IT Ticketing, Profile read/write,
  Manager, Handoff, extensible); priority/order and dependency edges come from the
  catalogue, not improvised.
- planner_facts.json: dependency edges now mirror the catalogue (Knowledge is the
  deflection foundation -> Ticketing recommends Knowledge; IT likewise; reads
  before writes), each sourced "ESS scenario catalogue". Replaces the earlier
  single unsourced edge.
- tests updated (kind recommends; catalogue source). 86 pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d eval after plan creation

After the plan is authored (Phases 1-4), the planner now hands the sponsor's captured scenarios to the eval skill to generate a first, theoretical (scenario-based) evaluation - generate-only, before anything is built. The planner only invokes the eval skill; it does not own or author eval content. Adds Phase 5 (evaluate.md), renumbers Capture to Phase 6, and documents the seed-vs-refine relationship to the topic-driven 'Generate evaluation tests' task (unchanged).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… briefs, per-category enabled-scenario capture

Rename the plan's human view to ESS-scenario-plan.md and make it editable: a Plan editor revises it directly (or via chat intent) and the planner reconciles the change back into plan.json via the CLI, asking where ambiguous (new src/skills/planner/edit.md). Adds update-task and remove-task CLI/model verbs so reconciliation can modify/delete tasks.

Enrich the task brief on start: when an assignee engages a task, render a detailed how-to - hand off to the owning kit skill (/setup, /connect, /create, /evaluate) or fetch the step's Learn page for portal/manual steps (register Entra app, provision env, publish). Mantra: enrich from Learn; descriptions carry the how, detailed steps are fetched fresh from the task's Learn anchor.

Capture the enabled scenarios per in-scope category (Context group scenarioCapability, grounded from the catalogue named list + Learn, OOB unless the editor pins an extensible one) so the theoretical eval reads topic-level scenarios off the plan and writes golden prompts per scenario.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… via setup

Q8 ("brand-new environment, or do you already have ESS running?") selected a
greenfield-vs-enrichment branch, but enrichment is a future seam-only path
(section 14) with no implementation, and the greenfield-vs-existing distinction
is already resolved by detection at execution time: the always-emitted setup
prerequisites are idempotent (/setup reads .local/config.json; the install
pre-check reports an already-installed ESS as PASSED), so an existing deployment
no-ops satisfied Tasks without asking. Removing Q8 also honors the interview's
"fewest questions / propose, don't interrogate" rule.

Remove Q8 from interview.md and design section 8.2, fix the 4-8 / "8 is one
branch" counters, and record the resolution in open-items section 17.9. Q8
stored no context key, so nothing downstream is orphaned.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…thing)

Bring the demo's early-eval idea onto the dev branch as a render-only preview: as soon as the interview captures scenarios + goals (Phase 2, before modelling), the planner renders the golden prompts grouped by scenario category so the sponsor sees the acceptance bar up front.

Render-only: it displays the prompts in chat but writes no file, creates no eval records, and pushes nothing. The eval skill (evaluations/create/SKILL.md) is left untouched - the planner does not own it and does not run its generate/scan/push pipeline. Actual generation stays with the topic-driven Generate evaluation tests task, later.

Reframes evaluate.md (Phase 5) from theoretical generate-only to eager render-only preview; adds the eager hook in interview.md; updates SKILL.md phase labels and the dev-design (10.2, worked example). Instruction-only; 89 tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gram

Add the end-to-end flow diagram (Mermaid, sec 3.1): sponsor + task-assignee lanes, the plan as shared state, the WeveNova tenant inventory box, and the numbered info-transfer arms including /setup WRITE, WeveNova READ (value/presence-only), and Dataverse live READ (200/403).

Align the spec to the flow: research now also scans the tenant inventory (7.8); capture (12) scans the WeveNova tenant inventory + controlled Dataverse APIs first (field visibility = value if safe-for-all else presence-only; access enforced by Dataverse 200/403, not the ADK), falling back to local observe (config.json) then ask; /setup artifacts persist to the tenant inventory and are read back from WeveNova (12.2). Reframe sec 14 from 'inventory is future' to a first-class absent-safe read/write seam (proactive /discover skip-existing stays future), and reconcile the sec 1 summary + sec 2 non-goals + sec 13 seam bullet.

Docs-only. The tagged .docx in Downloads is a stale export (baseline 72a24f8); this repo .md is the living spec on PR #220.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nt->include, empty->leave)

Clarify that research reads the tenant inventory in WeveNova (not Dataverse directly): if present, include it (detect existing deployment, pre-fill); if empty/absent, leave it and run Learn-only. Updates the sec 1 summary, sec 7.8, and the sec 3.1 flow-diagram research node.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tinct from the Dataverse APIs

Correct the conflation throughout: the tenant inventory is a WeveNova artifact (we are implementing) that STORES facts about the tenant; it is DISTINCT from the Dataverse APIs those facts are fetched from. The planner reads it as one source alongside Learn links and Dataverse (if available).

Flow diagram (sec 3.1): Dataverse now feeds the inventory (facts fetched from Dataverse 200/403 -> stored), and the planner READS the inventory (value if safe, else presence-only) rather than reading Dataverse directly. Reframes sec 1, sec 7.8 (the inventory is another source), sec 12.1/12.2/12.3, and sec 14 accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Nova does not fetch from Dataverse)

Correct the capture path: WeveNova does NOT reach into Dataverse. The assignee runs /discover, which fetches the tenant facts from the distinct Dataverse APIs (200/403, the caller's own access) and then calls WeveNova to store them in the tenant inventory; the planner reads the inventory (value if safe, else presence-only).

Rewire the sec 3.1 flow diagram (Dataverse -> /discover -> WeveNova inventory -> planner) and update the info-transfer arrows, sec 12.1/12.2/12.3, and sec 14 (the /discover run that POPULATES the inventory is part of this flow; only proactive skip-existing pruning is future).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…elds, permission)

Add a read edge from the sponsor's Research to the WeveNova tenant inventory (we read it during planning, present->include/empty->leave), and pose the open questions in the diagram: which fields does the inventory expose to the planner, and what permission (if any) is required to read them. Also recorded as sec 18 open question 12.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es) now, WeveNova MCP later

Reflect the envisioned interim model: /discover crawls Dataverse for system details and writes ids+names into .local/config.json (same local file /setup writes); the planner reads ids+names from config.json for now and pins them to the Plan as a task output/PlanArtifact. /discover separately owns persisting to the WeveNova inventory and defining its read surface; the read moves to a WeveNova MCP once that stabilises. This matches capture.py, which reads environmentId from .local/config.json today.

Rewire the sec 3.1 diagram (Dataverse -> /discover -> config.json -> planner; dashed /discover->inventory and inventory->planner-via-MCP), add a config.json node, and add a CAVEAT that the inventory needs a read surface exposing ids+names (which fields, what permission). Updated info-transfer, sec 7.8, sec 12.1/12.2/12.3, sec 14, and sec 18 #12.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…role tasks

Gap 1: /setup clones the deployed ESS agent into config.json; capture now pins it as an Agent PlanArtifact alongside the Environment. Adds capture.detect_agent, extends config_snapshot with the agent, wires cmd_capture_setup to pin both, adds Agent to ARTIFACT_KINDS, and updates capture.md + design doc SS12.

Gap 2: Workday setup is multi-role (App/Cloud App Admin SSO, Workday Administrator tenant, Environment Maker pack+connect, InfoSec/IT firewall) per setup/workday/tasks.md - it is never one integration-owner task. Rewrites model.md 'a task is not a skill steps' to split on every role boundary and read the checklist role: verbatim; updates the backbone tables + worked plan.json + walkthrough in the design doc to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…act), not env/agent-only

Replaces the env-only + agent-only detectors with a single generic capture.detect_config_artifacts: it diffs the whole .local/config.json and pins EVERY id+name (and any other artifact a skill recorded) as a PlanArtifact - the environment and cloned agent get recognised kinds/keys (Environment, Agent), and any other id-bearing object or list of objects is captured too (Connection, EntraApp, KnowledgeSource, or Custom). config_snapshot now returns the full config (deep copy) so the sweep sees every key; cmd_capture-setup pins all detected artifacts. Updates capture.md, model.md, and the design doc SS12/SS3.1 to frame capture as generic. Adds detect_config_artifacts tests (env+agent, generic connection/custom, list-of-objects, changed-only, empty).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nts, validated saves

Code correctness:
- capture: generic config sweep now requires a real before-snapshot (add snapshot-config + capture-setup --before-file); with no snapshot only the recognised env+agent are pinned so pre-existing config is never mis-attributed. Never sweep the 'agents' inventory list (no duplicate/historical agents). Add capture-setup --dry-run (preview before pinning).
- plan_model.claim_task rejects tasks that aren't an open role pool (never silently replace an owner / erase a role).
- tasks_for_person (Flow 2) excludes Completed tasks.
- render_summary shows met AND unmet scenario dependencies (scenario_dependency_status).
- validate flags artifacts whose producedByTaskId references an unknown task.
- cli._save validates before persisting (refuse to write an invalid plan).
- pin-output/capture-setup: reject malformed --attr; do not mark a task Completed while declared produces are unresolved (pin still persists).
- summary is read-only (no longer rewrites ESS-scenario-plan.md, so it can't clobber unreconciled edits).

Docs:
- register /planner in the repo-root wrong-folder redirect list.
- SKILL.md: /setup connects to an already-deployed env (does not create it).
- capture.md: planner invokes capture after /setup (setup flow has no hook).
- __init__/plan_model docstrings: RoleDirectory (not IRoleDirectory); no task 'action' field.
- edit.md: narrow the editable surface (description/produces/consumes are chat-intent, not table columns).
- research.md: research context is session notes today (no persisted sidecar); briefs re-read Learn live.
- scenario_catalogue.md: note the Handoff any-of prerequisite is a known check-deps limitation.
- fix add-task usage example (--skill removed).

Tests: +9 planner tests (claim rejection, Flow-2 completed exclusion, unresolved-produces, orphan-artifact validation, dry-run, read-only summary, attr validation, completion guard). 108 pass; ruff clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
harsheetjain and others added 29 commits August 31, 2026 17:49
The AgentConfiguration client core runs inside stdio MCP servers, where stdout is reserved for JSON-RPC frames. Printing the 'Opening browser for sign-in' notice to stdout could corrupt the protocol stream on the first uncached authentication and disconnect the MCP client. Route the human-facing notice to stderr instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the functional review threads on the WeveNova planner MCP/skill work:

- list_project_plan_tasks_for_caller: match role-pooled tasks only when
  Role-typed (assignedToRoleId + assignedToType eq 'Role') so a person's
  grounding role no longer leaks their tasks to every holder of that role (#14),
  and exclude Completed tasks from the caller's active view (#11).
- Mark create_agent_configuration_project (get-or-create) and attest_plan_role
  (upsert) idempotent via a new _CREATE_IDEMPOTENT_ANNOTATIONS so clients may
  safely retry them after an ambiguous failure (#21).
- Add mcp and httpx to scripts/requirements.txt (the only manifest the standard
  bootstrap installs) so the planner MCP server can start after setup (#9/#15).
- roles/SKILL.md: prefer the cached plan and, on multiple remote plans, ask the
  maker via a clickable choice instead of auto-picking the most recently
  updated one (#18).
- planner/sync.md: publish with a persisted idempotencyKey reused across
  retries so an ambiguous 5xx can no longer duplicate the plan (#19).
- roles/nudge.md: add a named-assignee visibility path for work pooled to
  non-attestable roles so it can no longer stall silently (#20).
- Rename the MCP surface modules planner.py/roles.py to
  planner_surface.py/roles_surface.py to end the sys.modules collision with the
  scripts/planner and scripts/roles packages, so the full suite collects in one
  pytest invocation (#5).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
src/mcp/agentconfig/mcp.server.json is a leftover contextual descriptor from the
pre-split MCP layout: its cwd points at src/mcp/agentconfig, which now holds only
the descriptor (the landing-page server.py lives in src/mcp/agentconfig_landing_page
after the #251 sync). The landing-page server is already registered as a default
in .vscode/mcp.defaults.json with the correct cwd, and nothing invokes
`mcp_config.py configure landing-page`, so the descriptor is redundant and would
only ever materialize a server that cannot launch.

This is a landing-page artifact rather than planner work, so remove it here; the
identical removal is reconciled on feature/landing-page-config in a separate PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(Entra + PowerPlatform)

The planner MCP hard-coded only the 3 External attestable roles and forced
provider="External", which wrongly rejected assigning Power Platform
Administrator (an Entra directory role) and every other provider-owned role.

Mirror the backend registry AttestableAuthorizationRoles.cs: 12 roles across
three providers (External, Entra, PowerPlatform). Each role carries its owning
provider, and attest_plan_role now derives the provider from the role instead
of hard-forcing External (the caller may still pass a provider, but it must
match the role's owner). External roles keep sending provider="External", so
existing behaviour is preserved.

Update the roles skill docs (SKILL.md, attest.md, nudge.md, planner/assign.md)
to defer to the live list_attestable_roles set instead of hard-denying anything
outside the 3 External roles, and to map "Power Platform admin" to the Entra
attestable id.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…PlanId

The planner backend now keeps at most one active plan per project (activating a
plan archives the previous one) and names it in Project.activePlanId. Drop the
obsolete multi-plan 'list plans and ask which to resume/attest' logic from the
planner and roles skills, the planner prompt, and sync.md, replacing it with a
single-plan resolution: use activePlanId when set, else take the lone
non-archived plan (an un-activated Draft) from list_project_plans, and never fall
back to 'the most recently updated'. Surface the same invariant in the get/create
project and list_project_plans MCP tool docstrings. Verified end-to-end via the
MCP tool surface against the live backend (create/get project, plan+task CRUD,
role-assigned task, and the 2nd-activation-archives-1st invariant).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…the plan's

sync.md step 6 told the agent to reuse the etag returned by update_project_plan
(the plan's etag) for the assignment mutation that follows activation. But a task
reassignment goes through update_project_plan_task, whose etag must be the task's
own etag, so reusing the plan etag guarantees a 412; a first role attestation
(create_role_assigned_project_plan_task) is a create and takes no etag at all.
Re-hydrate right after activation and record the assignment with the etag that
matches the call. Documentation-accuracy fix only; no tool or code behavior
changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The /roles person resolver requests only User.ReadBasic.All, whose basic projection does not grant jobTitle. A \ on it comes back empty at best and 403s the whole call at worst, so the field could never be reliably populated. Drop jobTitle from the Graph \, the candidate projection, the resolve-person disambiguation contract, and the corresponding mocks/tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin the WorkIQ preview server default from @latest to the current published 1.0.0 so maker environments get a reproducible people-fallback MCP instead of silently drifting when a new WorkIQ is published.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite the planner activation narrative to match backend-verified behavior:
a plan publishes as Draft with assignees baked in, a Draft's tasks are
read-only until Active, and the backend never auto-activates. Activation is
now an explicit step the sponsor confirms, not something triggered by the
"first role/task assignment".

- sync.md steps 5-6: ask the sponsor whether/when to activate; a Draft can't
  be updated in place (create_project_plan makes a new plan), so refine
  locally before publishing.
- assign.md: publish as Draft, then ask the sponsor to activate.
- SKILL.md: three spots updated to the confirm-then-activate model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add dispatcher-based evaluation creation, catalogue-grounded generation, deterministic CSV export, and folder-scoped quality validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Persist review markers through setup and scoped push, reconcile local and deployed state, and safely promote workspace sets before cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add deployed-set discovery, connected Copilot Studio profile selection, asynchronous run execution, history retrieval, and evidence-based result analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Return explicit reasons for pending reviews and locally completed reviews that have not been pushed, while keeping blocked sets unavailable for execution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Require structured test-set selection, offer makers self-edit or SME feedback paths, and clarify that reviewer recommendations return official edits and execution ownership to the maker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Limit reviewer actions to inspection and written recommendations while keeping all source edits, validation, push, and execution with the maker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l-wevenova-mcp

feat(agentconfig-mcp): WeveNova planner MCP for planner-skill integration
Refresh the configured agent and rebuild its baseline before listing review work, using the existing checkpointed refresh flow so reviewers do not need a separate pull step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite the planner's Markdown view (ESS-scenario-plan.md) from a flat
`group -> key: value` bullet dump into a readable, sectioned document:
an Overview (market, audience, jobs-to-be-done, business goals, pilot
bar), Scenarios in scope (each with its capabilities, backing system and
plain-language dependencies), Systems, plus the existing Scenario
dependencies, Tasks and Produced outputs ledgers.

The view stays grounded on plan.json (the local cache hydrated from the
persisted WeveNova plan) and is enriched from Microsoft Learn at
render/refresh time when a research-context.json corpus is present
alongside the plan. Enrichment is best-effort: the plan renders fully
without it. render_summary stays pure (optional research arg); the
sidecar load happens in write_summary/cmd_summary.

Update the planner skill instructions (edit.md, SKILL.md) to describe the
readable sections and the render-time Learn enrichment, and add tests for
the new sections, humanize/learn_links/read_research_context helpers, and
the preserved dependency-table and Tasks contracts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the original test case identifier as the display fallback when local component metadata is unavailable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve explicit review-block messages, require synchronized review completion, validate runtime configuration, clean collision-safe exports, and keep Python 3.11 compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Redirect the run command from the monorepo root and document generation, review, push, execution, history, and results behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ations

evaluations: add end-to-end authoring, review, and execution
WeveNova replaced the per-item upsert, the single-item DELETE and the bulk
reconcile endpoint with one POST .../syncInventory that takes the tenant's
entire inventory. Absence is now the delete verb: anything Active that the
payload omits is retired. There is no server-side guardrail, so the failure
direction inverted -- a partial crawl used to retire too little and would
now retire too much. Every safety property has to live in the client.

Contract migration:
- Replace upsert/retire/reconcile with a single sync_inventory call, and
  drop all ETag/If-Match preconditions and the 412 retry path with them.
- Add carry-forward: read the current inventory first and re-send verbatim
  every Active row belonging to a scope this run cannot vouch for, so a
  scope that failed, was truncated, or was never visited loses nothing.
- Let a scope retire by omission only when it is authoritative -- fully
  enumerated, no fatal error, nothing unmappable, not capped, and read
  tenant-wide for that kind.
- Withhold the sync entirely when the current inventory cannot be read, or
  when it holds a kind this build cannot round-trip faithfully. Never
  submit an empty payload.
- Enforce the client-side limits (400 items, 50 per kind, no duplicate
  kind:naturalKey) before sending rather than trading a round trip for a
  rejection.

Fix the long-POST timeouts that were leaving the run stranded:
- Tier the HTTP budgets (connect 10s / read 30s / sync 600s). One flat 30s
  covered both a paged GET and a sync that legitimately runs for minutes,
  so the sync always timed out and was re-sent five times onto a service
  still applying the first attempt.
- Derive the MCP RPC budget from the sync budget so the two can never be
  equal; whichever expires first owns the error, and the inner one is the
  informative one.
- Give the sync its own 2-attempt retry policy instead of inheriting the
  5-attempt read budget, where every attempt re-sends the whole payload.
- Skip the request entirely when the payload already matches the service,
  comparing on the wire form so a re-run over an unchanged tenant does no
  writes at all. Anything not provably equal still syncs.
- Stop backing off after the final attempt, which only spaced out a try
  that never happens.

Narrate the run on stderr with a heartbeat so a multi-minute sync is
visibly alive, and set expectations before the wait rather than after, so
a healthy run is not mistaken for a hung one and cancelled.

Add the WeveNova MCP server that fronts the Inventory API and resolves its
own token; it is the default write path for /discover.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert the evaluation lifecycle merge from feature/planner-skill so the work can be proposed independently against main.

This reverts merge commit 555d429.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l-evaluations

REVERT: evaluations: add end-to-end authoring, review, and execution
 Add /discover skill: admin-run tenant inventory discovery crawler
@amilandi

Copy link
Copy Markdown
Contributor

Telemetry gap: /planner doesn't emit ADK capability telemetry

Reviewed the branch for telemetry coverage. The new skill will not appear on the ADK Capability Usage donut (or any ADK cube). Every other maker-facing skill in solutions/ess-maker-skills/src/skills/ either emits capability.use in-process (setup, connect, discover, evaluate, backup/restore template configs, push, publish) or via a python scripts/emit_capability.py <cap> step in its SKILL.md (topics/{create,update,delete,review,test}, workflows/{create,update,delete,test}, cleanup, troubleshoot, onboarding, evaluations/{create,update,delete}). /planner does neither — I grepped the whole feature branch for emit_capability, adk_telemetry, and adk_capability and found zero hits across all 23 planner source files and 12 test files.

Consequence: when this merges as-is, every /planner run will be silently invisible on the dashboards used to track ADK adoption and capability usage. It won't even land in the unknown bucket — the emit call has to happen for that.

Recommended fix

Match the /discover pattern from PR #238 (Apurva's TestCapabilityTelemetry class is the model). It's a one-place, in-process emit right after argparse succeeds:

  1. Add "planner" to ADK_CAPABILITIES in solutions/ess-maker-skills/scripts/adk_telemetry.py (and to the comment header block). Note that PR Split capability taxonomy so every user-facing command has its own dashboard wedge #259 already has this pending — you may want to coordinate with that PR or with amilandi to avoid a merge conflict on ADK_CAPABILITIES.

  2. Emit once per invocation in scripts/planner/cli.py::main, right after parser.parse_args(argv):

    def main(argv: list[str] | None = None) -> int:
        _configure_io()
        parser = build_parser()
        args = parser.parse_args(argv)
    
        # Anonymous capability telemetry, emitted here — right after argparse
        # succeeds, so `--help` and bad arguments don't count — rather than at
        # the end. The "Capability Usage by Type" donut measures that a maker
        # *used* /planner, not whether the command happened to succeed; gating
        # the emit on a clean exit would silently undercount precisely the
        # runs worth looking into.
        try:
            import adk_telemetry
            # block=True: short-lived CLI process, emit synchronously so the
            # event isn't dropped when the interpreter exits and kills a
            # daemon thread.
            adk_telemetry.emit_capability_use("planner", block=True)
        except Exception:  # noqa: BLE001 — telemetry must never break the flow
            pass
    
        try:
            return args.func(args)
        # ...
  3. Do NOT also add a emit_capability.py planner step to SKILL.md. emit_capability_use() doesn't dedupe, so wiring both would double-count every run. This is called out in Apurva's SKILL.md guardrail for /discover — same reasoning applies here. scripts/planner/cli.py is always the entry point, so the in-process emit is sufficient.

  4. Add a test file tests/planner/test_cli_telemetry.py modeled on Apurva's TestCapabilityTelemetry from tests/scripts/test_discover_inventory.py:

    • test_planner_is_canonical_so_it_cannot_bucket_to_unknown — asserts "planner" in ADK_CAPABILITIES and normalize_capability("planner") == "planner".
    • test_a_run_emits_the_capability_exactly_once — spies emit_capability_use, runs cli.main(...), asserts exactly one ("planner", ...) call.
    • test_the_emit_is_synchronous — asserts block=True.
    • test_bad_arguments_do_not_count_as_usagemain(["nonexistent-cmd"]) raises SystemExit; assert no emit.
    • test_a_telemetry_failure_never_breaks_the_flow — monkeypatch emit_capability_use to raise; assert main(...) still returns 0.

Note on subcommand granularity

/planner has many subcommands (init, add-scenario, add-task, assign, claim, capture-setup, pin-output, research, summary, validate, sync, etc.). Emitting a single "planner" capability for all of them is the right first cut — it matches how the rest of the taxonomy works today (one value per maker-facing skill/command, not per sub-action). If PMs later want finer breakdown (e.g., "how many plans got past add-scenario?"), that's an additive change: add adk_capability as an already-present dimension and stamp the subcommand into a new planner_subcommand dimension on a follow-up telemetry event, without breaking the donut.

Happy to help review the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants