Skip to content

feat(experiments): refine launchdarkly-experiment-setup skill for new MCP tool shapes#70

Merged
hsadhvani merged 2 commits into
mainfrom
harsh/EXPT-4361-refine-experiment-setup-skill
May 29, 2026
Merged

feat(experiments): refine launchdarkly-experiment-setup skill for new MCP tool shapes#70
hsadhvani merged 2 commits into
mainfrom
harsh/EXPT-4361-refine-experiment-setup-skill

Conversation

@hsadhvani
Copy link
Copy Markdown
Contributor

@hsadhvani hsadhvani commented May 27, 2026

Summary

Closes EXPT-4361. Updates the experiment setup skill so the JSON payloads it teaches an agent to send match the actual LaunchDarkly MCP tool shapes (the LD REST API shapes used by gram-functions):

  • create-experiment now uses the required nested iteration object, with hypothesis, metrics, treatments (each with allocationPercent + a parameters array), a flags map keyed by flagKey with ruleId/flagConfigVersion, and primarySingleMetricKey. Documents optional top-level tags, methodology, dataSource, analysisConfig, plus iteration-level attributes, covariateId, canReshuffleTraffic.
  • update-experiment is described as mutableFieldsByStatus-aware: what's editable depends on iteration status, and skipped fields come back in the response.
  • Adds workflow steps for the new tools: save-and-start-experiment-iteration (mid-experiment design changes in one call) and stop-experiment-iteration (with winningTreatmentId / winningReason).
  • Drops the bogus reshuffle: true from the start step; mentions changeJustification instead.
  • Bumps skill version to 0.2.0 and adds a CHANGELOG entry.

Testing approaches

  • Static check against gram-functions tool inputSchemas — every JSON example field name, nesting, and type matches src/tools/experiments.ts in launchdarkly/gram-functions.
  • Recommended follow-up: install the skill locally as a Claude Code plugin and walk through create → start → save-and-start → stop against a test LD project once gram-functions is reachable from your MCP config.

Feature flags

N/A

… MCP tool shapes

Co-Authored-By: Claude <noreply@anthropic.com>
@hsadhvani hsadhvani requested a review from a team as a code owner May 27, 2026 21:35
@hsadhvani hsadhvani marked this pull request as draft May 27, 2026 21:36
Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 75/100 (3/4) -25
agentcontrol/configs-update 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (3/3) no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

@hsadhvani hsadhvani marked this pull request as ready for review May 29, 2026 17:06
@hsadhvani hsadhvani merged commit 673d22a into main May 29, 2026
11 checks passed
@hsadhvani hsadhvani deleted the harsh/EXPT-4361-refine-experiment-setup-skill branch May 29, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants