Skip to content

Consolidate all tests into one repo-level suite run on every publish #28

Description

@ElliotDrel

Problem

Tests are split across two homes and only cover a fraction of the pack, so a publish can go green while most skills are untested.

Tests live inside skills and ship to users. skills/estack-doc-review-viewer/e2etest.mjs and selftest.mjs sit in the skill folder, so npm pack bundles them — 43.3kB of test code downloaded by every user who installs elliot-stack. They are also invisible to pytest, which sets norecursedirs = skills, so nothing in CI runs them.

Coverage is thin. tests/ covers 3 of 23 skills: estack-flight-planner, estack-migrate-claude-session-history, estack-read-agent-history. The other 20 have no automated test at any level.

There is no single entry point. .github/workflows/publish.yml runs pytest -q, then two hand-listed commands for the migrate smoke tests, because those are a Python script and a Node script that pytest does not collect. Adding a test in a new language means editing the workflow, and anything not hand-added is silently skipped.

Nothing runs when a skill did not change. Gate 6 in .agents/skills/manage-e-stack/steps/prep.md runs the suite, but the suite itself only reaches the three covered skills. A regression in an untouched skill — caused by a shared script, a convention change, or a doc migration — surfaces after release.

What to build

One suite at the repo root that runs everything, every publish, regardless of what changed.

  • All tests move to tests/. Nothing test-related stays inside skills/. Move e2etest.mjs and selftest.mjs to tests/estack-doc-review-viewer/ and repoint their imports. Skill folders hold only what a user needs at runtime.
  • One command runs the whole suite and returns a single exit code. It must collect Node and Python tests together, so the two hand-listed migrate commands disappear from the workflow.
  • Every skill is covered by at least a smoke test — the skill loads, its frontmatter parses, and its scripts run --help without throwing. That alone would have caught the ENOENT risk found when skills moved to ~/.e-stack/, where tracker-tools.cjs had no mkdir anywhere in it.
  • The suite runs unconditionally. No change detection, no skipping unchanged skills.
  • prep.md gate 6 and publish.yml both call the one command, so local and CI runs cannot drift apart.

Verification

  • npm pack --dry-run lists no test file under skills/.
  • The publish workflow contains exactly one test step.
  • Removing a mkdir from any skill's write path fails the suite.
  • Every folder in skills/ has a corresponding folder in tests/.

Found during the v1.0.68 publish-readiness audit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions