Add fastly-stats skill

[zshenker]

Summary

Adds a new fastly-stats skill: a dedicated, self-contained reference for Fastly's stats/metrics APIs so an agent can get to the data without reading Fastly's docs — usable via the fastly CLI or by building raw URLs for curl/direct HTTP.

Stats were previously only lightly covered inside fastly-cli (references/stats.md, CLI-only). This skill goes deep on the full family of stats APIs across two hosts and three response shapes, with endpoints, query parameters, and response schemas documented inline.

What's included

skills/fastly-stats/
  SKILL.md                          # entrypoint: the two-hosts/three-shapes model,
                                    # auth + token safety, "choose the right API" table,
                                    # accurate CLI quick-reference, raw-URL templates,
                                    # workflows, POP-level diagnosis, limits
  references/
    historical-stats-api.md         # /stats* on api.fastly.com — from/to/by, flat envelope,
                                    # usage/billing (SI units, billable_units), regions
    inspector-api.md                # Origin & Domain Inspector, historical + real-time:
                                    # start/end/downsample, dimensions+values, cursor pagination
    realtime-api.md                 # rt.fastly.com /v1/channel polling model, Timestamp chaining
    fields.md                       # measurement/metric catalog (with TOC) + all_/compute_/waf_
    debugging.md                    # symptom-keyed troubleshooting

Also lists the skill in README.md. No manifest changes needed (skills are discovered by directory).

Design notes

• Two hosts: api.fastly.com (historical) vs rt.fastly.com (real-time).
• Three response shapes: classic flat {status, meta, msg, data}; Inspector dimensions+values with cursor pagination; real-time Data[].{recorded, aggregated, datacenter}.
• Fixes an inaccuracy in the existing CLI docs: the inspector subcommands use --downsample/--metric/--group-by/--limit/--cursor (not --by/--field).
• Complements the existing fastly-cli stats reference rather than replacing it (cross-referenced).
• Copy-paste CLI + curl templates throughout; no bundled scripts to maintain.
• Emphasizes the non-obvious pitfalls: NDJSON --json (use jq -s), decimal-SI vs binary units and raw-bandwidth-vs-billable-usage, UTC defaults, Origin/Domain Inspector as a paid add-on, real-time Timestamp chaining, cursor pagination, and preferring per-POP (datacenter) granularity (incl. flagging Shield POPs).

Validation

• make validate ✓ (markdownlint, yamllint, jsonlint, frontmatter checks, plugin/extension validate)
• make skillscheck (--strict) ✓ — 0 errors, 0 warnings

How it was tested

Developed and evaluated with the skill-creator workflow: with-skill vs. no-skill agent runs across three realistic tasks (cross-service bandwidth, live RPS/5xx watch, origin 5xx/latency), graded against explicit assertions. The skill scored 100% then 95% (against a stricter assertion set) vs. an 82% no-skill baseline, catching concrete errors the baseline made — mishandling NDJSON, missing cursor pagination, unprefixed Inspector metric names, and binary-GiB units.

🤖 Generated with Claude Code

[jedisct1]

I don't understand the purpose of that file.

It looks like a methodology essay on experimental design, with a thin Fastly costume.

It's a generic undergraduate lab-methods handout, telling the agent nothing about what Fastly actually returns., it seems to just tell the agent how to reason.

Figures seem to be based on some random test/example, and are not going to match a real users' account. None can be checked or reproduced.

I just teaches what the model already knows. There's zero unique Fastly content, so as Fastly documentation, it's not useful.

Can you clarify the intended purpose of that file?

[jedisct1]

This is an agent skill, for agents, not a markdown document for humans.

There's too much prose (just increasing token usage) and formatting like ** and * is useless to agents.

[jedisct1]

Evaluation https://github.com/fastly/fastly-agent-toolkit-evals/tree/main/additional-experiments/stats-pr12

Quoting the verdict:

1. Cut the size. 11,743 words to 4,465, 7 files to 4. Delete references/verification-method.md, merge the three API reference files into one, drop the DEN/IAD/BOS war stories and all 189 bold spans, and move the eight rules that actually decide correctness to the top of SKILL.md. Today "use decimal SI, not GiB" is a sub-bullet in the tenth section, and Part 2 shows it decides two of the five tasks.

2. Give every endpoint one owner. A session loads fastly, fastly-cli and fastly-stats together, so three copies of the /stats* table are three copies in the same context window. fastly-stats has the best version, so let it keep the material and cut the other two down to a pointer: observability.md 1,018 to 407 words, fastly-cli/references/stats.md 1,769 to 463. The routing data says the agent already behaves this way, opening fastly-stats and nothing else; the duplicates are paying rent for text nobody reads.

   This includes the unit, window and gauge rules that f6dc310 put into both neighbours. They are there because main has no fastly-stats to point at, and about 85 words of them are duplicated between observability.md and fastly-cli/references/stats.md. Merging PR Add fastly-stats skill #12 removes the reason: collapse them into fastly-stats, which owns the conventions, and revert both neighbours to pointer-only. skills-trim is already in that shape and scores 24/25 on both models, so nothing is lost by doing it.

3. Lead with the CLI. 15 curl invocations become 5, and the 5 that survive are the ones fastly stats genuinely cannot reach. Every avoidable curl costs a token substitution, a URL-encoded time string and a hand-written envelope walk, and PR Add fastly-stats skill #12 spends a whole debugging section on a failure that only exists because of the substitution.

4. Fix the three wrong claims in Part 4.4. An unentitled Inspector returns HTTP 200 with an empty array, not an error, and fastly-cli has said so correctly all along. from=1 day ago&by=day, the skill's own first worked call, returns zero rows. The realtime CLI example filters on .Data[]? and silently prints nothing. All three produce confident wrong answers rather than visible failures, which is the worst failure mode for a reference an agent trusts.

5. Narrow the description so three skills stop competing for the same question.

Recommendation: merge, but merge the trimmed version. The 82%-overlap number in Part 1 is an argument for consolidating ownership.

[jedisct1]

Token/wall time usage with Deepseek 4-Flash:

       skills mcp    trials   turns   wall s   prompt tok
       base   nomcp      25      15      182      176,173
       stats  nomcp      25      10      106      109,659
       trim   nomcp      25      11       85       93,445
       base   mcp        25      19      221      212,918
       stats  mcp        25      14      160      170,207
       trim   mcp        25      10       93       97,773

Trimming the fastly-stats skill significantly reduces wall time and price.