[geo-optimizer] Add robots.txt to docs site to explicitly allow AI bot crawling

[github-actions]

Source audit: GEO Audit Report — 2026-07-03
Audit date: 2026-07-03
Run: §28673522290

Finding

robots_txt score: 0/18 — found: false. All AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are in an ambiguous crawl-permission state. This single gap costs 18 points on every page across the 195-URL sitemap — the single largest score deficit in the entire docs site audit.

Why this matters

Without a robots.txt, AI crawlers cannot confirm they are explicitly permitted to index github.github.com/gh-aw/. This reduces citation probability in ChatGPT, Perplexity, and Google AI Overview because those engines deprioritize or skip sites with ambiguous bot permissions. Fixing this single file unlocks +18 points site-wide — the largest single-file gain available.

Suggested fix

1. Generate a compliant robots.txt using the geo-optimizer CLI:

   geo robots --base-url https://github.github.com/gh-aw/ > public/robots.txt

2. Verify the output explicitly allows at minimum: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, anthropic-ai.
3. Deploy the file so it is served at https://github.github.com/gh-aw/robots.txt.
4. Re-run the GEO audit (gh aw run geo-optimizer) to confirm the score improvement.

Note: The GitHub repo (github.com/github/gh-aw) already has an excellent robots.txt (15/18) with 27 AI bots allowed — the docs site just needs to match that.

Generated by 🌍 GEO Optimizer Daily Audit · 38.2 AIC · ⌖ 13.5 AIC · ⊞ 5K · ◷

• expires on Jul 10, 2026, 8:57 AM UTC-08:00