Skip to content

fix: align H1 headings with frontmatter title convention#36

Merged
waynesun09 merged 6 commits into
mainfrom
fix-experiment-h1-headings
Jul 2, 2026
Merged

fix: align H1 headings with frontmatter title convention#36
waynesun09 merged 6 commits into
mainfrom
fix-experiment-h1-headings

Conversation

@waynesun09

Copy link
Copy Markdown
Member

Summary

  • Update all 21 experiment H1 headings to match their frontmatter title: field
  • The numbering convention (PR feat: add experiment numbering and metadata conventions #35) set frontmatter titles to NNNN. Title format but left H1 headings with old inconsistent formats (Experiment 002:, PoC FullSend, ADR 0022, etc.)
  • The fullsend docs site sidebar extracts H1 for directory-based entries (fullsend-ai/fullsend#2969), so mismatched H1s cause wrong sidebar titles

Test plan

  • Verify no remaining frontmatter/H1 mismatches: for d in 0*/; do fm=$(grep -m1 "^title:" "$d/README.md" | sed 's/^title:\s*//' | tr -d '"'); h1=$(grep -m1 "^# " "$d/README.md" | sed 's/^# //'); [ "$fm" != "$h1" ] && echo "MISMATCH: $d"; done
  • Pre-commit hooks pass (lint-experiment-numbers, lint-experiment-frontmatter, lint-experiment-index)
  • After merging and bumping the submodule in fullsend, sidebar titles display correctly

The experiment numbering convention (PR #35) set frontmatter titles
to the `NNNN. Title` format but left H1 headings with their old
inconsistent formats (e.g. "Experiment 002:", "PoC FullSend",
"ADR 0022"). This causes the fullsend docs site sidebar to display
wrong titles since it falls back to H1 when rendering directory
entries.

Update all 21 experiment H1 headings to match their frontmatter
title field exactly.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@waynesun09 waynesun09 requested a review from a team as a code owner July 2, 2026 20:55
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 8:56 PM UTC · Ended 8:57 PM UTC
Commit: 0a95cac · View workflow run →

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Align experiment H1 headings with frontmatter titles

🐞 Bug fix 📝 Documentation 🕐 10-20 Minutes

Grey Divider

AI Description

• Update 21 experiment docs so the H1 exactly matches frontmatter title:.
• Fix docs-site sidebar titles that fall back to H1 for directory entries.
• Normalize legacy heading formats ("Experiment 002:", "PoC", "ADR") to NNNN. Title.
Diagram

graph TD
  A["Experiment markdown files"] --> B(["Docs site renderer"]) --> C{"H1 matches frontmatter title?"}
  C -->|"Yes"| D["Correct sidebar label"]
  C -->|"No"| E["Wrong sidebar label"]
  subgraph Legend
    direction LR
    _file["Document"] ~~~ _proc(["Process"]) ~~~ _dec{"Decision"}
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Update the docs site to prefer frontmatter title for directory entries
  • ➕ Eliminates reliance on H1 as a fallback source of truth
  • ➕ Prevents future sidebar regressions even if authors change headings
  • ➖ Requires changes in the consuming docs site repo and its release process
  • ➖ May not be possible if the renderer cannot access frontmatter for directory nodes
2. Add/strengthen a lint rule enforcing `title:` == H1
  • ➕ Prevents drift from reappearing
  • ➕ Keeps the convention explicit and automatically enforced in CI/pre-commit
  • ➖ Doesn't fix existing mismatches by itself
  • ➖ Adds maintenance cost to lint tooling and contributor workflow

Recommendation: The PR’s approach is the fastest and least invasive fix for the current regression: make the H1 match the established frontmatter NNNN. Title convention everywhere. As a follow-up, consider adding/confirming an automated lint check for title:/H1 equality to prevent future drift; updating the docs site to always prefer frontmatter is attractive but belongs in the consumer repo and may not be feasible depending on its renderer.

Files changed (21) +21 / -21

Documentation (21) +21 / -21
0001-agent-outage-fire-drill.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the legacy H1 format ("Experiment 003: …") with the normalized numbered title matching the frontmatter 'title:' field.

0001-agent-outage-fire-drill.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 to match the exact frontmatter 'title:' value, removing the older "Experiment 002:" prefix formatting.

0002-claude-scanner/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Normalizes the H1 to the 'NNNN. Title' convention so it matches the frontmatter title used by navigation tooling.

0003-scanner/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Shortens/normalizes the H1 text and number formatting to exactly match the frontmatter 'title:' field.

0004-meta-loop-self-improving-engine/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the prior PoC-style heading with the standardized numbered title matching frontmatter for consistent sidebar labels.

0005-agent-scoped-tools-triage/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 from an unnumbered "Experiment:" format to the normalized numbered title used in frontmatter.

0007-github-actions-agent-runtime-mvp/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Rewrites the H1 to match the frontmatter 'title:' exactly, ensuring consistent navigation titles.

0008-guardrails-eval/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Normalizes capitalization and numbering in the H1 so it matches the frontmatter title string used by site tooling.

0009-hermes-security-patterns/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Converts the H1 from the prior "Experiment:" phrasing to the standardized numbered title matching frontmatter.

0010-host-side-api-server/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Adjusts the H1 wording/capitalization to be identical to the frontmatter 'title:' value for stable sidebar rendering.

0011-integration-service-design-drift/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the longer "Experiment:" H1 with the normalized numbered title matching frontmatter, avoiding sidebar mismatch.

0012-model-armor-vs-agent-triage/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 to the 'NNNN. Title' convention and matches the frontmatter title used for navigation labels.

0013-openshell-policy-bypass/README.md

0014-openshell-sandbox-evaluation.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Normalizes the H1 from the previous "Experiment:" format to match the frontmatter numbered title exactly.

0014-openshell-sandbox-evaluation.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the prior "Experiment 002:" H1 prefix with the standardized numbered title matching the frontmatter field.

0015-prompt-injection-defense/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 from the older "Experiment 004:" style to the normalized numbered title consistent with frontmatter.

0016-promptfoo-eval/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Normalizes the H1 string (numbering and capitalization) so it matches the frontmatter title used by the docs sidebar.

0017-reasoning-monitor/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Changes the H1 to the standardized numbered format matching the frontmatter 'title:' for consistent navigation labeling.

0018-runner-hello-world/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 from the generic "Experiment:" label to the numbered title matching the frontmatter field.

0019-skills/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the prior "Experiment:" heading with the normalized numbered title to match the frontmatter string exactly.

0020-target-repo-skills/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Updates the H1 from the older ADR-style naming to the standardized numbered title matching frontmatter to avoid sidebar drift.

0021-tool-scoping/README.md

README.mdAlign H1 heading with frontmatter title +1/-1

Align H1 heading with frontmatter title

• Replaces the PoC-style H1 with the normalized numbered title matching the frontmatter 'title:' field for correct sidebar labels.

0022-claude-github-app-auth/README.md

Rename review-cache-experiment to 0023-review-cache, add required
frontmatter (title, status, topics), update H1 heading, and add
entry to README index table.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 8:57 PM UTC · Ended 9:01 PM UTC
Commit: 0a95cac · View workflow run →

@qodo-code-review

qodo-code-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 9 rules
✅ Skills: writing-how-to

Grey Divider


Action required

1. 0006 title/H1 mismatch ✓ Resolved 🐞 Bug ≡ Correctness
Description
0006-code-agent-evaluation/EXPERIMENT.md still has an H1 that does not match its frontmatter
title, so this experiment will remain inconsistent with the convention this PR establishes. This
likely preserves the exact navigation/title mismatch this PR aims to eliminate for at least one
experiment entry.
Code

0007-github-actions-agent-runtime-mvp/README.md[9]

+# 7. GitHub Actions agent runtime MVP
Relevance

⭐⭐⭐ High

Repo established strict experiment title conventions (PR#35); team previously fixed doc consistency
issues in experiment docs (PR#9).

PR-#35
PR-#9

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Experiment 0006’s frontmatter title and H1 diverge, while other experiment entry pages follow the
new convention where H1 equals the frontmatter title.

0006-code-agent-evaluation/EXPERIMENT.md[1-10]
0007-github-actions-agent-runtime-mvp/README.md[1-10]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The experiment entry `0006-code-agent-evaluation/EXPERIMENT.md` has frontmatter `title: "6. Code agent evaluation"` but the document H1 is `# Code Agent Evaluation Experiment`. This breaks the convention updated in this PR where experiment entry pages have H1 identical to the frontmatter title.

## Issue Context
Other experiment entry pages updated by this PR now use `# N. ...` headings that match their frontmatter title exactly, but experiment 0006 still uses an older H1 format.

## Fix Focus Areas
- 0006-code-agent-evaluation/EXPERIMENT.md[1-10]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

2. Stale experiment link labels 🐞 Bug ⚙ Maintainability
Description
Several docs still refer to experiments using the old label format (e.g., “Experiment 002”) even
though the experiment pages are now titled as “2. …”. This leaves cross-references that no longer
match the destination page titles/H1s.
Code

0002-claude-scanner/README.md[9]

+# 2. Claude-based ADR drift scanner
Relevance

⭐⭐ Medium

No prior evidence on updating cross-doc link labels; PR#35 set numbering conventions but didn’t
enforce references.

PR-#35

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The destination experiment page title/H1 now uses the 2. ... format, but multiple other documents
still refer to it as “Experiment 002”, demonstrating inconsistent naming after this PR’s convention
change.

0002-claude-scanner/README.md[1-10]
0003-scanner/README.md[23-29]
0001-agent-outage-fire-drill.md[100-103]
0004-meta-loop-self-improving-engine/README.md[140-155]
0011-integration-service-design-drift/README.md[257-264]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
After renaming experiment entry H1s/titles to the `N. Title` convention, multiple in-repo references still use the legacy `Experiment 00N` wording. This makes cross-references inconsistent with the new canonical experiment titles.

## Issue Context
Examples include relationship sections and comparison headings that still say “Experiment 002”, “Experiment 003”, etc., while the target pages now start with “2.”, “3.”, etc.

## Fix Focus Areas
- 0001-agent-outage-fire-drill.md[100-103]
- 0003-scanner/README.md[23-29]
- 0004-meta-loop-self-improving-engine/README.md[140-155]
- 0011-integration-service-design-drift/README.md[257-264]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment thread 0007-github-actions-agent-runtime-mvp/README.md
---

# Experiment 002: Claude-based ADR Drift Scanner
# 2. Claude-based ADR drift scanner

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

2. Stale experiment link labels 🐞 Bug ⚙ Maintainability

Several docs still refer to experiments using the old label format (e.g., “Experiment 002”) even
though the experiment pages are now titled as “2. …”. This leaves cross-references that no longer
match the destination page titles/H1s.
Agent Prompt
## Issue description
After renaming experiment entry H1s/titles to the `N. Title` convention, multiple in-repo references still use the legacy `Experiment 00N` wording. This makes cross-references inconsistent with the new canonical experiment titles.

## Issue Context
Examples include relationship sections and comparison headings that still say “Experiment 002”, “Experiment 003”, etc., while the target pages now start with “2.”, “3.”, etc.

## Fix Focus Areas
- 0001-agent-outage-fire-drill.md[100-103]
- 0003-scanner/README.md[23-29]
- 0004-meta-loop-self-improving-engine/README.md[140-155]
- 0011-integration-service-design-drift/README.md[257-264]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Add validation to lint-experiment-frontmatter that enforces H1
heading matches the frontmatter title field exactly. Prevents
future drift between the two after the convention was established.

Also fixes 0006-code-agent-evaluation which had a pre-existing
mismatch caught by the new check.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 9:01 PM UTC · Ended 9:04 PM UTC
Commit: 0a95cac · View workflow run →

Align with repo convention — all experiments use README.md as the
primary document. Update cross-references in RECOMMENDATION.md (4)
and invoke-variant.sh (1).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 9:04 PM UTC · Completed 9:14 PM UTC
Commit: 59cfec8 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

Review

Verdict: approve · 1 low finding

Clean consistency fix that aligns H1 headings with frontmatter titles across 21 experiments, standardizes the review-cache-experiment directory to 0023-review-cache, and adds linter enforcement via a new extract_h1() function. The latest commit (05bd450e) addresses all three medium findings from the prior review:

  • extract_h1() frontmatter skipping — now correctly tracks in_frontmatter state between --- delimiters before matching H1 headings
  • Stale EXPERIMENT.md self-reference in 0006-code-agent-evaluation/README.md — updated to README.md
  • Stale directory name in 0023-review-cache/README.md — updated to 0023-review-cache/

The prior low-severity finding about the dead EXPERIMENT.md fallback in lint-experiment-numbers was also resolved — the fallback has been removed. All remaining EXPERIMENT.md references in the repo (RECOMMENDATION.md, invoke-variant.sh) are updated in this PR.

Findings

1. No linked issue for non-trivial change

Severity: low · Category: missing-authorization · File: N/A

This PR touches 43 files and adds linter validation but has no linked issue. The work clearly follows from merged PR #35 (experiment numbering conventions) and is architecturally sound, fitting the conventions documented in AGENTS.md. The scope is well-defined and internally consistent.


Dimensions reviewed: correctness, security, intent & coherence, style/conventions, documentation currency. Cross-repo contracts skipped (no exported interfaces modified). No security findings.


Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

  • AGENTS.md

Labels: PR updates documentation headings and adds documentation linting enforcement.

Previous run

Review

Verdict: comment · 3 medium, 2 low findings

Clean consistency fix that aligns H1 headings with frontmatter titles across 21 experiments, standardizes the review-cache-experiment directory to 0023-review-cache, and adds linter enforcement. The changes are well-scoped and follow the conventions established in the merged PR #35. Three medium findings are worth addressing but do not block merge.

Findings

1. extract_h1() does not skip YAML frontmatter — latent false-positive risk

Severity: medium · Category: logic error · File: hack/lint-experiment-frontmatter, line 65

The new extract_h1() function scans from line 1 without skipping the YAML frontmatter block (delimited by ---). YAML comments start with #, so a line like # some note inside frontmatter would match the ^#\s+(.+)$ regex and be returned as the H1 heading instead of the actual markdown heading. No current experiment files have YAML comments in frontmatter, but any added in the future would trigger a false-positive lint failure.

Remediation: Track whether scanning is inside the frontmatter block (between --- delimiters) and only start matching after the frontmatter ends. Alternatively, reuse parse_frontmatter() to find where frontmatter ends and start scanning from that offset.

2. Stale self-reference to EXPERIMENT.md in file structure diagram

Severity: medium · Category: stale reference · File: 0006-code-agent-evaluation/README.md (renamed from EXPERIMENT.md), line 856

The file structure diagram still contains ├── EXPERIMENT.md # This document. Since this PR renames the file to README.md, the self-reference is now incorrect.

Remediation: Update line 856 to ├── README.md # This document.

3. Stale directory name in file structure diagram

Severity: medium · Category: stale reference · File: 0023-review-cache/README.md (renamed from review-cache-experiment/), line ~210

The file structure diagram still shows review-cache-experiment/ as the directory name. After the rename to 0023-review-cache/, this reference is stale.

Remediation: Update the directory name in the file structure diagram to 0023-review-cache/.

4. No linked issue for non-trivial change

Severity: low · Category: missing-authorization · File: N/A

This PR touches 42 files and adds linter validation but has no linked issue. The work clearly follows from the merged PR #35 (experiment numbering conventions) and is architecturally sound, but AGENTS.md's "surgical changes" principle suggests explicit authorization for non-trivial work.

5. Dead EXPERIMENT.md fallback in lint-experiment-numbers

Severity: low · Category: stale reference · File: hack/lint-experiment-numbers, lines 85–86

The lint-experiment-numbers script (not modified by this PR) still has EXPERIMENT.md as a fallback candidate. After this PR renames the only file using that convention, no experiment uses EXPERIMENT.md. The fallback is harmless dead code but could be cleaned up.


Dimensions reviewed: correctness, security, intent & coherence, style, documentation currency. Cross-repo contracts skipped (no exported interfaces modified). No security findings.


Labels: PR updates documentation and linting for experiment naming conventions.

Trailing whitespace, ruff lint fixes (13 errors), and ruff format
on review-cache experiment files.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 9:17 PM UTC · Ended 9:19 PM UTC
Commit: 0a95cac · View workflow run →

- extract_h1() now skips YAML frontmatter to avoid matching comments
- Fix stale EXPERIMENT.md self-reference in 0006 file tree
- Fix stale review-cache-experiment/ directory name in 0023 file tree
- Remove dead EXPERIMENT.md fallback from both lint scripts

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:20 PM UTC · Completed 9:28 PM UTC
Commit: 05bd450 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment documentation Improvements or additions to documentation labels Jul 2, 2026
@waynesun09 waynesun09 added this pull request to the merge queue Jul 2, 2026
Merged via the queue into main with commit 68c6a3b Jul 2, 2026
4 checks passed
@waynesun09 waynesun09 deleted the fix-experiment-h1-headings branch July 2, 2026 22:18
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 10:20 PM UTC · Completed 10:26 PM UTC
Commit: 05bd450 · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

PR #36 aligned H1 headings with frontmatter titles across 21 experiments, renamed a non-conforming experiment directory, and added a lint check to enforce H1/title consistency. The fullsend-ai review agent performed well, catching 3 medium-severity bugs (frontmatter-skipping in extract_h1, stale file references, dead code fallback) that the author fixed before merge. No human review occurred despite the agent correctly flagging AGENTS.md as a protected path requiring human approval — the PR was self-merged. Two improvement opportunities identified: enforcing protected-path review requirements, and adding test coverage for the lint scripts whose bugs were caught only by the review agent.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation requires-manual-review Review requires human judgment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant