Skip to content

Test that the code examples in the docs run and match their output - #244

Merged
stefankoegl merged 1 commit into
masterfrom
claude/youthful-mayer-kn8ntk
Oct 27, 2026
Merged

stefankoegl merged 1 commit into
masterfrom
claude/youthful-mayer-kn8ntk

Conversation

@stefankoegl

Copy link
Copy Markdown
Owner

Add DocumentationTests to tests.py:

  • test_tutorial runs doc/tutorial.rst as a doctest.
  • test_commandline runs the "$ " shell sessions of the bash code blocks
    in doc/commandline.rst against the CLI of this checkout and compares
    stdout with the documented output. "$ cat FILE" provides the example
    input files.

The new tests uncovered several broken examples, which are fixed:

  • tutorial: multi-line statements lacked "..." continuation prompts,
    from_string() was called with a placeholder string, results were shown
    for assignments, and the result of from_string() was discarded.
  • commandline: jsondiff output used an outdated operation order and key
    order, jsondiff prints nothing (not "[]") for equal files, and the
    usage texts were outdated.

simplejson is added to the dev requirements for the tutorial examples, and
the doc sources are shipped in the sdist, which already includes tests.py.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01PewzpkVb4tA4eWEkCzH9vT

Add DocumentationTests to tests.py:

* test_tutorial runs doc/tutorial.rst as a doctest.
* test_commandline runs the "$ " shell sessions of the bash code blocks
  in doc/commandline.rst against the CLI of this checkout and compares
  stdout with the documented output. "$ cat FILE" provides the example
  input files.

The new tests uncovered several broken examples, which are fixed:

* tutorial: multi-line statements lacked "..." continuation prompts,
  from_string() was called with a placeholder string, results were shown
  for assignments, and the result of from_string() was discarded.
* commandline: jsondiff output used an outdated operation order and key
  order, jsondiff prints nothing (not "[]") for equal files, and the
  usage texts were outdated.

simplejson is added to the dev requirements for the tutorial examples, and
the doc sources are shipped in the sdist, which already includes tests.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PewzpkVb4tA4eWEkCzH9vT

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are focused, preserve existing CLI test behavior, and have no unresolved blocking findings.

No open issues 🎉

What changed in this PR

Adds automated checks for documentation examples so the tutorial and command-line examples stay consistent with the library and CLI.

Changes:

  • Runs tutorial examples as doctests and checks documented shell-session output using a shared CLI runner.
  • Corrects tutorial examples, CLI output, and usage text.
  • Adds the tutorial’s development dependency and includes documentation sources in source distributions.
File Description
tests.py Adds documentation tests and extracts a shared CLI runner.
requirements-dev.txt Adds simplejson for tutorial examples.
MANIFEST.in Includes documentation sources in source distributions.
doc/​tutorial.rst Makes examples executable and corrects displayed results.
doc/​commandline.rst Updates CLI usage text and expected output.

Review effort: Balanced

@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 100.0%. remained the same — claude/youthful-mayer-kn8ntk into master

@stefankoegl
stefankoegl merged commit 3c8240c into master Oct 27, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants