End to end tests - #568
Merged
Merged
End to end tests#568
Conversation
Each subdirectory of src/test/resources/e2e is a test case that runs the program in a subprocess and compares its output against goal files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Master converted build.gradle to build.gradle.kts, so port the end-to-end test configuration (JUnit dependencies, the system properties that the test harness reads, and the JaCoCo agent argument for the subprocess) to the Kotlin DSL. Make EndToEndTest package-private, as pmdTest requires. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* sortLines: do not assume that the output ends with a line separator. * normalize: do not substitute for `user.home` when the JVM could not determine it and set it to "?". * run: kill a subprocess that does not finish, rather than blocking the build forever. * A failed postcheck reports postcheck.sh's own output, not the program's. * Remove JVM warnings from standard error only; a line of the program's own standard output might start with "WARNING: ". * A failure message states the versions of the programs that the test case requires, since a goal file may record the exact wording of their messages. * Reject a test case directory that contains an unrecognized file, or a goal file `expected-postcheck` without `postcheck.sh`. * Rename `getRequiredProperty` to `propertyOrDefault`, which is what it does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test harness: * Remove JAVA_TOOL_OPTIONS, _JAVA_OPTIONS, and JDK_JAVA_OPTIONS from the subprocess environment. A JVM that inherits them prints a "Picked up ..." line to standard error, which made every test fail. * Leave the files of at most 5 failing tests in place, and say in the failure message when a directory was deleted instead. Test cases: * Add `requires git` to the 38 test cases that run git. Without git on the PATH they failed in setup.sh rather than being skipped, and their failure messages did not state the git version. * pull-real-hg: add `sort-output`. hg writes this message to both standard output and standard error, and the program merges the two streams with one thread apiece, so the order varied from run to run. Build: * Disable the JaCoCo agent in the test JVM, which runs none of the program's code; it already runs in the subprocess. * Fence the generated command-line option documentation off from spotless. google-java-format 1.36.1 reflowed it to one or two words per line. * Correct the comment about how the CI job matrix chooses a test JVM. Checks: * prek.toml: exclude the test cases' fixture files, which the previous glob misnamed, but keep checking their shell scripts. * Drop the CircleCI hook and its install step; there is no .circleci directory. Narrow the .claude/settings.json permissions back to the project: Edit(./**) rather than Edit(*), and one WebFetch domain rather than all of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Program: * Merge a subprocess's standard output and standard error a whole line at a time. PumpStreamHandler pumps each stream in a thread of its own, and ByteArrayOutputStream.write is atomic per chunk rather than per line, so one stream's output could land in the middle of a line of the other's. Sorting the output of pull-real-hg cannot repair that, so that test could fail. Test harness: * Do not let a problem with deleting a failing test's temporary directory replace the test's own failure message. A subprocess that the harness killed for running too long, or a process that such a subprocess started, may still be creating files there. * Wait for a killed subprocess to die, which makes that less likely. * When regenerating goal files, print the name of each test case that is skipped because a program it requires is not installed; regeneration leaves such a test case's goal files unchanged. Say so in the class documentation and in the README, too. Build: * Delete build/jacoco/test.exec before running the tests. Disabling the JaCoCo extension for the test JVM also disabled Gradle's own deletion of that file, and the subprocesses append to it, so coverage data accumulated across runs: a report credited tests that had since been changed or deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # .claude/settings.json # prek.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.