urtest: Convert AuTest tests to pytest - #13545
Draft
bneradt wants to merge 7 commits into
Draft
Conversation
AuTest's stalled file-format support leaves replay tests tied to thin Python registration wrappers. Collect replay YAML directly so every scenario has an independent pytest result and sandbox. Add ATS-specific DNS, Proxy Verifier, Traffic Server, log, and metric lifecycle support, then convert all 96 ATSReplayTest registrations to .test.yaml files. Keep bespoke AuTests in place and run both suites from the existing autest targets.
bneradt
marked this pull request as draft
August 13, 2026 18:37
bneradt
changed the title
bneradt
changed the title
ATS end-to-end coverage depends on the stagnant AuTest framework and splits replay and bespoke tests across different runner abstractions. This makes the suite harder to maintain, schedule, and debug. This patch replaces AuTest with pytest-owned Uranium tests. Replay YAML is collected directly, while a native scenario layer preserves existing bespoke test behavior with isolated processes, readiness checks, cleanup, filtering, sharding, and Fedora container execution. It also adds procedural ATS and curl fixtures, converts the first basic tests to ordinary pytest, and updates the build, CI, documentation, and test layout around the urtest interface.
bneradt
force-pushed
the
pytest-replay-tests
branch
from
August 14, 2026 17:29
1b8339e to
678cb46
Compare
added 2 commits
August 14, 2026 14:41
Procedural Uranium tests need reusable process ownership and curl targeting that works for multiple ATS instances and Unix sockets. The test tooling layout and command line also made framework code difficult to distinguish from the ATS test inventory. This patch adds fixture-owned ATS factories and ATS-aware curl requests, including UDS support, and converts representative basic coverage to the procedural API. It moves the harness and its unit tests under tests/tools/uranium, updates imports, and lets pytest own selection, parallelism, and collection options while the wrapper handles Docker execution.
The remaining compatibility tests still depended on AuTest process orchestration, which prevented the Uranium suite from being fully native and made parallel execution unreliable. This removes the compatibility backend and converts each test to direct Proxy Verifier replay metadata or a native pytest scenario. It also hardens process startup, logging, reload timestamps, DNS, and timeout handling so the complete suite runs reliably with eight workers. The test guide and runner documentation now describe the replay-first workflow and pytest-native selection and parallelism.
added 3 commits
August 17, 2026 10:42
The initial pytest migration left explicitly disabled scenarios as permanent skips and made native curl commands cumbersome to read and write. Repository guidance also retained obsolete AuTest conventions. This patch restores the opt-in scenarios behind a pytest manual marker and adds --run-manual for deliberate execution. It also changes Curl to parse one shell-style argument string, updates every call site, and documents parameter and timeout expectations. This adds collection and service regression coverage and refreshes the Uranium documentation to match the native pytest workflow.
Procedural Uranium support had accumulated unrelated process, client, and assertion behavior in one large module, making changes difficult to isolate while scenarios depended on its public import path. The converted log filename scenario could also read asynchronously written logs before every destination had flushed. This patch addresses this by moving focused implementations into a services package behind the existing tools.uranium.services facade and updating internal imports, documentation, and facade coverage. It also waits for each expected log record before asserting so parallel runs do not race ATS logging.
Replay processing crosses pytest collection, YAML validation, and runtime execution, but the framework directory did not explain those boundaries. Closely related modules were therefore difficult to distinguish. This patch addresses this by documenting the direct replay and procedural call flows, each module and supporting directory, and guidance for placing new framework code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft for the all-in end-to-end test migration discussed on the
original pytest replay PR. It supersedes #13035 without depending on changes to
the AuTest project.
Summary
.test.yaml.test.pydefinitions as one pytest item,using the pinned AuTest package only as a compatibility backend
sharding, isolated sandboxes/ports, and shared/exclusive serial-test locking
e2etest:e2etest.sh, CMake options andtargets, presets, documentation, environment variables, and the replay
e2etest:metadata keyEach replay file remains an independent item. A feature with several scenarios
should therefore keep separate
scenario.test.yamlfiles unless the sessionsintentionally share ATS state.
Developer workflow
Run one test from the source tree:
The same
-fworkflow selects native YAML or compatibility Python items andsupports globs and multiple names.
-k,-m,-j,--list, and other pytestselection options remain available. A configured tree generates
<build>/tests/e2etest.shfor the fastest warmed-up single-test path.Container behavior
The source entry point defaults to
ci.trafficserver.apache.org/ats/fedora:44, performs an incremental dedicatedbuild/install, and runs the selected tests there. It launches Docker with the
CI-relevant
--init, host networking, andSYS_PTRACEcapability, but does notuse
--privilegedor mount the Docker socket.Automatic direct execution requires both a container marker and exact Fedora
44 identification, so the existing Jenkins container does not attempt Docker
in Docker.
--run-in-dockerand--no-run-in-dockeroverride the decision.Container-local short
/tmpsandboxes avoid Unix-socket path and bind-mountproblems while regular diagnostics are copied back to the build tree.
CI compatibility
Temporary deprecated aliases retain
ci-fedora-autest,autest.sh, the oldCMake options/targets, and the Jenkins
parallel_modefeature probe. This letsthe current external pipeline execute the unified pytest inventory immediately;
the pipeline can rename its stage, preset, and command in a follow-up. Existing
SHARD/SHARDCNTvalues now shard all pytest items rather than only.test.pyfiles.
Validation
contains one unrelated untracked replay in addition to this PR's 626 items)
passed concurrently