Skip to content

Add scheduled full test suite runs every 6 hours#5045

Open
kenyonj wants to merge 1 commit intomainfrom
schedule-full-test-suite
Open

Add scheduled full test suite runs every 6 hours#5045
kenyonj wants to merge 1 commit intomainfrom
schedule-full-test-suite

Conversation

@kenyonj
Copy link
Contributor

@kenyonj kenyonj commented Feb 13, 2026

Summary

  • Add schedule trigger to run the full test suite every 6 hours
  • Scheduled and workflow_dispatch runs execute all test types regardless of file changes
  • PR and merge_group events continue using incremental diff-based testing

Changes

  • .github/workflows/test.yml:
    • Add schedule trigger with cron: "0 */6 * * *"
    • Add run_type step to detect scheduled/manual runs
    • Skip git diff steps during scheduled runs (no PR context)
    • Run all test types unconditionally during scheduled runs

Why

This ensures collection rename issues are caught regularly (every 6 hours) without blocking every PR with expensive API checks. Combined with the hourly collections-renames workflow, this provides comprehensive coverage while keeping PR CI fast.

Add a schedule trigger (every 6 hours) to ensure the full test suite
runs regularly even when there are no PRs. This complements the
incremental PR-based testing by providing a safety net for catching
collection renames and other issues.

Scheduled and workflow_dispatch runs execute all test types regardless
of file changes, while PR and merge_group events continue to use the
incremental diff-based approach.
Copilot AI review requested due to automatic review settings February 13, 2026 20:44
@kenyonj kenyonj requested a review from a team as a code owner February 13, 2026 20:44
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds scheduled full test suite runs every 6 hours to complement the existing hourly collection-renames workflow, ensuring comprehensive testing coverage while keeping PR-based CI fast. The implementation uses GitHub Actions' schedule trigger with a cron expression and introduces logic to differentiate between incremental (PR/merge) and full (scheduled/manual) test runs.

Changes:

  • Add schedule trigger with cron: "0 */6 * * *" to run tests every 6 hours
  • Introduce run type detection that treats schedule and workflow_dispatch events as full runs
  • Modify git diff steps to skip during full runs (since there's no PR context)
  • Update Ruby setup and test execution conditionals to always run during full runs

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 62 to +65
if: |
(matrix.test_type == 'topics' && steps.topics.outputs.changed) ||
(matrix.test_type == 'collections' && steps.collections.outputs.changed) ||
(matrix.test_type == 'all' && steps.all.outputs.changed)
(matrix.test_type == 'topics' && (steps.topics.outputs.changed || steps.run_type.outputs.full_run == 'true')) ||
(matrix.test_type == 'collections' && (steps.collections.outputs.changed || steps.run_type.outputs.full_run == 'true')) ||
(matrix.test_type == 'all' && (steps.all.outputs.changed || steps.run_type.outputs.full_run == 'true'))
Copy link

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current implementation relies on a subtle Ruby behavior that may not be immediately obvious. When git diff steps are skipped during scheduled runs, the environment variables (TOPIC_FILES, COLLECTION_FILES, TEST_ALL_FILES) are set to empty strings. The test helpers check if ENV.fetch("TEST_ALL_FILES", false), and since empty strings are truthy in Ruby, this causes the wildcard pattern to be returned correctly.

However, this behavior is fragile and unclear:

  1. It relies on empty strings being truthy in Ruby (only nil and false are falsy)
  2. It depends on GitHub Actions evaluating undefined step outputs as empty strings
  3. The intended behavior is not explicit in the workflow

Consider making this more explicit and robust by either:

  1. Setting TEST_ALL_FILES to a clear sentinel value like "true" or "1" when full_run is true
  2. Adding a comment explaining this intentional behavior
  3. Using a dedicated full_run environment variable that the test helpers check explicitly

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant