tests: parallel batching for packages when running unit tests

[parthea]

A legacy bottleneck in the unit test presubmit causes a timeout/crash after 4.5 hours (log) when testing all packages. This is blocking PR #17585 (Python 3.15 testing for all packages).

This PR introduces dynamic parallel execution, dropping runtime from several hours to under 19 minutes.

Key Changes

• Dynamic Matrix Parallelism (unittest.yml): Replaced the sequential test loop with a dynamic GitHub Actions matrix. Parallel coverage artifacts are named with unique indices (.coverage-${{ matrix.python }}-${{ matrix.batch-index }}) to prevent overwrites.

• Deterministic Isolation & Chunking (ci/get_batches.py): Added a centralized Python script to partition packages into matrix lanes:

• Guaranteed Isolation: Heavy packages (google-cloud-compute, google-cloud-compute-v1beta, google-cloud-dialogflow, google-cloud-dialogflow-cx, google-cloud-retail) bypass the grouping logic and automatically get their own dedicated runners.

• Standard Chunking: All other packages are chunked evenly into batches of 10.

• Targeted Execution (ci/run_conditional_tests.sh): Updated the script to accept explicit package lists passed directly from the matrix slice.

• Footprint Verification: Added a final validation job unittest-runtime-result to verify that all expected matrix lanes reported successful outcomes, protecting against silent runner drops.

Performance Impact
Before: ~4.5 hours (Timeout)

After: < 19 minutes

[gemini-code-assist]

Code Review

This pull request updates ci/run_conditional_tests.sh to allow passing specific package paths as optional arguments to run tests only on those directories, defaulting to the entire packages directory if no arguments are provided. The review feedback highlights two key improvements: first, making the check for the "packages" directory robust against trailing slashes (e.g., "packages/") by stripping them before comparison; second, replacing the fragile parsing of ls with safer, more idiomatic Bash globbing.

[parthea]

/gemini review

[gemini-code-assist]

Code Review

This pull request updates ci/run_conditional_tests.sh to allow passing specific package paths as optional arguments to restrict testing to those directories, falling back to the default behavior of scanning the packages directory if no arguments are provided. The review feedback identifies two robustness issues: first, handling non-existent directory arguments by validating their existence and exiting early; second, preventing issues with unmatched glob patterns when a directory is empty by verifying that each resolved path actually exists before processing.

[parthea]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a batching mechanism for CI testing via a new Python script, updates the conditional test runner script to accept specific package paths, and lowers the test coverage threshold for the Django Spanner package. The review feedback highlights a critical bug in the batch slicing logic that can cause packages to run in multiple batches, recommends reverting the coverage threshold reduction, and suggests quoting array expansions in the shell script to prevent word-splitting issues.