Skip to content

chore: [WIP] experimental parallel kokoro system test #17608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
chalmerlowe wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

chore: [WIP] experimental parallel kokoro system test #17608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
692e1ff
feat: enable parallel system tests in Kokoro and trigger 4 packages
chalmerlowe Jul 1, 2026
461580e
refactor: refine gcloud isolation with local variable
chalmerlowe Jul 1, 2026
b8795e4
chore: add VM diagnostics and temporarily skip sqlalchemy-bigquery
chalmerlowe Jul 7, 2026
54f8a72
chore: revert sqlalchemy-bigquery trigger
chalmerlowe Jul 7, 2026
4206da2
chore: add 10 more packages to parallel test trigger
chalmerlowe Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 132 additions & 4 deletions .kokoro/system.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,20 @@ RETVAL=0

pwd

echo "=== KOKORO VM SCOUTING ==="
echo "CPU Count:"
nproc
echo "--------------------------"
echo "Detailed CPU Info:"
lscpu | grep -E "^(Model name|CPU\(s\)|Thread\(s\) per core|Core\(s\) per socket)"
echo "--------------------------"
echo "Memory Status:"
free -h
echo "--------------------------"
echo "Disk Usage:"
df -h /
echo "=========================="

run_package_test() {
local package_name=$1
local package_path="packages/${package_name}"
Expand All @@ -46,6 +60,10 @@ run_package_test() {
# Inherit NOX_SESSION from environment to allow configs (like prerelease.cfg) to pass it in
local NOX_SESSION="${NOX_SESSION}"

# ISOLATION: Create a unique gcloud config dir for this run
local gcloud_config_dir=$(mktemp -d -t "gcloud-config-${package_name}-XXXXXX")
local CLOUDSDK_CONFIG="${gcloud_config_dir}"
Comment on lines +64 to +65

@gemini-code-assist gemini-code-assist Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since set -e is active, if any command fails during the test execution, the function/subshell will exit immediately, and the manual cleanup at the end of the function will be bypassed, leaking the temporary directory.

Using a trap on EXIT ensures that the temporary directory is cleaned up regardless of how the subshell exits. We should use double quotes in the trap definition so that the local variable gcloud_config_dir is expanded immediately, as the local variable will be out of scope when the subshell actually exits.

Additionally, if package_name contains any slashes, mktemp will fail because the template cannot contain directory separators. Sanitizing the package name first prevents this issue.

Suggested change
local gcloud_config_dir=$(mktemp -d -t "gcloud-config-${package_name}-XXXXXX")
local CLOUDSDK_CONFIG="${gcloud_config_dir}"
local safe_package_name=$(echo "${package_name}" | tr '/' '_')
local gcloud_config_dir=$(mktemp -d -t "gcloud-config-${safe_package_name}-XXXXXX")
trap "rm -rf '${gcloud_config_dir}'" EXIT
local CLOUDSDK_CONFIG="${gcloud_config_dir}"


echo "------------------------------------------------------------"
echo "Configuring environment for: ${package_name}"
echo "------------------------------------------------------------"
Expand Down Expand Up @@ -80,7 +98,7 @@ run_package_test() {
esac

# Export variables for the duration of this function's sub-processes
export PROJECT_ID GOOGLE_APPLICATION_CREDENTIALS NOX_FILE NOX_SESSION
export PROJECT_ID GOOGLE_APPLICATION_CREDENTIALS NOX_FILE NOX_SESSION CLOUDSDK_CONFIG
export GOOGLE_CLOUD_PROJECT="${PROJECT_ID}"

gcloud auth activate-service-account --key-file="$GOOGLE_APPLICATION_CREDENTIALS"
Expand All @@ -94,12 +112,15 @@ run_package_test() {
set -e
popd > /dev/null

rm -rf "${gcloud_config_dir}"
return $res
Comment on lines +115 to 116

@gemini-code-assist gemini-code-assist Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

With the trap on EXIT configured at the start of the function, we can safely remove the manual rm -rf call here. We should also unregister the trap using trap - EXIT before returning to prevent it from running again if the shell continues.

Suggested change
rm -rf "${gcloud_config_dir}"
return $res
trap - EXIT
return $res

}

# A file for running system tests
system_test_script="${PROJECT_ROOT}/.kokoro/system-single.sh"

PACKAGES_TO_TEST=()

# Run system tests for each package with directory packages/*/tests/system
for path in `find 'packages' \
\( -type d -wholename 'packages/*/tests/system' \) -o \
Expand All @@ -113,6 +134,11 @@ for path in `find 'packages' \
package_name=${package_name%%/*}
package_path="packages/${package_name}"

if [[ "$package_name" == "sqlalchemy-bigquery" ]]; then
echo "Skipping sqlalchemy-bigquery for diagnostics."
continue
fi

# Determine if we should skip based on git diff
# We always check for changes in these specific versioning/config files
files_to_check=(
Expand Down Expand Up @@ -147,10 +173,112 @@ for path in `find 'packages' \
set -e

if [[ "${package_modified}" -gt 0 || "$KOKORO_BUILD_ARTIFACTS_SUBDIR" == *"continuous"* ]]; then
# Call the function - its internal exports won't affect the next loop
run_package_test "$package_name" || RETVAL=$?
PACKAGES_TO_TEST+=("$package_name")
else
echo "No changes in ${package_name} and not a continuous build, skipping."
fi
done
exit ${RETVAL}

# Parallel Execution Logic
MAX_JOBS=${MAX_JOBS:-4}
active_jobs=0
declare -A job_pids
declare -A job_pkgs
failed_packages=()
passed_packages=()
Comment on lines +182 to +188

@gemini-code-assist gemini-code-assist Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To avoid duplicating the complex process reaping logic, we can extract it into a helper function reap_finished_jobs. This improves maintainability and readability.

# Parallel Execution Logic
MAX_JOBS=${MAX_JOBS:-4}
active_jobs=0
declare -A job_pids
declare -A job_pkgs
failed_packages=()
passed_packages=()

reap_finished_jobs() {
  for pid in "${!job_pids[@]}"; do
    if ! kill -0 "$pid" 2>/dev/null; then
      wait "$pid" && status=0 || status=$?
      finished_pkg=${job_pkgs[$pid]}
      if [ "$status" -eq 0 ]; then
        echo "✔ [SUCCESS] ${finished_pkg}"
        passed_packages+=("$finished_pkg")
      else
        echo "✘ [FAILURE] ${finished_pkg} (Exit Code: ${status})"
        failed_packages+=("$finished_pkg")
      fi
      unset "job_pids[$pid]"
      unset "job_pkgs[$pid]"
      active_jobs=$((active_jobs - 1))
    fi
  done
}


# Temporary directory for clean log segregation
LOG_DIR=$(mktemp -d -t test-logs-XXXXXX)
# Clean up logs on exit
trap 'rm -rf "$LOG_DIR"' EXIT

if [ ${#PACKAGES_TO_TEST[@]} -eq 0 ]; then
echo "No packages to test."
exit 0
fi

echo "=================================================="
echo "Starting parallel test execution for ${#PACKAGES_TO_TEST[@]} packages"
echo "Concurrency limit: ${MAX_JOBS}"
echo "=================================================="

for pkg in "${PACKAGES_TO_TEST[@]}"; do
# Maintain concurrency limit
while [ "$active_jobs" -ge "$MAX_JOBS" ]; do
for pid in "${!job_pids[@]}"; do
if ! kill -0 "$pid" 2>/dev/null; then
wait "$pid" && status=0 || status=$?
finished_pkg=${job_pkgs[$pid]}
if [ "$status" -eq 0 ]; then
echo "✔ [SUCCESS] ${finished_pkg}"
passed_packages+=("$finished_pkg")
else
echo "✘ [FAILURE] ${finished_pkg} (Exit Code: ${status})"
failed_packages+=("$finished_pkg")
fi
unset "job_pids[$pid]"
unset "job_pkgs[$pid]"
active_jobs=$((active_jobs - 1))
fi
done
sleep 0.1
done
Comment on lines +207 to +225

@gemini-code-assist gemini-code-assist Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use the extracted reap_finished_jobs helper function here to simplify the concurrency limit loop.

Suggested change
while [ "$active_jobs" -ge "$MAX_JOBS" ]; do
for pid in "${!job_pids[@]}"; do
if ! kill -0 "$pid" 2>/dev/null; then
wait "$pid" && status=0 || status=$?
finished_pkg=${job_pkgs[$pid]}
if [ "$status" -eq 0 ]; then
echo "✔ [SUCCESS] ${finished_pkg}"
passed_packages+=("$finished_pkg")
else
echo "✘ [FAILURE] ${finished_pkg} (Exit Code: ${status})"
failed_packages+=("$finished_pkg")
fi
unset "job_pids[$pid]"
unset "job_pkgs[$pid]"
active_jobs=$((active_jobs - 1))
fi
done
sleep 0.1
done
while [ "$active_jobs" -ge "$MAX_JOBS" ]; do
reap_finished_jobs
sleep 0.1
done


safe_pkg_name=$(echo "$pkg" | tr '/' '_')
log_file="${LOG_DIR}/${safe_pkg_name}.log"

echo "Spawning tests for ${pkg}..."
run_package_test "$pkg" > "$log_file" 2>&1 &
pid=$!
job_pids["$pid"]=$pid
job_pkgs["$pid"]=$pkg
active_jobs=$((active_jobs + 1))
done

# Reap remaining processes
while [ "$active_jobs" -gt 0 ]; do
for pid in "${!job_pids[@]}"; do
if ! kill -0 "$pid" 2>/dev/null; then
wait "$pid" && status=0 || status=$?
finished_pkg=${job_pkgs[$pid]}
if [ "$status" -eq 0 ]; then
echo "✔ [SUCCESS] ${finished_pkg}"
passed_packages+=("$finished_pkg")
else
echo "✘ [FAILURE] ${finished_pkg} (Exit Code: ${status})"
failed_packages+=("$finished_pkg")
fi
unset "job_pids[$pid]"
unset "job_pkgs[$pid]"
active_jobs=$((active_jobs - 1))
fi
done
sleep 0.1
done
Comment on lines +239 to +257

@gemini-code-assist gemini-code-assist Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use the extracted reap_finished_jobs helper function here to simplify the final reaping loop.

Suggested change
while [ "$active_jobs" -gt 0 ]; do
for pid in "${!job_pids[@]}"; do
if ! kill -0 "$pid" 2>/dev/null; then
wait "$pid" && status=0 || status=$?
finished_pkg=${job_pkgs[$pid]}
if [ "$status" -eq 0 ]; then
echo "✔ [SUCCESS] ${finished_pkg}"
passed_packages+=("$finished_pkg")
else
echo "✘ [FAILURE] ${finished_pkg} (Exit Code: ${status})"
failed_packages+=("$finished_pkg")
fi
unset "job_pids[$pid]"
unset "job_pkgs[$pid]"
active_jobs=$((active_jobs - 1))
fi
done
sleep 0.1
done
while [ "$active_jobs" -gt 0 ]; do
reap_finished_jobs
sleep 0.1
done


echo ""
echo "=================================================="
echo " TEST RUN SUMMARY "
echo "=================================================="
echo "Total tested: ${#PACKAGES_TO_TEST[@]}"
echo "Passed: ${#passed_packages[@]}"
echo "Failed: ${#failed_packages[@]}"
echo "=================================================="

if [ ${#failed_packages[@]} -gt 0 ]; then
echo ""
echo "!!! DETAILED LOGS FOR FAILED PACKAGES !!!"
for pkg in "${failed_packages[@]}"; do
safe_pkg_name=$(echo "$pkg" | tr '/' '_')
log_file="${LOG_DIR}/${safe_pkg_name}.log"
echo "--------------------------------------------------"
echo "LOGS FOR: ${pkg}"
echo "--------------------------------------------------"
[ -f "$log_file" ] && cat "$log_file"
echo ""
done
exit 1
fi

echo "All tests passed successfully!"
exit 0
2 changes: 2 additions & 0 deletions packages/django-google-spanner/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Copyright 2020 Google LLC
# Trigger parallel system test run

#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
Expand Down
2 changes: 2 additions & 0 deletions packages/google-api-core/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Copyright 2018 Google LLC
# Trigger parallel system test run

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
4 changes: 2 additions & 2 deletions packages/google-auth/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Copyright 2014 Google Inc.
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,8 +16,7 @@
import io
import os

from setuptools import find_namespace_packages
from setuptools import setup
from setuptools import find_namespace_packages, setup

cryptography_base_require = [
"cryptography >= 38.0.3",
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-bigquery/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Copyright 2018 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-bigtable/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion packages/google-cloud-core/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Copyright 2018 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -17,7 +18,6 @@

import setuptools


# Package metadata.

name = "google-cloud-core"
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-datastore/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-error-reporting/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-firestore/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-logging/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
2 changes: 2 additions & 0 deletions packages/google-cloud-ndb/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Copyright 2018 Google LLC
# Trigger parallel system test run

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-pubsub/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
1 change: 1 addition & 0 deletions packages/google-cloud-storage/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2026 Google LLC
# Trigger parallel system test run
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down
Loading