CI/CD: Add performance regression tests using QE infrastructure #798

juliannguyen4 · 2025-06-25T20:11:48Z

TODO

tail test container in gha
rm rclone credentials if not used

.github/workflows/tests.yml

codecov-commenter · 2025-06-25T20:37:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.00%. Comparing base (cd0e783) to head (a22ca22).
⚠️ Report is 2 commits behind head on dev.

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #798   +/-   ##
=======================================
  Coverage   82.00%   82.00%           
=======================================
  Files          99       99           
  Lines       14698    14698           
=======================================
  Hits        12053    12053           
  Misses       2645     2645

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

.github/workflows/tests.yml

…in permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

…aces, so rm them

.github/workflows/run-qe-test.yml

…ed the test with testctl version def256f

.github/workflows/run-qe-test.yml

+        repository: citrusleaf/qe-tools
+        token: ${{ secrets.CLIENT_BOT_PAT }}
+        path: qe-tools
+        sparse-checkout: |
+          bin/test-enqueue
+        sparse-checkout-cone-mode: false
+
+    - name: Add test-enqueue to PATH
+      run: echo "$(realpath qe-tools/bin)" >> $GITHUB_PATH
+
+    - uses: jfrog/setup-jfrog-cli@v4
+      env:
+        JF_URL: ${{ secrets.JFROG_PLATFORM_URL }}
+        JF_ACCESS_TOKEN: ${{ secrets.JFROG_ACCESS_TOKEN }}
+
+    - run: jf rt download qe-go-dev-local/testctl.amd64.linux qe-tools/bin/testctl
+    - run: chmod u+x ./testctl
+      working-directory: qe-tools/bin
+
+    - uses: actions/checkout@v4
+      with:
+        path: aerospike-client-python
+        sparse-checkout: |
+          test/${{ env.TEST_CONFIG_FILE_NAME }}
+        sparse-checkout-cone-mode: false
+
+    - name: Add AWS credentials
+      run: |
+        mkdir -p ~/.aws
+        cd ~/.aws
+        sections=("default" "qe")
+        for section in ${sections[@]};
+        do
+          cat <<-EOF >> credentials
+          [$section]
+          aws_access_key_id = ${{ secrets.QE_TEST_ENQUEUE_AWS_ACCESS_KEY_ID }}
+          aws_secret_access_key = ${{ secrets.QE_TEST_ENQUEUE_AWS_SECRET_ACCESS_KEY }}
+          region = us-west-1
+        	EOF
+        done
+      shell: bash
+
+    # - run: testctl --version
+
+    # test-enqueue hides testctl's output, so this can be helpful for debugging
+    # - run: testctl --config staging run 901b1178-0159-452d-9b36-64d455e568f2 --deployment perf_client
+    #   working-directory: aerospike-client-python/test/
+
+    - name: Allows us to get the exact test run id that was created by test-enqueue.
+      run: echo TESTCTL_USER=$(uuidgen) >> $GITHUB_ENV
+
+    - run: yq -i '.definitions.std.[0] = "${{ inputs.test-scenario }}"' ${{ env.TEST_CONFIG_FILE_NAME }}
+      working-directory: aerospike-client-python/test/
+
+    - name: Enqueue test run and fail fast if unable to enqueue
+      run: |
+        enqueue_count=$(test-enqueue --config staging --deployment perf_client ${{ env.TEST_CONFIG_FILE_NAME }} | grep "COUNT:" | sed 's/COUNT\://' | xargs)
+        if [[ "$enqueue_count" != "1" ]]; then
+          echo "We expected 1 test run to be enqueued, but $enqueue_count was actually enqueued."
+          exit 1
+        fi
+      working-directory: aerospike-client-python/test/
+
+    - name: Wait for test run to finish
+      id: poll-run-id
+      run: |
+        while true; do
+          # Color messes up the run_id's string
+          # Use xargs to trim whitespace from testctl's output
+          run_id=$(${{ env.TESTCTL_BINARY_NAME }} --config staging ps --no-color --user ${{ env.TESTCTL_USER }} -n 1 -status completed --columns RUN_ID | sed '1d' | xargs)
+          if [[ -z "$run_id" ]]; then
+            echo "Test run has not finished yet..."
+            sleep 2
+          else
+            echo "Test run $run_id has finished."
+            break
+          fi
+        done
+        echo "run_id=$run_id" >> "$GITHUB_OUTPUT"
+
+    # Downloading builds doesn't work here, so we disable it
+    - run: ${{ env.TESTCTL_BINARY_NAME }} --config staging download --skip-build-download ${{ steps.poll-run-id.outputs.run_id }}
+
+    - name: Print test run logs
+      run: |
+        cd ${{ steps.poll-run-id.outputs.run_id }}*/work/test_0
+        echo "stdout:"
+        cat stdout
+        echo "stderr:"
+        cat stderr
+
+    - name: Set Github job status
+      run: |
+        run_status=$(${{ env.TESTCTL_BINARY_NAME }} --config staging ps --no-color --user ${{ env.TESTCTL_USER }} -n 1 -status completed --columns STATUS | sed '1d' | xargs)
+        echo $run_status
+        if [[ "$run_status" == "Failure" ]]; then
+          exit 1
+        else
+          exit 0
+        fi


To fix the problem, add a permissions block to the workflow file to explicitly set the minimum required permissions for the GITHUB_TOKEN. Since the workflow does not appear to require any write access to repository contents, the safest default is to set permissions: contents: read at the top level of the workflow. This will apply to all jobs unless overridden. The change should be made near the top of the file, after the name: (if present) or after the on: block, and before the jobs: block. No additional imports or definitions are required.

…ssion-tests-using-asmikas-qe-env

This reverts commit 3eec83a.

This reverts commit 919d0f0.

finish step 1

3990aec

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

.github/workflows/tests.yml Fixed Show fixed Hide fixed

juliannguyen4 added 2 commits June 25, 2025 13:13

fix

8874083

do this way

cb602a9

2nd repo probably needs to be cd'd into

af6406e

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

.github/workflows/tests.yml Fixed Show fixed Hide fixed

juliannguyen4 added 16 commits June 25, 2025 14:31

forgot yaml

1a0bc99

install testctl

48b3360

fix

c221ab7

use jf cli instead

ad02e19

+x

89a502a

chmod inside new shell

c383595

.local/bin doesnt exist in PATH?

47f93c9

just install in default loc

97d75f1

just install in qe-tools/bin

46b25fc

debug

e1a8f44

action only supports env, but we need the cred file

7b3950a

fix

7c34a09

debug

01cb334

show version

41165de

add qe section

5b14066

print out test runs

276537a

github-advanced-security bot found potential problems Jun 26, 2025

View reviewed changes

.github/workflows/tests.yml Fixed Show fixed Hide fixed

juliannguyen4 and others added 6 commits June 27, 2025 14:26

Potential fix for code scanning alert no. 76: Workflow does not conta…

1518ff3

…in permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

try running custom test now

07c701a

use test definition registered in bob

a2060f1

use env var to always get the right test run id

a10524a

poll test run, then download and print test results

6b8cab4

print cmds for debugging

a661af2

juliannguyen4 added 11 commits July 1, 2025 19:47

use heredoc to add aws credentials

173d5fd

heredoc contents maybe needs to be tabbed?

682a668

leading tabs since this is in a for loop

891afb3

rm leading space up to first EOF. <<- will not remove leading whitesp…

a0ac884

…aces, so rm them

rm leading space for 2nd EOF

92dc1c4

print debug

85bfa06

fix

aeca494

hide debugging

b361c5e

add workflow to enqueue qe tests

4b208c0

rm automatic trigger

044416e

fix order

0f0e9b5

github-advanced-security bot found potential problems Jul 2, 2025

View reviewed changes

.github/workflows/run-qe-test.yml Fixed Show fixed Hide fixed

juliannguyen4 added 4 commits July 2, 2025 09:40

fix

1a0d69d

print stderr as well

38c6df3

use echo, set -x does not print in the right order

2278a1a

use rclone to fetch binaries from google drive

d494d8e

github-advanced-security bot found potential problems Jul 16, 2025

View reviewed changes

.github/workflows/run-qe-test.yml Fixed Show fixed Hide fixed

juliannguyen4 added 2 commits July 16, 2025 09:14

wrong syntax

51a80df

fix incorrect names for binaries

0983843

github-advanced-security bot found potential problems Jul 16, 2025

View reviewed changes

.github/workflows/run-qe-test.yml Fixed Show fixed Hide fixed

juliannguyen4 added 3 commits July 16, 2025 09:17

debug print

d7e1136

use the prev version of testctl and test-enqueue. we already register…

c498a5f

…ed the test with testctl version def256f

fix

63c6c3e

github-advanced-security bot found potential problems Jul 16, 2025

View reviewed changes

juliannguyen4 added 6 commits July 25, 2025 10:47

Merge remote-tracking branch 'origin/dev' into cicd-performance-regre…

bcbe93a

…ssion-tests-using-asmikas-qe-env

update redirected links. double timeout delay

3eec83a

Revert "update redirected links. double timeout delay"

9a84c25

This reverts commit 3eec83a.

replace redirected link

919d0f0

Revert "replace redirected link"

a22ca22

This reverts commit 919d0f0.

tail logs

9c10e6c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI/CD: Add performance regression tests using QE infrastructure #798

CI/CD: Add performance regression tests using QE infrastructure #798

Uh oh!

juliannguyen4 commented Jun 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

codecov-commenter commented Jun 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Uh oh!

@@ -8,2 +8,5 @@
+            permissions:
+              contents: read
             jobs:

CI/CD: Add performance regression tests using QE infrastructure #798

Are you sure you want to change the base?

CI/CD: Add performance regression tests using QE infrastructure #798

Uh oh!

Conversation

juliannguyen4 commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Uh oh!

juliannguyen4 commented Jun 25, 2025 •

edited

Loading

codecov-commenter commented Jun 25, 2025 •

edited

Loading