Add Azure VM and AKS integration test jobs to the test-artifacts workflow - #2216
Open
movence wants to merge 13 commits into
Open
Add Azure VM and AKS integration test jobs to the test-artifacts workflow#2216movence wants to merge 13 commits into
movence wants to merge 13 commits into
Conversation
movence
added
the
ready for testing
Inline AzureVMIntegrationTest job that provisions an Azure VM, installs the pre-built agent .deb via SSH, and runs the Go integration test from the test repo's terraform/azure/vm directory.
Transaction Search (CloudWatchLogs trace segment destination) is required for the OTLP endpoint but breaks legacy PutTraceSegments-based tests in us-west-2. Since the setting is per-region, isolate the Azure test by running it in us-east-2 where Transaction Search is enabled, and revert us-west-2 to the XRay destination so existing xray_test/otlp_test jobs pass as before.
Adds AKS-default-otel job that provisions an AKS cluster, deploys the CWA DaemonSet from the pre-built ECR image, generates OTLP load via a hostNetwork Job, and validates delivery via CloudWatch APIs.
The AKS job referenced a setup-go commit SHA that does not exist, causing the job to fail at action resolution. Use the same v4.3.0 pin as the other jobs in this workflow.
The pull secret is generated inside terraform via the aws_ecr_authorization_token data source, so the step output and the terraform variable are no longer consumed.
- AzureVM job now only fires when the filter matches ./test/azure/vm or the broader ./test/azure (same pattern as the AKS job). - Pass github_test_repo and github_test_repo_branch to terraform destroy for both AzureVM and AKS jobs so state files resolve correctly during teardown.
movence
force-pushed
the
feature/azure-integration-test-workflow
branch
from
July 24, 2026 15:37
69c92b8 to
e79c5fe
Compare
- AKS: fail-fast ECR image pre-check before ~60-min cluster provision - Harden runner-IP fetch: https/checkip.amazonaws.com, -f/--retry/--max-time, IPv4 validation - Move terraform var interpolations into env: blocks (no shell injection surface) - Add discrete Get Runner IP step; reuse cached output in AKS apply/destroy - Align AKS Go to ~1.26.4, job key to AKSIntegrationTest, if: always() style
jefchien
reviewed
Aug 3, 2026
Comment on lines
+2028
to
+2037
| -var="region=us-east-2" \ | ||
| -var="runner_ip=$RUNNER_IP" \ | ||
| -var="cwa_github_sha=$BUILD_ID" \ | ||
| -var="agent_deb_path=$AGENT_DEB_PATH" \ | ||
| -var="github_test_repo=$TEST_REPO_URL" \ | ||
| -var="github_test_repo_branch=$TEST_REPO_BRANCH" \ | ||
| -var="azure_resource_group=$AZURE_RESOURCE_GROUP" \ | ||
| -var="azure_vnet_name=$AZURE_VNET_NAME" \ | ||
| -var="azure_oidc_provider_arn=$AZURE_OIDC_PROVIDER_ARN" \ | ||
| -var="azure_token_audience=$AZURE_TOKEN_AUDIENCE" || true |
Contributor
There was a problem hiding this comment.
nit: Do we need to pass in all of the variables? Doesn't the terraform state already have them? None of the other tests seem to need this. Without them, we wouldn't need most of the env vars (except for the ARM_* ones).
Comment on lines
+2005
to
+2007
| - name: Fail if apply failed | ||
| if: steps.terraform_apply.outcome == 'failure' | ||
| run: exit 1 |
Contributor
There was a problem hiding this comment.
Do we need to have
- name: "[WIP] ✅ Passed"
if: steps.terraform_apply.outcome == 'success' && matrix.arrays.wip
run: echo "::notice::WIP test passed"
- name: "[WIP] ⚠️ Failed (Overruled)"
if: steps.terraform_apply.outcome == 'failure' && matrix.arrays.wip
run: |
echo "::warning::WIP test failed but allowed to continue"
echo "### ⚠️ WIP Test Failed (Overruled)" >> $GITHUB_STEP_SUMMARY
- name: Fail if not WIP
if: steps.terraform_apply.outcome == 'failure' && !matrix.arrays.wip
run: exit 1
to match the other tests?
| AZURE_VNET_NAME: ${{ vars.AZURE_VNET_NAME }} | ||
| AZURE_OIDC_PROVIDER_ARN: ${{ vars.AZURE_OIDC_PROVIDER_ARN }} | ||
| AZURE_TOKEN_AUDIENCE: ${{ vars.AZURE_TOKEN_AUDIENCE }} | ||
| run: | |
Contributor
There was a problem hiding this comment.
Should we reuse nick-fields/retry to match the other destroy steps?
- name: Terraform destroy
if: always()
uses: nick-fields/retry@14672906e672a08bd6eeb15720e9ed3ce869cdd4 # v2.9.0
env:
MATRIX_TERRAFORM_DIR: ${{ matrix.arrays.terraform_dir }}
with:
max_attempts: 3
timeout_minutes: 8
retry_wait_seconds: 5
command: |
if [ -n "$MATRIX_TERRAFORM_DIR" ]; then
cd "$MATRIX_TERRAFORM_DIR"
else
cd terraform/eks/addon/gpu
fi
terraform destroy -auto-approve
The AKS image pre-check ran without an explicit region, so it inherited the job's us-east-2 credentials while the build only publishes to us-west-2. It would have failed every run; it had not executed yet, since the last AKS run predates it. Both destroy steps now use nick-fields/retry like the other suites, which also drops the `|| true` that was hiding teardown failures. The VM destroy no longer passes cwa_github_sha, agent_deb_path or the test-repo vars: those feed provisioners, which do not run on destroy. The rest stay because runner_ip has no default and the vnet, subnet and OIDC-provider data sources are read during the destroy plan. Both suites are marked work-in-progress so failures report without blocking CI, matching the per-test wip flag the generator sets. These jobs are not matrix driven, so the flag is declared on the job rather than in the test matrix. Teardown failures still fail the job, as they do for the other suites.
movence
force-pushed
the
feature/azure-integration-test-workflow
branch
from
August 4, 2026 14:24
ec720d8 to
fad384f
Compare
The review comment asked for the three-step wip block the other suites use, not for these suites to be marked work-in-progress. An earlier attempt read it the other way and marked them WIP, which the run history does not support: Azure VM is 5/5 and AKS 4/5, and that one AKS failure was a terraform ordering bug in the test repo rather than flakiness. The jobs now carry a single-entry matrix with wip false, so `Fail if not WIP` is the branch that fires and the suites still gate merges, with no [WIP] label. The expressions match the other jobs exactly, and the name prefix is derived from the flag instead of hardcoded, so there is one value to flip if Azure provisioning does turn flaky later.
jefchien
approved these changes
Aug 5, 2026
|
|
||
| - name: Login ECR | ||
| id: login-ecr | ||
| uses: aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076 # v2.0.1 |
Contributor
There was a problem hiding this comment.
nit: All other workflows use a different pinned version
Paamicky
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
No CI coverage existed for the agent running on non-AWS hosts (Azure VM / AKS) with the default OTel config. The test suites live in the test repo (companion PR: aws/amazon-cloudwatch-agent-test#730) but need workflow jobs to provision and run them.
Description of changes
Adds two jobs to
test-artifacts.yml(workflow-only change):AzureVM-default-otel— terraform-provisions an Azure VM, installs the built.deb, and runs the on-VM Go test suite (test/azure/vm).AKS-default-otel— terraform-provisions an AKS cluster, deploys the agent DaemonSet from the integration-test ECR image, and runs the runner-side Go test suite (test/azure/aks).Both jobs gate on
test_dir_filtercontaining their test path (a./test/azurefilter runs both), authenticate to Azure via OIDC, and always runterraform destroyon teardown. TheLogin ECRstep's registry output feeds the image repo; the AKS pull-secret token is minted inside terraform, so no docker config JSON passes through shell interpolation.Latest revision (
8a016dd3)Workflow-safety hardening on top of the green run below; no test logic changed.
aws ecr describe-images) before the ~60-min cluster provision, mirroring the VM job's.debcheck.https://checkip.amazonaws.comwith-f --retry --max-time+ IPv4 validation (was uncheckedcurl ifconfig.me).-varinterpolations into stepenv:blocks (removes the shell-injection surface).Get Runner IPstep reused by apply + destroy, Go pinned~1.26.4, job key renamed toAKSIntegrationTest.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Latest green run: 30584238599 — validated at
f039ce4cagainst test-repo38d1c2a5, first attempt. Both jobs passed:build_idset to the branch HEAD, so the ECR image under test is built from that exact workflow state.Worth noting
failureeven when every created job succeeds — pre-existing onmain, where ~12 jobs lack the!= '[]'matrix guard the ITAR/CN jobs use. Adding that guard would fix the badge; can be a follow-up.CreateManifest(which needsBuildMSI-2022) to have published the multi-arch tag; the new ECR pre-check now fails fast with a clear error instead of a lateErrImagePull— re-run the failed build jobs rather than debugging AKS.AZURE_CLIENT_ID/AZURE_CLIENT_SECRET/AZURE_TENANT_ID/AZURE_SUBSCRIPTION_IDsecrets, andAZURE_RESOURCE_GROUP/AZURE_VNET_NAME/AZURE_OIDC_PROVIDER_ARN/AZURE_TOKEN_AUDIENCEvariables.