Skip to content

Terraform finds the emulator on its Azure host, and the Windows CI builds with BuildKit - #57

Merged
paolosalvatori merged 2 commits into
mainfrom
fix-terraform-metadata-host-and-windows-buildx
Nov 19, 2026
Merged

paolosalvatori merged 2 commits into
mainfrom
fix-terraform-metadata-host-and-windows-buildx

Conversation

@paolosalvatori

Copy link
Copy Markdown
Collaborator

Motivation

The manual run of this repository's Samples CI on 2026-11-19 (every sample on Linux and Windows, run 44497987605) failed 13 of its 45 jobs against localstack/localstack-azure:latest, for two unrelated reasons.

  • The Terraform cluster jobs, on Linux and Windows. localstack/localstack-pro pull request 10527 makes the emulator serve Azure, its metadata endpoint included, only on azure.localhost.localstack.cloud and on the real Azure hosts. terraform/deploy.sh hardcoded localhost.localstack.cloud:4566 as the metadata host of azurerm, so terraform plan failed with retrieving metadata from endpoint "https://localhost.localstack.cloud:4566": performing request: expected 200 OK but got 404. The scripts and the Bicep deployment were not affected, because the Azure CLI already targets https://azure.localhost.localstack.cloud:4566/.
  • Every Windows Python job. 04-push-docker-image.sh failed with NotFound: content digest sha256:37fb86c2a767...: not found. Since 2026-11-16 the emulator image is built on the same python:3.13-slim as the Python samples, but it holds the Debian layer of that image under another digest. The Windows job built with the legacy builder, because the docker CLI of the Windows runner has no buildx plugin. With the containerd image store of the Docker Engine 29 the job runs in WSL, that builder does not download a base layer whose content another image already unpacked, so docker push cannot find it. BuildKit downloads every layer of the image it stores.

Fixes SMF-1179 and SMF-1180

Changes

  • terraform/deploy.sh and tutorials/terraform/tags-labels-taints/deploy.sh: on the emulator, the metadata host of azurerm is the host and port of the Resource Manager endpoint the Azure CLI targets (az cloud show --query endpoints.resourceManager), azure.localhost.localstack.cloud:4566 with lstk, instead of a hardcoded localhost.localstack.cloud:4566. Nothing changes on Azure, where the scripts pass no metadata host.
  • terraform/README.md, both providers.tf files and the tutorial's variables.tf: the comments and descriptions name the new host.
  • .github/workflows/run-samples.yml: the Windows job installs the buildx plugin, v0.38.2 pinned with its SHA-256 like yq and sqlcmd, so docker build uses BuildKit, as on Docker Desktop and on the Linux runner.
  • README.md, Windows section: build the images with BuildKit, and what content digest ...: not found means.

Tests

  • The push failure, reproduced and fixed in a throwaway docker:29.1.3-dind (Docker 29.1.3 with the containerd image store, as in the Windows job), building FROM python:3.13-slim plus one RUN and pushing to a local registry:3:
    • After docker pull localstack/localstack-azure:latest, with the legacy builder: the base image pull downloads no layer, and the push fails with the CI's NotFound: content digest sha256:37fb86c2a767...: not found.
    • The same with BuildKit: the build downloads 37fb86c2a767, and the push succeeds.
    • Without the emulator image, with the legacy builder: the push succeeds.
    • A BuildKit rebuild after the failed legacy push: the push succeeds, which is the recovery the README describes.
  • Terraform, against localstack/localstack-azure:latest sha256:1c0fd9ad... (the image of the failed run), with the steps of the Linux job: terraform/deploy.sh, tests/test-diagnostic-settings.sh, the web-app-managed-identity Python scripts 01, 02, 04 and 05, and tests/test-vacation-planner.sh all passed. On the same emulator, terraform plan of the tags-labels-taints tutorial planned its 16 resources with the derived host, and failed with the CI's 404 with the old one.
  • CI: this pull request's run covers every Linux job. Run 44513075583, started manually on this branch with platform Windows, covers the 22 Windows jobs: in progress, the result follows in a comment.
  • bash -n, pre-commit run (zizmor, gitleaks) and actionlint 1.7.7 (with shellcheck on the run blocks) pass. shellcheck reports the same six notes in terraform/deploy.sh as on main.
  • Not run on Azure: on Azure the scripts pass no metadata host, so that path is unchanged.

Related

  • localstack/localstack-pro pull request 10527 (ENG-1266): the host-only routing.
  • SMF-1178: the published LocalStack images compress the Debian layer of their python base image a second time, which is why the emulator image holds it under another digest. Fixing the publishing pipeline would remove the trigger for every user of the legacy builder, but this repository no longer depends on it.
  • localstack-samples/localstack-azure-samples pull request 154 fixes the same metadata host in its api-management-function-app sample.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FvNuJMQFHqFMvJqLV82s3h

paolosalvatori and others added 2 commits November 19, 2026 12:13
Since localstack-pro #10527 the emulator serves Azure, its metadata endpoint
included, only on azure.localhost.localstack.cloud and the real Azure hosts,
so the azurerm provider got a 404 from the localhost.localstack.cloud:4566
metadata host that terraform/deploy.sh and the tags-labels-taints tutorial
hardcoded, and the Terraform cluster jobs failed at terraform plan. Both
scripts now take the host and port of the Resource Manager endpoint the
Azure CLI targets, azure.localhost.localstack.cloud:4566 with lstk, so
Terraform always reaches the emulator where the Azure CLI does.

Fixes SMF-1179

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FvNuJMQFHqFMvJqLV82s3h
The docker CLI of the Windows runner image has no buildx plugin, so the
samples' docker build fell back to the legacy builder on the Docker Engine
29 in WSL, whose containerd image store skipped downloading the Debian
layer of python:3.13-slim that the emulator image holds under another
digest. docker push then failed with "content digest ...: not found" in
every Windows Python job. The job now installs the buildx plugin, pinned
with its SHA-256, so it builds with BuildKit like Docker Desktop and the
Linux runner, and the README tells Windows users to do the same.

Fixes SMF-1180

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FvNuJMQFHqFMvJqLV82s3h
Copilot AI balanced review requested due to automatic review settings November 19, 2026 11:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Terraform emulator metadata and Windows CI BuildKit changes.

No open issues 🎉

What changed in this PR

This PR addresses Terraform metadata discovery failures against the Azure emulator and Windows CI image-push failures caused by the legacy Docker builder.

Changes:

  • Derive Terraform’s emulator metadata host from the Azure CLI Resource Manager endpoint, leaving real Azure behavior unchanged.
  • Install a version- and checksum-pinned buildx plugin in Windows CI to enable BuildKit.
  • Update Terraform configuration descriptions and Windows troubleshooting guidance.
File Description
tutorials/​terraform/​tags-labels-taints/​variables.tf Update the metadata-host description to name the Azure emulator host.
tutorials/​terraform/​tags-labels-taints/​providers.tf Correct the emulator host in provider comments.
tutorials/​terraform/​tags-labels-taints/​deploy.sh Derive the metadata host from the active Resource Manager endpoint.
terraform/​README.md Document how the emulator metadata host is selected.
terraform/​providers.tf Correct the emulator host in provider comments.
terraform/​deploy.sh Derive the metadata host from the active Resource Manager endpoint.
README.md Explain BuildKit setup and recovery from missing-layer push failures on Windows.
.github/​workflows/​run-samples.yml Install and verify the pinned Windows buildx plugin.

Review effort: Balanced

@paolosalvatori

paolosalvatori commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Windows results, from run 44513075583 on this branch (started manually, platform Windows, every sample): all 22 Windows jobs passed. They include every job that failed in run 44497987605:

  • All 11 Python jobs. For example, in web-app-managed-identity (python) the job installs buildx v0.38.2 (docker-buildx.exe: OK), BuildKit downloads 37fb86c2a767 (29.86 MB) during the build, the image index with its provenance attestation is pushed to the emulated ACR, and the web test passes.
  • Terraform cluster: web-app-managed-identity (python).

With this pull request's Linux run (23 of 23 jobs), every job that failed on 2026-11-19 now passes.

@paolosalvatori
paolosalvatori merged commit 664d162 into main Nov 19, 2026
49 checks passed
@paolosalvatori
paolosalvatori deleted the fix-terraform-metadata-host-and-windows-buildx branch November 20, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants