Terraform goes to the network when all information is already available locally - `terraform init` with lock file and broken `network_mirror` and providers already installed / available in provider cache directory

[fredden]

Terraform Version

% docker run --rm -it hashicorp/terraform:latest version
Terraform v1.13.1
on linux_amd64

% docker run --rm -it hashicorp/terraform:1.13 version
Terraform v1.13.1
on linux_amd64

Terraform Configuration Files

terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.7"
    }
  }
}

Debug Output

/demo # TF_LOG=trace terraform init
2025-09-03T14:24:58.102Z [INFO]  Terraform version: 1.13.1
2025-09-03T14:24:58.102Z [DEBUG] using github.com/hashicorp/go-tfe v1.74.1
2025-09-03T14:24:58.102Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.24.0
2025-09-03T14:24:58.102Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2025-09-03T14:24:58.102Z [DEBUG] using github.com/zclconf/go-cty v1.16.3
2025-09-03T14:24:58.102Z [INFO]  Go runtime version: go1.24.5
2025-09-03T14:24:58.102Z [INFO]  CLI args: []string{"terraform", "init"}
2025-09-03T14:24:58.102Z [TRACE] Stdout is a terminal of width 237
2025-09-03T14:24:58.102Z [TRACE] Stderr is a terminal of width 237
2025-09-03T14:24:58.102Z [TRACE] Stdin is a terminal
2025-09-03T14:24:58.102Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2025-09-03T14:24:58.102Z [INFO]  Loading CLI configuration from /root/.terraformrc
2025-09-03T14:24:58.103Z [DEBUG] Explicit provider installation configuration is set
2025-09-03T14:24:58.103Z [TRACE] Selected provider installation method cliconfig.ProviderInstallationNetworkMirror("https://broken.example.com/") with includes [] and excludes []
2025-09-03T14:24:58.103Z [INFO]  CLI command args: []string{"init"}
Initializing the backend...
2025-09-03T14:24:58.103Z [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2025-09-03T14:24:58.103Z [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2025-09-03T14:24:58.103Z [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2025-09-03T14:24:58.103Z [TRACE] Meta.Backend: instantiated backend of type <nil>
2025-09-03T14:24:58.103Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2025-09-03T14:24:58.104Z [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/random v3.7.2 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/random/3.7.2/linux_amd64
2025-09-03T14:24:58.104Z [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/random/3.7.2/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/random 3.7.2
2025-09-03T14:24:58.144Z [DEBUG] checking for provisioner in "."
2025-09-03T14:24:58.145Z [DEBUG] checking for provisioner in "/bin"
2025-09-03T14:24:58.145Z [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2025-09-03T14:24:58.145Z [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2025-09-03T14:24:58.145Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2025-09-03T14:24:58.145Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2025-09-03T14:24:58.145Z [TRACE] statemgr.Filesystem: read nil snapshot
Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
2025-09-03T14:24:58.145Z [DEBUG] Querying available versions of provider registry.terraform.io/hashicorp/random at network mirror https://broken.example.com/
2025-09-03T14:24:58.145Z [DEBUG] GET https://broken.example.com/registry.terraform.io/hashicorp/random/index.json
2025-09-03T14:24:58.145Z [TRACE] HTTP client GET request to https://broken.example.com/registry.terraform.io/hashicorp/random/index.json
2025-09-03T14:24:58.244Z [ERROR] GET https://broken.example.com/registry.terraform.io/hashicorp/random/index.json request failed: Get "https://broken.example.com/registry.terraform.io/hashicorp/random/index.json": dial tcp: lookup broken.example.com on 8.8.8.8:53: no such host
2025-09-03T14:24:58.244Z [DEBUG] GET https://broken.example.com/registry.terraform.io/hashicorp/random/index.json: retrying in 1s (1 left)
2025-09-03T14:24:59.244Z [INFO]  Previous request to the remote registry failed, attempting retry.
2025-09-03T14:24:59.244Z [TRACE] HTTP client GET request to https://broken.example.com/registry.terraform.io/hashicorp/random/index.json
2025-09-03T14:24:59.339Z [ERROR] GET https://broken.example.com/registry.terraform.io/hashicorp/random/index.json request failed: Get "https://broken.example.com/registry.terraform.io/hashicorp/random/index.json": dial tcp: lookup broken.example.com on 8.8.8.8:53: no such host
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/random: failed to query provider mirror https://broken.example.com/ for registry.terraform.io/hashicorp/random: the request failed after 2 attempts, please try
│ again later: Get "https://broken.example.com/registry.terraform.io/hashicorp/random/index.json": dial tcp: lookup broken.example.com on 8.8.8.8:53: no such host
│ 
│ To see which modules are currently depending on hashicorp/random and what versions are specified, run the following command:
│     terraform providers
╵
/demo # 

Expected Behavior

Initializing provider plugins...
- Reusing previous version of hashicorp/random from dependency lock file
- Using previously-installed hashicorp/random v3.7.2

Actual Behavior

Initializing provider plugins...
- Reusing previous version of hashicorp/random from dependency lock file
[DEBUG] Querying available versions of provider registry.terraform.io/hashicorp/random at network mirror https://unavailable/
[DEBUG] GET https://unavailable/registry.terraform.io/hashicorp/random/index.json
...
Error: Failed to query available provider packages

Steps to Reproduce

These steps have been written to be carried out within an isolated environment, like docker run --rm -it --entrypoint /bin/sh hashicorp/terraform:1.13. One of the steps overwrites ~/.terraformrc, so please be careful if running this on your primary system.

1. docker run --rm -it --entrypoint /bin/sh hashicorp/terraform:1.13
2. Create main.tf with the supplied required_providers configuration. This is an example and the bug / behaviour is not specific to this provider.
3. Run terraform init. This should work without any errors.
4. Set up a broken network mirror. It's important that this does not point to a working service.

   cat > ~/.terraformrc <<EOF
   provider_installation {
     network_mirror {
       url = "https://broken.example.com/"
     }
   }
   EOF

5. Run terraform init. This fails, but I think it should succeed.

Additional Context

While working on a Terraform set-up, our network mirror went down. I ran terraform init before the outage, so I knew that I had all the providers installed locally. (I was able to run several 'plan' and 'apply' commands.) After changing a module, I updated the "ref=" version number in the source line for the module, and then ran terraform init to download the latest version of the module. The command failed attempting to retrieve providers that it already had available (both in the configured provider cache and) locally.

The reproducer that I made for this issue doesn't use a cache provider (for simplicity), but if this is confirmed as a bug / desired behaviour change, then please do make this work for cases where the required version (as dictated by the lock file) is available in the provider cache directory.

I have attempted to look into the code to work out why the network was being hit for this action, but wasn't able to figure it out quickly.
I have searched for existing issues regarding this behaviour, but did not find any.

References

No response

Generative AI / LLM assisted development?

No response

[apparentlymart]

FWIW what is described here is the currently-intended behavior, because the cache directory is a cache of the provider packages themselves rather than the metadata about them. Terraform re-requests the metadata to decide whether it's okay to keep using the entry in the cache.

However, I can certainly understand the argument that Terraform should treat the cache as valid if running terraform init without -upgrade and if the lock file already agrees with the package that's in the cache. Unless the lock file has been tampered with in some way -- which the system already assumes is not the case in other ways -- agreement between the lock file and the cache is a good signal that the currently-selected version was available in the upstream provider source at some point, and should therefore still be there.

One potential hazard here is when using the direct installation method (which is the default when no other settings are configured) this metadata request gives the remote registry an opportunity to return a warning/error if e.g. the selected provider has been retracted due to a significant bug, and so skipping that metadata lookup would cause that remote change to be undetected. Network mirrors cannot include extra metadata like that so that is arguably moot for network mirror sources, but as far as I know the provider installer is not currently designed to make different decisions cache validity decisions depending on which installation method is selected: the subsystem that decides to ask the installation methods for metadata is unaware of which concrete installation methods are selected.

(I'm just sharing the context above in the hope that it's helpful. I don't have any strong opinion about what, if anything, to change here.)