timeout while waiting for state to become 'success' (timeout: 2m0s)

[erose96]

#777 attempted to fix this issue but it persists in my environment.

I do not believe this is an issue caused by the rate limit.

Here is the section of the debug log with where the exception in the title occurs:

2023-12-04T16:13:52.730Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState timeout after 2m0s: timestamp=2023-12-04T16:13:52.730Z
2023-12-04T16:13:52.730Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState starting 30s refresh grace period: timestamp=2023-12-04T16:13:52.730Z
2023-12-04T16:13:57.308Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState timeout after 2m0s: timestamp=2023-12-04T16:13:57.308Z
2023-12-04T16:13:57.308Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState starting 30s refresh grace period: timestamp=2023-12-04T16:13:57.308Z
2023-12-04T16:13:57.524Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState timeout after 2m0s: timestamp=2023-12-04T16:13:57.523Z
2023-12-04T16:13:57.524Z [WARN]  provider.terraform-provider-pagerduty_v3.2.2: WaitForState starting 30s refresh grace period: timestamp=2023-12-04T16:13:57.524Z
2023-12-04T16:14:22.732Z [ERROR] provider.terraform-provider-pagerduty_v3.2.2: WaitForState exceeded refresh grace period: timestamp=2023-12-04T16:14:22.731Z
2023-12-04T16:14:22.732Z [ERROR] vertex "module.{pagerduty_service_name}" error: timeout while waiting for state to become 'success' (timeout: 2m0s)
2023-12-04T16:14:22.733Z [ERROR] vertex "module.{pagerduty_service_name} (expand)" error: timeout while waiting for state to become 'success' (timeout: 2m0s)

The 200 that occurs right before this indicates the rate limit is not about to be hit:

Ratelimit-Limit: 960
Ratelimit-Remaining: 919
Ratelimit-Reset: 58

The WaitForState messages in the logs makes me think it's related to an issue upstream in the terraform-plugin-sdk. A fix was submitted for that issue a few years ago but was never reviewed.

See past issues: #765 #760

[tgoodsell-tempus]

Could be useful to introduce these to the operations as well, for additional control: https://developer.hashicorp.com/terraform/language/resources/syntax#operation-timeouts

[ingwarsw]

Whats strange with thats issue is that it (in our case) works from our personal computers (100% of cases passes) but fails (>95% cases fails) from github action.

And seems that with each fail it have "random" number of failed items, so its maybe related to some PG rate limiting at host level or something?
And just had even stranger error

│ Error: Get "https://api.pagerduty.com/users/XXX": read tcp 10.1.0.4:33696->44.237.102.140:443: read: connection reset by peer

Overall this issue is annoying as hell.

[austinpray-mixpanel]

Whats strange with thats issue is that it (in our case) works from our personal computers (100% of cases passes) but fails (>95% cases fails) from github action.

Same here. Our developers apply terraform via a github action and we are seeing the same thing.

[gunzy83]

Whats strange with thats issue is that it (in our case) works from our personal computers (100% of cases passes) but fails (>95% cases fails) from github action.

We have just ran into this as well with our first GH actions deploy using an scoped OAuth client credential (app) which only this one project is using for one deployment at a time.

No issues during development on local machine with multiple deploys and tear downs of the stack but going to staging and prod with this errored. I retried the staging job twice (the second time after waiting while reading issues on Github) and then the prod one went through.

Seems I may have got lucky on Github actions with a new runner or exit IP... there may be an undocumented IP address based limit in play?

[imjaroiswebdev]

@erose96 are you facing this issue in a local machine or inside a GH action runner as @ingwarsw describes? Additionally, could any of you please provide a example of the TF code facing this issue for me to try replicating it? So, I can come up ASAP with a solution. Thanks in advance folks!

[ingwarsw]

To test if its GH (network) issue i have created self hosted runner.
And the same pipeline now works 100% of cases.. while if I use GH runners it fails in 99% of cases (it randomply passes from time to time)...

I will try to create simple case run yesterday.
But should be easy to catch..
In most cases it fails with

pagerduty_tag_assignment

Something like that failed on second run.

locals {
  teams = {
    "a"  = "aa",
    "a1" = "aa1",
#    "a2" = "aa2",
#    "a3" = "aa3",
#    "a4" = "aa4",
#    "a5" = "aa5",
#    "a6" = "aa6",
#    "a7" = "aa7",
  }
}

import {
  id = "escalation_policies.xxx.yyy"
  to = pagerduty_tag_assignment.test["a"]
}
import {
  id = "escalation_policies.xxx.yyy"
  to = pagerduty_tag_assignment.test["a1"]
}


resource "pagerduty_tag" "tf_managed" {
  label = "test-me"
}

resource "pagerduty_team" "tf_teams" {
  for_each    = local.teams
  name        = each.key
  description = each.value
}

resource "pagerduty_tag_assignment" "test" {
  for_each = local.teams
  tag_id      = pagerduty_tag.tf_managed.id
  entity_type = "teams"
  entity_id   = pagerduty_team.tf_teams[each.key].id
}

provider "pagerduty" {
  token      = var.pagerduty_api_token
  user_token = var.pagerduty_user_api_token
}

variable "pagerduty_api_token" {
  type        = string
  description = "api token for pagerduty"
}

variable "pagerduty_user_api_token" {
  type        = string
  description = "api user token for pagerduty"
}

output "test" {
  value = pagerduty_tag_assignment.test
}```

[austinpray-mixpanel]

To test if its GH (network) issue i have created self hosted runner. And the same pipeline now works 100% of cases.. while if I use GH runners it fails in 99% of cases (it randomply passes from time to time)...

We are also testing moving our terraform actions to self-hosted runners and are monitoring to see if the timeouts go away

[gunzy83]

@ingwarsw legend, you just saved me from testing a self-hosted runner.

I have even seen an error on this:

data "pagerduty_vendor" "datadog" {
  name = "Datadog"
}

which fails after 5mins of spinning. Only seen this on Github actions, local machines work 100% of the time.

[imjaroiswebdev]

Hey folks! I prepared this repository for trying to replicate the error and after several intends (New commits and Actions re-runs), I can that tell I haven't had success 😅

If I captured correctly what you all being noting, the repository meets following condition for trying the reproduce the error:

• Terraform code project using PagerDuty TF provider.
• Terraform code is executed inside the GH Action Runner.
• I used PD Tags as @ingwarsw said.

On top of that, I added verbose (secured) logging for debugging the error and at the end find out what's going on.

As you being pointing out, locally the TF plan/apply works flawlessly, even in TF Cloud runners too (I did the test just in case).

Therefore, I would really appreciate if any you could submit a few PRs, so you can help me to replicate this error and find the culprit please 🙏🏽. I'll do my best staying tune and promptly merging your PRs till We reproduce the error and hopefully catch the bug in the logs. Thanks in advance for your help and patience.

[austinpray-mixpanel]

@imjaroiswebdev can you try adding a bunch of user / team lookups? We are suspicious that our pagerduty schedule definitions cause a bunch of cascading requests having to look up each user by email and such

[austinpray-mixpanel]

Here's a sanitized example of how we define teams and schedules.

locals {
  team = "DevInfra"
  members = [
    "bogus1@pagerduty.com",
    "bogus2@pagerduty.com",
    "bogus3@pagerduty.com",
    "bogus4@pagerduty.com",
    "bogus5@pagerduty.com",
  ]
  start = "2023-11-27T14:30:00-07:00"
  manager = "bogus1@pagerduty.com"
}

resource "pagerduty_team" "default" {
  name = local.team
}

data "pagerduty_user" "team" {
  for_each = toset(local.members)
  email    = each.key
}

data "pagerduty_user" "manager" {
  email = local.manager
}

resource "pagerduty_schedule" "default" {
  name      = "${local.team} schedule"
  time_zone = "America/Los_Angeles"

  layer {
    name                         = "${local.team} Ops Leads"
    start                        = local.start
    rotation_virtual_start       = local.start
    rotation_turn_length_seconds = 60 * 60 * 24 * 7
    users                        = [for member in local.members : data.pagerduty_user.team[member].id]
  }
  teams = [pagerduty_team.default.id]
}

resource "pagerduty_escalation_policy" "default" {
  name  = "${local.team} Escalation Policy"
  teams = [pagerduty_team.default.id]

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.default.id
    }
  }
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "user_reference"
      id   = data.pagerduty_user.manager.id
    }
  }
}

edit: PR imjaroiswebdev/pd-tfprovider-issue-780-experiment#1

[imjaroiswebdev]

Hey @austinpray-mixpanel thank you very much for your help, however, this configuration wasn't enough to replicate the error look 😩

[gunzy83]

Hey @austinpray-mixpanel thank you very much for your help, however, this configuration wasn't enough to replicate the error look 😩

We appreciate the effort @imjaroiswebdev. Are you able to check internally if there is any rate limiting at the host/ip level in addition to the new rate limiting rules published publically last year? That may explain this issue better than a standard reproduction.

I have only had 1/5 new deployments fail since I posted, however that job was failing repeatedly on pagerduty_vendor datasource until I waited another hour to retry. Our account is so small with very little API use so far (we are not hitting the documented limits) but this kind of random flakiness will kill any notion of packaging Pagerduty service config with app deployment code if we want reliable automated deploys.

[imjaroiswebdev]

I finally was able to reproduce the issue here, I decided to re-run the job until it failed because of this. I believe the last time I didn't try it enough. I just meant to update you all for letting you know I'm researching further into this with other engineering teams to catch the culprit and get back to you with a solution, workaround or something 💪🏽