Occasional error updating SSL endpoint

[mars]

Since getting CI running, I've observed that one of our flakey tests is for the SSL certificate heroku_cert resource.

Terraform Version

Terraform v0.12.0-dev
+ provider.heroku v1.7.3

Affects Terraform v0.11.x versions too.

Affected Resources

• heroku_addon, plan = "ssl:endpoint"
• heroku_cert

The cert explicitly depends on the addon's state.

Terraform Configuration Files

From the test file:

resource "heroku_app" "foobar" {
  name = "%s"
  region = "eu"
}

resource "heroku_addon" "ssl" {
  app = "${heroku_app.foobar.name}"
  plan = "ssl:endpoint"
}

resource "heroku_cert" "ssl_certificate" {
  app = "${heroku_app.foobar.name}"
  depends_on = ["heroku_addon.ssl"]
  certificate_chain="${file("%s")}"
  private_key="${file("%s")}"
}

Debug Output

Flakey test failures, like this one when running on my local machine:

--- FAIL: TestAccHerokuCert_US (31.24s)
    testing.go:568: Step 1 error: errors during apply:
        
        Error: Error updating SSL endpoint: Patch https://api.heroku.com/apps/tftest-w46lc0i9f9/ssl-endpoints/0d5e210b-bef4-4c72-9af3-ef59b4fdc3c9: Could not complete the action. Please try again later.
        
          on /var/folders/zb/774_2qrx4d1drt38ldz4rgw1xcwwj8/T/tf-test293153547/main.tf line 16:
          (source code not available)
        

In TeamCity CI they look like this:

--- FAIL: TestAccHerokuCert_US (31.54s)
  testing.go:518: Step 1 error: Error applying: 1 error occurred:
      * heroku_cert.ssl_certificate: 1 error occurred:
      * heroku_cert.ssl_certificate: Error updating SSL endpoint: Patch https://api.heroku.com/apps/tftest-ewmrcxf20l/ssl-endpoints/61b273c4-469b-48f8-8f5b-4018e16963ca: Could not complete the action. Please try again later.

And sometimes the errors are a 503 status:

--- FAIL: TestAccHerokuCert_EU (38.78s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
        	* heroku_cert.ssl_certificate: 1 error occurred:
        	* heroku_cert.ssl_certificate: Error creating SSL endpoint: Post https://api.heroku.com/apps/tftest-1u2fl41n43/ssl-endpoints: encountered an error : 503 Service Unavailable

Expected Behavior

The SSL certificate heroku_cert should be uploaded successfully to the SSL endpoint heroku_addon.

Actual Behavior

Sometimes it fails, throwing this "try again later" message.

Steps to Reproduce

1. Set yourself up for testing

2. Run this test over & over again, eventually it will fail:

   make testacc TEST="./heroku/" TESTARGS='"-run=TestAccHerokuCert_US"'
   

Important Factoids

Perhaps the certificate resource should be patient with SSL endpoint 😇

[talbright]

Does this seem like another use case for adding a delay, similar to #158?

[mars]

Yes, a wait that continues waiting under these error conditions would be the fix, but waiting after the platform returns errors seems like a bad idea because it should fail when errors are encountered.

Fortunately I believe the underlying platform issue was fixed last week. These errors have not appeared in almost a week. Continuing to observe and will close this issue once the certain of the fix.

[talbright]

Awesome. If not let me know, I can work on a fix. I finally have some breathing room back 😰

[mars]

It's been a week since we've seen this error in daily CI runs, so closing since it seems to be fixed by Heroku!