Change from APM to Integrations server fail after update

[kuisathaverat]

APM service is defined as apm on 7.x and as integrations_server on 8.x. If you create a 7.x deployment wit apm then you update to 8.x and finally you replace apm by integrations_server this latest change fails with the following error:

╷
│ Error: failed updating deployment
│ 
│   with ec_deployment.main,
│   on main.tf line 24, in resource "ec_deployment" "main":
│   24: resource "ec_deployment" "main" {
│ 
│ api error: 1 error occurred:
│       * clusters.cluster_plan_state_error: APM is not in managed mode, Integrations Server payload cannot be used
│ (resources.integrations_server[0])
│ 
│ 

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

The deployment must replace APM with the integrations service

Current Behavior

The changes in the plan must happen, and the apm service must be replaced by the integrations_server

  # ec_deployment.main will be updated in-place
  ~ resource "ec_deployment" "main" {
      - apm                    = {
          - elasticsearch_cluster_ref_id          = "main-elasticsearch" -> null
          - http_endpoint                         = "http://REDACTED.apm.us-west2.gcp.elastic-cloud.com:80" -> null
          - https_endpoint                        = "https://REDACTED.apm.us-west2.gcp.elastic-cloud.com:443" -> null
          - instance_configuration_id             = "gcp.apm.n2.68x32x45" -> null
          - instance_configuration_version        = 1 -> null
          - latest_instance_configuration_id      = "gcp.apm.n2.68x32x45" -> null
          - latest_instance_configuration_version = 1 -> null
          - ref_id                                = "main-apm" -> null
          - region                                = "gcp-us-west2" -> null
          - resource_id                           = "REDACTED" -> null
          - size                                  = "1g" -> null
          - size_resource                         = "memory" -> null
          - zone_count                            = 1 -> null
        } -> null
        id                     = "90a1739e540e5fa3ebf72063ebeb4d98"
      + integrations_server    = {
          + elasticsearch_cluster_ref_id          = "main-elasticsearch"
          + endpoints                             = (known after apply)
          + http_endpoint                         = (known after apply)
          + https_endpoint                        = (known after apply)
          + instance_configuration_id             = (known after apply)
          + instance_configuration_version        = (known after apply)
          + latest_instance_configuration_id      = (known after apply)
          + latest_instance_configuration_version = (known after apply)
          + ref_id                                = "main-integrations_server"
          + region                                = (known after apply)
          + resource_id                           = (known after apply)
          + size                                  = (known after apply)
          + size_resource                         = "memory"
          + zone_count                            = (known after apply)
        }
        name                   = "test-update"
        # (11 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform definition

Create 7.17.26 deployment

variable "ess_apikey" {
  type = string
}
terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.12.2"
    }
  }
}
provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = var.ess_apikey
  verbose = true
}
resource "ec_deployment" "main" {
  name = "test-update"
  alias = "ltest-update"
  region                 = "gcp-us-west2"
  version                = "7.17.26"
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    hot = {
      autoscaling = {}
      zone_count = 1
    }
    strategy = "rolling_all"
  }
  apm = {}
  kibana = {
  }
}

Update to 8.18.0

variable "ess_apikey" {
  type = string
}
terraform {
  required_version = ">= 0.12.29"
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.12.2"
    }
  }
}
provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = var.ess_apikey
  verbose = true
}
resource "ec_deployment" "main" {
  name = "test-update"
  alias = "ltest-update"
  region                 = "gcp-us-west2"
  version                = "8.18.0"
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    hot = {
      autoscaling = {}
      zone_count = 1
    }
    strategy = "rolling_all"
  }
  apm = {}
  kibana = {
  }
}

Replace APM with the Integrations server

variable "ess_apikey" {
  type = string
}
terraform {
  required_version = ">= 0.12.29"
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.12.2"
    }
  }
}
provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = var.ess_apikey
  verbose = true
}
resource "ec_deployment" "main" {
  name = "test-update"
  alias = "ltest-update"
  region                 = "gcp-us-west2"
  version                = "8.18.0"
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    hot = {
      autoscaling = {}
      zone_count = 1
    }
    strategy = "rolling_all"
  }
  integrations_server = {}
  kibana = {
  }
}

Steps to Reproduce

1. Create a deployment with 7.17.26 and APM (see terraforms above)
2. Update the deployment to 8.18.0 (see terraforms above)
3. Update the deployment to use integrations server instead of APM (see terraforms above)

Context

Possible Solution

Remove the APM, and after that, add the integrations server, but this has a problem: you hit the same bug if you try to update the integrations server settings. So every time you have to change the setting on the Integrations server, you have to delete it and add it again.

Delete APM/Integration server

variable "ess_apikey" {
  type = string
}
terraform {
  required_version = ">= 0.12.29"
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.12.2"
    }
  }
}
provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = var.ess_apikey
  verbose = true
}
resource "ec_deployment" "main" {
  name = "test-update"
  alias = "ltest-update"
  region                 = "gcp-us-west2"
  version                = "8.18.0"
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    hot = {
      autoscaling = {}
      zone_count = 1
    }
    strategy = "rolling_all"
  }
  kibana = {
  }
}

Add the integrations server again after removing the APM/Integration server

variable "ess_apikey" {
  type = string
}
terraform {
  required_version = ">= 0.12.29"
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.12.2"
    }
  }
}
provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = var.ess_apikey
  verbose = true
}
resource "ec_deployment" "main" {
  name = "test-update"
  alias = "ltest-update"
  region                 = "gcp-us-west2"
  version                = "8.18.0"
  deployment_template_id = "gcp-storage-optimized"
  elasticsearch = {
    autoscale = false
    hot = {
      autoscaling = {}
      zone_count = 1
    }
    strategy = "rolling_all"
  }
  integrations_server = {}
  kibana = {
  }
}

Your Environment

• Version used: 0.12.2
• Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: ECH production
• Environment name and version (e.g. Go 1.9):
• Server type and version:
• Operating System and version:
• Link to your project:

[kuisathaverat]

There is a manual workaround to make the plan pass after the update. You have to follow the manual steps at Switch an Elastic Cloud cluster to the APM integration