Opensearch: missing fields during update cluster configuration

[acharviakou]

Describe the bug
Cluster can be created fine and all fields looks good
Unfortunately during update manifest some fields are missing
For example:

autoTuneOptions:
    desiredState: DISABLED
    useOffPeakWindow: false

clusterConfig:
 multiAZWithStandbyEnabled: false

Additionally can not update windowStartTime, but it can be related to previous issue

offPeakWindowOptions:
    enabled: true
    offPeakWindow:
      windowStartTime:
        hours: 10
        minutes: 0

Assumption: logic to handle these fields is missing in the following method
https://github.com/aws-controllers-k8s/opensearchservice-controller/blob/main/pkg/resource/domain/hooks.go#L281

Steps to reproduce
kubectl apply (change account id; region name; sg, subnet ids and another personal data)

apiVersion: opensearchservice.services.k8s.aws/v1alpha1
kind: Domain
metadata:
  annotations:
    meta.helm.sh/release-name: test-service
    meta.helm.sh/release-namespace: test
  labels:
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: test-service
    app.kubernetes.io/part-of: test-service
    helm.sh/release: test-service
  name: os-d-test
  namespace: test
spec:
  accessPolicies: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT_ID:role/pod-role/iamr-d-test-REGION_NAME"},"Action":"es:*","Resource":"arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test/*"}]}'
  advancedOptions:
    override_main_response_version: "true"
    rest.action.multi.allow_explicit_index: "true"
  advancedSecurityOptions:
    anonymousAuthEnabled: false
    enabled: false
    internalUserDatabaseEnabled: false
  aimlOptions:
    naturalLanguageQueryGenerationOptions:
      desiredState: DISABLED
  autoTuneOptions:
    desiredState: DISABLED
    useOffPeakWindow: false
  clusterConfig:
    coldStorageOptions:
      enabled: false
    dedicatedMasterEnabled: false
    instanceCount: 2
    instanceType: t3.small.search
    warmEnabled: false
    multiAZWithStandbyEnabled: false
    zoneAwarenessConfig:
      availabilityZoneCount: 2
    zoneAwarenessEnabled: true
  cognitoOptions:
    enabled: true
    identityPoolID: REGION_NAME:f60f0f18-0230-4e3c-88fb-45244342
    roleARN: arn:aws:iam::ACCOUNT_ID:role/service-role/iamr-all-cognito-es-access
    userPoolID: REGION_NAME_nCtuRHSPS
  domainEndpointOptions:
    customEndpointEnabled: false
    enforceHTTPS: true
    tlsSecurityPolicy: Policy-Min-TLS-1-0-2019-07
  ebsOptions:
    ebsEnabled: true
    volumeSize: 10
    volumeType: gp2
  encryptionAtRestOptions:
    enabled: true
    kmsKeyID: arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/c92a7817-0622-438d-a06c-234c0b08e333
  engineVersion: OpenSearch_1.3
  ipAddressType: ipv4
  name: os-d-test
  nodeToNodeEncryptionOptions:
    enabled: false
  offPeakWindowOptions:
    enabled: true
    offPeakWindow:
      windowStartTime:
        hours: 0
        minutes: 0
  softwareUpdateOptions:
    autoSoftwareUpdateEnabled: false
  tags:
  - key: Name
    value: os-d-test
  - key: Service
    value: test-service
  - key: CreatedBy
    value: EKS ACK
  vpcOptions:
    securityGroupIDs:
    - sg-234234234234
    - sg-534524523423
    subnetIDs:
    - subnet-23423445
    - subnet-65356554

State of object in k8s

apiVersion: opensearchservice.services.k8s.aws/v1alpha1
kind: Domain
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"opensearchservice.services.k8s.aws/v1alpha1","kind":"Domain","metadata":{"annotations":{"meta.helm.sh/release-name":"test-service","meta.helm.sh/release-namespace":"test"},"labels":{"app.kubernetes.io/instance":"test-service","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"test-service","app.kubernetes.io/part-of":"test-service","helm.sh/release":"test-service"},"name":"os-d-test","namespace":"test"},"spec":{"accessPolicies":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::ACCOUNT_ID:role/pod-role/iamr-d-test-REGION_NAME\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test/*\"}]}","advancedOptions":{"override_main_response_version":"true","rest.action.multi.allow_explicit_index":"true"},"advancedSecurityOptions":{"anonymousAuthEnabled":false,"enabled":false,"internalUserDatabaseEnabled":false},"aimlOptions":{"naturalLanguageQueryGenerationOptions":{"desiredState":"DISABLED"}},"autoTuneOptions":{"desiredState":"DISABLED","useOffPeakWindow":false},"clusterConfig":{"coldStorageOptions":{"enabled":false},"dedicatedMasterEnabled":false,"instanceCount":2,"instanceType":"t3.small.search","multiAZWithStandbyEnabled":false,"warmEnabled":false,"zoneAwarenessConfig":{"availabilityZoneCount":2},"zoneAwarenessEnabled":true},"cognitoOptions":{"enabled":true,"identityPoolID":"REGION_NAME:f60f0f18-0230-4e3c-88fb-45244342","roleARN":"arn:aws:iam::ACCOUNT_ID:role/service-role/iamr-all-cognito-es-access","userPoolID":"REGION_NAME_nCtuRHSPS"},"domainEndpointOptions":{"customEndpointEnabled":false,"enforceHTTPS":true,"tlsSecurityPolicy":"Policy-Min-TLS-1-0-2019-07"},"ebsOptions":{"ebsEnabled":true,"volumeSize":10,"volumeType":"gp2"},"encryptionAtRestOptions":{"enabled":true,"kmsKeyID":"arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/c92a7817-0622-438d-a06c-234c0b08e333"},"engineVersion":"OpenSearch_1.3","ipAddressType":"ipv4","name":"os-d-test","nodeToNodeEncryptionOptions":{"enabled":false},"offPeakWindowOptions":{"enabled":true,"offPeakWindow":{"windowStartTime":{"hours":0,"minutes":0}}},"softwareUpdateOptions":{"autoSoftwareUpdateEnabled":false},"tags":[{"key":"Name","value":"os-d-test"},{"key":"Service","value":"test-service"},{"key":"CreatedBy","value":"EKS ACK"}],"vpcOptions":{"securityGroupIDs":["sg-234234234234","sg-534524523423"],"subnetIDs":["subnet-65356554","subnet-23423445"]}}}
    meta.helm.sh/release-name: test-service
    meta.helm.sh/release-namespace: test
  creationTimestamp: "2024-11-26T14:25:55Z"
  finalizers:
  - finalizers.opensearchservice.services.k8s.aws/Domain
  generation: 2
  labels:
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: test-service
    app.kubernetes.io/part-of: test-service
    helm.sh/release: test-service
  name: os-d-test
  namespace: test
  resourceVersion: "541481280"
  uid: 0d0532b2-83fb-4e23-841c-64371ab12f60
spec:
  accessPolicies: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT_ID:role/pod-role/iamr-d-demo-test-REGION_NAME"},"Action":"es:*","Resource":"arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test/*"}]}'
  advancedOptions:
    override_main_response_version: "true"
    rest.action.multi.allow_explicit_index: "true"
  advancedSecurityOptions:
    anonymousAuthEnabled: false
    enabled: false
    internalUserDatabaseEnabled: false
  aimlOptions:
    naturalLanguageQueryGenerationOptions:
      desiredState: DISABLED
  autoTuneOptions:
    useOffPeakWindow: false
  clusterConfig:
    coldStorageOptions:
      enabled: false
    dedicatedMasterEnabled: false
    instanceCount: 2
    instanceType: t3.small.search
    multiAZWithStandbyEnabled: false
    warmEnabled: false
    zoneAwarenessConfig:
      availabilityZoneCount: 2
    zoneAwarenessEnabled: true
  cognitoOptions:
    enabled: true
    identityPoolID: REGION_NAME:f60f0f18-0230-4e3c-88fb-45244342
    roleARN: arn:aws:iam::ACCOUNT_ID:role/service-role/iamr-all-cognito-es-access
    userPoolID: REGION_NAME_nCtuRHSPS
  domainEndpointOptions:
    customEndpointEnabled: false
    enforceHTTPS: true
    tlsSecurityPolicy: Policy-Min-TLS-1-0-2019-07
  ebsOptions:
    ebsEnabled: true
    volumeSize: 10
    volumeType: gp2
  encryptionAtRestOptions:
    enabled: true
    kmsKeyID: arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/c92a7817-0622-438d-a06c-234c0b08e333
  engineVersion: OpenSearch_1.3
  ipAddressType: ipv4
  name: os-d-test
  nodeToNodeEncryptionOptions:
    enabled: false
  offPeakWindowOptions:
    enabled: true
    offPeakWindow:
      windowStartTime:
        hours: 0
        minutes: 0
  softwareUpdateOptions:
    autoSoftwareUpdateEnabled: false
  tags:
  - key: Name
    value: os-d-test
  - key: Service
    value: test-service
  - key: CreatedBy
    value: EKS ACK
  vpcOptions:
    securityGroupIDs:
    - sg-534524523423
    - sg-234234234234
    subnetIDs:
    - subnet-23423445
    - subnet-65356554
status:
  ackResourceMetadata:
    arn: arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test
    ownerAccountID: "ACCOUNT_ID"
    region: REGION_NAME
  changeProgressDetails:
    changeID: 355c77ae-2f43-4d40-a03a-56313e7aedd2
    configChangeStatus: Completed
    initiatedBy: CUSTOMER
    lastUpdatedTime: "2024-11-26T14:48:28Z"
    startTime: "2024-11-26T14:25:57Z"
  conditions:
  - lastTransitionTime: "2024-11-26T14:49:04Z"
    status: "True"
    type: ACK.ResourceSynced
  created: true
  deleted: false
  domainID: ACCOUNT_ID/os-d-test
  domainProcessingStatus: Active
  endpoints:
    vpc: vpc-os-d-test-t6es6vxnq46l42xeerwere3.REGION_NAME.es.amazonaws.com
  processing: false
  serviceSoftwareOptions:
    automatedUpdateDate: "1970-01-01T00:00:00Z"
    cancellable: false
    currentVersion: OpenSearch_1_3_R20241003-P1
    description: There is no software update available for this domain.
    newVersion: ""
    optionalDeployment: true
    updateAvailable: false
    updateStatus: COMPLETED
  snapshotOptions:
    automatedSnapshotStartHour: 0
  upgradeProcessing: false

Run kubectl apply for the same manifest
Status k8s object after second apply

apiVersion: opensearchservice.services.k8s.aws/v1alpha1
kind: Domain
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"opensearchservice.services.k8s.aws/v1alpha1","kind":"Domain","metadata":{"annotations":{"meta.helm.sh/release-name":"test-service","meta.helm.sh/release-namespace":"test"},"labels":{"app.kubernetes.io/instance":"test-service","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"test-service","app.kubernetes.io/part-of":"test-service","helm.sh/release":"test-service"},"name":"os-d-test","namespace":"test"},"spec":{"accessPolicies":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::ACCOUNT_ID:role/pod-role/iamr-d-test-REGION_NAME\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test/*\"}]}","advancedOptions":{"override_main_response_version":"true","rest.action.multi.allow_explicit_index":"true"},"advancedSecurityOptions":{"anonymousAuthEnabled":false,"enabled":false,"internalUserDatabaseEnabled":false},"aimlOptions":{"naturalLanguageQueryGenerationOptions":{"desiredState":"DISABLED"}},"autoTuneOptions":{"desiredState":"DISABLED","useOffPeakWindow":false},"clusterConfig":{"coldStorageOptions":{"enabled":false},"dedicatedMasterEnabled":false,"instanceCount":2,"instanceType":"t3.small.search","multiAZWithStandbyEnabled":false,"warmEnabled":false,"zoneAwarenessConfig":{"availabilityZoneCount":2},"zoneAwarenessEnabled":true},"cognitoOptions":{"enabled":true,"identityPoolID":"REGION_NAME:f60f0f18-0230-4e3c-88fb-45244342","roleARN":"arn:aws:iam::ACCOUNT_ID:role/service-role/iamr-all-cognito-es-access","userPoolID":"REGION_NAME_nCtuRHSPS"},"domainEndpointOptions":{"customEndpointEnabled":false,"enforceHTTPS":true,"tlsSecurityPolicy":"Policy-Min-TLS-1-0-2019-07"},"ebsOptions":{"ebsEnabled":true,"volumeSize":10,"volumeType":"gp2"},"encryptionAtRestOptions":{"enabled":true,"kmsKeyID":"arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/c92a7817-0622-438d-a06c-234c0b08e333"},"engineVersion":"OpenSearch_1.3","ipAddressType":"ipv4","name":"os-d-test","nodeToNodeEncryptionOptions":{"enabled":false},"offPeakWindowOptions":{"enabled":true,"offPeakWindow":{"windowStartTime":{"hours":0,"minutes":0}}},"softwareUpdateOptions":{"autoSoftwareUpdateEnabled":false},"tags":[{"key":"Name","value":"os-d-test"},{"key":"Service","value":"test-service"},{"key":"CreatedBy","value":"EKS ACK"}],"vpcOptions":{"securityGroupIDs":["sg-234234234234","sg-534524523423"],"subnetIDs":["subnet-23423445","subnet-65356554"]}}}
    meta.helm.sh/release-name: test-service
    meta.helm.sh/release-namespace: test
  creationTimestamp: "2024-11-26T14:25:55Z"
  finalizers:
  - finalizers.opensearchservice.services.k8s.aws/Domain
  generation: 4
  labels:
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: test-service
    app.kubernetes.io/part-of: test-service
    helm.sh/release: test-service
  name: os-d-test
  namespace: test
  resourceVersion: "541482020"
  uid: 0d0532b2-83fb-4e23-841c-64371ab12f60
spec:
  accessPolicies: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT_ID:role/pod-role/iamr-d-demo-test-REGION_NAME"},"Action":"es:*","Resource":"arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test/*"}]}'
  advancedOptions:
    override_main_response_version: "true"
    rest.action.multi.allow_explicit_index: "true"
  advancedSecurityOptions:
    anonymousAuthEnabled: false
    enabled: false
    internalUserDatabaseEnabled: false
  aimlOptions:
    naturalLanguageQueryGenerationOptions:
      desiredState: DISABLED
  autoTuneOptions:
    desiredState: DISABLED
  clusterConfig:
    coldStorageOptions:
      enabled: false
    dedicatedMasterEnabled: false
    instanceCount: 2
    instanceType: t3.small.search
    warmEnabled: false
    zoneAwarenessConfig:
      availabilityZoneCount: 2
    zoneAwarenessEnabled: true
  cognitoOptions:
    enabled: true
    identityPoolID: REGION_NAME:f60f0f18-0230-4e3c-88fb-45244342
    roleARN: arn:aws:iam::ACCOUNT_ID:role/service-role/iamr-all-cognito-es-access
    userPoolID: REGION_NAME_nCtuRHSPS
  domainEndpointOptions:
    customEndpointEnabled: false
    enforceHTTPS: true
    tlsSecurityPolicy: Policy-Min-TLS-1-0-2019-07
  ebsOptions:
    ebsEnabled: true
    volumeSize: 10
    volumeType: gp2
  encryptionAtRestOptions:
    enabled: true
    kmsKeyID: arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/c92a7817-0622-438d-a06c-234c0b08e333
  engineVersion: OpenSearch_1.3
  ipAddressType: ipv4
  name: os-d-test
  nodeToNodeEncryptionOptions:
    enabled: false
  offPeakWindowOptions:
    enabled: true
    offPeakWindow:
      windowStartTime:
        hours: 0
        minutes: 0
  softwareUpdateOptions:
    autoSoftwareUpdateEnabled: false
  tags:
  - key: Name
    value: os-d-test
  - key: Service
    value: test-service
  - key: CreatedBy
    value: EKS ACK
  vpcOptions:
    securityGroupIDs:
    - sg-234234234234
    - sg-534524523423
    subnetIDs:
    - subnet-23423445
    - subnet-65356554
status:
  ackResourceMetadata:
    arn: arn:aws:es:REGION_NAME:ACCOUNT_ID:domain/os-d-test
    ownerAccountID: "ACCOUNT_ID"
    region: REGION_NAME
  changeProgressDetails:
    changeID: 355c77ae-2f43-4d40-a03a-56313e7aedd2
  conditions:
  - lastTransitionTime: "2024-11-26T14:50:45Z"
    status: "False"
    type: ACK.ResourceSynced
  created: true
  deleted: false
  domainID: ACCOUNT_ID/os-d-test
  domainProcessingStatus: Active
  endpoints:
    vpc: vpc-os-d-test-t6es6vxnq46l42xeerwere3.REGION_NAME.es.amazonaws.com
  processing: false
  serviceSoftwareOptions:
    automatedUpdateDate: "1970-01-01T00:00:00Z"
    cancellable: false
    currentVersion: OpenSearch_1_3_R20241003-P1
    description: There is no software update available for this domain.
    newVersion: ""
    optionalDeployment: true
    updateAvailable: false
    updateStatus: COMPLETED
  snapshotOptions:
    automatedSnapshotStartHour: 0
  upgradeProcessing: false

Compare autoTuneOptions and clusterConfig.multiAZWithStandbyEnabled

Controller logs

2024-11-26T15:17:39.513Z	DEBUG	ackrt	> r.Sync	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	>> r.resetConditions	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	<< r.resetConditions	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	>> rm.ResolveReferences	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	<< rm.ResolveReferences	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	>> rm.EnsureTags	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	<< rm.EnsureTags	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	>> rm.ReadOne	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.513Z	DEBUG	ackrt	>>> rm.sdkFind	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.659Z	DEBUG	ackrt	<<< rm.sdkFind	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.660Z	DEBUG	ackrt	<< rm.ReadOne	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.660Z	DEBUG	ackrt	>> r.updateResource	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.660Z	INFO	ackrt	desired resource state has changed	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12, "diff": [{"Path":{"Parts":["Spec","AutoTuneOptions","DesiredState"]},"A":"DISABLED","B":null},{"Path":{"Parts":["Spec","AutoTuneOptions","UseOffPeakWindow"]},"A":null,"B":false},{"Path":{"Parts":["Spec","ClusterConfig","MultiAZWithStandbyEnabled"]},"A":null,"B":false}]}
2024-11-26T15:17:39.660Z	DEBUG	ackrt	>>> rm.Update	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:39.660Z	DEBUG	ackrt	>>>> rm.customUpdateDomain	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<<<< rm.customUpdateDomain	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<<< rm.Update	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12, "latest": {}}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	INFO	ackrt	updated resource	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<< r.updateResource	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>> r.lateInitializeResource	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>>> rm.LateInitialize	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	no late initialization required.	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<<< rm.LateInitialize	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<< r.lateInitializeResource	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>> r.ensureConditions	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	<< r.ensureConditions	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	< r.Sync	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	requeueing resource after finding resource synced condition false	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	> r.patchResourceStatus	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.228Z	DEBUG	ackrt	>> kc.Patch (status)	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.242Z	DEBUG	ackrt	patched resource status	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12, "json": "{\"metadata\":{\"resourceVersion\":\"541493925\"},\"spec\":{\"tags\":[{\"key\":\"Name\",\"value\":\"os-d-test\"},{\"key\":\"Service\",\"value\":\"test-service\"},{\"key\":\"CreatedBy\",\"value\":\"EKS ACK\"},\"status\":{\"changeProgressDetails\":{\"configChangeStatus\":null,\"initiatedBy\":null,\"lastUpdatedTime\":null,\"startTime\":null},\"conditions\":[{\"lastTransitionTime\":\"2024-11-26T15:17:43Z\",\"status\":\"False\",\"type\":\"ACK.ResourceSynced\"}]}}"}
2024-11-26T15:17:43.242Z	DEBUG	ackrt	<< kc.Patch (status)	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.242Z	DEBUG	ackrt	< r.patchResourceStatus	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12}
2024-11-26T15:17:43.242Z	DEBUG	ackrt	requeue needed after error	{"kind": "Domain", "namespace": "test", "name": "os-d-test", "account": "ACCOUNT_ID", "role": "", "region": "REGION_NAME", "is_adopted": false, "generation": 12, "error": "temporary out of sync, reconcile after some time", "after": "30s"}
2024-11-26T15:17:43.247Z	DEBUG	exporter.field-export-reconciler	error did not need requeue	{"error": "the source resource is not synced yet"}

Expected outcome
A concise description of what you expected to happen.
I expect to see in cluster the same manifest in yaml file I am going to apply, in k8s and cluster state in AWS

Environment
K8s 1.31
eks.12
OpenSearch domain
public.ecr.aws/aws-controllers-k8s/opensearchservice-controller:1.0.0

[ShacharMalachi]

Hi,

Is there any update/estimation regarding providing a fix for this bug?

Thanks!
Shachar.

[acharviakou]

aws-controllers-k8s/opensearchservice-controller#78

[jlbutler]

Fixed by aws-controllers-k8s/opensearchservice-controller#89

[acharviakou]

Hi guys
After update controller to latest version I see a lot of logs(3h) like

{"level":"info","ts":"2025-06-02T10:33:41.070Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Domain","namespace":"test","name":"os-d-demo-test","account":"***","role":"","region":"eu-west-1","is_adopted":false,"generation":2958,"diff":[{"Path":{"Parts":["Spec","AutoTuneOptions","DesiredState"]},"A":"DISABLED","B":"DISABLE_IN_PROGRESS"}]}
{"level":"info","ts":"2025-06-02T10:50:21.185Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Domain","namespace":"test","name":"os-d-demo-test","account":"***","role":"","region":"eu-west-1","is_adopted":false,"generation":2958,"diff":[{"Path":{"Parts":["Spec","AutoTuneOptions","DesiredState"]},"A":"DISABLED","B":"DISABLE_IN_PROGRESS"}]}
{"level":"info","ts":"2025-06-02T11:07:01.321Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Domain","namespace":"test","name":"os-d-demo-test","account":"***","role":"","region":"eu-west-1","is_adopted":false,"generation":2958,"diff":[{"Path":{"Parts":["Spec","AutoTuneOptions","DesiredState"]},"A":"DISABLED","B":"DISABLE_IN_PROGRESS"}]}
{"level":"info","ts":"2025-06-02T11:23:41.486Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Domain","namespace":"test","name":"os-d-demo-test","account":"***","role":"","region":"eu-west-1","is_adopted":false,"generation":2958,"diff":[{"Path":{"Parts":["Spec","AutoTuneOptions","DesiredState"]},"A":"DISABLED","B":"DISABLE_IN_PROGRESS"}]}

aws cli output return

aws opensearch describe-domain-config --domain-name os-d-demo-test
"AutoTuneOptions": {
            "Options": {
                "DesiredState": "DISABLED",
                "RollbackOnDisable": "NO_ROLLBACK",
                "MaintenanceSchedules": [],
                "UseOffPeakWindow": false
            },
            "Status": {
                "CreationDate": "2025-02-05T17:59:14.027000+03:00",
                "UpdateDate": "2025-02-05T18:19:46.670000+03:00",
                "UpdateVersion": 9,
                "State": "DISABLE_IN_PROGRESS",
                "PendingDeletion": false
            }
        },

As for me controller behaves correctly.
Is it something should be fixed by AWS opensearch service team?

[acharviakou]

Reply from AWS support

The DISABLE_IN_PROGRESS state for Auto-Tune is likely occurring because T2 and T3 instance types are not supported for the Auto-Tune feature. When a domain has unsupported instance types in either master or data nodes, the Auto-Tune functionality becomes disabled. Since the cluster is using t3.small instances, this explains the why it is in disable state.

However, coming to your questions here are few insights - 

1. Why AWS api return DISABLE_IN_PROGRESS status?
From the backend tools I can confirm that the Auto-tune is disabled as t3.small instance type doesn't support Auto-tune functionality. API sometimes returns the status as DISABLED or DISABLE_IN_PROGRESS for t2 and t3 instances.

2. Is it expected behaviour? How we can debug/fix such issue? 
This is a expected behaviour with t2 and t3 instance type. Also, there is no operational impact of this status as it is not in use and not supported.