Installation on vSphere - ERROR Error: rpc error: code = Unavailable desc = transport is closing #309

zombiefish · 2020-08-20T15:38:05Z

Describe the bug
Installation fails with 'Unavailable description'

Version
openshift-install 4.5.0-0.okd-2020-08-12-020541
built from commit 699277bb61706731d687b9e40700ebf4630b0851
release image quay.io/openshift/okd@sha256:6974c414be62aee4fde24fe47ccfff97c2854ddc37eb196f3f3bcda2fdec17b4

How reproducible
100% reporducible

Command Line
$ openshift-install create cluster --dir=/export/Projects/OKD/VMware --log-level=info
? SSH Public Key /home/XXX/.ssh/id_rsa.pub
? Platform vsphere
? vCenter vcenter.XXX.com
? Username [email protected]
? Password [? for help] ********
INFO Connecting to vCenter vcenter.XXX.com
INFO Defaulting to only available datacenter: XXX
? Cluster Containers
? Default Datastore Lenovo1_ISCSI_RAID5_16TB
? Network OKD (10.10.22%2f24)
? Virtual IP Address for API 10.10.22.100
? Virtual IP Address for Ingress 10.10.22.101
? Base Domain XXX.com
? Cluster Name okd
? Pull Secret [? for help] *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************INFO Obtaining RHCOS image file from 'https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200629.3.0/x86_64/fedora-coreos-32.20200629.3.0-vmware.x86_64.ova?sha256=172f299a3e28be360740ff437a5ea9bfc246f52ea8f313d4138c5d16fd4b11e1'
INFO The file was found in cache: /home/XXX
/.cache/openshift-installer/image_cache/062bfe3785d26fa220e2e6e72d1b3562. Reusing...
INFO Creating infrastructure resources...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

openshift_install.log
openshift_install_state.json.txt

vrutkovs · 2020-08-21T13:46:24Z

Last log lines:

time="2020-08-20T11:23:00-04:00" level=debug msg="vsphere_tag_category.category: Creating..."
time="2020-08-20T11:23:00-04:00" level=error
time="2020-08-20T11:23:00-04:00" level=error msg="Error: could not create category: 400 Bad Request: {\"type\":\"com.vmware.vapi.std.errors.already_exists\",\"value\":{\"messages\":[]}}"
time="2020-08-20T11:23:00-04:00" level=error
time="2020-08-20T11:23:00-04:00" level=error msg="  on ../../../../../tmp/openshift-install-732522033/main.tf line 54, in resource \"vsphere_tag_category\" \"category\":"
time="2020-08-20T11:23:00-04:00" level=error msg="  54: resource \"vsphere_tag_category\" \"category\" {"
time="2020-08-20T11:23:00-04:00" level=error
time="2020-08-20T11:23:00-04:00" level=error
time="2020-08-20T11:23:00-04:00" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply Terraform: failed to complete the change"

It appears the tag category already exists?

yukselao · 2020-10-14T22:59:16Z

any solution for this problem?

yukselao · 2020-10-14T22:59:45Z

I got similar error message. I am trying to install openshift 4.5 on vsphere platform.

TownGeekAus · 2020-12-15T01:38:43Z

Same here INFO Creating infrastructure resources...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

TownGeekAus · 2020-12-15T01:39:22Z

openshift 4.6
DEBUG vsphere_tag.tag: Creating...
DEBUG vsphere_tag.tag: Creation complete after 0s [id=urn:vmomi:InventoryServiceTag:32f172ad-d760-4c1d-822d-79c054140f24:GLOBAL]
DEBUG vsphereprivate_import_ova.import: Creating...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

jlacasse · 2021-01-19T22:20:24Z

Same thing for me:

DEBUG vsphere_folder.folder[0]: Creation complete after 0s [id=group-v308]
DEBUG vsphereprivate_import_ova.import: Creating...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

killergoalie · 2021-01-20T18:10:23Z

Are any of you using ESXi 6.0? I'm seeing the same error with vcenter 6.7 and esxi hosts 6.0 (can't upgrade hosts due to hardware limitation)

AndrewProwse · 2021-01-20T21:56:37Z

Unfortunately I never got this to work using the included tools. I was on ESXi 6.0. I ended up writing my own scripts, and use my own haproxy, tftpd server (to automate the grub command line needs for bootstrap, manager and worker), and web server (hold ignition files)to get this mess automated.

leewx95 · 2021-01-24T12:47:33Z

For my scenario, I first run the script without any issue until i hit network error when POST-ing the vmdk into the ESXi host inside my vCenter cluster. As ESXi host connection was not part of the test, and was not informed in the OKD documentation, I didn't check the connection.

Error of first attempt.

On my second attempt, something seems to be created in vCenter which aborts the installer-provisioned script. Error for 2nd attempt as below.

root@okd-dhcp:# ./openshift-install create cluster --dir=/opt/openshift/ --log-level=info
INFO Creating infrastructure resources...
ERROR
ERROR Error: could not create category: 400 Bad Request: {"type":"com.vmware.vapi.std.errors.already_exists","value":{"messages":[]}}
ERROR
ERROR   on ../tmp/openshift-install-549359176/main.tf line 54, in resource "vsphere_tag_category" "category":
ERROR   54: resource "vsphere_tag_category" "category" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

```root@okd-dhcp:# pwd

JaimeMagiera · 2021-01-24T16:41:14Z

If on previous attempts the installer got so far as to create the category, it would need to be deleted before running the installer again.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vcenterhost.doc/GUID-BA3D1794-28F2-43F3-BCE9-3964CB207FB6.html

https://registry.terraform.io/providers/hashicorp/vsphere/latest/docs/resources/tag_category

sysThematic · 2021-03-03T16:39:16Z

Hi all,
I've the same problem on Vsphere 7.0

DEBUG vsphere_folder.folder[0]: Creating...
DEBUG vsphere_folder.folder[0]: Creation complete after 0s [id=group-v3180]
DEBUG vsphereprivate_import_ova.import: Creating...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

openshift-bot · 2021-06-02T07:41:27Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

evertmulder · 2021-06-28T17:25:14Z

I also had the error when the template is being created. Note tags were created without issues, that seems another unrelated issue.

DEBUG vsphereprivate_import_ova.import: Creating...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR

It turned out I used a datastore that was not available for the cluster I was installing to. Changing to a datastore that is available for the cluster, the problem was fixed.

Perhaps this will help someone, it is a bit unclear from the logs.

openshift-bot · 2021-07-28T17:46:32Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2021-08-27T20:52:22Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2021-08-27T20:52:41Z

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Likqez · 2025-01-10T16:01:26Z

I also had the error when the template is being created. Note tags were created without issues, that seems another unrelated issue.
DEBUG vsphereprivate_import_ova.import: Creating...
ERROR
ERROR Error: rpc error: code = Unavailable desc = transport is closing
ERROR
ERROR
It turned out I used a datastore that was not available for the cluster I was installing to. Changing to a datastore that is available for the cluster, the problem was fixed.

Perhaps this will help someone, it is a bit unclear from the logs.
@evertmulder

I am having the same issue, do you still know how u knew the datastore was not available for that cluster?
I'm using RH and not OKD version and thus filed an issue here
openshift/installer#9352

jcpowermac · 2025-01-10T16:14:39Z

@Likqez since this issue is closed can you create a new one? Also attach the installer logs, this isn't enough information to go on.

vrutkovs added the platform/vsphere label Aug 21, 2020

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 2, 2021

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 28, 2021

openshift-ci bot closed this as completed Aug 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation on vSphere - ERROR Error: rpc error: code = Unavailable desc = transport is closing #309

Installation on vSphere - ERROR Error: rpc error: code = Unavailable desc = transport is closing #309

zombiefish commented Aug 20, 2020

vrutkovs commented Aug 21, 2020

yukselao commented Oct 14, 2020

yukselao commented Oct 14, 2020

TownGeekAus commented Dec 15, 2020

TownGeekAus commented Dec 15, 2020 •

edited

Loading

jlacasse commented Jan 19, 2021

killergoalie commented Jan 20, 2021

AndrewProwse commented Jan 20, 2021

leewx95 commented Jan 24, 2021 •

edited

Loading

JaimeMagiera commented Jan 24, 2021

sysThematic commented Mar 3, 2021

openshift-bot commented Jun 2, 2021

evertmulder commented Jun 28, 2021 •

edited

Loading

openshift-bot commented Jul 28, 2021

openshift-bot commented Aug 27, 2021

openshift-ci bot commented Aug 27, 2021

Likqez commented Jan 10, 2025 •

edited

Loading

jcpowermac commented Jan 10, 2025 •

edited

Loading

Installation on vSphere - ERROR Error: rpc error: code = Unavailable desc = transport is closing #309

Installation on vSphere - ERROR Error: rpc error: code = Unavailable desc = transport is closing #309

Comments

zombiefish commented Aug 20, 2020

vrutkovs commented Aug 21, 2020

yukselao commented Oct 14, 2020

yukselao commented Oct 14, 2020

TownGeekAus commented Dec 15, 2020

TownGeekAus commented Dec 15, 2020 • edited Loading

jlacasse commented Jan 19, 2021

killergoalie commented Jan 20, 2021

AndrewProwse commented Jan 20, 2021

leewx95 commented Jan 24, 2021 • edited Loading

JaimeMagiera commented Jan 24, 2021

sysThematic commented Mar 3, 2021

openshift-bot commented Jun 2, 2021

evertmulder commented Jun 28, 2021 • edited Loading

openshift-bot commented Jul 28, 2021

openshift-bot commented Aug 27, 2021

openshift-ci bot commented Aug 27, 2021

Likqez commented Jan 10, 2025 • edited Loading

jcpowermac commented Jan 10, 2025 • edited Loading

TownGeekAus commented Dec 15, 2020 •

edited

Loading

leewx95 commented Jan 24, 2021 •

edited

Loading

evertmulder commented Jun 28, 2021 •

edited

Loading

Likqez commented Jan 10, 2025 •

edited

Loading

jcpowermac commented Jan 10, 2025 •

edited

Loading