Kubernetes VM don't have internet connection

[raihanjp]

ISSUE TYPE

• Bug Report
• Improvement Request

COMPONENT NAME

CloudStack Kubernetes Provider

CLOUDSTACK VERSION

4.18.1

OS / ENVIRONMENT

Ubuntu 22.04

SUMMARY

Can't pull any image

Failed to pull image "registry.k8s.io/metrics-server/metrics-server:v0.6.4": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/metrics-server/metrics-server:v0.6.4": failed to resolve reference "registry.k8s.io/metrics-server/metrics-server:v0.6.4": failed to do request: Head "https://registry.k8s.io/v2/metrics-server/metrics-server/manifests/v0.6.4": dial tcp: lookup registry.k8s.io on 8.8.8.8:53: read udp 10.1.1.231:39706->8.8.8.8:53: i/o timeout

Ping from control node
cloud@skripsi-control-18c618bc8bd:$ ping -c 1 google.com
ping: unknown host
cloud@skripsi-control-18c618bc8bd:$ ping -c 1 registry.k8s.io
ping: unknown host

STEPS TO REPRODUCE

Fresh kubernetes cluster installation
Kubernetes version: https://download.cloudstack.org/testing/cks/setup-1.27.8.iso
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

EXPECTED RESULTS

metrics-server-5d875656f5-flz97                       1/1     Running 

ACTUAL RESULTS

metrics-server-5d875656f5-flz97                       0/1     ImagePullBackOff

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@raihanjp

which type of zone (basic/advanced/advanced with security group) and network (isolated, vpc tier, shared) do you use ?

can you run the same ping commands from CloudStack virtual router of the network ?

[raihanjp]

@weizhouapache
-I'm using advanced zone and DefaultNetworkOfferingforKubernetesService which is isolated.

-Still can't ping when i ran the same ping commands on the virtual router of the network

[raihanjp]

I installed cloudstack exactly following this tutorial https://rohityadav.cloud/blog/cloudstack-kvm/ adapt with my network configuration

[weizhouapache]

@weizhouapache -I'm using advanced zone and DefaultNetworkOfferingforKubernetesService which is isolated.

-Still can't ping when i ran the same ping commands on the virtual router of the network

@raihanjp
what's the default egress policy ? is there any egress rules ?
can you ping the gateway of public IPs in the virtual router ?

[raihanjp]

@weizhouapache

• The default egress policy is allow and there's no any egress rules
• I can't ping the gateway of the public IP, but i can ping 8.8.8.8
  

[weizhouapache]

@weizhouapache

• The default egress policy is allow and there's no any egress rules
• I can't ping the gateway of the public IP, but i can ping 8.8.8.8

@raihanjp
all looks good.
now the problem is, you can ping 8.8.8.8, but not for google.com, right ?

[raihanjp]

@weizhouapache
Yes i can't ping google.com or registry.k8s.io or any domain. I want to troubleshoot this because i tought it will solve the problem i'm facing that is failed to pull image

[weizhouapache]

@weizhouapache Yes i can't ping google.com or registry.k8s.io or any domain. I want to troubleshoot this because i tought it will solve the problem i'm facing that is failed to pull image

It looks like a DNS issue
Do you use 8.8.8.8 as the dns server ?

[raihanjp]

Yes, i use 8.8.8.8 as the dns server

[raihanjp]

if my systemvm can ping google.com and registry.k8s.io, the problem must be on the kubernetes cluster creation process right? or do you think there could be a misconfiguration on the apache cloudstack installation?

[weizhouapache]

if my systemvm can ping google.com and registry.k8s.io, the problem must be on the kubernetes cluster creation process right? or do you think there could be a misconfiguration on the apache cloudstack installation?

ping/dns should work in the CloudStack virtual router (it is not in the scope of CKS configuration)

can you run more commands in the VR ?

cat /etc/resolv.conf
nslookup google.com

[raihanjp]

the resolv.conf only have this and nslookup can't ping google.com

nameserver 8.8.8.8

i added

nameserver 192.168.10.1

finally the VR can ping and nslookup google.com, but the VR itself is not pingable and it can't ping it's own ip address. and the kubernetes still can't pull the image, i've tried with other image and it still has the same error. (s-1-vm = system VM & r-11-VM = VR)

Does this ip route on the system vm and the VR matter? the VR name changed from the last image i post after i reboot the VR

[weizhouapache]

@raihanjp
you can add 192.168.10.1 as a dns server in the zone setting.

when you add nameserver in /etc/resolv.conf manually , please restart dnsmasq service in VR and retry.

[raihanjp]

• do you mean this by adding the dns server in the zone setting? And should the use.external.dns in the global setting set to "true"?
• i've restart dnsmasq service in VR, i just noticed if i reboot the VR, the config on resolv.conf is back to default only "nameserver 8.8.8.8"