The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?"

[FraGThieF]

Hello!
First of all thank you for this guide!
Learn so many new things, but now i am stucked and do not know how to proceed.

I try to get it up and running on rasperry pi's, 3 Masters, and 4 Workers.

Error message after lunching the playbook:

fatal: k3s-m3]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:01.384371", "end": "2022-03-31 22:13:11.580652", "msg": "non-zero return code", "rc": 1, "start": "2022-03-31 22:13:10.196281", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}

fatal: [k3s-m2]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:01.599374", "end": "2022-03-31 22:13:21.948957", "msg": "non-zero return code", "rc": 1, "start": "2022-03-31 22:13:20.349583", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}

fatal: [k3s-m1]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:01.694997", "end": "2022-03-31 22:14:06.756075", "msg": "non-zero return code", "rc": 1, "start": "2022-03-31 22:14:05.061078", "stderr": "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}

What can I do? Where do I have to search? what could the error be?

Thanks for any help that you can give me.

[timothystewart6]

Did you get latest? Also are you sure you are using the right interface name in your variables? Can you show them here?

[FraGThieF]

THX for your tutorials and help.

Yes, i got the latest,

Interface is eth0

---

k3s_version: v1.23.4+k3s1
ansible_user: FraG
#ansible_pass: *******
systemd_dir: /etc/systemd/system

interface which will be used for flannel

flannel_iface: "eth0"

apiserver_endpoint is virtual ip-address which will be configured on each master

apiserver_endpoint: "192.168.0.190"

k3s_token is required masters can talk together securely

this token should be alpha numeric only

k3s_token: "++++++++++"

extra_server_args: "--no-deploy servicelb --no-deploy traefik --write-kubeconfig-mode 644 default-not-ready-toleration-seconds=30 --kube-apiserver-arg default-unreachable-toleration-seconds=30 --kube-controller-arg node-monitor-period=20s --kube-controller-arg node-monitor-grace-period=20s --kubelet-arg node-status-update-frequency=5s"
extra_agent_args: "--kubelet-arg node-status-update-frequency=5s"

change these to your liking, the only required one is--no-deploy servicelb

#extra_server_args: "--no-deploy servicelb --no-deploy traefik"
#--write-kubeconfig-mode 644"
#extra_agent_args: ""
#"--kubelet-arg node-status-update-frequency=5s"

image tag for kube-vip

kube_vip_tag_version: "v0.4.2"

image tag for metal lb

metal_lb_speaker_tag_version: "v0.12.1"
metal_lb_controller_tag_version: "v0.12.1"

metallb ip range for load balancer

metal_lb_ip_range: "192.168.0.180-192.168.0.189"

[timothystewart6]

I don't see anything odd. I would try removing all server args except required, reset, and try it again

[mrimp]

Expand your hard disks... On all nodes...

Probably should make a note Tim. You do say this in the video!
Thx for your work!

[yankeeinlondon]

I too ran into this problem. I have double checked that the hosts.ini file indeed does match up to the IP addresses which are running. I am running 2 physical servers (both x86) with 3 back-planes and 2 workers.

[yankeeinlondon]

Before doing this I actually switched my Ubuntu template to the prior video that Tim did and made sure that the both username/password and SSH keys are consistent across all VMs.

[svartis]

I am encountering the same issue as well when I try to run this on rasperry pi's.
With Raspberry Pi OS Lite 64 bit I can't get it to work at all.
But with Ubuntu I can get 1 master up and running with nodes.
But when I try with 2 or 3 masters I get stuck with the same issue.

[timothystewart6]

Do all machines have the same time zones, same ssh keys, and are able to communicate with each other? Are you using passwordless sudo? If if not you might have to pass in additional flags like --user user --ask-become I would also remove any additional args from your nodes. These can be problematic on slow machines.

[wlwwt]

I am encountering the same issue on a setup provisioned with vagrant. Here is the stacktrace in verbose mode

FAILED - RETRYING: [172.20.20.11]: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).Result was: { "attempts": 1, "changed": false, "cmd": [ "k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}" ], "delta": "0:00:02.058195", "end": "2022-04-03 19:31:08.887394", "invocation": { "module_args": { "_raw_params": "k3s kubectl get nodes -l \"node-role.kubernetes.io/master=true\" -o=jsonpath=\"{.items[*].metadata.name}\"", "_uses_shell": false, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": false } }, "msg": "non-zero return code", "rc": 1, "retries": 21, "start": "2022-04-03 19:31:06.829199", "stderr": "time=\"2022-04-03T19:31:06Z\" level=info msg=\"Acquiring lock file /var/lib/rancher/k3s/data/.lock\"\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": [ "time=\"2022-04-03T19:31:06Z\" level=info msg=\"Acquiring lock file /var/lib/rancher/k3s/data/.lock\"", "The connection to the server localhost:8080 was refused - did you specify the right host or port?" ], "stdout": "", "stdout_lines": [] }

[FraGThieF]

Expand your hard disks... On all nodes...

Probably should make a note Tim. You do say this in the video! Thx for your work!

How big should the hard disk be? ATM it is 86% free?`

Do all machines have the same time zones, same ssh keys, and are able to communicate with each other? Are you using passwordless sudo? If if not you might have to pass in additional flags like --user user --ask-become I would also remove any additional args from your nodes. These can be problematic on slow machines.

Yes, yes, yes, yes.

I have tried it with only the first do neccessary args and run into the same issue again.

[timothystewart6]

can you please paste your all.yaml along with OS you are running these on? it's hard to tell what you are using vs what you copied and pasted from docs.

[svartis]

Ok, so I was able to solve my issue.
So to be able to post my full all.yaml I changed my k3s token from "K10908c1e28096f6665800342ac1bd8962df701dca5519d73b878a89d1921b432b4" to "mynotsosecrettoken" and this fixed my issue.
So now I can run 2 masters and 2 workers that are running on Ubuntu on raspberry pi.

I have also done a reset and verified that the old token was causing the issue.
And by doing another reset and changing it to the new token I was able to successfully run 2 masters and 2 workers again.

After some digging in logs I found this error line. So I was a bit unlucky with the token I had set:

Apr 04 03:04:03 master01 k3s[5136]: time="2022-04-04T03:04:03+02:00" level=fatal msg="failed to normalize token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

k3s_version: v1.23.4+k3s1
# this is the user that has ssh access to these machines
ansible_user: pi
systemd_dir: /etc/systemd/system

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "192.168.90.30"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "mynotsosecrettoken"

# change these to your liking, the only required one is--no-deploy servicelb
extra_server_args: "--no-deploy servicelb --no-deploy traefik --write-kubeconfig-mode 644"
extra_agent_args: ""

# image tag for kube-vip
kube_vip_tag_version: "v0.4.3"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.12.1"
metal_lb_controller_tag_version: "v0.12.1"

# metallb ip range for load balancer
metal_lb_ip_range: "192.168.90.80-192.168.90.81"

[jpconstantineau]

Which version of Ubuntu for Raspberry PI do you use? 20 or 21? (Assuming 64 bits server)

[yankeeinlondon]

I had some odd disk storage issues on one of my physical devices but coming out of that I'm still experiencing the problem. I do have some initial success with the first master server:

and then I get this message which I don't quite know how to read:

All three masters are now engaging and there is a "change" but it's writing text out to stderr seems like potentially a bad sign. This is then followed by the slow beat of an unhappy service:

Diagnostics at this point are:

• Load balancer VIP times out on ping (192.168.100.200)
• Attempts to ping individual masters works without issue
  

Final output is:

My all.yml file is:

---
k3s_version: v1.23.5-rc5+k3s1
# this is the user that has ssh access to these machines
ansible_user: ken
become: true
systemd_dir: /etc/systemd/system

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "192.168.100.200"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "xxxxxxxxxxxxxxxx"

# change these to your liking, the only required one is--no-deploy servicelb
extra_server_args: "--no-deploy servicelb --no-deploy traefik --write-kubeconfig-mode 644 --kube-api-server-arg default-not-ready-toleration-seconds=30 --kube-apiserver-arg default-unreachable-toleration-seconds=30 --kube-controller-arg node-monitor-period=20s --kube-controller-arg node-monitor-grace-period=20s --kubelet-arg node-status-update-frequency=5s"
extra_agent_args: "--kubelet-arg node-status-update-frequency=5s"

# image tag for kube-vip
kube_vip_tag_version: "v0.4.3"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.12.1"
metal_lb_controller_tag_version: "v0.12.1"

# metallb ip range for load balancer
metal_lb_ip_range: "192.168.100.80-192.168.100.89"

[timothystewart6]

please try without all of the args on extra_server_args and extra_agent_args please use the default

[yankeeinlondon]

no noticeable difference when I comment out the extra_* options. I'm going to do one more full redo of your steps in the notes to make sure I'm not being an idiot. I'm a highly functioning ADHD nutcase so sometimes I skip things ... hopefully not here but worth checking.

[yankeeinlondon]

Oh, i also traversed all nodes -- master and worker -- and ran k3s check-status and all came back with a "pass" status.

[yankeeinlondon]

Not surprisingly trying to check on the nodes in the cluster failed as the service is not running:

[yankeeinlondon]

I can -- however -- manually start the masters with sudo k3s server & and they do stay up as this 404 error shows:

In the long trail of logs that come from starting these servers the only errors I'm seeing are the following:

could this be indicative of a missing network dep to support websockets?

[yankeeinlondon]

Finally, note that in this current state, I can reach the active node directly at 192.168.100.204 but the load balancing VIP seems to be up too as I get the same results hitting it on 192.168.100.200

[yankeeinlondon]

Now I find this very odd ... even though I set the configuration as you did in your video ... I am not able to run the kubectl commands without sudo:

Even more concerning, each of the masters is aware of only itself rather than the cluster at large.

[svartis]

Regarding my issue running on Raspberry Pi OS Lite 64 bit.
I had to manually add " cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory" to the cmdline.txt file.
So it looks like ansible is not adding it as expected.

[jpconstantineau]

Quick update: Running the runbook on a cluster of Raspberry PI 4 with Raspberry Pi OS Lite 64 bits (latest as of Jan 28 2022) - uses Debian Buulseye) fails on the "Verify that all nodes actually joined (check k3s-init.service if this fails)]" step.

[svartis]

Did the latest run apply the cgroup update to cmdline.txt file?
It worked fine for my setup with Debian Bullseye during testing of the PR.

[svartis]

I did a quick retest from the master branch with fresh installs of Debian Bullseye and I could still get it to work.
Can you see in your log output that i detects Debian Bullseye?

Cluster result:

I am using this args in all.yml:

# change these to your liking, the only required one is--no-deploy servicelb
extra_server_args: "--no-deploy servicelb --no-deploy traefik --write-kubeconfig-mode 644"
extra_agent_args: ""

And I could see in the log that it detected the correct OS and ran the tasks:

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Bullseye)] ***************************************************************************************************************
fredag 08 april 2022  11:34:32 +0200 (0:00:00.089)       0:00:53.003 ********** 
ok: [192.168.90.51]
ok: [192.168.90.52]
ok: [192.168.90.61]
ok: [192.168.90.62]

TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***********************************************************************************************************************************
fredag 08 april 2022  11:34:32 +0200 (0:00:00.130)       0:00:53.134 ********** 
included: /home/sm/Code/k3s-ansible/k3s-ansible/roles/raspberrypi/tasks/prereq/Raspbian.yml for 192.168.90.51, 192.168.90.52, 192.168.90.61, 192.168.90.62 => (item=/home/sm/Code/k3s-ansible/k3s-ansible/roles/raspberrypi/tasks/prereq/Raspbian.yml)

TASK [raspberrypi : Activating cgroup support] ******************************************************************************************************************************************************
fredag 08 april 2022  11:34:32 +0200 (0:00:00.176)       0:00:53.311 ********** 
changed: [192.168.90.52]
changed: [192.168.90.61]
changed: [192.168.90.51]
changed: [192.168.90.62]

TASK [raspberrypi : Install iptables] ***************************************************************************************************************************************************************
fredag 08 april 2022  11:34:33 +0200 (0:00:00.715)       0:00:54.027 ********** 
changed: [192.168.90.52]
changed: [192.168.90.51]
changed: [192.168.90.62]
changed: [192.168.90.61]

TASK [raspberrypi : Flush iptables before changing to iptables-legacy] ******************************************************************************************************************************
fredag 08 april 2022  11:34:54 +0200 (0:00:21.022)       0:01:15.049 ********** 
ok: [192.168.90.61]
ok: [192.168.90.51]
ok: [192.168.90.62]
ok: [192.168.90.52]

TASK [raspberrypi : Changing to iptables-legacy] ****************************************************************************************************************************************************
fredag 08 april 2022  11:34:55 +0200 (0:00:01.037)       0:01:16.086 ********** 
changed: [192.168.90.51]
changed: [192.168.90.61]
changed: [192.168.90.52]
changed: [192.168.90.62]

TASK [raspberrypi : Changing to ip6tables-legacy] ***************************************************************************************************************************************************
fredag 08 april 2022  11:34:56 +0200 (0:00:00.827)       0:01:16.914 ********** 
changed: [192.168.90.51]
changed: [192.168.90.52]
changed: [192.168.90.61]
changed: [192.168.90.62]

RUNNING HANDLER [raspberrypi : reboot] **************************************************************************************************************************************************************
fredag 08 april 2022  11:34:57 +0200 (0:00:00.667)       0:01:17.581 ********** 
changed: [192.168.90.61]
changed: [192.168.90.51]
changed: [192.168.90.62]
changed: [192.168.90.52]

[jpconstantineau]

The /boot/cmdline.txt did get updated with the cgroup.

[jpconstantineau]

For some reason the k3s.service is nowhere to be found... I'll investigate a bit later and re-run it all and get the log files

[timothystewart6]

[timothystewart6]

Just moved it over to discussions :)

[jpconstantineau]

Starting again with the latest release (RPI did one today).

Not much changed.... At least the release didn't seem to make things more complicated.

Using DCHP reservations for PI addresses so that every redeploy is the same.
Using the exact same image for each node (Raspberry PiOS Lite (64 Bits) Released 2022-04-04)
Connecting using passwordless ssh

Here is the printout when running the playbook.

PLAY [k3s_cluster] **************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host 192.168.50.53 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.53]
[WARNING]: Platform linux on host 192.168.50.54 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.54]
[WARNING]: Platform linux on host 192.168.50.51 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.51]
[WARNING]: Platform linux on host 192.168.50.55 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.55]
[WARNING]: Platform linux on host 192.168.50.52 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.52]
[WARNING]: Platform linux on host 192.168.50.57 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.57]
[WARNING]: Platform linux on host 192.168.50.56 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [192.168.50.56]

TASK [prereq : Set SELinux to disabled state] ***********************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [prereq : Enable IPv4 forwarding] ******************************************************************************************************************************************************************************************************
changed: [192.168.50.55]
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.54]
changed: [192.168.50.56]
changed: [192.168.50.57]
changed: [192.168.50.53]

TASK [prereq : Enable IPv6 forwarding] ******************************************************************************************************************************************************************************************************
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.55]
changed: [192.168.50.54]
changed: [192.168.50.56]
changed: [192.168.50.57]
changed: [192.168.50.53]

TASK [prereq : Add br_netfilter to /etc/modules-load.d/] ************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [prereq : Load br_netfilter] ***********************************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [prereq : Set bridge-nf-call-iptables (just to be sure)] *******************************************************************************************************************************************************************************
skipping: [192.168.50.51] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.51] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.52] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.52] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.53] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.53] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.54] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.54] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.55] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.55] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.56] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.56] => (item=net.bridge.bridge-nf-call-ip6tables)
skipping: [192.168.50.57] => (item=net.bridge.bridge-nf-call-iptables)
skipping: [192.168.50.57] => (item=net.bridge.bridge-nf-call-ip6tables)

TASK [prereq : Add /usr/local/bin to sudo secure_path] **************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [download : Download k3s binary x64] ***************************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]


TASK [download : Download k3s binary arm64] *************************************************************************************************************************************************************************************************
changed: [192.168.50.52]
changed: [192.168.50.54]
changed: [192.168.50.55]
changed: [192.168.50.51]
changed: [192.168.50.56]
changed: [192.168.50.53]
changed: [192.168.50.57]

TASK [download : Download k3s binary armhf] *************************************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [raspberrypi : Test for raspberry pi /proc/cpuinfo] ************************************************************************************************************************************************************************************
ok: [192.168.50.53]
ok: [192.168.50.55]
ok: [192.168.50.54]
ok: [192.168.50.52]
ok: [192.168.50.51]
ok: [192.168.50.57]
ok: [192.168.50.56]

TASK [raspberrypi : Test for raspberry pi /proc/device-tree/model] **************************************************************************************************************************************************************************
ok: [192.168.50.51]
ok: [192.168.50.53]
ok: [192.168.50.54]
ok: [192.168.50.52]
ok: [192.168.50.55]
ok: [192.168.50.56]
ok: [192.168.50.57]

TASK [raspberrypi : Set raspberry_pi fact to true] ******************************************************************************************************************************************************************************************
ok: [192.168.50.51]
ok: [192.168.50.52]
ok: [192.168.50.53]
ok: [192.168.50.54]
ok: [192.168.50.55]
ok: [192.168.50.56]
ok: [192.168.50.57]

TASK [raspberrypi : Set detected_distribution to Raspbian] **********************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Buster)] *********************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [raspberrypi : Set detected_distribution_major_version] ********************************************************************************************************************************************************************************
skipping: [192.168.50.51]
skipping: [192.168.50.52]
skipping: [192.168.50.53]
skipping: [192.168.50.54]
skipping: [192.168.50.55]
skipping: [192.168.50.56]
skipping: [192.168.50.57]

TASK [raspberrypi : Set detected_distribution to Raspbian (ARM64 on Debian Bullseye)] *******************************************************************************************************************************************************
ok: [192.168.50.52]
ok: [192.168.50.51]
ok: [192.168.50.53]
ok: [192.168.50.54]
ok: [192.168.50.55]
ok: [192.168.50.56]
ok: [192.168.50.57]


TASK [raspberrypi : execute OS related tasks on the Raspberry Pi] ***************************************************************************************************************************************************************************
included: /mnt/c/Data/git/k3s-ansible/roles/raspberrypi/tasks/prereq/Raspbian.yml for 192.168.50.51, 192.168.50.52, 192.168.50.53, 192.168.50.54, 192.168.50.55, 192.168.50.56, 192.168.50.57

TASK [raspberrypi : Activating cgroup support] **********************************************************************************************************************************************************************************************
changed: [192.168.50.51]
changed: [192.168.50.53]
changed: [192.168.50.52]
changed: [192.168.50.55]
changed: [192.168.50.54]
changed: [192.168.50.56]
changed: [192.168.50.57]

TASK [raspberrypi : Install iptables] *******************************************************************************************************************************************************************************************************
changed: [192.168.50.51]
changed: [192.168.50.53]
changed: [192.168.50.54]
changed: [192.168.50.52]
changed: [192.168.50.55]
changed: [192.168.50.56]
changed: [192.168.50.57]

TASK [raspberrypi : Flush iptables before changing to iptables-legacy] **********************************************************************************************************************************************************************
ok: [192.168.50.52]
ok: [192.168.50.51]
ok: [192.168.50.54]
ok: [192.168.50.53]
ok: [192.168.50.55]
ok: [192.168.50.56]
ok: [192.168.50.57]

TASK [raspberrypi : Changing to iptables-legacy] ********************************************************************************************************************************************************************************************
changed: [192.168.50.53]
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.54]
changed: [192.168.50.55]
changed: [192.168.50.56]
changed: [192.168.50.57]

TASK [raspberrypi : Changing to ip6tables-legacy] *******************************************************************************************************************************************************************************************
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.53]
changed: [192.168.50.55]
changed: [192.168.50.54]
changed: [192.168.50.56]
changed: [192.168.50.57]

RUNNING HANDLER [raspberrypi : reboot] ******************************************************************************************************************************************************************************************************
changed: [192.168.50.53]
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.54]
changed: [192.168.50.55]



PLAY [master] *******************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [192.168.50.51]
ok: [192.168.50.52]
ok: [192.168.50.53]

TASK [k3s/master : Clean previous runs of k3s-init] *****************************************************************************************************************************************************************************************
ok: [192.168.50.52]
ok: [192.168.50.53]
ok: [192.168.50.51]

TASK [k3s/master : Clean previous runs of k3s-init] *****************************************************************************************************************************************************************************************
ok: [192.168.50.52]
ok: [192.168.50.53]
ok: [192.168.50.51]

TASK [k3s/master : Create manifests directory on first master] ******************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Copy vip rbac manifest to first master] **********************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Copy vip manifest to first master] ***************************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Copy metallb namespace manifest to first master] *************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Copy metallb ConfigMap manifest to first master] *************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Copy metallb main manifest to first master] ******************************************************************************************************************************************************************************
skipping: [192.168.50.52]
skipping: [192.168.50.53]
changed: [192.168.50.51]

TASK [k3s/master : Init cluster inside the transient k3s-init service] **********************************************************************************************************************************************************************
changed: [192.168.50.51]
changed: [192.168.50.52]
changed: [192.168.50.53]

TASK [k3s/master : Verify that all nodes actually joined (check k3s-init.service if this fails)] ********************************************************************************************************************************************
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).

... more cut out...

FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (1 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (1 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (1 retries left).
fatal: [192.168.50.52]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.690759", "end": "2022-04-08 22:00:04.575916", "msg": "non-zero return code", "rc": 1, "start": "2022-04-08 22:00:03.885157", "stderr": "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
fatal: [192.168.50.53]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.684863", "end": "2022-04-08 22:00:06.133263", "msg": "non-zero return code", "rc": 1, "start": "2022-04-08 22:00:05.448400", "stderr": "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
fatal: [192.168.50.51]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.736497", "end": "2022-04-08 22:00:07.474801", "rc": 0, "start": "2022-04-08 22:00:06.738304", "stderr": "", "stderr_lines": [], "stdout": "raspberrypi", "stdout_lines": ["raspberrypi"]}

TASK [k3s/master : Kill the temporary service used for initialization] **********************************************************************************************************************************************************************
changed: [192.168.50.52]
changed: [192.168.50.53]
changed: [192.168.50.51]

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
192.168.50.51              : ok=26   changed=16   unreachable=0    failed=1    skipped=10   rescued=0    ignored=0
192.168.50.52              : ok=20   changed=10   unreachable=0    failed=1    skipped=16   rescued=0    ignored=0
192.168.50.53              : ok=20   changed=10   unreachable=0    failed=1    skipped=16   rescued=0    ignored=0
192.168.50.54              : ok=15   changed=8    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0
192.168.50.55              : ok=15   changed=8    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0
192.168.50.56              : ok=15   changed=8    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0
192.168.50.57              : ok=15   changed=8    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0

[jpconstantineau]

@svartis See above. I am still getting the issue on my rpi cluster.

@timothystewart6 Sorry about the direct ping and reply to wrong message... didn't really want to copy back up... Perhaps we want to create a RPI specific discussion...

[timothystewart6]

@svartis See above. I am still getting the issue on my rpi cluster.

@timothystewart6 Sorry about the direct ping and reply to wrong message... didn't really want to copy back up... Perhaps we want to create a RPI specific discussion...

I would recommend creating your own thread in discussions if you need help or if it's a bug open an issue.

[timothystewart6]

This was fixed with a new reset task that cleans up the VIP