pai node : Failed to connect to apiserver

[poetryben88]

on pai node , run：systemctl status kubelet

● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-11-04 16:02:04 CST; 7s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 22413 (kubelet)
Tasks: 26 (limit: 4915)
CGroup: /system.slice/kubelet.service
└─22413 /usr/local/bin/kubelet --logtostderr=true --v=2 --node-ip=192.168.50.191 --hostname-override=pai-worker1 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/etc/kubernet

11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.495889 22413 feature_gate.go:216] feature gates: &{map[]}
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.496000 22413 plugins.go:103] No cloud provider specified.
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.496029 22413 server.go:541] No cloud provider specified: "" from the config file: ""
11月 04 16:02:04 pai-worker1 kubelet[22413]: E1104 16:02:04.497419 22413 bootstrap.go:238] Unable to read existing bootstrap client config: invalid configuration: no configuration has been provided
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.497444 22413 bootstrap.go:117] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.499277 22413 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.521611 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
11月 04 16:02:06 pai-worker1 kubelet[22413]: I1104 16:02:06.534133 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
11月 04 16:02:08 pai-worker1 kubelet[22413]: I1104 16:02:08.827977 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

on master node:
root@pai-master:/home/fxkj# kubectl get node
NAME STATUS ROLES AGE VERSION
pai-master Ready master 25d v1.15.11
pai-worker1 NotReady 25d v1.15.11

kubectl describe nodes pai-worker1
Name: pai-worker1
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=pai-worker1
kubernetes.io/os=linux
pai-worker=true
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 09 Oct 2021 16:54:41 +0800
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

---

NetworkUnavailable False Sat, 09 Oct 2021 17:07:06 +0800 Sat, 09 Oct 2021 17:07:06 +0800 CalicoIsUp Calico is running on this node
MemoryPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 192.168.50.191
Hostname: pai-worker1
Capacity:
cpu: 20
ephemeral-storage: 490691512Ki
github.com/fuse: 110
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65525284Ki
nvidia.com/gpu: 4
pods: 110
Allocatable:
cpu: 19900m
ephemeral-storage: 452221296711
github.com/fuse: 110
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65172884Ki
nvidia.com/gpu: 4
pods: 110
System Info:
Machine ID: 9f90e16ee38c4434aaa145f48b54e4f1
System UUID: 478f53c6-1797-cf67-ba8e-04d4c45b392c
Boot ID: 33833536-aa28-4ffa-9e26-4c8166727767
Kernel Version: 5.4.0-42-generic
OS Image: Ubuntu 18.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.7
Kubelet Version: v1.15.11
Kube-Proxy Version: v1.15.11
PodCIDR: 10.200.0.128/25
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE

---

default job-exporter-xmtd4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
default log-manager-ds-tm2sj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
default node-exporter-7kbhz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
kube-system calico-kube-controllers-9f59f7848-bh9q4 30m (0%) 100m (0%) 64M (0%) 256M (0%) 22d
kube-system calico-node-27b29 150m (0%) 300m (1%) 64M (0%) 500M (0%) 21d
kube-system coredns-67db989964-m2mnj 0 (0%) 0 (0%) 0 (0%) 1000Mi (1%) 22d
kube-system dns-autoscaler-77486c4479-2f9th 20m (0%) 0 (0%) 10Mi (0%) 0 (0%) 22d
kube-system k8s-host-device-plugin-daemonset-bdx55 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
kube-system kube-proxy-wjtrm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
kube-system nvidia-device-plugin-daemonset-hfkt5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

---

cpu 200m (1%) 400m (2%)
memory 138485760 (0%) 1804576k (2%)
ephemeral-storage 0 (0%) 0 (0%)
github.com/fuse 0 0
nvidia.com/gpu 0 0
Events:

Available GPU nodes is 0;