KWOK service are crashing on host with big amount of running KWOK clusters.

[RomanBudnyk]

Hi colleagues,

first of all - thanks for creating and supporting KWOK tool, this is very helpful!

We are trying to emulate a big amount of running cluster (our goal - 10K, but for nearest future is 200).
For this purpose, we've chosen KWOK solution and created host with 32 cpus, 64 gb ram and 80 gb storage.
there is no load on running clusters, we were operating with only one of them (creating pods, etc.).

but after some time we cannot create any pods:

root@kwok-small:~# k --context kwok-test1 get deployments.apps
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
fake-pod   0/1     0            0           60s

what we were trying to do to fix the issue:

1. cluster restart - does not help although services looks like running;
2. node reboot - does not help as well.

the only way that is working for us now - is to recreate KWOK clusters, but this cause a big issue as it is corrupting all underlying processes.

please help with the following:

1. why is this happening?
2. is there are any ways to prevent this? (bigger VM, lesser KWOK clusters, etc.)

I am attaching cluster logs here, please check. For me it looks like a big amount of clusters could be the issue, but cannot be 100% sure.

will wait for any answers, it will be very helpful!

Regards,
Roman

logs.zip

[wzshiming]

From your logs, it looks like you created the cluster using binary runtime, and it looks like the machines went dormant causing the kube-controller and kube-scheduler to quit,

But at the moment, there is no keep-alive mechanism for binary runtime, so you need to restart using kwokctl start cluster.

[RomanBudnyk]

hi @wzshiming ,

thanks for your reply. what can you suggest in our case? should we use docker for our needs? (I've tried it before and saw KWOK can create containers for each KWOK service).

Thanks!

[wzshiming]

Yep, It's okay to use Docker, or you can use kwokctl snapshot to save and restore the cluster.

https://kwok.sigs.k8s.io/docs/user/kwokctl-snapshot/#etcd-snapshot

[RomanBudnyk]

thanks, we will study these possibilities! Cheers!