kwok managed pods do not respect finalizers and are deleted immediately

[tuoyikuan]

How to use it?

• kwok
• kwokctl --runtime=docker (default runtime)
• kwokctl --runtime=binary
• kwokctl --runtime=nerdctl
• kwokctl --runtime=kind

What happened?

After a pod with non-empty finalizers has be scheduled to a kwok managed node, it will be deleted immediately even through no one external removes the finalizers.
If you watch such a pod from client view (for example by kubectl get pod mypod -o yaml -w --output-watch-events=true) , you will find the pod resource seems first got all finalizers removed and then deleted by the apiserver.
like below (with irrelevant part dropped):

object:
  apiVersion: v1
  kind: Pod
  metadata:
    creationTimestamp: "2024-01-19T14:33:14Z"
    finalizers:
    - mytest/myfinalizer
  spec: ...
  status: ...
type: ADDED
---
object:
  apiVersion: v1
  kind: Pod
  metadata:
    deletionGracePeriodSeconds: 30
    deletionTimestamp: "2024-01-19T14:35:50Z"
    finalizers:
    - mytest/myfinalizer
  spec: ...
  status: ...
type: MODIFIED
---
object:
  apiVersion: v1
  kind: Pod
  metadata:
    deletionGracePeriodSeconds: 30
    deletionTimestamp: "2024-01-19T14:35:50Z"
  spec: ...
  status: ...
type: MODIFIED
---
object:
  apiVersion: v1
  kind: Pod
  metadata:
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2024-01-19T14:35:20Z"
  spec: ...
  status: ...
type: MODIFIED
---
object:
  apiVersion: v1
  kind: Pod
  metadata:
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2024-01-19T14:35:20Z"
  spec: ...
  status: ...
type: DELETED

What did you expect to happen?

The kwok managed pods respect the custom finalizers and do not clean them until their owner removes it, just like normal pod (or any k8s api resource).

How can we reproduce it (as minimally and precisely as possible)?

1. Create a pod and schedule it to a kwok fake node.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80
  nodeName: kwok-fake-node
EOF

2. Add the pod with some custom finalizers
   kubectl patch pod nginx -p '{"metadata": {"finalizers": ["mytest/myfinalizer"]}}'
3. Delete the pod while the finalizers are still there, and you will find the pod is deleted immediately.
   kubectl delete pod nginx

Anything else we need to know?

No response

Kwok version

$ kwok --version
kwok version v0.3.0 go1.20.3 (linux/amd64)

OS version

```console

On Linux:

$ cat /etc/os-release

paste output here

$ uname -a

paste output here

On Darwin:

$ uname -a

paste output here

Darwin C02F52S4ML85 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64 x86_64

On Windows:

C:> wmic os get Caption, Version, BuildNumber, OSArchitecture

paste output here

</details>

[wzshiming]

apiVersion: kwok.x-k8s.io/v1alpha1
kind: Stage
metadata:
  name: pod-delete
spec:
  resourceRef:
    apiGroup: v1
    kind: Pod
  selector:
    matchExpressions:
    - key: '.metadata.deletionTimestamp'
      operator: 'Exists'
    - key: '.metadata.finalizers'
      operator: 'DoesNotExist'
  next:
    delete: true

Using this instead of the original should be the effect you want. See more

[wzshiming]

/remove-kind bug
/kind support

[tuoyikuan]

Oh.. Thanks a lot~