k8s: guest-pull: "Liveness probe" test "fails"

[fidencio]

This is a weird one, as the test itself doesn't fail, but we can clearly see this in the bats output:

#   Warning  Failed     3s                kubelet            Error: failed to create containerd task: failed to create shim task: failed to mount /run/kata-containers/f0ec86fb156a578964007f7773a3ccbdaf60023106634fe030f039e2e154cd11/rootfs to /run/kata-containers/liveness/rootfs, with error: ENOENT: No such file or directory: unknown

Full bats output:

ok 1 Liveness probe
# [bats-exec-test:59] INFO: k8s configured to use runtimeclass
# [bats-exec-test:282] INFO: Adding allow all policy to Pod from /home/sdp/actions-runner/_work/kata-containers/kata-containers/tests/integration/kubernetes/runtimeclass_workloads_work/probe-pod-liveness.yaml
# pod/liveness-exec created
# pod/liveness-exec condition met
#     Liveness:       exec [cat /tmp/healthy] delay=3s timeout=1s period=3s #success=1 #failure=3
#   Warning  Unhealthy  7s (x3 over 13s)  kubelet            Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
# Name:                liveness-exec
# Namespace:           kata-containers-k8s-tests
# Priority:            0
# Runtime Class Name:  kata
# Service Account:     default
# Node:                984fee00bd70.jf.intel.com/10.23.153.204
# Start Time:          Sun, 19 May 2024 08:19:36 -0700
# Labels:              test=liveness
# Annotations:         io.containerd.cri.runtime-handler: kata-qemu-tdx
#                      io.katacontainers.config.agent.policy:
#                        cGFja2FnZSBhZ2VudF9wb2xpY3kKCmRlZmF1bHQgQWRkQVJQTmVpZ2hib3JzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgQWRkU3dhcFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENsb3...
# Status:              Running
# IP:                  10.42.0.40
# IPs:
#   IP:  10.42.0.40
# Containers:
#   liveness:
#     Container ID:  containerd://7b004441e89a8bfea1e57331537c5e86a55deec192eb86086d08cc568036067c
#     Image:         quay.io/prometheus/busybox:latest
#     Image ID:      quay.io/prometheus/busybox@sha256:dfa54ef35e438b9e71ac5549159074576b6382f95ce1a434088e05fd6b730bc4
#     Port:          <none>
#     Host Port:     <none>
#     Args:
#       /bin/sh
#       -c
#       touch /tmp/healthy; echo "Check status"; sleep 6; rm -rf /tmp/healthy; echo "Check dead"; sleep 12
#     State:          Waiting
#       Reason:       CrashLoopBackOff
#     Last State:     Terminated
#       Reason:       StartError
#       Message:      failed to create containerd task: failed to create shim task: failed to mount /run/kata-containers/f0ec86fb156a578964007f7773a3ccbdaf60023106634fe030f039e2e154cd11/rootfs to /run/kata-containers/7b004441e89a8bfea1e57331537c5e86a55deec192eb86086d08cc568036067c/rootfs, with error: ENOENT: No such file or directory: unknown
#       Exit Code:    128
#       Started:      Wed, 31 Dec 1969 16:00:00 -0800
#       Finished:     Sun, 19 May 2024 08:20:04 -0700
#     Ready:          False
#     Restart Count:  2
#     Liveness:       exec [cat /tmp/healthy] delay=3s timeout=1s period=3s #success=1 #failure=3
#     Environment:    <none>
#     Mounts:
#       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k8rss (ro)
# Conditions:
#   Type                        Status
#   PodReadyToStartContainers   True
#   Initialized                 True
#   Ready                       False
#   ContainersReady             False
#   PodScheduled                True
# Volumes:
#   kube-api-access-k8rss:
#     Type:                    Projected (a volume that contains injected data from multiple sources)
#     TokenExpirationSeconds:  3607
#     ConfigMapName:           kube-root-ca.crt
#     ConfigMapOptional:       <nil>
#     DownwardAPI:             true
# QoS Class:                   BestEffort
# Node-Selectors:              katacontainers.io/kata-runtime=true
# Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
#                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
# Events:
#   Type     Reason     Age               From               Message
#   ----     ------     ----              ----               -------
#   Normal   Scheduled  31s               default-scheduler  Successfully assigned kata-containers-k8s-tests/liveness-exec to 984fee00bd70.jf.intel.com
#   Normal   Pulled     23s               kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 345ms (345ms including waiting)
#   Normal   Started    21s               kubelet            Started container liveness
#   Warning  Unhealthy  7s (x3 over 13s)  kubelet            Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
#   Normal   Killing    7s                kubelet            Container liveness failed liveness probe, will be restarted
#   Normal   Pulled     7s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 389ms (389ms including waiting)
#   Warning  Failed     5s                kubelet            Error: failed to create containerd task: failed to create shim task: the file /bin/sh was not found: unknown
#   Normal   Pulling    5s (x3 over 23s)  kubelet            Pulling image "quay.io/prometheus/busybox:latest"
#   Normal   Pulled     4s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 342ms (342ms including waiting)
#   Normal   Created    4s (x3 over 23s)  kubelet            Created container liveness
#   Warning  Failed     3s                kubelet            Error: failed to create containerd task: failed to create shim task: failed to mount /run/kata-containers/f0ec86fb156a578964007f7773a3ccbdaf60023106634fe030f039e2e154cd11/rootfs to /run/kata-containers/liveness/rootfs, with error: ENOENT: No such file or directory: unknown
#   Warning  BackOff    1s (x3 over 3s)   kubelet            Back-off restarting failed container liveness in pod liveness-exec_kata-containers-k8s-tests(b1a980bf-a5b3-479d-97c2-ebdb45773eff)
# pod "liveness-exec" deleted