Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeshark with Bottlerocket? #1498

Open
boatmisser opened this issue Feb 8, 2024 · 1 comment
Open

Kubeshark with Bottlerocket? #1498

boatmisser opened this issue Feb 8, 2024 · 1 comment
Assignees

Comments

@boatmisser
Copy link

Describe the bug
Has anyone gotten kubeshark to work with bottlerocket?
I can't get the worker pods to startup; get a cgroup error in sniffer logs and the pod refuses connections on 30001, no other errors in the worker logs.
The tracer is even worse; ended up turning that off : error="field GoCryptoTlsAbi0Read: program go_crypto_tls_abi0_read: apply CO-RE relocations: can't read types: type id 11360: unknown kind: Unknown (19)"

Provide more information
EKS (1.28)
Bottlerocket OS 1.17.0 (aws-k8s-1.28)
6 nodes but only binding 1 via afifinity
AWS CNI with NetworkPolicy enabled (v1.16.2-eksbuild.1)
kubeshark (v52.1.30)

To Reproduce
Steps to reproduce the behaviour:

  1. Run kubeshark tap
  2. Get pods
  3. See pods are not starting up.
  4. Describe failing pod - :30001: connect: connection refused
  5. Check logs - sniffer has 1 ERR; tracer has 1 FTL
    Expected behaviour
    kubeshark-worker pod starts up successfully.

Logs
kubeshark-worker -- describe

  Warning  BackOff    2m39s (x2 over 2m40s)  kubelet            Back-off restarting failed container tracer in pod kubeshark-worker-daemon-set-6z7qh_default(158a19c6-503d-483d-81ca-9a5c722229df)
  Warning  Unhealthy  2m37s (x4 over 2m39s)  kubelet            Readiness probe failed: dial tcp 10.205.90.30:30001: connect: connection refused
  Warning  Unhealthy  2m37s (x3 over 2m39s)  kubelet            Liveness probe failed: dial tcp 10.205.90.30:30001: connect: connection refused

kubeshark-worker -- get

    lastState:
      terminated:
        containerID: containerd://f25251dc0806402781da65661e6706695b2678c3a12a58e4384a6ebe7df43b5a
        exitCode: 2
        finishedAt: "2024-02-08T01:21:19Z"
        reason: Error
        startedAt: "2024-02-08T01:21:12Z"
    name: sniffer
    lastState:
      terminated:
        containerID: containerd://69330d411c37155edea8300c16c276ae0d09613537052c59e73e838201e29f24
        exitCode: 1
        finishedAt: "2024-02-08T01:20:46Z"
        reason: Error
        startedAt: "2024-02-08T01:20:46Z"
    name: tracer

kubeshark-worker -- sniffer

2024-02-08T01:18:13Z INF main.go:73 > Starting worker...
2024-02-08T01:18:13Z ERR main.go:87 > While setting the memory limit error="Unable to parse the cgroup file"
2024-02-08T01:18:13Z INF main.go:95 > Memory limit is set to limit=800000000
2024-02-08T01:18:13Z INF main.go:100 > Starting worker...
2024-02-08T01:18:13Z INF misc/data.go:24 > Set the data directory to: data-dir=/app/data/ip-10-205-90-30.ap-southeast-2.compute.internal
2024/02/08 01:18:14 [DEBUG] GET http://kubeshark-hub:80/resolver/history
2024/02/08 01:18:15 [DEBUG] GET http://kubeshark-hub:80/resolver/history

kubeshark-worker -- tracer

2024-02-08T01:17:55Z INF tracer/misc/data.go:17 > Set the data directory to: data-dir=data
2024-02-08T01:17:55Z INF tracer/main.go:47 > Starting tracer...
2024-02-08T01:17:55Z INF tracer/tracer.go:39 > Initializing tracer (chunksSize: 40960000) (logSize: 4096)
2024-02-08T01:17:55Z INF tracer/tracer.go:53 > Detected Linux kernel version: 6.1.61
2024-02-08T01:17:55Z FTL tracer/main.go:68 > Couldn't initialize the tracer: error="field GoCryptoTlsAbi0Read: program go_crypto_tls_abi0_read: apply CO-RE relocations: can't read types: type id 11360: unknown kind: Unknown (19)"

kubeshark-worker -- load-pf-ring

No pf_ring module found for the current kernel version 6.1.61
Falling back to AF_PACKET

No dump logs when running with kubeshark tap --set dump-logs=true

@alongir
Copy link
Member

alongir commented Feb 20, 2024

@boatmisser can you try this release: https://github.com/kubeshark/kubeshark/releases/tag/v52.1.50 ?
I didn't try with: bottlerocket, however, issues seem familiar and are likely to be resolved in the latest patch release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants