Not working in CircleCI

[Pithikos]

How to use it?

• kwok
• kwokctl --runtime=docker (default runtime)
• kwokctl --runtime=binary
• kwokctl --runtime=nerdctl
• kwokctl --runtime=kind

What happened?

Kwok setup does not behave the same on CircleCI server as local Linux setup (following same instructions).

What did you expect to happen?

I expect to be able and create a cluster and kubectl as I do locally following exactly the same instructions.

How can we reproduce it (as minimally and precisely as possible)?

Start a CircleCI SSH session based on a workflow like below:

version: 2.1

orbs:
  kubernetes: circleci/kubernetes@1.3.1
  
jobs:

  run-tests:
    docker:
      - image: cimg/python:3.10.2
    steps:
      - setup_remote_docker
      - kubernetes/install-kubectl
      - run:
          name: Pull KWOK image
          command: docker pull registry.k8s.io/kwok/cluster:v0.3.0-k8s.v1.27.3
      - checkout
      - run:
          name: Run tests
          command: echo Testing complete.


workflows:
  version: 2
  run-unit-tests:
    jobs:
      - run-tests

Try to run Kwok (once SSHed)

# Install kwok & kwokctl
wget -O kwok -c "https://github.com/kubernetes-sigs/kwok/releases/download/v0.4.0/kwok-linux-amd64"
wget -O kwokctl -c "https://github.com/kubernetes-sigs/kwok/releases/download/v0.4.0/kwokctl-linux-amd64"
chmod +x kwok
chmod +x kwokctl
sudo mv kwok /usr/local/bin/kwok
sudo mv kwokctl /usr/local/bin/kwokctl

# Start kwok cluster
kwokctl create cluster --name=kwok
kubectl config use-context kwok-kwok
kubectl get all

Output:

E0123 10:35:02.157227    2238 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32766/api?timeout=32s": dial tcp 127.0.0.1:32766: connect: connection refused
E0123 10:35:02.157439    2238 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32766/api?timeout=32s": dial tcp 127.0.0.1:32766: connect: connection refused
E0123 10:35:02.158706    2238 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32766/api?timeout=32s": dial tcp 127.0.0.1:32766: connect: connection refused
E0123 10:35:02.158852    2238 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32766/api?timeout=32s": dial tcp 127.0.0.1:32766: connect: connection refused
E0123 10:35:02.160113    2238 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32766/api?timeout=32s": dial tcp 127.0.0.1:32766: connect: connection refused
The connection to the server 127.0.0.1:32766 was refused - did you specify the right host or port?

Anything else we need to know?

No response

Kwok version

$ kwok --version
kwok version v0.4.0 go1.20.7 (linux/amd64)

$ kwokctl --version
kwokctl version v0.4.0 go1.20.7 (linux/amd64)

OS version

```console # On Linux: $ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal $ uname -a Linux alto 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ```

[wzshiming]

Try the following steps using binary file to create a cluster without docker

wget -O kwokctl -c "https://github.com/kubernetes-sigs/kwok/releases/download/v0.4.0/kwokctl-linux-amd64"
chmod +x kwokctl
sudo mv kwokctl /usr/local/bin/kwokctl

kwokctl create cluster --runtime=binary
kubectl get all

[Pithikos]

Try the following steps using binary file to create a cluster without docker

wget -O kwokctl -c "https://github.com/kubernetes-sigs/kwok/releases/download/v0.4.0/kwokctl-linux-amd64"
chmod +x kwokctl
sudo mv kwokctl /usr/local/bin/kwokctl

kwokctl create cluster --runtime=binary
kubectl get all

I tried the binary and it works ok for basic cases.

However my application uses Google's client library to communicate with Kube API, so it kind of blows up. E.g. for checking the namespaces it gives

...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=32764): Max retries exceeded with url: /api/v1/namespaces (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unsupported certificate purpose (_ssl.c:1129)')))
...

I wonder if the issue with how I setup the whole thing or if it's just not meant to work like that. Since I can see the Docker container for kwok running fine with docker ps, just that I can't communicate with it via kubectl

[Pithikos]

Played more with this. Works if executor is set to machine instead of docker.

E.g.

version: 2.1

orbs:
  kubernetes: circleci/kubernetes@1.3.1
  
jobs:
  run-tests:
    machine:
      image: ubuntu-2004:current  # THIS LINE
    steps:
      - kubernetes/install-kubectl
      - checkout
      ..

[wzshiming]

Nice, looks like it was caused by a bad internet connection then.

[Pithikos]

Oh it's not a connection issue per se. It's apparently on how CircleCI environment is setup. https://circleci.com/docs/using-docker/#docker-benefits-and-limitations

With docker it actually spawns everything in an unprivilged and stripped container environment is my understanding. With machine it acts more as if it is running like locally on a barebones machine. Both have docker & utils installed apparently, but docker environment has some gotchas due to limitations

[wzshiming]

/remove-kind bug
/kind support