-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Registry TLS configuration from registries.yaml is only honored for mirror endpoints #9839
Comments
Can you confirm that you are not using a custom containerd config template? Can you provide the output of |
I have not touched the template at all. I also inspected the containerd toml and compared everything that seemed relevant to a backup from an earlier version and everything was identical. I do not have the containerd log anymore. Are you unable to reproduce this behavior in 1.29.3+k3s1? 🤔 If absolutely need be I can destroy my cluster and build from scratch, but that should be the last resort. EDIT: the cluster is up and running on 1.29.2+k3s1 with traffic going to/from. It's disruptive for me to test this on the same metal. I can try on another machine, but so can anyone :) it would be nice to see if anyone else can reproduce this |
According to the containerd docs at https://github.com/containerd/containerd/blob/release/1.7/docs/hosts.md, all the host fields are valid at the root level:
This is what k3s generates: root@systemd-node-1:/# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://172-17-0-7.sslip.io/v2"
capabilities = ["pull", "resolve", "push"]
ca = ["/usr/local/share/ca-certificates/registry.crt"] However, containerd fails to load that:
Apparently it goes looking for at least one As a workaround, we can generate an empty host section; the following works properly: root@systemd-node-1:/# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://172-17-0-7.sslip.io/v2"
capabilities = ["pull", "resolve", "push"]
ca = ["/usr/local/share/ca-certificates/registry.crt"]
[host] I can address this in the next release. In the mean time, if you do not currently specify a port in your registry namespace, you should be able to work around the issue with something like this in your registries.yaml: mirrors:
172-17-0-7.sslip.io:
endpoint:
- https://172-17-0-7.sslip.io:443
configs:
"172-17-0-7.sslip.io:443":
tls:
ca_file: /usr/local/share/ca-certificates/registry.crt Note use of a port in the endpoint to force it to generate a host entry in the hosts.toml. |
|
Thank you very much for going through the work to reproduce this, @brandond! |
Using
|
Validated on master branch with version v1.29.4-rc1+k3s1Environment DetailsInfrastructure
Node(s) CPU architecture, OS, and Version:
Cluster Configuration:
Config.yaml:
registries.yaml:
test-image.yaml:
Testing Steps
Replication Results:
Pod Events:
Validation Results:
Check the hosts.toml for host section:
|
Environmental Info:
K3s Version:
Node(s) CPU architecture, OS, and Version:
Cluster Configuration:
1 controlplane node running etcd, 5 worker nodes, all matching raspberry pi computers
Describe the bug:
configuring
/etc/rancher/k3s/registries.yaml
with the bare minimum for a private registry with a self-signed cert no longer works, but downgrading to 1.29.2+k3s1 allows it to work again without any other changes.Steps To Reproduce:
/etc/rancher/k3s/registries.yaml
as abovelatest
channel (currently version1.29.3+k3s1
)certificate signed by unknown authority
error emitted by containerd, captured inkubectl describe pod $pod_name
eventsExpected behavior:
to see the image pull correctly as it did in the previous release :)
Actual behavior:
errors related to tls verification and failed pulls
Additional context / logs:
not to lead you down a rabbit hole, but perhaps this is related? #9341
The text was updated successfully, but these errors were encountered: