Summary
The major theme of this kube-router release is to make kube-router fully conformant with the end-to-end upstream sig-network tests. These are tests that the Kubernetes project provides for testing significant networking features and becoming conformant with the tests means that the networking functionality of kube-router is more consistent with other network providers like kube-proxy, Calico, and Cilium.
Major New Features / Fixes
- kube-router's init container now installs CNI plugins when they are missing. The location for these plugins can be controlled by setting the environment variable
HOST_BIN_PATHwithin the init container, but will default to/opt/cni/bin. This is something that all major Kubernetes networking providers do, but kube-router has not done in the past. For more information on CNI plugins, please see: https://github.com/containernetworking/plugins - kube-router now implements
.spec.healthCheckNodePortwhich has long been a part of the Kubernetes service specification, but kube-router hasn't implemented it until now. This port is meant to give visibility about whether or not an endpoint for a service exists on a node to workloads outside the Kubernetes cluster. If a service endpoint is on the node, then this port returns a 200 HTTP response, otherwise it returns a 503 HTTP response if no endpoint exists on the node. By choosing a node that contains an endpoint, a client can ensure source IP preservation. - Hairpinning in kube-router is now fixed. kube-router now has a dedicated controller to ensuring that the
vethinterface inside the container is in promiscuous mode. This used to be handled bykubeletand then it was handled bydocker-shimand then it was removed entirely. As far as I can tell, it was never ported tocontainerdorcri-o. Without this functionality, return traffic ends up getting black-holed before it is routed outside of the container. For more details see: 0f3714b - Users can now specify
--metrics-addrto choose the IP address that kube-router listens on when providing Prometheus metrics - Users can now specify custom protocol timeouts for IPVS services exposed by kube-router
--service-tcp-timeout- (default:0spreserves system value, typically 900 seconds)--service-tcpfin-timeout- (default:0spreserves system value, typically 120 seconds)--service-udp-timeout- (default:0spreserves system value, typically 300 seconds)
- kube-router now abides by the service label
service.kubernetes.io/service-proxy-name. Setting this label to something other thankube-routerwill result in kube-router ignoring the service - kube-router now honors
spec.internalTrafficPolicyand implementsspec.externalTrafficPolicycorrectly. For more information see: https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-policies for more information
Backward Compatibility Breaks
kube-router now serves traffic to endpoints that that have a status of either ready or serving. See: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#conditions for more information on the differences between these conditions. Before this release, kube-router only served traffic to ready endpoints, which improperly ignored pods that were in Terminating status. This change makes kube-router compliant with the upstream standards.
The release switches from Alpine base 3.18 to 3.19 which changes the iptables backend from iptables to nftables. This is a pretty big change, since the two backends are not mixable. The kube-router project recommends coupling this upgrade with a rolling restart of the nodes in your cluster so that the iptables rules get flushed alongside the upgrade.
Other Notes
- If you're running kube-router in a container and use hairpinning, make sure that
hostPID: trueis set on your kube-router pod. This is the only way that kube-router is able to properly set thevethinterface into promiscuous mode. This attribute has now been added to all of the official kube-router DeaemonSet YAML files
Major Dependency Updates
This release also updates:
- Alpine base container from
3.18to3.19 - Go from
v1.20tov1.21 - Kubernetes from
v1.28tov1.29
Contributions
Thanks to:
- @nextsux - For improving overlay tunnel device binding
- @jpiper - For adding the ability to configure an IP to bind metrics to
- @mrueg - For always keeping our dependencies up to date
- @AlexisDucastel - For testing the CNI plugin installation functionality
- @hakman - For helping us improve the upstream e2e tests and providing kOps support and feedback
Changelog (from v2.0.1 - v2.1.0)
- a7d7ac5 - feat(alpine): update 3.18 -> 3.19
<Aaron U'Ren> - 5b56e6c - feat(goreleaser): update v1.21.2 -> v1.24.0
<Aaron U'Ren> - a648800 - feat(moq): update v0.3.2 -> v0.3.4
<Aaron U'Ren> - 46a1b17 - feat(go): upgrade 1.20.13 -> 1.21.7 + dep update
<Aaron U'Ren> - c356289 - feat(lint): update markdownlint v0.37.0 -> 0.39.0
<Aaron U'Ren> - 47fe189 - feat(lint): update golangci-lint and fix lint errors
<Aaron U'Ren> - 9a136c1 - feat(NSC): implement NodePort Health Check
<Aaron U'Ren> - 7aec8d0 - doc(NSC): add comment for hairpin controller
<Aaron U'Ren> - 959022f - feat(NSC): add endpoint statuses to internal struct
<Aaron U'Ren> - 16daa08 - feat(NSC): add endpoints that are ready or serving
<Aaron U'Ren> - efddb2e - Bump dependencies
<Manuel Rüger> - 3a1efa0 - dual_stack_issue_report.md: remove issue report template
<Aaron U'Ren> - dda7ce6 - doc(cni): CNI requirements and install process
<Aaron U'Ren> - a57a226 - feat(cni): add binaries to host when missing
<Aaron U'Ren> - 98eea79 - doc(user-guide.md): add info about traffic policies
<Aaron U'Ren> - fcd21b4 - feat: fully support service traffic policies
<Aaron U'Ren> - 8404260 - feat: increase unit test coverage
<Aaron U'Ren> - 24505f0 - fact(service_endpoints_sync.go): standardize error handling
<Aaron U'Ren> - d3cf4d1 - feat(NSC): add / clarify log messages
<Aaron U'Ren> - d757f49 - feat(NSC): honor headless label
<Aaron U'Ren> - 8afdee8 - fact(NSC): differentiate headless services
<Aaron U'Ren> - 30d3769 - fact(NSC): update Errorf syntax
<Aaron U'Ren> - a0fe844 - feat(NSC): honor service-proxy-name label
<Aaron U'Ren> - ced5102 - feat(NSC): add IPVS service timeouts
<Aaron U'Ren> - 47290a7 - build(deps): bump k8s.io/client-go from 0.28.3 to 0.28.4
<dependabot[bot]> - 27f2283 - build(deps): bump k8s.io/cri-api from 0.28.3 to 0.28.4
<dependabot[bot]> - a6de8cb - build(deps): bump github.com/aws/aws-sdk-go from 1.48.10 to 1.48.16
<dependabot[bot]> - eb462ba - feat(linux_networking.go): add more error info
<Aaron U'Ren> - eb26288 - fix(manifests): add hostPID for hairpin
<Aaron U'Ren> - aebaa48 - fix(NSC): handle endpoint slice ready nil
<Aaron U'Ren> - 0f3714b - fix(hairpin): set hairpin_mode for veth iface
<Aaron U'Ren> - 3ea293d - build(deps): bump github.com/aws/aws-sdk-go from 1.47.9 to 1.48.10
<dependabot[bot]> - 4b863e9 - build(deps): bump k8s.io/api from 0.28.3 to 0.28.4
<dependabot[bot]> - f575d31 - build(deps): bump github.com/osrg/gobgp/v3 from 3.20.0 to 3.21.0
<dependabot[bot]> - 724588c - build(deps): bump golang.org/x/net from 0.18.0 to 0.19.0
<dependabot[bot]> - 448f8b6 - build(deps): bump github.com/aws/aws-sdk-go from 1.45.24 to 1.47.9
<dependabot[bot]> - 0209ac5 - build(deps): bump google.golang.org/grpc from 1.58.3 to 1.59.0
<dependabot[bot]> - b00a997 - build(deps): bump github.com/onsi/gomega from 1.29.0 to 1.30.0
<dependabot[bot]> - 595d3e4 - build(deps): bump golang.org/x/net from 0.17.0 to 0.18.0
<dependabot[bot]> - fcf0ad9 - prometheus metrics: add option to specify listen address
<Jason Piper> - 961c63b - build(deps): bump github.com/onsi/gomega from 1.27.10 to 1.29.0
<dependabot[bot]> - b13cedc - build(deps): bump k8s.io/client-go from 0.28.2 to 0.28.3
<dependabot[bot]> - 49314ab - build(deps): bump k8s.io/cri-api from 0.28.2 to 0.28.3
<dependabot[bot]> - e0861fc - build(deps): bump github.com/osrg/gobgp/v3 from 3.19.0 to 3.20.0
<dependabot[bot]> - 44d8108 - build(deps): bump github.com/docker/docker
<dependabot[bot]> - 66890d5 - feat: Disable binding overlay tunnels to specific device
<Martin -nexus- Mlynář> - f41defc - build(deps): bump google.golang.org/grpc from 1.58.2 to 1.58.3
<dependabot[bot]> - 9d6b647 - doc(user-guide): remove bootkube, add k3s
<Aaron U'Ren> - e8be179 - doc(RELEASE.md): add more information
<Aaron U'Ren> - a00778e - feat(.goreleaser.yml): add prerelease and header to release
<Aaron U'Ren> - eb30de9 - lint: add markdownlint to linting
<Aaron U'Ren> - dad4991 - doc: update and format all documentation
<Aaron U'Ren> - 6442689 - doc(user-guide.md): update formatting
<Aaron U'Ren> - dbd741a - doc(introduction.md): update and format
<Aaron U'Ren> - 4379104 - doc(generic.md): update and format
<Aaron U'Ren> - 1957907 - doc(kubeadm.md): update and format
<Aaron U'Ren>