Summary
v2.12.0 is mostly a reliability release. It changes what kube-router's health check means so that failing syncs no longer get kube-router restarted, adds metrics to cover those failures instead, adds CNI configuration templates, and brings quite a few bug and security fixes to the Network Policy, Network Services, and Network Routes Controllers.
User-facing changes:
/healthznow only reports liveness - controllers now heartbeat at the start and end of every sync whether it succeeded or not, so a sync that keeps failing (a rule the kernel rejects, a route that can't be installed, a BGP peer that won't come up) no longer restarts kube-router into aCrashLoopBackOff. A failed initial sync in the Network Services Controller is also no longer fatal.- CNI configuration templates - the new
KUBE_ROUTER_CNI_CONF_TEMPLATE_FILEenv var has kube-router build its CNI config from a template and write the finished file itself, so the container runtime can't pick up a half-finished config on a freshly booted node (which showed up as transient "no IP ranges specified" errors). Also, a.conflistCNI config without abridgeplugin now fails at startup with a clear error instead of causing a panic later on. See the user guide and the example manifest. - BFD improvements - GoBGP v4.10.0 fixes a bug where a stale BFD administrative reset could kill the next BGP session and leave the peer flapping indefinitely (#2189), and the new
kube_router_bfd_peer_infometric exposes the session state of each BFD peer. - Network policy fixes:
- Egress rules with a named port and no
topeers used to allow any destination on the resolved port number, and are now limited to pods that actually expose that port. Pods could also end up in the wrong named port entries. - Some chain and set names could collide. Because the fix changes chain and set names, expect kube-router to recreate them on the first sync after upgrading (the old ones are cleaned up automatically).
- Full syncs on the iptables backend are a lot cheaper on nodes with many pods, since kube-router no longer copies and scans the whole filter table once per pod (in our benchmark with 80 pods, a sync went from about 150 ms and 90 MB to about 0.5 ms and 0.2 MB).
- Egress rules with a named port and no
- Service proxy fixes:
- Services whose namespace, name, and port name joined into the same key (port
http-altonweband portaltonweb-http, for example) collided, which left one without a VIP and had the other load balancing to both services' pods. - DSR setup now only enters the network namespace of pods in the service's own namespace, where before a selectorless service could point it at pods in other namespaces or at the host itself.
- Services whose namespace, name, and port name joined into the same key (port
- Consistent backends for hashing schedulers - endpoints are now sorted for the
sh,dh, andmhschedulers so that the same client reaches the same backend no matter which node it lands on. Existing IPVS services keep their current order, so check out the troubleshooting doc if you need to reset them. - BGP peer passwords no longer leak into logs - when the
kube-router.io/peersannotation failed to parse, the logged error could include the surrounding YAML, base64 encoded passwords and all. - Example manifests refreshed - the daemonsets in
daemonset/now gate liveness behind a startup probe, usednsPolicy: DefaultandcniVersion1.0.0, and drop RBAC grants kube-router no longer needs. The proxy-only and firewall-only manifests now use a service account instead of a host kubeconfig and leave the CNI config alone. If you've based your deployment on any of these, be sure to diff them before upgrading.
Notable maintenance changes:
- Go - 1.26.10 -> 1.27.3
- Kubernetes client libraries - v0.36.3 -> v0.37.2
- GoBGP - v4.8.0 -> v4.10.0
Contributions
Thanks to @twz123, @mrueg, @rkojedzinszky, @damex, @spicysharkco, and @cwaynej for their contributions to this release.
A special welcome to @spicysharkco and @cwaynej on their first contributions!
Thanks as well to @catherinetcai for moving our CI over to Blacksmith. And thanks to Blacksmith and Greptile for generously supporting open source projects like kube-router.
Changelog
- 6e325be - doc(e2e): note that the suite needs docker 28.0+
<Aaron U'Ren> - 1713f69 - fix(Makefile): run prep-release checks against new pins
<Aaron U'Ren> - 073bab8 - fix(e2e-netpol.sh): build the matrix test binary only once
<Aaron U'Ren> - 7ea0095 - test(e2e): check a completed pod joined the policy's sets
<Aaron U'Ren> - b3d3a7f - test(e2e): retry set reads that fail on the active backend
<Aaron U'Ren> - d61f779 - fix(e2e-netpol.sh): use shasum when sha256sum is missing
<Aaron U'Ren> - 2471dd8 - fix(e2e-netpol.sh): load images for the node platform only
<Aaron U'Ren> - 8b30f59 - fix(go.mod): pin kube-openapi to client-go's version
<Aaron U'Ren> - ff02c83 - fix(_typos.toml): don't spellcheck go.mod
<Aaron U'Ren> - c274167 - chore(go.mod): update all deps to latest
<Aaron U'Ren> - f35f6a3 - chore(doctoc): update documentation table-of-contents
<Aaron U'Ren> - 6c6ea7b - fix(Makefile): always rebuild the runtime image stage
<Aaron U'Ren> - bde6ff0 - chore(lint): fix linter errors from upgrade
<Aaron U'Ren> - f6efa33 - fix(Makefile): stop docker steps on ctrl-c
<Aaron U'Ren> - 89fa87d - feat(e2e-netpol.sh): run matrix suites at once when there's room
<Aaron U'Ren> - 76cf312 - test(e2e): wait for default-deny to admit new pods before using them
<Aaron U'Ren> - ecf2021 - test(e2e): run the chain GC spec in parallel with the rest
<Aaron U'Ren> - cc23e0c - test(e2e): don't make deleted test pods wait out a grace period
<Aaron U'Ren> - ac82b30 - test(e2e): wait for blocks to land instead of sleeping
<Aaron U'Ren> - bb7d6d6 - test(e2e): check pod phase every 250ms instead of every 2s
<Aaron U'Ren> - d1af5f5 - test(e2e): preload the test image and warm up nodes first
<Aaron U'Ren> - 8537979 - fix(Makefile): quote E2E_FOCUS and E2E_SKIP for regexes with spaces
<Aaron U'Ren> - 07dc26f - feat(e2e-netpol.sh): always use the kind version CI declares
<Aaron U'Ren> - f56af50 - fix(dependency-updater): pin multi-arch docker tags
<Aaron U'Ren> - 6c0c753 - feat(Makefile): run every test level in prep-release
<Aaron U'Ren> - c22d44d - feat(e2e): add e2e-netpol-all to run the full matrix
<Aaron U'Ren> - 128bf1f - test(e2e): check kube-router's sets for a completed pod's IPs
<Aaron U'Ren> - 3dba6e7 - test(e2e): make the dual-stack specs enforce each family
<Aaron U'Ren> - 31e5d7a - test(e2e): confirm enforcement before checking allow paths
<Aaron U'Ren> - 86422fe - test(e2e): run servers and clients on separate nodes
<Aaron U'Ren> - 03c3fe4 - fix(e2e-netpol.sh): dump debug info when a run fails
<Aaron U'Ren> - 99afae8 - feat(docker): add .dockerignore to trim the build context
<Aaron U'Ren> - bbc2674 - fix(bgp_policies.go): call the export policy an export policy in logs
<Aaron U'Ren> - 8e453fd - chore(kind): update 0.32.0 -> 0.33.0
<Aaron U'Ren> - 34016d4 - fix(ci_env.go): only write the go version to GO_VERSION
<Aaron U'Ren> - dda0dd9 - chore(grype): update 0.117.0 -> 0.121.0
<Aaron U'Ren> - 91d7db9 - chore(goreleaser): update 2.17.1 -> 2.19.0
<Aaron U'Ren> - 7228d7c - chore(typos): update 1.49.0 -> 1.51.1
<Aaron U'Ren> - 3f9e811 - chore(lint): update 2.12.2 -> 2.15.0
<Aaron U'Ren> - 9055049 - chore(kind): update node 1.36.1 -> 1.37.0
<Aaron U'Ren> - 1dc6510 - chore(alpine): update 2.24.3 to latest sha
<Aaron U'Ren> - e83a6cf - chore(go): update 1.26.10 -> 1.27.3
<Aaron U'Ren> - e1c58d1 - chore(build): update openscorecard-action GHA 2.4.4 -> 2.5.0
<Aaron U'Ren> - 99e6737 - chore(build): update codeql-action GHA 4.37.7 -> 4.39.1
<Aaron U'Ren> - 538c25c - chore(build): update ssetup-buildx-action GHA 4.3.0 -> 4.4.1
<Aaron U'Ren> - c8c5190 - chore(build): update login-action GHA 4.6.0 -> 4.7.0
<Aaron U'Ren> - 4ee1e44 - build(deps): bump golang.org/x/net from 0.58.0 to 0.62.0
<dependabot[bot]> - 30d82b0 - build(deps): bump anchore/sbom-action from 0.24.2 to 0.24.3
<dependabot[bot]> - fe79ce0 - build(deps): bump github.com/moby/moby/client from 0.6.0 to 0.6.2
<dependabot[bot]> - 300318b - build(deps): bump goreleaser/goreleaser-action from 7.2.3 to 7.3.0
<dependabot[bot]> - 4167b72 - test(NPC): add name collision and format tests
<Aaron U'Ren> - 8eb9e3f - fix(NPC): prevent chain and set name collisions
<Aaron U'Ren> - 0ee763b - fact(NPC): consolidate netpol object naming into names.go
<Aaron U'Ren> - 541d1be - fix(NSC): stop services with matching joined names from sharing a service ID
<spicysharkco> - e75e29e - Update GoBGP version to v4.10.0
<Richard Kojedzinszky> - dc168be - feat(gobgp): add kube_router_bfd_peer_info metric
<Roman Kuzmitskii> - 3dac4d6 - fix(NSC): add a couple more DSR endpoint guards
<Aaron U'Ren> - 562073f - fix(NSC): skip DSR endpoints that don't resolve to a service pod
<Manuel Rüger> - 0b551bb - fix(NSC): only enter netns of service namespace pods for DSR
<Manuel Rüger> - ecf5008 - fix(NPC): drop the log-drop check that copied the filter table per pod
<spicysharkco> - 14fdff7 - doc(NPC): add todo about potential future performance improvement
<Aaron U'Ren> - 526f95c - fix(NPC): don't share pod IPs between named port entries
<Manuel Rüger> - 664ca2c - fix(NPC): scope egress named ports with empty "to" to exposing pods
<Manuel Rüger> - 4fcb687 - doc(README.md): fix logo whitespacing
<Aaron U'Ren> - e1f1379 - fix(utils): don't include YAML source in Base64String unmarshal errors
<Manuel Rüger> - d73ed10 - fix(NRC): don't include peer annotation source in YAML parse errors
<Manuel Rüger> - aa95dbf - fix(NSC): treat nil EndpointSlice port names as unnamed, range check ports
<Manuel Rüger> - e5766b8 - fix(NSC): skip EndpointSlice ports with nil port or name
<Manuel Rüger> - a22f712 - build(deps): bump github.com/onsi/gomega from 1.43.0 to 1.44.0
<dependabot[bot]> - 558c973 - build(deps): bump github.com/aws/aws-sdk-go-v2/config
<dependabot[bot]> - 4eb7df4 - fix(NSC): sort endpoints for hashing schedulers
<Aaron U'Ren> - d31b7c8 - fix(NSC): shuffle endpoints once per service
<Aaron U'Ren> - 5ee6433 - test(NSC): add tests for buildEndpointSliceInfo()
<Aaron U'Ren> - 9468df1 - fix(NSC): use math/rand/v2 for endpoint shuffle
<Aaron U'Ren> - 64ee40f - test(NSC): ensure that shuffle() actually randomizes endpoints
<Aaron U'Ren> - 39781b2 - doc(README): add Greptile logo to acknowledgements
<Aaron U'Ren> - 28d687e - ops(ci): Migrate to Blacksmith CI (#2178)
<Cat C> - 6f84583 - build(deps): bump github/codeql-action/upload-sarif
<dependabot[bot]> - 81cfd59 - build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.0 to 1.47.1
<dependabot[bot]> - 02710b3 - build(deps): bump docker/setup-qemu-action from 4.2.0 to 4.4.0
<dependabot[bot]> - a206026 - build(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1
<dependabot[bot]> - aab49db - fact(health): consolidate RunSync logic
<Aaron U'Ren> - 23cc61e - fix(health): address PR feedback
<Aaron U'Ren> - 74d16a0 - fix: make remaining controller heartbeats consistent
<Aaron U'Ren> - 87d2bcd - fix(NSC): propagate sync errors and beat unconditionally
<Aaron U'Ren> - ed283c4 - fix(NPC): beat unconditionally and record sync results
<Aaron U'Ren> - 5f61d23 - feat(metrics): add controller sync result metrics
<Aaron U'Ren> - 9d175a7 - build(deps): bump github.com/onsi/ginkgo/v2 from 2.32.1 to 2.33.0
<dependabot[bot]> - 2949873 - build(deps): bump github.com/aws/aws-sdk-go-v2/config
<dependabot[bot]> - 1f92344 - build(deps): bump github/codeql-action/upload-sarif
<dependabot[bot]> - 33a52e7 - build(deps): bump github.com/moby/moby/client from 0.5.1 to 0.6.0
<dependabot[bot]> - f9aec26 - build(deps): bump docker/build-push-action from 7.3.0 to 7.4.0
<dependabot[bot]> - b69c2d0 - build(deps): bump github.com/aws/smithy-go from 1.28.1 to 1.28.2
<dependabot[bot]> - f0b94ed - feat: Support bootstrapping CNI configs from a template file
<Tom Wieczorek> - f41c008 - fix: Reject CNI conflists without a bridge plugin
<Tom Wieczorek> - 3888df1 - chore: Dedupe the CNI config tests
<Tom Wieczorek> - 08abfc1 - chore: Use getPodCIDRsMapFromCNISpec in CNI Config tests
<Tom Wieczorek> - 1278b26 - chore: Use error strings in CNI config table tests
<Tom Wieczorek> - c601548 - chore: Deprecate CNINetworkConfig's IsConfList
<Tom Wieczorek> - 773b676 - chore: Rollup the CNI config marshalling roundtrip test
<Tom Wieczorek> - d7a4a5f - chore: Introduce cniConfContent for CNI config tests
<Tom Wieczorek> - 0dcd0fd - chore: Replace tabs with spaces in test CNI configs
<Tom Wieczorek> - dfa767a - chore: Use temp dir facility from Go's test framework in the CNI config test
<Tom Wieczorek> - 194514f - build(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.2
<dependabot[bot]> - d3e469f - build(deps): bump github.com/aws/aws-sdk-go-v2/config
<dependabot[bot]> - 3f2c775 - build(deps): bump github.com/onsi/gomega from 1.42.1 to 1.43.0
<dependabot[bot]> - ef2af38 - build(deps): bump github.com/moby/moby/client from 0.5.0 to 0.5.1
<dependabot[bot]> - 3ccaf20 - build(deps): bump github/codeql-action/upload-sarif
<dependabot[bot]> - 92a2563 - build(deps): bump anchore/sbom-action from 0.24.0 to 0.24.2
<dependabot[bot]> - 7ee9fc9 - feat(cni): update cniVersion 0.3.0 -> 1.0.0
<Aaron U'Ren> - 57d6ab2 - feat(daemonset): modernize pod specs
<Aaron U'Ren> - f8c1912 - feat(daemonset): add RBAC to the legacy service manifests
<Aaron U'Ren> - bfa50a8 - fact(daemonset): drop dead RBAC grants
<Aaron U'Ren> - 9411710 - fix(daemonset): correct kubeconfig hostPath mounts
<Aaron U'Ren> - 19d9b08 - fix(daemonset): gate liveness behind a startup probe
<Aaron U'Ren> - 3828703 - doc(README.md): add e2e-netpol badge
<Aaron U'Ren> - 4093eea - doc(README.md): replace retired Go Report Card badge
<Aaron U'Ren> - 3b1ad28 - build(deps): bump github.com/aws/aws-sdk-go-v2/service/ec2
<dependabot[bot]> - 14d1473 - build(deps): bump github.com/onsi/ginkgo/v2 from 2.25.3 to 2.32.1
<dependabot[bot]> - bdde891 - build(deps): bump actions/stale from 10.4.0 to 11.0.0
<dependabot[bot]> - a63492f - build(deps): bump actions/setup-go from 6.5.0 to 7.0.0
<dependabot[bot]> - b3b66a0 - build(deps): bump docker/setup-buildx-action from 4.2.0 to 4.3.0
<dependabot[bot]> - a10924f - build(deps): bump golang.org/x/net from 0.57.0 to 0.58.0
<dependabot[bot]> - 4374f0d - build(deps): bump github.com/aws/aws-sdk-go-v2/feature/ec2/imds
<dependabot[bot]> - 6f3af15 - build(deps): bump the k8s-dependencies group with 4 updates
<dependabot[bot]> - 993555c - fact(NSC): enumerate local IPs once per nodeport sync
<Aaron U'Ren> - 076f735 - fact(NSC): reuse caller's local IP map in isValidKubeRouterServiceArtifact
<Chris Jewell>