Summary of Changes
Major Changes:
- Add a readinessProbe to the kvstoremesh container that reports initial synchronization status to support configuring a separate, initial rate-limit to be used while synchronizing. Both clustermesh-apiserver and kvstoremesh now use a high initial rate-limit to decrease start time. (#30361, @thorn3r)
- bpf: introduce encrypted overlay datapath support (#31073, @ldelossa)
- multicast: add CLIs to manage multicast BPF maps (#31355, @harsimran-pabla)
- policy/k8s: Add support for CIDRGroupRef in IngressDeny and EgressDeny (#30933, @pippolo84)
- This adds a new policy field, EnableDefaultDeny, which permits the creation of network polices that do not drop non-matching traffic. (#30572, @squeed)
Minor Changes:
- Add "node-map-max" to allow configuring nodemap size. (#31407, @tommyp1ckles)
- Add helm values.schema.json file for validating supplied values for correct type. (#30631, @ubergesundheit)
- Add line numbers and file names to all metrics in 'cilium-dbg bpf metrics list' (#30972, @ti-mo)
- Add support for ClusterIP service advertisement with BGP Control Plane (#30963, @chaunceyjiang)
- Add support for ExternalIP service advertisement with BGP Control Plane (#31245, @chaunceyjiang)
- agent: add several new flags to control Cilium's datapath events notifications (#30063, @mvisonneau)
- Allow the Host Firewall and IPv6 BPF masquerading to be used together. (#31511, @qmonnet)
- Allows for using AWS SGs in the ingress section of rules. (#30708, @Alex-Waring)
- bgpv1: Add Local internalTrafficPolicy support for ClusterIP advertisements (#31442, @chaunceyjiang)
- bgpv1: BGP Control Plane metrics (#31469, @YutaroHayakawa)
- bugtool: Collect hubble metrics (#31533, @chancez)
- Change Node IPAM to select all nodes if externalTrafficPolicy=Cluster and add
nodeipam.cilium.io/match-node-labels
annotation (#31406, @MrFreezeex) - cleanup: Remove deprecated values for KPR (#31286, @sayboras)
- cni: use default logger with timestamps. (#31014, @tommyp1ckles)
- envoy: Add support for exposing Envoy Admin API (#30655, @sayboras)
- feat: Add the http return code to metric api_processed_total (#31227, @vipul-21)
- Fix Cilium default values for EKS when Cilium clustermesh-apiserver LoadBalancer fails to create NLB with AWS Load Balancer Controller with syntax error. (#31329, @oshangalwaduge)
- Fixes a bug where ToFQDN IPs may be garbage collected too early, disrupting existing connections. (#31205, @squeed)
- fqdn: avoid expensive sort/unique of names during GC (#30920, @tklauser)
- GatewayAPI supports to setting the number of trusted loadbalancer hops (#30662, @chaunceyjiang)
- helm: Bump minimum k8s version to v1.21+ (#31648, @sayboras)
- ingress: Allow strict kube-proxy-replacement (#31284, @sayboras)
- Introduce
cilium-dbg encrypt flush --stale
flag to remove XFRM states and policies with stale node IDs. (#31159, @pchaigno) - labelsfilter: Always apply Cluster entity specific identity-relevant label (#31178, @soggiest)
- Only detach Cilium-owned legacy XDP programs when XDP is disabled (#31654, @ti-mo)
- pkg/kvstore/allocator: Standardize usage of logfields (#30526, @antonipp)
- Remove helm option
enable-remote-node-identity
after being deprecated in v1.15. (#31228, @doniacld) - Support IPv4 fragmentation for service backends. (#31364, @julianwiedmann)
- This allows the initialDelaySeconds option to be configured. This allows users running larger clusters to extend the time it takes for preflight to become ready. (#30495, @chaunceyjiang)
- WG: Improve L7 checks (#31299, @brb)
Bugfixes:
- bpf: use
bpf_htons
instead of using shift (#31247, @chez-shanpu) - Cilium allows selecting 'lo' as a device again. (#31200, @bimmlerd)
- cilium-health: Fix broken retry loop in
cilium-health-ep
controller (#31622, @gandro) - cni: Allow text-ts log format value (#31686, @sayboras)
- cni: Use batch endpoint deletion API in chaining plugin (#31456, @sayboras)
- envoy: register secret syncer even if only CEC is enabled (#31447, @mhofstetter)
- Fix a bug in the StateDB library that may have caused stale read after write. This may have potentially affected the L2 announcements feature and the node address selection. (#31164, @joamaki)
- Fix a bug that could cause local packet delivery to be skipped, leading to lower performance, when IPsec was enabled and
--devices
provided. (#31345, @pchaigno) - Fix a bug where pod label updates are not reflected in endpoint labels in presence of filtered labels. (#31395, @tklauser)
- Fix the logic of the api-server connectivity check for the kubernetes probe (#31019, @tkna)
- fix: Delegated ipam not configure ipv6 if ipv6 disabled in agent (#31104, @tamilmani1989)
- Fixed issue when updated nodes were being reported with unknown connectivity status in health report (#30917, @marseel)
- Fixed issue with assigning 0 nodeID when corresponding bpf map run out of space.
Potentially it could have impacted connectivity in large clusters (>4k nodes) with IPSec or Mutual Auth enabled.
Otherwise, it was merely generating unnecessary error log messages. (#31380, @marseel) - fqdn: Fixed bug that caused DNS Proxy to be overly restrictive on allowed DNS selectors. (#31328, @nathanjsweet)
- gateway-api: Ensure hostname check when set on both the HTTPRoute and the Gateway Listener (#30686, @cjvirtucio87)
- gateway-api: fixed RequestRedirect picks wrong port with multiple listeners (#31361, @chaunceyjiang)
- gateway-api: Retrieve LB service from same namespace (#31271, @sayboras)
- gateway-api: shorten the length of the value of the svc's label. (#31292, @chaunceyjiang)
- helm: Update pod affinity for cilium-envoy (#31150, @sayboras)
- hubble/relay: Fix certificate reloading in PeerManager (#31376, @glrf)
- hubble: fix parsing of invalid HTTP URLs (#31100, @kaworu)
- Hubble: fix traffic direction and is reply when IPSec is enabled (#31211, @kaworu)
- ingress/gateway-api: sort virtual hosts in CEC (#31493, @mhofstetter)
- ingress/gateway-api: stable envoy listener filterchain sort-order (#31572, @mhofstetter)
- k8s/utils: correctly filter out labels in StripPodSpecialLabels (#31421, @tklauser)
- metric: Avoid memory leak/increase in cilium-agent (#31714, @sayboras)
- metrics: Disable prometheus metrics by default (#31144, @joestringer)
- operator: fix errors/warnings metric. (#31214, @tommyp1ckles)
- Updated Kernel parsing to handle single and double digit kernel version as well (#30699, @MeherRushi)
CI Changes:
- Additionally test host firewall + KPR disabled in E2E tests (#30914, @giorio94)
- AKS: avoid overlapping pod and service CIDRs (#31504, @bimmlerd)
- bgpv1: avoid object tracker vs informer race (#31010, @bimmlerd)
- bgpv1: fix Test_PodIPPoolAdvert flakiness (#31365, @rastislavs)
- bgpv2/ci: added watch reactor for bgp cluster config (#31381, @harsimran-pabla)
- bpf: fix go testdata check in ci (#31419, @mhofstetter)
- Checkout the target branch, instead of the default one, on pull_request based GHA test workflows (#31198, @giorio94)
- ci-e2e: Add e2e test with WireGuard + Host Firewall (#31594, @qmonnet)
- ci-e2e: Add matrix for bpf.tproxy and ingress-controller (#31272, @sayboras)
- ci/ipsec: Print more info to debug credentials removal check failures (#31652, @qmonnet)
- ci: Bump lvh-kind ssh-startup-wait-retries (#31387, @YutaroHayakawa)
- ci: check license of third party Go dependencies (#31129, @rolinh)
- ci: fail container scans on vulnerability scan results (#31092, @ferozsalam)
- contrib/scripts: Remove false positives from check-go-testdata.sh (#31089, @dylandreimerink)
- deflake endpointmanager tests (#31488, @bimmlerd)
- Drop legacy and superseded test from the Ginkgo suite (#31411, @giorio94)
- Drop the remaining references to the CILIUM_CLI_MODE environment variable in GHA workflows. (#31199, @giorio94)
- gateway-api: Enable GRPCRoute conformance tests (#31055, @sayboras)
- gh/workflows: Add IPsec key rotation action and use it in ci-eks / ci-ipsec-e2e (#29704, @brb)
- gh: workflows: clarify reference to issue #23283 (#31118, @julianwiedmann)
- gha: disable fail-fast on integration tests (#31420, @giorio94)
- gha: fix coredns logs retrieval in conformance-clustermesh (#31509, @giorio94)
- gha: Remove manual device setting (#31435, @sayboras)
- gha: retrieve additional coredns-related troubleshooting info (#31384, @giorio94)
- introduce ARM github workflows (#31196, @aanm)
- ipam: deepcopy interface resource correctly. (#26998, @tommyp1ckles)
- k8s_install.sh: specify the CNI version (#31182, @aanm)
- loader: fix issue where errors cancelled compile cause error logs. (#30988, @tommyp1ckles)
- Make BPF unit tests reproducible (#31526, @ti-mo)
- Make testdata build output more stable by reducing header includes (#31644, @ti-mo)
- renovate: temporarily do not update GoBGP (#31123, @rastislavs)
- slices: don't modify missed input slice in test (#31119, @bimmlerd)
- test/verifier: Keep existing environment when running make (#31632, @gentoo-root)
- test/verifier: Sort BPF program names for stable output (#31617, @gentoo-root)
- test: Update KPR value in ipsec upgrade jobs (#31649, @sayboras)
- update azure k8s versions (#31220, @brlbil)
- workflows: Cover IPsec encrypted overlay mode in end-to-end tests (#31637, @pchaigno)
- workflows: Debug info for key rotations (#31627, @pchaigno)
- workflows: ipsec-e2e: add missing key types for some configs (#31636, @julianwiedmann)
Misc Changes:
- Add monitor aggregation for all events related to packets ingressing to the network-facing device. (#31015, @learnitall)
- Add the documentation for using
serviceAdvertisements
(#31331, @chaunceyjiang) - agent: Remove redundant pod spec checks (#31105, @aditighag)
- agent: Wrap propagating errors from proxy wait group (#31398, @aditighag)
- all: remove repetitive words (#31566, @deterclosed)
- api: Upgrade go-swagger version to v0.30.5 (#31647, @sayboras)
- Avoid depending on sysctl in the kind.sh script for IPv6 determination (#31180, @giorio94)
- bgpv1: Adjust ConnectionRetryTimeSeconds to 1 in component tests (#31218, @YutaroHayakawa)
- bgpv1: Disable PodCIDR Reconciler for unsupported IPAM modes (#31181, @YutaroHayakawa)
- bgpv2: fix operator flaky test cases (#31255, @harsimran-pabla)
- bgpv2: Introducing pod cidr reconciler for bgpv2. (#30815, @harsimran-pabla)
- bgpv2: introducing PodIPPool reconciler (#31546, @harsimran-pabla)
- bgpv2: remove automatic bgp peering policy translation to new BGP CRDs. (#31252, @harsimran-pabla)
- bpf,config: Add ENABLE_LOCAL_REDIRECT_POLICY macro (#31098, @aditighag)
- bpf: add node_key to alignchecker (#31393, @julianwiedmann)
- bpf: Don't skip local delivery for plain-text packets when IPsec is enabled (#31193, @pchaigno)
- bpf: host: optimize from-host's ICMPv6 path (#31127, @julianwiedmann)
- bpf: lxc: also set from_tunnel for IPv6 CT entries (#30877, @julianwiedmann)
- bpf: nodeport: add nodeport_rev_dnat_ingress_ipv4_hook infra (#31244, @jibi)
- bpf: nodeport: clean up ct_state usage in nodeport_lb*() (#31427, @julianwiedmann)
- bpf: nodeport: don't forward host id in nodeport_lb4 (#31120, @jibi)
- bpf: nodeport: simplify CT entry validation in nodeport_lb*() (#31165, @julianwiedmann)
- bpf: update unreachable-tailcall.o after updating CILIUM_BUILDER_IMAGE (#31412, @mhofstetter)
- bpf: xdp: remove unused set_encrypt_dip() (#31367, @julianwiedmann)
- bugtool: Capture memory fragmentation info from /proc (#30966, @pchaigno)
- cec: move config property 'envoy-config-timeout' into hive config (#31086, @mhofstetter)
- chore(deps): update all github action dependencies (main) (#31282, @renovate[bot])
- chore(deps): update all github action dependencies (main) (#31443, @renovate[bot])
- chore(deps): update all github action dependencies (main) (#31573, @renovate[bot])
- chore(deps): update all github action dependencies (main) (#31697, @renovate[bot])
- chore(deps): update all github action dependencies (main) (patch) (#31130, @renovate[bot])
- chore(deps): update all lvh-images main (main) (patch) (#31131, @renovate[bot])
- chore(deps): update all lvh-images main (main) (patch) (#31230, @renovate[bot])
- chore(deps): update all lvh-images main to bpf-next-20240309.012251 (main) (patch) (#31276, @renovate[bot])
- chore(deps): update all lvh-images main to bpf-next-20240315.012542 (main) (patch) (#31440, @renovate[bot])
- chore(deps): update all-dependencies (main) (#31275, @renovate[bot])
- chore(deps): update cilium/cilium-cli action to v0.16.0 (main) (#31281, @renovate[bot])
- chore(deps): update cilium/little-vm-helper action to v0.0.17 (main) (#31695, @renovate[bot])
- chore(deps): update dependency cilium/cilium-cli to v0.16.0 (main) (#31171, @renovate[bot])
- chore(deps): update dependency cilium/cilium-cli to v0.16.3 (main) (#31386, @renovate[bot])
- chore(deps): update dependency cilium/cilium-cli to v0.16.4 (main) (#31673, @renovate[bot])
- chore(deps): update docker.io/library/golang:1.22.1 docker digest to 0b55ab8 (main) (#31438, @renovate[bot])
- chore(deps): update gcr.io/distroless/static-debian11:nonroot docker digest to 55c6361 (main) (#31439, @renovate[bot])
- chore(deps): update github/codeql-action action to v3.24.8 (main) (#31479, @renovate[bot])
- chore(deps): update go to v1.22.1 (main) (#31277, @renovate[bot])
- chore(deps): update golangci/golangci-lint docker tag to v1.57.1 (main) (#31576, @renovate[bot])
- chore(deps): update golangci/golangci-lint docker tag to v1.57.2 (main) (#31696, @renovate[bot])
- chore(deps): update hubble cli to v0.13.2 (main) (#31320, @renovate[bot])
- chore(deps): update module github.com/go-jose/go-jose/v3 to v3.0.3 [security] (main) (#31241, @renovate[bot])
- chore: update json-mock image source in examples (#31373, @loomkoom)
- cilium, bpf: pkts/byte count conversion for ct (#31087, @borkmann)
- cilium-dbg: listing load-balancing configurations displays L7LB proxy port (#31503, @mhofstetter)
- cilium: Enable plain IPIP/IP6IP6 termination (#31213, @borkmann)
- config: Remove unused
ENCRYPT_IFACE
macro (#31323, @pchaigno) - container/bitlpm: Add Lookup Boolean Return Value (#31037, @nathanjsweet)
- contrib: Add installation script for tools in devcontainer (#31534, @fujitatomoya)
- controller: Add and use lookup function for controllers (#31236, @christarazi)
- datapath, bpf: Remove unnecessary IPsec code (#31344, @pchaigno)
- dev: Enable IPv6 system setting for devcontainer environment. (#31268, @fujitatomoya)
- doc,bgpv1: Add some failure scenarios (#31249, @YutaroHayakawa)
- doc,bgpv1: Bootstrapping BGP CPlane failure scenario doc (#31153, @YutaroHayakawa)
- doc,bgpv1: More failure scenario and wording improvement (#31470, @YutaroHayakawa)
- doc: Clarified GwAPI KPR prerequisites (#31366, @PhilipSchmid)
- doc: Document APAC community meeting (#31461, @YutaroHayakawa)
- docs: aks: avoid overlapping service and pod CIDRs (#31543, @bimmlerd)
- docs: Correct dynamic hubble exporter sample configs example (#31445, @littlesheng19)
- docs: Document
No node ID found
drops in case of remote node deletion (#31635, @pchaigno) - docs: Fix 'kubectl exec' invocations (quotes, double dash separator) in example script kafka-sw-gen-traffic.sh (#30462, @saintdle)
- docs: Fix profiling related debugging instructions (#31044, @aditighag)
- docs: Fix various typos in README.rst (#31072, @payneInTheBrian)
- docs: ipsec: document native-routing + Egress proxy case (#31478, @julianwiedmann)
- docs: Suggest using operator logs for troubleshooting (#31500, @simonfelding)
- docs: Update link to cilium/ebpf's list of eBPF program types (#31699, @haiyuewa)
- docs: Update link to USERS.md in README from RAW Github to standard Github UI (#30589, @ondrejsika)
- docs: Warn on key rotations during upgrades (#31437, @pchaigno)
- Document the process for disabling workflows (#31603, @michi-covalent)
- Downgrade L2 Neighbor Discovery failure log to Debug (#31179, @YutaroHayakawa)
- endpointmanager: Improve health reporter messages when stopped (#31231, @christarazi)
- envoy: Bump golang version to 1.21.8 (#31224, @sayboras)
- envoy: cleanup istio specifics (#31448, @mhofstetter)
- envoy: move config values from global config into hive cell (#31351, @mhofstetter)
- envoy: Remove deprecated runtime key logs (#31108, @sayboras)
- envoy: support configurable Envoy base id in embedded mode (#31449, @mhofstetter)
- fix 'mismatch' typos in error messages (#31660, @julianwiedmann)
- Fix helm template for hubble-relay prometheus annotations (#31253, @glrf)
- Fix running tests locally in kind. (#31234, @gentoo-root)
- fix(deps): update all go dependencies main (main) (#31112, @renovate[bot])
- fix(deps): update all go dependencies main (main) (#31278, @renovate[bot])
- fix(deps): update all go dependencies main (main) (#31441, @renovate[bot])
- fix(deps): update all go dependencies main (main) (#31462, @renovate[bot])
- fix(deps): update google.golang.org/genproto/googleapis/rpc digest to a219d84 (main) (#31305, @renovate[bot])
- fix(deps): update google.golang.org/genproto/googleapis/rpc digest to c811ad7 (main) (#31322, @renovate[bot])
- fix(deps): update module github.com/docker/docker to v25.0.5+incompatible [security] (main) (#31531, @renovate[bot])
- gateway-api: Replace deprecated status (#31111, @sayboras)
- helm: Remove pipe in value comments to avoid breaking Helm reference (#31588, @qmonnet)
- helm: update nodeinit image using renovate (#31641, @tklauser)
- hive/cell/health: don't warn when reporting on stopped reporter. (#31262, @tommyp1ckles)
- hubble/relay/server: remove unused Server.stop chan (#31560, @tklauser)
- Ignore kvstore node events for the local node, to avoid unnecessarily increasing the ipcache_errors_total (cannot_overwrite_by_source) metric. (#31399, @giorio94)
- images/builder: get rid of annoying git ownership warnings (#31538, @ti-mo)
- images: bump cni plugins to v1.4.1 (#31347, @aanm)
- Improve compatibility with LLVM 17. (#31403, @gentoo-root)
- Improve compatibility with LLVM 17. (#31459, @gentoo-root)
- Improve insertNodeNeighbor behavior to report health (#29415, @derailed)
- Improve LocalNodeStore.Get() performance and fix possible deadlock (#31013, @giorio94)
- ingress/gateway-api: stable address order for Ingress hostnetwork listener addresses (#31477, @mhofstetter)
- ingress: sort all shared ingresses during model generation (#31494, @mhofstetter)
- ingress: Update docs with network policy example (#31060, @sayboras)
- IPAM: Refactors Node API Types to Support Separate IP Families (#30684, @danehans)
- ipam: Remove unused variable (#31401, @christarazi)
- ipcache: Remove synchronous CIDR identity allocation (#31311, @gandro)
- iptables: Manage IP sets independently with the stateDB reconciler (#31099, @pippolo84)
- iptables: Simplify proxy rules removing ingress/egress flag (#31068, @pippolo84)
- iptables: Unit tests cleanup (#31368, @pippolo84)
- kind: reset sysctl net.ipv4.ip_unprivileged_port_start to 1024 (#31370, @mhofstetter)
- lint: Remove temp variable in the 'for' loop (#31523, @sayboras)
- loader: add message if error is ENOTSUP (#31413, @kkourt)
- lxcmap: Fix comment about byte-order (#31362, @joestringer)
- Make it clear USERS.md should be production use cases (#31316, @xmulligan)
- Makefiles: Allow external input for go build/test/clean flags. (#29646, @wanlin31)
- Miscellaneous cleanups around node discovery (#31397, @giorio94)
- modularize node discovery (#31589, @dylandreimerink)
- multicast: modify list operations from iterator to batch lookup. (#31562, @harsimran-pabla)
- node: add support for injection of optional ipset filter (#31550, @giorio94)
- node: Replace ipv[46]MasqAddrs with Table[NodeAddress] (#30457, @joamaki)
- pkg/ip: Updates PrefixToIps() to Limit the Number of Returned IPs (#30921, @danehans)
- policy/k8s: Refactor and move
ToServices
translation to policy package (#31062, @gandro) - policy: Fix missing labels from SelectorCache selectors (#31358, @christarazi)
- Prepare for release v1.16.0-pre.0 (#31121, @aanm)
- proxy: configurable portrange (#31556, @mhofstetter)
- proxy: remove unused ifaces and code for proxy <-> endpoint interaction (#31547, @mhofstetter)
- README: Update releases (#31665, @thorn3r)
- Remove
HAVE_LARGE_INSN_LIMIT
(#31094, @dylandreimerink) - Remove Istio ambient compatibility blurb (#31525, @bleggett)
- Remove old bpf feature probes (#31096, @dylandreimerink)
- Remove tcx links created by Cilium 1.16 onwards (#31553, @ti-mo)
- renovate: Drop references to Cilium 1.12 (#31148, @joestringer)
- renovate: separate major.minor.patch for lvh images (#31126, @aanm)
- secret-sync: improve logging (#31415, @mhofstetter)
- signal: remove spare debug logs (#31723, @tklauser)
- stream: Relocate to cilium/stream (#30846, @joamaki)
- update readme with 1.16.0-pre.0 (#31128, @aanm)
Docker Manifests
cilium
quay.io/cilium/cilium:v1.16.0-pre.1@sha256:f822fed7e9ab9ef9251e3e21eaf6d4d5179a6b5831e147c3ab1caaa3f9b17b79
clustermesh-apiserver
quay.io/cilium/clustermesh-apiserver:v1.16.0-pre.1@sha256:6489a11ebdf28be5238842afaea4e5e2a9628e8c4fb66d712b3998fb1bfa034b
docker-plugin
quay.io/cilium/docker-plugin:v1.16.0-pre.1@sha256:0540dce44dc09dd54cbb1a665736664913dc242b9bca261fb138b8ac6de3aa8e
hubble-relay
quay.io/cilium/hubble-relay:v1.16.0-pre.1@sha256:80a213c50bc9915b73950c2efbbc04a32ab2df5058e0d5afe86c64d83a59cc2d
operator-alibabacloud
quay.io/cilium/operator-alibabacloud:v1.16.0-pre.1@sha256:9237c6dfc208e5f76c01922932d3c568f269356f485076a62c9a503d1af76710
operator-aws
quay.io/cilium/operator-aws:v1.16.0-pre.1@sha256:bf75d57fcfd1fb0b6ad8c6257e0758872278609847640fc4245cd04be139d7fd
operator-azure
quay.io/cilium/operator-azure:v1.16.0-pre.1@sha256:099fb5537d294bdf41755f93acbf8c6e2ecbca162b139028b4897f2904e04e4b
operator-generic
quay.io/cilium/operator-generic:v1.16.0-pre.1@sha256:73e8c7a415dfd3c6bb166848248c719ced5db53123c0f29c77e08771d1ec8400
operator
quay.io/cilium/operator:v1.16.0-pre.1@sha256:eb3303b6290ee9b06da28c383a65c680d03bc2028f6bdc046d5f1494eb5a485c