This is the first beta of the 1.29 release.
Significant changes
Deferred deletion / pruning phase
Some infrastructure changes are potentially disruptive to the continued
operation of the cluster. For the most disruptive operations, particularly
those that break rolling-update of the cluster, we have started to use deferred
deletion to minimize the impact. For example, on AWS we create a second NLB
during the kops update
phase when we cannot change the NLB directly.
kops update
will report that a --prune
is needed. To minimize disruption,
we recommend you perform this after a rolling-update, for example:
kops update $MYCLUSTER --yes --admin
kops rolling-update $MYCLUSTER --yes
kops update $MYCLUSTER --yes --admin --prune # NEW!
Deferred deletion is currently used to safely introduce security groups for NLBs on AWS,
and to move to an internal load balancer for kops-controller
on GCP.
Initial OpenTelemetry Support
We are starting to add (experimental) support for OpenTelemetry,
in particular Tracing support. Setting OTEL_EXPORTER_OTLP_TRACES_FILE
will write a trace file which can then be read by the traceserver program.
More information and options are described in docs/opentelemetry.md.
The tracing data is not expected to be particularly useful for end-users in
this release; the (non-standard) recording approach is instead intended to
work well with our Prow end-to-end testing system so that developers can
optimize kOps.
Please note: this is not telemetry in the "phone-home" sense.
The kOps project does not collect data from your machine. As an
open-source project we do not even want to collect any of your data.
Currently the only OpenTelemetry backend supported is writing to a
filesystem (and it is opt-in). In future you will be able to configure
other OpenTelemetry backends, but this data will only be sent if
you enable OpenTelemetry, and only sent to where you configure.
AWS
-
Network Load Balancers in front of the Kubernetes API and bastion hosts now
have a security group attached. These security groups are used for security group rules
allowing incoming traffic to the NLBs as well as traffic between the NLBs and their target
instances. -
Posts event data to URL upon instance interruption action in aws-node-termination-handler with
WEBHOOK_URL
.
GCP
-
As of Kubernetes version 1.29, credentials for private GCR/AR repositories will be handled by the out-of-tree credential provider. This is an additional binary that each instance downloads from the assets repository.
-
We now use a private load-balancer for in-cluster traffic on GCP, which allows us
to use network tags to restrict access only to the cluster nodes.
Breaking changes
-
kops toolbox dump
limits the number of nodes dumped to 500 by default. Use--max-nodes
to override. -
Support for Kubernetes version 1.23 has been removed.
Known Issues
- The Amazon VPC CNI is now compatible with Ubuntu 22.04. Fix applied via kubernetes/kops#16313.
Deprecations
-
Support for Kubernetes version 1.24 is deprecated and will be removed in kOps 1.30.
-
Support for Kubernetes version 1.25 is deprecated and will be removed in kOps 1.31.
-
Support for AWS Classic Load Balancer for API is deprecated and should not be used for newly created clusters.
-
All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.
What's Changed
kops validate cluster
improvements by @upodroid in #16187- gce: Remove custom resolver by @hakman in #16189
- skip_regex.go: kube-router add back in service afinity test by @aauren in #16188
- chore(channels): bump k8s and ubuntu ami versions in alpha channel by @moshevayner in #16190
- Update Calico to v3.27.0 by @hakman in #16192
- Disable Statefulsets provisioning from CL2 Load Tests by @hakuna-matatah in #16172
- Add cert-manager resource types to kubetest2-kops artifacts by @rifelpet in #16193
- Parallelize k8s resource dumps with kops toolbox dump by @rifelpet in #16196
- Include pod logs in toolbox dump by @rifelpet in #16198
- Update k8s.io/* to v0.29.0 by @hakman in #16199
- Update dependencies by @github-actions in #16201
- test: Print the
create cluster
command by @hakman in #16202 - scale-test: Add feature flag for creating a single nodes instance group by @hakman in #16203
- Dump previous and current container logs separately by @hakman in #16200
- aws: Set AWS_REGION env var for ebs-csi-node and ebs-csi-driver by @hakman in #16206
- aws: Add option for setting QPS and Burst for EBS CSI Driver by @hakman in #16207
- Spotinst: Bump controller version to 1.0.97 by @yehielnetapp in #16208
- feat: add us-west zone for hetzner by @finzzz in #16209
- Check if kubeconfig exists before dumping resources by @hakman in #16205
- Promote alpha to stable by @moshevayner in #16210
- aws: Use instance metadata to get warm pool state by @rifelpet in #16213
- Dump and redact secrets by @rifelpet in #16211
- Update to Cilium 1.14.5 by @hakman in #16214
- Allow override of the DNS domain used by the tests. by @ameukam in #16217
- aws: Retrieve instance info only when max pods is not set by @hakman in #16216
- Add permission needed for service-linked role creation by @ameukam in #16219
- Remove kube-system cert-manager webhook exclusion by @rifelpet in #16221
- Jaeger tracing visualizer improvements by @rifelpet in #16220
- Update dependencies by @github-actions in #16224
- aws: Set provider ID when starting kubelet by @hakman in #16223
- scale-test: Reduce validation count and interval by @hakman in #16225
- aws: Update EBS CSI driver to v1.26.0 by @hakman in #16227
- Add option for setting CCM ConcurrentNodeSyncs by @hakman in #16228
- aws: Skip deleting ASG instances without volumes by @hakman in #16229
- Make cluster deletion configurable by @hakman in #16231
- Bump actions/dependency-review-action from 3.1.4 to 3.1.5 by @dependabot in #16232
- scale-test: Use single nodes instance group for AWS by @hakman in #16204
- Bump kubetest2 by @ameukam in #16234
- test: Improve cluster deletion defaults by @hakman in #16236
- Replace
k8s.io/utils/strings/slices
withgolang.org/x/exp/slices
by @hakman in #16238 - aws: Update EBS CSI driver by @hakman in #16239
- aws: Use
domain
instead ofvpc
when renderingaws_eip
by @hakman in #16237 - Bump GCP terraform provider to latest by @rifelpet in #16242
- docs: fix broken example command by @markusleh in #16243
- Update dependencies by @github-actions in #16244
- Revert "aws: Skip deleting ASG instances without volumes" by @hakman in #16246
- Update Go to v1.21.6 by @hakman in #16245
- Prefer external endpoints when building kubeconfig by @justinsb in #16248
- aws: Terminate ASG instances in batches of 100 instances by @hakman in #16251
- aws: Ignore InvalidRouteTableID.NotFound errors during cluster deletion by @hakman in #16252
- aws: fix maxPods when cilium ipam=eni is used by @argusua in #16253
- Update containerd to v1.7.12 by @hakman in #16257
- Switch to GCS url for upgrades tests by @ameukam in #16258
- Use dns=none for newly created clusters including for AWS and GCE by @hakman in #16262
- Update aws-sdk-go to v1.49.24 by @ameukam in #16263
- test: Set num-nodes flag by @upodroid in #16176
- Refactor: Replace ForAPIServer with WellKnownServices by @justinsb in #15829
- gce: fix nlb firewall rules, operations and alias network subnets by @upodroid in #16265
- build(deps): bump actions/dependency-review-action from 3.1.5 to 4.0.0 by @dependabot in #16267
- openstack: Include kube-apiserver controlplane ports in dns=none by @zetaab in #16271
- Increase CCM workers to speed up node bootstrap process by @hakuna-matatah in #16256
- Add 1.28 release notes to docs menu by @yurrriq in #16275
- Add support to configure HPA Controller concurrent syncs flag in HPA/KCM Controller by @hakuna-matatah in #16277
- Add support to configure Job Controller concurrent syncs flag in Job… by @hakuna-matatah in #16280
- Refactor: Plumb context through GCE firewallRule methods by @justinsb in #16281
- Fix dumping logs for GCE scale tests by @upodroid in #16266
- Add
boskos-resource-type
flag to use different GCE projects for scale/gpu testing by @upodroid in #16268 - OpenStack: update CSI images by @zetaab in #16283
- toolbox dump: output correct type for target groups by @justinsb in #16285
- chore(channels): bump k8s and ubuntu ami versions in alpha channel by @moshevayner in #16284
- Fix: support comparison of int types in dry-run by @justinsb in #16290
- refactor: NetworkLoadBalancer Name should match Name tag by @justinsb in #16288
- tweak: Set Scheme on NLB tasks for public load balancers by @justinsb in #16289
- refactor: Introduce runTests helper method into aws tests by @justinsb in #16292
- Refactor: Move NLB listing function into awsup by @justinsb in #16295
- chore(networking): bump aws cni to 1.16.2 by @moshevayner in #16297
- Revert "Don't set LimitNoFile for containerd systemd unit file" by @zetaab in #16300
- Update runc & containerd by @zetaab in #16302
- chore(channels): promote alpha to stable by @moshevayner in #16306
- refactor: wait for load balancer readiness using a private field by @justinsb in #16294
- Add GCE scale testing on kops by @upodroid in #16181
- fix(nodeup): set
MACAddressPolicy=none
when using AWS VPC CNI by @moshevayner in #16313 - Upgrade AWS Load Balancer Controller to v2.7.0 by @yurrriq in #16316
- Update to cilium 1.15 by @zadjadr in #16315
- feat: added image minimum and maximum gc age by @Lerentis in #16318
- build(deps): bump peter-evans/create-pull-request from 5.0.2 to 6.0.0 by @dependabot in #16322
- Update dependencies by @hakman in #16323
- Skip gen-cli-docs on depup by @hakman in #16321
- Refactor: Split out NLB Listener into its own task by @justinsb in #16299
- refactor: Drop TargetGroups from NetworkLoadBalancer task by @justinsb in #16324
- Dont set -num-nodes on karpenter-managed clusters by @rifelpet in #16325
- docs: Remove warning about Amazon VPC CNI not being compatible with Ubuntu 22.04 by @moshevayner in #16326
- Set LimitNOFILE to 1048576 instead of
infinity
by @dims in #16329 - azure: Migrate to the new SDK version by @hakman in #16286
- Update dependencies by @github-actions in #16331
- Set KUBECONFIG for LBC's ginkgo tests by @rifelpet in #16334
- Docs: fix typos in office hours page by @justinsb in #16337
- clockmock: Add more methods that take a context by @justinsb in #16338
- Move DNS topology setup earlier in cluster creation by @rifelpet in #16342
- deletion: tolerate concurrent SQS queue deletion by @justinsb in #16341
- Cleanup import of the same package in tests by @justinsb in #16343
- validation: Allow overlap of pod/node CIDR and service CIDR by @justinsb in #16344
- Include /etc/hosts coredns mounts for dns=none clusters by @rifelpet in #16347
- azure: Replace lb.ForAPIServer with lb.WellKnownServices by @hakman in #16348
- Add support for AL2023 AMI to use Amazon VPC CNI by @dims in #16350
- aws: Post event data to URL upon instance interruption action by @voriol in #16009
- Refactor IAM Policy Builder by @rifelpet in #16351
- create command: remove example docs say is not implemented yet. by @jrabbit in #16308
- target group: refactor discovery into awsup by @justinsb in #16339
- Use IAM Policy Builder for SQS Queue Policy by @rifelpet in #16353
- refactor: Introduce DeletionProcessingMode by @justinsb in #16293
- Update Go to v1.22.0 by @hakman in #16346
- Update dependencies by @github-actions in #16357
- azure: Avoid spurious changes in VirtualNetwork by @hakman in #16358
- Generate revisions of NLB objects, and introduce cleanup phase by @justinsb in #16356
- gce: Update GCE storage service scope to DevstorageFullControlScope to resolve permission error. by @sl1pm4t in #16355
- add support for devcontainer by @remyleone in #16186
- azure: Mark a few tasks as implementing HasAddress by @justinsb in #16359
- Set --dns=none on upgrade tests from older kops versions by @rifelpet in #16360
- build(deps): bump actions/dependency-review-action from 4.0.0 to 4.1.0 by @dependabot in #16361
- Update Cilium to v1.15.1 by @hakman in #16362
- Fix bash conditional pattern matching in upgrade script by @rifelpet in #16364
- devcontainer: update go version, use features by @justinsb in #16365
- Skip known-failing test on most e2e jobs by @rifelpet in #16368
- aws: Update EBS CSI driver to v1.28.0 by @hakman in #16369
- doc/aws: Add space before the k8s slack url by @tungbq in #16370
- Skip hostname test for all aws jobs by default by @rifelpet in #16373
- Migrate many-addons e2e template to dns=none by @rifelpet in #16374
- Update dependencies by @github-actions in #16375
- gce: match IP addresses including subnet where relevant by @justinsb in #16380
- chore: update dependencies in submodules by @justinsb in #16372
- GCE: Use internal load balancer for node to control-plane traffic by @justinsb in #16379
- Skip hostname e2e test on digitalocean by @rifelpet in #16381
- build(deps): bump actions/dependency-review-action from 4.1.0 to 4.1.3 by @dependabot in #16384
- gce: Limit health check names to 63 chars by @hakman in #16385
- gce: Limit backend names to 63 chars by @hakman in #16386
- Update NVIDIA Container Toolkit URL by @elezar in #16387
- Install nerdctl and crictl on nodes by @h3poteto in #16383
- Continue attemps to dump artifacts in
toolbox dump
by @rifelpet in #16389 - chore: update boilerplate.py to recognize new build tags by @justinsb in #16390
- Add validation to help users move from usePolicyConfigMap by @hakman in #16391
- Experimental limited support for cluster-api by @justinsb in #15522
- Update dependencies by @github-actions in #16392
- build(deps): bump peter-evans/create-pull-request from 6.0.0 to 6.0.1 by @dependabot in #16393
- e2e tests: When upgrading, wait for the new configuration by @justinsb in #16395
- Fix shellcheck warnings for bootstrap script by @hakman in #16394
- Update dependencies by @github-actions in #16397
- Use github.com/go-viper/mapstructure/v2 by @ameukam in #16402
- docs: Update relnotes for 1.29 for deferred deletion by @justinsb in #16404
- aws: Expose port 8443 when using NLB with a custom certificate by @justinsb in #16403
- gce: Change default storage class to balanced-csi by @sl1pm4t in #16269
- gce: Set node IP Alias range to match NodeCIDRMaskSize by @sl1pm4t in #16272
- Release 1.29.0-beta.1 by @justinsb in #16406
New Contributors
- @finzzz made their first contribution in #16209
- @markusleh made their first contribution in #16243
- @argusua made their first contribution in #16253
- @Lerentis made their first contribution in #16318
- @voriol made their first contribution in #16009
- @jrabbit made their first contribution in #16308
- @tungbq made their first contribution in #16370
Full Changelog: v1.29.0-alpha.3...v1.29.0-beta.1