Highlights
Ray History Server (beta)
The Ray History Server is graduating to beta. Introduced as alpha in KubeRay v1.6, the History Server lets you view the Ray Dashboard and debug ephemeral clusters (such as those managed by RayJob) even after they have been terminated.
Since v1.6, the History Server has received major improvements:
- Performance and scalability: lazy loading, an LRU byte-based session cache, a disk-first event storage pipeline, and event compression significantly reduce memory usage for both the collector and the server.
- Ray token authentication support for both the collector and live-cluster proxying.
- Better UI and API coverage: a cluster selection page, task logs, and Serve / placement group / Ray Data endpoints polled by default.
- History Server images are now published to quay.io/kuberay (
historyserver:v1.7.0andcollector:v1.7.0)
Try it here: History Server Quick Start Guide.
Automatic History Server Sidecar Configuration (alpha)
The operator can now automatically inject the History Server event collector sidecar into your RayCluster’s pods, no manual sidecar configuration needed. Enable the RayClusterHistoryServer feature gate on the operator and set spec.historyServerOptions on your RayCluster:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: raycluster-historyserver
spec:
historyServerOptions:
collectorOptions:
image: quay.io/kuberay/collector:v1.7.0
See ray-cluster.historyserver.yaml for a full example.
⚠️ Warning: RayClusterHistoryServer is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterHistoryServer=true
RayService Incremental Upgrade graduates to Beta
The RayServiceIncrementalUpgrade feature gate is now beta and enabled by default. This feature allows RayService to perform zero-downtime upgrades by gradually shifting traffic to the new cluster using the Kubernetes Gateway API, instead of requiring 2x resources for a blue/green switch.
Since its Alpha release, the feature has received several improvements for more reliable and robust upgrades, including:
- Rollback support: Users can now revert the RayService spec mid-upgrade, with safe traffic shifting between the pending and active clusters.
- Improved reliability: Improved capacity management, traffic migration, failure handling, and cluster cleanup during incremental upgrades.
- E2E test coverage: Expanded test scenarios to improve coverage.
mTLS support via cert-manager (alpha)
KubeRay v1.7 can automatically provision and rotate certificates for mutual TLS between RayCluster’s pods using cert-manager.
⚠️ Warning: RayClusterMTLS is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterMTLS=true
Enable mTLS feature with following to try this feature:
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: raycluster-mtls
spec:
rayVersion: '2.55.1'
tlsOptions:
enabled: true
See ray-cluster.mtls.yaml for a full example.
NetworkPolicy support for RayCluster (alpha)
KubeRay v1.7 adds a new spec.networkPolicy field to RayCluster for network isolation. When set, the operator creates separate NetworkPolicies for head and worker pods, with configurable modes (DenyAll, DenyAllIngress, DenyAllEgress) and per-worker-group ingress/egress rules. Intra-cluster pod-to-pod traffic is always permitted.
Try it here: Network Policy Quick Start Guide.
⚠️ Warning: RayClusterNetworkPolicy is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=RayClusterNetworkPolicy=true
Below is an example of enabling network policy feature:
apiVersion: ray.io/v1
kind: RayCluster
spec:
networkPolicy:
mode: DenyAll
head:
ingressRules:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: <prometheus-namespace>
ports:
- port: 8080
protocol: TCP
See ray-cluster.network-policy-deny-all.yaml for an example.
GCS fault tolerance with embedded storage (alpha)
GCS fault tolerance no longer requires an external Redis. With the GCSFaultToleranceEmbeddedStorage feature gate enabled, you can persist GCS metadata in an embedded RocksDB store backed by a persistent volume on the head Pod:
spec:
gcsFaultToleranceOptions:
backend: rocksdb
storage:
size: 1Gi
⚠️ Warning: GCSFaultToleranceEmbeddedStorage is an Alpha feature and is disabled by default. To enable it, set the feature gate on the kuberay-operator:
--feature-gates=GCSFaultToleranceEmbeddedStorage=true
See ray-cluster.embedded-gcs-ft.yaml for a full example.
Other Notable Features
- RayCluster now supports built-in Ingress configuration via
headGroupSpec.ingressOptionswithhost,path,pathType, andtlsfields. - RayCronJob now supports the
spec.timeZonefield, and child RayJobs use deterministic names to prevent duplicate firings. - The WorkerGroup API includes a new
priorityfield for scheduler integrations. - A new alpha feature gate
SidecarSubmitterRestartprevents premature job termination during transient head Pod resource spikes in RayJob sidecar mode. - The operator configuration supports
defaultPodAnnotationsanddefaultPodLabelsapplied to all Ray Pods. - Memory usage improvements in the operator: Pod informer cache selectors and more efficient pod resource calculation.
kubectl ray logssupport--gke-linkto print a Cloud Logging link on GKE.- Upgraded to Go 1.26; Grafana dashboards updated for Ray 2.56.
Breaking Changes
- Helm: in the kuberay-operator chart,
metrics.serviceMonitor.selectorhas been renamed tometrics.serviceMonitor.additionalLabels, and empty labels are no longer emitted (#4979). Update your values file if you set this field. - History Server (alpha to beta): configuration has changed since the v1.6 alpha. S3 credentials now use the standard AWS environment variables (
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, ...) instead ofAWS_S3ID/AWS_S3SECRET/AWS_S3TOKEN(#4665), the runtime-class flag has been replaced by--storage-backend(#5085), and the on-storage event layout has changed (#4670). Redeploy the collector and history server together on v1.7 and see the updated setup guide.
Deprecations
- The
ray.io/v1alpha1API version is now marked deprecated (#5122). Please migrate toray.io/v1.
Changelog
- Disable RayMultiHostIndexing feature for TestReconcile_Multihost_Replicas by @Future-Outlier in #4583
- Disable the field alignment lint check by @jinbum-kim in #4560
- Mark RC releases as pre-release in GoReleaser by @Future-Outlier in #4590
- [RayCluster] Add example YAML for manually enabling Ray k8s auth by @andrewsykim in #4582
- [RayService] Rollback Support for Incremental Upgrades by @ryanaoleary in #4109
- [RayService] Promote Incremental Upgrade Feature to Beta by @ryanaoleary in #4599
- Revert "[RayService] Promote Incremental Upgrade Feature to Beta" by @Future-Outlier in #4602
- [Github Action] Skip krew-index update for pre-release tags by @Future-Outlier in #4587
- [Helm] update ray-cluster chart to apiVersion: v2 by @mboersma in #4593
- Update kind configs and docs to v1.29.0 by @mboersma in #4595
- [Helm] remove ingress template support for k8s < 1.19 by @mboersma in #4591
- [Helm] Update ray-cluster default resource values by @Future-Outlier in #4588
- [Chore] Use RayCluster name as ServiceAccount name for RBAC authentication by @JiangJiaWei1103 in #4611
- [History Server] Add guide to push image to Google Artifact Registry for History Server images by @chiayi in #4618
- [History Server] Fix API response format to match Ray Dashboard frontend schema by @Future-Outlier in #4615
- [Test][Release] Change upgrade test version to test upgrade from 1.5.1 to 1.6.0 by @Future-Outlier in #4623
- [History Server] Typo and minor fix for image guide readme by @chiayi in #4621
- [Fix] Align error msg with the head and serve svc behavior by @JiangJiaWei1103 in #4614
- [Refactor] [history server] Rename event collector to align with REP by @JiangJiaWei1103 in #4574
- [fix] Fix flaky RayJob e2e test by cleaning up resources after fail subtest by @seanlaii in #4603
- [Fix] [Flaky] Use head node ID to get event_JOBS logs by @JiangJiaWei1103 in #4626
- [Test] Add load tests and behavioral checks to incremental upgrade E2E by @JiangJiaWei1103 in #4541
- Remove FieldIndexer on PodUID that is not used in raycluster controller by @troy0820 in #4629
- [KubeRay DashBoard] Fix history server rayjob yaml and raycluster yaml cause error by @CheyuWu in #4640
- [History Server] Add Google Cloud storage history server deployment example by @chiayi in #4641
- Update the codegen script to be vendor-aware by @jinbum-kim in #4394
- [History Server] Add Google Cloud Storage RayCluster manifest example for History Server by @chiayi in #4642
- [History Server] Fix image & env var for Google Cloud Storage example yaml. by @chiayi in #4655
- [Helm] use service.ports list for apiserver ingress backend port by @mboersma in #4609
- [ray-operator] add example YAML for using Redis as a sidecar with persistence by @andrewsykim in #4657
- [historyserver] Fix S3 client to use default credential chain when static credentials are not provided by @ashutosh1807 in #4653
- [Fix] [Flaky] Insert firewall rules to the correct table by @JiangJiaWei1103 in #4662
- [ray-operator] remove TODO for setting service account token audiences by @andrewsykim in #4666
- [Refactor] Use IsJobDeploymentTerminal for EndTime update by @JiangJiaWei1103 in #4646
- [Refactor] [history server] Align
tasks/timelinegeneration with Ray Dashboard by @AndySung320 in #4546 - [Fix] Fail RayService if StepSizePercent exceeds MaxSurgePercent by @JiangJiaWei1103 in #4663
- [historyserver] Migrate S3 credentials to standard AWS environment variables by @ashutosh1807 in #4665
- [History Server] Change RayClusterID on collector side to RayClusterNamespace by @chiayi in #4673
- [Fix] [RayService] Check serve label before iptables injection in upgrade test by @Future-Outlier in #4677
- Example high throughput LLM serving RayService by @spencer-p in #4684
- [Fix] Add timeout to Ray HTTP proxy client for k8s proxy mode by @JiangJiaWei1103 in #4680
- feat(historyserver): upgrade Aliyun OSS Go SDK from v1 to v2 by @ChenYi015 in #4687
- CARRY: refactor isopenshift check by @laurafitzgerald in #4365
- [kubectl plugin] kubectl ray logs --link=gke by @spencer-p in #4683
- chore(samples): remove unmaintained RayJob Modin example YAML by @EagleLo in #4700
- Fix file matching in Aliyun OSS GetContent by @utafrali in #4692
- [Docs] Clarify per-cluster serve service behavior by @JiangJiaWei1103 in #4699
- [RayService][e2e] fix flaky test in getting HTTPRoute by @fscnick in #4701
- [History Server] Fix /logical/actors response format to match Ray Dashboard API by @win5923 in #4563
- [RayService][e2e] fix flaky test TestAutoscalingRayService by @AndySung320 in #4702
- [golangci-lint] Enable deprecated API warnings linter by @mboersma in #4605
- [Config]: standardize quoting style in Kubernetes manifests (#4353) by @DejusDevspace in #4359
- [PodPool-VK] add main entry function for cache pod manager (#4250) by @hzyfox in #4584
- Quick fix for kubectl ray logs --link=gke by @spencer-p in #4725
- [Docs] Add History Server local development guide by @JiangJiaWei1103 in #4719
- fix delete volcano PodGroup unconditionally by @fscnick in #4669
- [History Server] refactor event collector to use path/filepath for local FS paths by @400Ping in #4514
- Avoid nil dereference by @dentiny in #4737
- Fix Issue #4686 (rayservice): update RayCluster when Suspend toggles by @lorriexingfang in #4739
- Include error in panic by @dentiny in #4747
- [Refactor][history server] Replace all hardcoded root temporary paths with path constants by @400Ping in #4548
- [ray-operator] Export additional client go metrics by @spencer-p in #4764
- fix: include metadata in RayJob submit command when using clusterSelector by @EagleLo in #4616
- RayService incremental upgrade reliability improvements by @ryanaoleary in #4601
- [Fix] [History Server] Normalize hex ID for data integrity by @JiangJiaWei1103 in #4775
- [Test] Fix race in DELETE_RAYJOB_CR_AFTER_JOB_FINISHES envtest by @marosset in #4790
- Avoid dashboard hard-code namespace to apiserver by @dentiny in #4794
- Upgrade to golang v1.26 from v1.25 by @hango880623 in #4597
- [RayJob] revert 4160 background goroutine get job info by @fscnick in #4766
- Go 1.26 and K8s 1.36 client-go updates by @marosset in #4703
- [RayJob][volcano] consistent naming on the return of calculatePodGroupParams by @fscnick in #4734
- Add Pod informer cache selector to reduce memory usage by @AndySung320 in #4761
- [RayJob] Add GCS health wait to K8sJobMode submitter by @seanlaii in #4798
- fix: HistoryServer unit tests failure caused by ID normalization in ConvertBase64ToHex by @kenchung285 in #4769
- Update community meeting info in README.md by @marosset in #4803
- [RayJob] Replace DefaultReadinessProbeFailureThreshold with RayDashboardGCSHealthCheckTimeoutSeconds by @seanlaii in #4800
- update doc string for K8sControllerRuntimeCacheSelectors by @AndySung320 in #4802
- [kubectl-plugin][Refactor] Standardize namespace resolution to respect kubeconfig context. by @kash2104 in #4367
- [Doc] Update text-summarizer sample to ray-ml 2.49.2 by @kevin85421 in #4811
- Fix dashboard error display by @dentiny in #4806
- [Autoscaler][Bug] inject RAY_enable_autoscaler_v2=false when user set autoscalerOptions.version=v1 by @justinyeh1995 in #4810
- [history server] Display history task logging by @dentiny in #4818
- [CI] Add History Server Build and Test to Github Actions by @kenchung285 in #4776
- Add RayService finalizer by @yaroslava-serdiuk in #4807
- [Docs] Remove comments in JSON by @JiangJiaWei1103 in #4826
- [apiserver][test] Trust E2E_API_SERVER_RAY_IMAGE verbatim (closes #4346) by @1fanwang in #4770
- [CI] Enable dependabot for github actions by @dentiny in #4848
- Feat/add autoscaler start cmd by @fscnick in #4835
- [ray-operator] Allow disabling log_monitor via rayStartParams by @raulchen in #4857
- [1/N] [History Server Beta] [Feat] Lazy loading by @JiangJiaWei1103 in #4795
- Bump the all-actions group across 1 directory with 16 updates by @dependabot[bot] in #4854
- chore(deps): bump idna from 3.10 to 3.15 in /clients/python-client by @dependabot[bot] in #4861
- chore(deps): bump next from 15.4.10 to 15.5.18 in /dashboard by @dependabot[bot] in #4862
- chore(deps): bump ip-address from 10.0.1 to 10.2.0 in /dashboard by @dependabot[bot] in #4870
- [History Server] Add compression library for history server by @chiayi in #4828
- [ray-operator] support overriding ulimit open files with env var by @andrewsykim in #4860
- [RayService][Kueue] Support top-level Spec.Suspend for zero-downtime upgrade by @kevin85421 in #4841
- chore(deps): bump azure/setup-helm from 4.3.0 to 5.0.0 in the all-actions group by @dependabot[bot] in #4880
- Remove cpu limit for rayservice e2e test by @AndySung320 in #4859
- [History Server] Restructure metadir file structure to include owner kind and name by @chiayi in #4670
- [RayService][Kueue] Support top-level
Spec.Suspendfor incremental upgrade by @kevin85421 in #4881 - [ray-operator] Migrate from deprecated scheme.Builder to runtime.NewSchemeBuilder by @marosset in #4801
- [RayCronJob] Use deterministic child RayJob names to prevent duplicate firings by @tson1111 in #4888
- delete unused experimental folder by @andrewsykim in #4890
- [Chore][CI] Upgrade Ray version to 2.55.0 to support entrypointLabelSelector by @my-vegetable-has-exploded in #4884
- [RayJob][SidecarMode] Introduce Retry Mechanisms when K8s version >= 1.35 by @justinyeh1995 in #4399
- [CI] Make build-start-operator.sh fail fast on setup errors by @EagleLo in #4893
- Implement finalizer timeout for Ray Clusters with Redis FT by @ns-sundar in #4836
- [History Server] Followup fixes/updates for updating cluster-metadata dir by @chiayi in #4898
- [fix] fixing test issues in TestDeletionStrategy by @marosset in #4906
- [history server]Add cmdline flags to make qps and burst configurable by @KunWuLuan in #4821
- [History server] History server to cache cluster info by @chiayi in #4856
- [CI][Python Client] Retry operator build and extend readiness wait for transient flake by @EagleLo in #4873
- [History Server] [Refactor] Move and validate API server QPS and burst defaults by @JiangJiaWei1103 in #4912
- Update high throughput Ray Serve LLM samples by @spencer-p in #4917
- Update observedGeneration during suspend lifecycle in RayService by @AndySung320 in #4897
- [2/N] [History Server Beta] [Feat] LRU cache by @JiangJiaWei1103 in #4796
- [History Sever] Use compression library in collector and historyserver by @chiayi in #4899
- [ray-operator] add sample for using agent-sandbox for sandboxing by @vicentefb in #4911
- [History server] fix mismatch func signature by @fscnick in #4930
- [Feature] Add NetworkIsolation support to RayClusters by @chipspeak in #4638
- chore: upgrade controller-runtime from pseudo-version to v0.24.1 by @chipspeak in #4941
- [ray-operator] Clean up batch scheduler resources on suspend by @wyj15 in #4940
- [Feat] Add RayCronJob timezone support by @machichima in #4762
- [ray-operator] CalculateMinResources should respect workerGroupSpec.Suspend by @aditya-786 in #4958
- Fix dangling RayJob in HTTP mode by @dentiny in #4916
- [kuberay-apiserver] do not set auth env vars by default by @andrewsykim in #4889
- ray-operator: add limit reader for serve actor health checks by @andrewsykim in #4961
- [ray-operator] CalculatePodResource should not mutate the input PodSpec by @aditya-786 in #4959
- [ray operator] update api version for agent sandbox artifacts by @vicentefb in #4966
- chore(deps): bump golang.org/x/net from 0.49.0 to 0.55.0 by @dependabot[bot] in #4969
- ray-operator: optimize memory efficiency when calculating pod resources by @andrewsykim in #4953
- [Grafana] Update dashboards to Ray 2.56 and fix auto-load ConfigMap size limit by @kevin85421 in #4977
- [helm][kuberay-operator] Don't emit empty ServiceMonitor labels; rename selector -> additionalLabels by @kevin85421 in #4979
- [ray-operator] Migrate from deprecated GetEventRecorderFor to GetEventRecorder by @marosset in #4855
- Add
NewClusterWithIncrementalUpgradeRayService rollback load tests by @ryanaoleary in #4915 - [History Server] Add default and "latest" path to
/enter_clusterby @chiayi in #4907 - [History server] Add quay release for history server by @chiayi in #4980
- [History server] Fix history server symlinks in log collector by @chiayi in #4995
- [Feature][History Server] Add cluster selection page by @chiayi in #4905
- set restart never on worker pod if autoscaler v1 by @fscnick in #5012
- Add Fake TPU e2e Autoscaling Test Cases by @ryanaoleary in #2279
- Fix: Register RayJob and RayService webhooks in main.go by @vic-comm in #4991
- [Refactor] Use RayNodeTypeLabelKey constant in CalculateAvailableRepl… by @dalaoqi in #4950
- [ray-operator][CI] Replace GetGatewayIP with GetGatewayHost in incremental e2e tests. by @AlterHoodie in #4933
- Update apiserver and apiserversdk to use v1 events api by @marosset in #5008
- Updating issue and PR templates by @marosset in #4564
- feat: add defaultPodAnnotations and defaultPodLabels to Configuration by @alyssa1303 in #4987
- [Refactor] rename network isolation to network policy by @machichima in #5016
- [CI] Add additional_dependencies to eslint pre-commit hook by @alimaazamat in #5025
- [RayService] Rename TargetClusterChanged condition to DesiredClusterSpecChanged by @ryanaoleary in #5022
- [Feature] Final Built-In Ingress Support with Host, Path, PathType, and TLS Fields by @alimaazamat in #4901
- [GCS FT] Support embedded RocksDB storage backend for GCS fault tolerance by @jhasm in #5013
- ray-operator: enable newexpr linter, replace ptr.To with new() by @yjaw in #5019
- [Feature] Add mTLS Support via Cert Manager to RayCluster by @kryanbeane in #4566
- [Feature][History server] Add history server api to kuberay for injecting collector sidecar by @chiayi in #4982
- [History Server] Update history server to retrieve nodeid and move leftover session logs programmatically by @chiayi in #4983
- [CI] Deflaky rayjob_test.go e2e tests by @win5923 in #5047
- Include the priority field in the WorkerGroup API by @gangli113 in #4942
- [History server] Add env vars for history server and fix for history server e2e test by @chiayi in #5050
- [CI] Deflaky rayservice_incremental_upgrade_test e2e tests by @win5923 in #5046
- Sample config for using Platform Events feature by @richabanker in #5001
- [History Server] Set owner info cookie and update logs file structure by @chiayi in #4918
- Add RayJob sample for JAX TPU profiling by @richabanker in #5059
- log warning if autoscaler v1 restart policy is not never by @fscnick in #5014
- [RayService][IncrementalUpgrade] Fix fast rollback behavior for RayService upgrades by @ryanaoleary in #4989
- [fix][networkpolicy] Add networkpolicy feature-gate to helm values by @chipspeak in #5049
- [CI] Deflake TestRayServiceSuspendDuringIncrementalUpgrade by @yjaw in #5024
- hot fix TargetClusterChanged to DesiredClusterSpecChanged by @justinyeh1995 in #5063
- [Feat] Different network policy rules across worker groups by @machichima in #5028
- [history server] Fix job event naming, graceful collector exit, and add default helm value by @chiayi in #5067
- [History server] Implement disk-first event storage pipeline to reduce collector memory by @KunWuLuan in #4874
- [Feature][history server] Support ray auth token mode for live cluster proxy by @CheyuWu in #4520
- [RayService][Test] Add additional incremental upgrade rollback functional test cases by @ryanaoleary in #5010
- Add additional KubeRay autoscaler e2e tests by @ryanaoleary in #2761
- [Bug][RayService] Reset Ready=Initializing on resume so initializing-timeout fires by @EagleLo in #4894
- [Refactor] Extract mTLS issuer reconciliation helper by @vincent-626 in #5057
- test: replace fixed wait in RayCluster scale-down test by @Smallfu666 in #5045
- feat: add flags to separate logging and event collection by @vic-comm in #5011
- [CI] Fix master build: use events.NewFakeRecorder in RayService unit test by @Future-Outlier in #5084
- [History server] Update history server examples and defaults by @chiayi in #5080
- [RayService][e2e] Fix misplaced max_ongoing_requests in incremental-upgrade serveConfigV2 by @justinyeh1995 in #5107
- [History Server] Replace "runtime class name" flag and env var with "storage backend" by @machichima in #5085
- [3/N] [History Server Beta] [Feat] Byte-based cache by @JiangJiaWei1103 in #4960
- [Feature] [history server] Support token auth in the history server collector by @CheyuWu in #5078
- [RayService] Promote Incremental Upgrade Feature to Beta by @win5923 in #5044
- [history server][CI] fix broken test build on master by @win5923 in #5116
- [History Server] Tune CPU, memory, and event logging for cold loads by @Future-Outlier in #5095
- [History Server] Poll Serve, placement group, and Ray Data endpoints by default by @Future-Outlier in #5093
- Bump x/net version used by ray-operator to latest (0.57.0) to address CVEs by @jeff303 in #5073
- [Flaky test] [History Server] Cache and retry for helm pull prometheus by @machichima in #5136
- [History Server] Support Ray 2.56 task log info by @win5923 in #5125
- [History server] Update
RAY_ROOT_DIRTOSTORAGE_ROOT_DIRand examples to use root of the bucket path as root by @chiayi in #5121 - [CRD] Mark ray.io/v1alpha1 as a deprecated API version by @dstrodtman in #5122
- [history server] rename session-cache-max-bytes to session-cache-max-memory by @win5923 in #5139
- [release-1.7] cherry picks for v1.7.0 by @machichima in #5159
- [release-1.7] cherry picks for v1.7.0 by @machichima in #5163
- update KubeRay version to 1.7.0 by @machichima in #5176
Full Changelog: v1.6.2...v1.7.0