⚠️💥 UPCOMING BREAKING CHANGES 💥⚠️

Starting from next server release 1.30.0, for security reasons, Temporal docker images will be slimmed down and we are taking away the binaries and packages that don’t strictly need to be included. This includes:

temporalio/server

• temporal CLI - included in admin-tools
• tctl and tctl-authorization-plugin - both are deprecated CLIs
• dockerize - used for templating the configuration, functionality that is now inlined in the server codebase
• curl - not part of the Temporal distribution

temporalio/admin-tools

• tctl and tctl-authorization-plugin
• python3
• libev
• curl
• jq
• yq
• mysql-client
• postgresql-client
• expat
• tini
• cqlsh

Task queue fairness - pre-release

Description:

Task queue fairness allows you to control the execution order of workflows, activities, and child workflows within a single task queue by assigning fairness keys and weights. Note that priority keys take precedence over fairness assignments.

Fairness can be attached to workflows and activities using the latest versions of most SDKs. In order for priority to take effect on the server, you need to switch to set the dynamic config matching.enableFairness to true either on specific task queues, namespaces, or globally.

⚠️ Turning the feature on/off will cause currently backlogged tasks to be lost; which can cause workflows to be stuck. This limitation will be lifted in future releases.

See more usage details here: Temporal - Task Queue Fairness Guide (Pre-Release)

Rollout operational task: (schema upgrade)

Versioning improvement

Bug fixes

• RampingVersionPercentageChangedTime is now visible from the Routing Config and can be accessed via a DescribeDeployment call (#8089)
• Correct the drainage status of a deactivated version when it gets rollbacked (#8119)
• Fix max deployment count check to not block new versions (#7841)
• Do not set sticky queue if a deployment transition is ongoing (#7852)
• Do not bypass task generation when version changes during a deployment transition (#7890)
• Make unsupported and deprecated Deployment API’s return Unimplemented error (#8009)
• Update DeploymentName from Override and fix batch UpdateOptions serialization bug (#7910)

Improvements

• Queries on drained + poller-less version to respond with descriptive error message (#7946)
• SetWorkerDeploymentManager: Help coordinate multiple tenant writes to the k8s worker controller without overwriting each others changes (#8278)
  Allow people to revert a task queue back to the un-versioned state without needing to set a 0% ramped non-nil Ramping Version (#8172)
• Allow users to be able to opt-in to activating versions that have not yet had any pollers (#8254)

Cloud rollout operational task:

Task queue config

Description:

Task queues can be now configured via the new UpdateTaskQueueConfig endpoint. It allows configuring (1) the queue’s maximum requests per second and (2) default maximum requests per second for fairness keys (feature is in pre-release). For example, using the Temporal CLI 1.5.0 or later:

temporal task-queue config set --task-queue foo --task-queue-type bar --queue-rate-limit 100 --queue-rate-limit-reason "throttling"

Note that the rate limit for the task queue from the API takes precedence over the rate limit set via the worker’s TaskQueueActivitiesPerSecond option. If the API rate limit is unset again, it will fall back to the worker’s rate limit again, if set. Otherwise the system’s default limit is applied.

Authorizer

Description: Added a new dynamic config (frontend.exposeAuthorizerErrors) to control whether the frontend authorization interceptor should propagate errors returned by the Authorizer component as-is or wrap them with a PermissionDenied service error. Default is false, meaning all errors will be wrapped with PermissionDenied, which matches current behavior to avoid breaking any custom Authorizer implementers.

Eager workflow start public release (on by default)

Description: Eager workflow start is a latency optimization for worker and a starter a colocated in the same process. If the starter (client) requests eager execution, and a worker slot is available, the client will request the server to start the workflow eagerly. If permitted (the dynamic
config is on, and there's no start delay), the first workflow task will be returned inline to be processed by the colocated worker.

This feature is now on by default and the EagerExecutionAccepted flag has been added to WorkflowExecutionStartedEventAttributes for debugging purposes (#8056)

To turn this feature off set the dynamic config system.enableEagerWorkflowStart to false.

Activity and workflow metrics changes

Description: A number of activity and workflow metrics were added and activity_e2e_latency has been deprecated. (#8196, #8185)

Deprecated Metrics

• activity_e2e_latency → Deprecated, replaced with better named activity_start_to_close_latency

New Metrics

• activity_start_to_close_latency: Per attempt latency from activity start to close
• activity_schedule_to_close_latency: End-to-end duration, including retries and backoff.
• activity_success: Number of succeeded activities
• activity_fail: Number of final failures for activities
• activity_timeout: Incremented on the final activity timeout tagged by timeout_type
• activity_task_fail: Failures for activities including retries
• activity_task_timeout: Number of activity attempt timeouts, tagged by timeout_type.
• activity_cancel: Number of canceled activities
• workflow_duration: End to end latency of workflow

Batch operation improvement

Description: Removed the internal BatchParams go struct in favor of a protobuf struct for safer serialization. Optimized the batch operation processing with proactive page fetching, removing the worker wait for new pages to be completed. (#8144, #8081).

Batch Activity Reset and Update Options

Description: Support activity reset and update-options on the server side. (#8061)

Nexus

Description:

• Fixed a bug where the standby outbound task executor could sometimes return a NamespaceNotActive error.
• Fixed a data race in Nexus disallowed headers dynamic config.
• All remote frontend HTTP calls will always attempt to use HTTP2 by default.
• Changed the default to true for dynamic config component.nexusoperations.recordCancelRequestCompletionEvents . This config will be removed in a future release.
• Fixed a bug for forwarded Nexus operation completion HTTP requests that contained a failure. The failure will now be reconstructed instead of reusing the original HTTP request body.
• Added logic to forward Nexus HTTP requests using the same dispatch type (by endpoint or by namespace+task queue) as the original request. This behavior is behind a dynamic config, frontend.nexusForwardRequestUseEndpointDispatch, because endpoints currently do not support replication and therefore forwarding requests by endpoint will not work out of the box.
• The original HTTP request headers will now be passed through as-is, without sanitization, for forwarded requests.
• Bug fix: a new workflow task will be generated for NexusOperationCancelRequestCompleted|Failed events.

Visibility

Description: ScanWorkflowExecutions has been removed from Visibility. The API is still available, but it simply calls ListWorkflowExecutions.

Worker Insights

Description: Worker insights is an umbrella project that addresses the operational complexities and toil of worker management (tuning performance knobs, scaling workers). Our end goal is to automate this process for our users. As a first step, this release adds the capability to propagate the worker state (configuration + SDK metrics) to the server via a heartbeat mechanism. This state is stored in-memory on the matching servers, and can be queried by the user.

To achieve this, this release introduces 3 APIs:

1. RecordWorkerHeartbeat: To send heartbeat to the matching service. Used by SDK.
2. ListWorkers: To query the state of 1 or more workers that match a predicate.
3. DescribeWorker: To query the state of a specific worker.

These APIs are disabled by default.

Flags are: frontend.WorkerHeartbeatsEnabled and frontend.ListWorkersEnabled.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.29.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

Full Changelog: v1.28.1...v1.29.0