github temporalio/temporal v1.30.1
on GitHub

5 hours ago

⚠️💥 DOCKER IMAGE BREAKING CHANGES 💥⚠️

Starting in this release, for security reasons, Temporal Docker images have been slimmed down, and we have removed binaries and packages that do not strictly need to be included. This includes:

temporalio/admin-tools

Removed tools:

  • bash
  • python3
  • libev
  • curl
  • jq
  • yq
  • mysql-client
  • postgresql-client
  • expat
  • tini
  • cqlsh
  • tctl
  • tctl-authorization-plugin

Added tools:

  • temporal-elasticsearch-tool

temporalio/server

Removed components:

  • temporal CLI
  • tctl and tctl-authorization-plugin (both deprecated)
  • dockerize
  • curl
  • bash

dockerize → embedded sprig

Previous behavior:

  • dockerize processed a persistent config template
  • The template lived in the image but could be overridden via volume mounts
  • dockerize provided sprig + custom helper functions
  • The server binary would search a set of paths for the rendered config file

New behavior:

  • sprig templating is now built directly into the server binary
  • The default config template is embedded in the binary
  • dockerize is fully removed as an image dependency
  • TEMPORAL_SERVER_CONFIG_FILE_PATH is used to reference the config file. When not specified, the embedded template is processed.
  • # enable-template is required at the top of the config file to enable sprig templating.

References:

Image deprecations

The following images are deprecated and will no longer receive updates:

  • temporalio/auto-setup
    • For local development, we recommend using the CLI dev server.
    • See Samples Server for examples of how to use different persistence and visibility stores.
  • temporalio/base-admin-tools
  • temporalio/base-server
  • temporalio/base-ci-builder
  • temporalio/base-builder

⚠️💥 Helm Chart BREAKING CHANGES 💥⚠️

Due to the Docker image breaking changes above, you must now use a minimum Helm chart version of temporal-0.73.2.

Configuration Options

Value Description Default
server.useEntrypointScript Use an entrypoint script that auto-detects dockerize vs sprig false
server.configMapsToMount Which config template to mount: dockerize, sprig, or both dockerize
server.setConfigFilePath Set TEMPORAL_SERVER_CONFIG_FILE_PATH env var (required for sprig) false

Configuration for Different Scenarios

Option 1: Support both pre-1.30 and 1.30+ images (recommended for CI/testing) 

server:
  useEntrypointScript: true
  configMapsToMount: "both"
  setConfigFilePath: false

Option 2: 1.30+ only (sprig templating) 

server:
  image:
    tag: "1.30.1" # or later
  useEntrypointScript: false
  configMapsToMount: "sprig"
  setConfigFilePath: true

Option 3: Pre-1.30 only (dockerize, deprecated) 

server:
  image:
    tag: "1.29.3" # or earlier
  useEntrypointScript: false
  configMapsToMount: "dockerize"
  setConfigFilePath: false

⚠️ Behavioral / Potentially Breaking Change

Retry behavior of PermissionDenied for Nexus operations

The retry behavior for PermissionDenied errors when scheduling a Nexus operation has been updated.

  • Previously, these errors were not retried. With this change, PermissionDenied is now considered retryable.
  • Impact:
    • OSS users who inject their own custom Nexus registries may notice different retry patterns for PermissionDenied errors.
  • We are considering adding a dynamic config flag to control this behavior.

metrics.Tag interface-to-struct conversion

metrics.Tag is changing from an interface to a concrete type, and tag Key() and Value() reader methods are being replaced with exported Key and Value fields. metrics.Handler methods that accept tags now only accept the concrete type.

If you build your own temporal server with a custom metrics handler temporal.WithCustomMetricsHandler(metricsHandler), you need to update that handler to read metrics.Tag keys and values via exported Key and Value fields. If you provide a custom metrics.Tag interface implementation, you need to replace that implementation with a concrete metrics.Tag, for example by using metrics.StringTag.

⚠️ Worker Versioning

Warning

Consider upgrading your existing server version to 1.29.3 before upgrading to 1.30.1 to preserve backward-compatibility guarantees when using worker versioning.

  • Improved reliability and safety:
    • Made APIs significantly more robust and scalable by propagating routing config to task queues asynchronously.
      • [⚠️ Behavioral Change] In cases where you want to wait until all partitions of all task queues are aware of your latest Routing Config change (i.e., latest Current or Ramping Version) before taking an action, you must now ensure that WorkerDeploymentInfo.RoutingConfigUpdateState == ROUTING_CONFIG_UPDATE_STATE_COMPLETED after SetCurrentVersion or SetRampingVersion API calls complete. Previously, those APIs returned success only after that condition was true, but now propagation completes asynchronously. In general, users need not be concerned with this change, but this behavior change is being noted for awareness.
    • [⚠️ Behavioral Change] Reject Versioning Override for versions that do not exist.
    • New safety mechanism (revision number) to guarantee that auto-upgrade workflows never accidentally use an outdated version if they switch task queue partitions.
  • Improved observability and ease of use:
    • New TemporalUsedWorkerVersions search attribute, which shows which Worker Deployment Versions a workflow has used during its lifetime.
    • Improved accuracy and tagging of metrics.
    • Improved error messages (added Worker Deployment name, Build ID, and other info to error messages where relevant).
    • WorkflowExecutionOptionsUpdated history event now contains the identity of the client that changed the options (i.e., to set/unset a Versioning Override).
  • New capabilities:
    • LastCurrentTime timestamp in version info tells you whether a version was ever promoted to Current in the past. This enables accelerated rollout of versions that have been Current in the recent past by allowing the controller to detect whether a rollout is actually a rollback.
    • If a workflow is already Pinned, you can now set a Pinned Versioning Override without specifying a pinned version (easier for batch jobs setting Pinned Versioning Override on a large batch of workflows running on different versions).
  • Bug fixes:

Task Queue Priority & Fairness Public Preview

  • Task Queue Priority and Fairness (initially introduced in 1.29) are now in Public Preview.
  • To enable Priority on a task queue, namespace, or globally, set dynamic config matching.useNewMatcher to true.
  • To enable Fairness on a task queue, namespace, or globally, set dynamic configs matching.useNewMatcher, matching.enableFairness, and matching.enableMigration to true.

Examples:
Priority (task queue scoped) 

matching.useNewMatcher:
  - value: true
    constraints:
      namespace: "my-namespace"
      taskQueueName: "my-task-queue"

Fairness (task queue scoped): 

matching.useNewMatcher:
  - value: true
    constraints:
      namespace: "my-namespace"
      taskQueueName: "my-task-queue"
matching.enableFairness:
  - value: true
    constraints:
      namespace: "my-namespace"
      taskQueueName: "my-task-queue"
matching.enableMigration:
  - value: true
    constraints:
      namespace: "my-namespace"
      taskQueueName: "my-task-queue"
  • Changes since 1.29:
    • Fairness weight overrides can be set through the API.
    • Priority metadata can be updated on existing workflows and activities through UpdateWorkflowExecutionOptions and UpdateActivityOptions. Affected tasks are rescheduled.
    • The approximate_backlog_count metric now has a task_priority label.
    • Polls to sticky queues are redirected to higher-priority backlog tasks on normal partitions. This is enabled by default, but can be disabled with dynamic configs matching.priorityBacklogForwarding and matching.ephemeralDataUpdateInterval.
    • Fairness can be turned on for active task queues without losing tasks. Previously backlogged tasks are dispatched first. Ensure dynamic config matching.enableMigration is enabled before, or at the same time as, enabling fairness.
    • Fairness can be configured to switch on automatically on usage of fairness keys. Enable dynamic config matching.autoEnableV2.

Degraded Workflow Visibility

  • system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute is a new dynamic config value to turn on degraded workflow visibility. Setting this to 0 turns off the feature entirely; any positive integer value becomes the number of consecutive workflow task failures needed to add the TemporalReportedProblems search attribute.
  • TemporalReportedProblems is a new search attribute to identify workflows that are not making progress. The search attribute is a KeywordList with two entries: a cause and a category. The cause is either WorkflowTaskFailed or WorkflowTaskTimedOut. If the cause is WorkflowTaskFailed, the category is one of the following values. If the cause is WorkflowTaskTimedOut, the category is always ScheduleToClose.
  • See documentation here

External payloads

history.externalPayloadsEnabled is a new dynamic config value that enables server-side support for the claim-check pattern, where payloads are stored in external storage outside of history. When enabled, the server produces the aggregate size and count of external payloads and returns ExternalPayloadSizeBytes and ExternalPayloadCount on the DescribeWorkflowExecution call. This feature depends on claim-check pattern support in the Temporal SDK, which is currently under development.

Visibility with OpenSearch

Temporal can now run with OpenSearch 2+ as a visibility store. You only need to set up Temporal config as you would for Elasticsearch (see the provided Elasticsearch config template).

Increasing the maximum number of custom search attributes

You can now change the maximum number of custom search attributes when using a SQL database as the visibility store. Previously, there was a hard-coded amount of pre-allocated fields that could be used to create custom search attributes. Now, you can change the maximum number of pre-allocated fields.

Example: assuming you have the default schema provided by Temporal and you want to increase the number of Int-type custom search attributes from 3 to 5 and Keyword-type from 10 to 12, follow the steps below:

  1. Change your DB schema by adding the necessary columns.

    1. MySQL (check the existing schema definition, and copy the syntax for the respective search attribute type): 

      ALTER TABLE custom_search_attributes
  ADD COLUMN Int04     BIGINT       GENERATED ALWAYS AS (search_attributes->"$.Int04"),
  ADD COLUMN Int05     BIGINT       GENERATED ALWAYS AS (search_attributes->"$.Int05"),
  ADD COLUMN Keyword11 VARCHAR(255) GENERATED ALWAYS AS (search_attributes->>"$.Keyword11"),
  ADD COLUMN Keyword12 VARCHAR(255) GENERATED ALWAYS AS (search_attributes->>"$.Keyword12");

ALTER TABLE custom_search_attributes
  ADD INDEX by_int_04     ON custom_search_attributes (namespace_id, Int04),
  ADD INDEX by_int_05     ON custom_search_attributes (namespace_id, Int05),
  ADD INDEX by_keyword_11 ON custom_search_attributes (namespace_id, Keyword11),
  ADD INDEX by_keyword_12 ON custom_search_attributes (namespace_id, Keyword12);

    2. PostgreSQL (check the existing schema definition, and copy the syntax for the respective search attribute type): 

      ALTER TABLE executions_visibility
  ADD COLUMN Int04     BIGINT       GENERATED ALWAYS AS ((search_attributes->'Int04')::bigint),
  ADD COLUMN Int05     BIGINT       GENERATED ALWAYS AS ((search_attributes->'Int05')::bigint),
  ADD COLUMN Keyword11 VARCHAR(255) GENERATED ALWAYS AS (search_attributes->>'Keyword11'),
  ADD COLUMN Keyword12 VARCHAR(255) GENERATED ALWAYS AS (search_attributes->>'Keyword12');

ALTER TABLE executions_visibility
  ADD INDEX by_int_04     ON executions_visibility (namespace_id, Int04,     (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id),
  ADD INDEX by_int_05     ON executions_visibility (namespace_id, Int05,     (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id),
  ADD INDEX by_keyword_11 ON executions_visibility (namespace_id, Keyword11, (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id),
  ADD INDEX by_keyword_12 ON executions_visibility (namespace_id, Keyword12, (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id);

  2. Modify the Temporal config file: 

    persistence:
  visibilityStore: ...

visibility:
  persistenceCustomSearchAttributes:
    Int: 5
    Keyword: 12

  3. Restart Temporal.

You can only increase the maximum number of pre-allocated fields. Reducing the maximum number in the config is a no-op. Do not drop columns from your database table.

Visibility Query Converter

We are introducing a new experimental query converter to replace the existing ones. Currently, there are two query converter implementations: one for Elasticsearch, which builds the query search body; and one for SQL, which mainly validates the query. The new query converter unifies these implementations such that each Visibility Store implementation can implement the StoreQueryConverter interface to build its query object without needing to implement parsing and validation.

The new unified query converter is not 100% backward compatible with the existing query converter for Elasticsearch. In other words, if you use Elasticsearch as your visibility store, some queries might not work.

In particular, the new query converter performs stricter validation of query clauses and does not allow comparisons between a search attribute and a value of a different type. For example, the existing query converter for Elasticsearch allows comparing a Keyword-type search attribute with an integer (e.g., WorkflowType = 123), but the new query converter returns an error.

Since this is still experimental, the new unified query converter is disabled by default and can be enabled by setting the dynamic config system.visibilityEnableUnifiedQueryConverter: true. Once the new query converter is production-ready, it will be enabled by default (projected: v1.31.0), and the old ones will be deprecated and removed (projected: v1.32.0).

Elasticsearch tool

Temporal now provides an experimental temporal-elasticsearch-tool to manage visibility templates and indexes. The tool supports the same authentication and AWS request-signing features as the Temporal server. See the README for usage details.

Internal Nexus Callback Routing

This simplifies Nexus callback configuration by introducing internal routing for worker-targeted operations, eliminating the need for callback URL templates and allowlist configuration in normal deployments.

New useSystemCallbackURL Toggle (component.nexusoperations.useSystemCallbackURL)

  • When enabled, worker-targeted Nexus operations use a fixed temporal://system callback URL instead of requiring a configured URL template
  • External endpoint targets continue to use template-generated URLs
  • Default: false (will change to true in a future release)

Automatic temporal://system Allowlisting

  • The temporal://system URL is now always permitted by callback address validation, regardless of configured component.callbacks.allowedAddresses rules
  • No explicit allowlist entry is needed for internal worker callbacks

Dynamic Config Examples

Before (required configuration): 

system.enableNexus:
  - value: true
component.nexusoperations.callback.endpoint.template:
  - value: http://localhost:7233/namespaces/.NamespaceName/nexus/callback
component.callbacks.allowedAddresses:
  - value:
      - Pattern: "*"
        AllowInsecure: true

After (zero config for worker targets): 

system.enableNexus:
  - value: true
component.nexusoperations.useSystemCallbackURL:
  - value: true

Migration Notes

When useSystemCallbackURL is enabled, your HTTPCallerProvider must route internal requests using the Source and Token headers, overriding the path to /nexus/callback. See components/callbacks/request.go for the routing implementation.

Dynamic Config Changes

Nexus

  • component.nexusoperations.limit.operation.timeout.min has been renamed to component.nexusoperations.limit.request.timeout.min, and given a default value of 500ms.
  • A new config, component.nexusoperations.limit.dispatch.task.timeout.min, has been added with a default value of 1s. This is the minimum time remaining for a request to be dispatched to the handler worker. If the remaining request timeout is less than this value, a timeout error is returned. Working in conjunction with MinRequestTimeout, both configs help ensure that the server has enough time to complete a Nexus request.

Helpful links to get you started with Temporal

Temporal Docs
Server
Samples Server
Helm Chart

Docker images

Server
Admin-Tools

Full Changelog: v1.30.0...v1.30.1

Check out latest releases or
releases around temporalio/temporal v1.30.1

Don't miss a new temporal release

NewReleases is sending notifications on new releases.

Get notifications