Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to release v1.19, you must upgrade your storage schema version to the following:

• MySQL schema version 1.9
• PostgreSQL schema version 1.9
• Cassandra schema version 1.7

Use the appropriate schema upgrade tool to upgrade your schema version.
For details, see https://docs.temporal.io/cluster-deployment-guide#upgrade-server.

Note that schema version is not the same as the storage type (database) version.

Golang version

• Upgraded golang to 1.19.

tctl

• tctl release with v1.17.2 (https://github.com/temporalio/tctl/releases/tag/v1.17.2).

Metric

A major refactoring has been done in the metrics code. Both metrics.Client and metrics.Scope are replaced by metrics.metricsHandler.

Customized metricsHandler is also supported by using temporal.WithCustomMetricsHandler .

Deprecated metrics

All metrics that have the suffix _per_tl are deprecated.

Improvements and fixes

Clean up mutable state when Workflow is closed

When the Workflow is closed, you can trigger a cleanup to remove Memos and Search Attributes from the mutable state. This helps reducing the size of the mutable state for closed Workflows. However, because the cleanup deletes data, it’s best to pair Advanced Visibility with Elasticsearch to ensure that Search Attributes can be retrieved. This feature is controlled by the dynamic config history.visibilityProcessorEnableCloseWorkflowCleanup.

Count limits for pending Child Workflows, Activities, Signals, and cancellation requests

This change adds a new system protection limit for the maximum number of pending Child Workflows, pending Activities, pending Signals to external Workflows, and pending requests to cancel external Workflows.
Operators can set different values for those new limits via the dynamic config. The default is currently 50,000 for each value.

limit.numPendingChildExecutions.error
limit.numPendingActivities.error
limit.numPendingSignals.error
limit.numPendingCancelRequests.error

Maximum number of concurrent pollers

The dynamic config frontend.namespaceCount controls how many concurrent pollers are allowed to connect to Temporal Server. Currently, this limit applies to the total number of pollers; that is, Activity plus Workflow pollers. In v1.19.0, this limit now applies to the number of pollers per type; that is, Activity and Workflow pollers won’t compete with each other for a connection to Temporal Server.

Batch deletion

The batch delete operation is now support in the batch operation API. Use BatchOperationDeletion option in the StartBatchOperationRequest for a batch delete operation.

History task processing

Default implementation

• Host-level Task scheduler is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler is true.
• Multi-cursor queue implementation is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnableMultiCursor is true.

Rate limiting

• Added priority rate limiter for Task processing. By default this rate limiter is disabled but can be enabled by setting history.taskSchedulerEnableRateLimiter to true. The rate limiter should be enabled only when multi-cursor queue implementation is enabled.
• The rate is controlled by history.taskSchedulerMaxQPS for per-host limit and history.taskSchedulerNamespaceMaxQPS for per-Namespace limit. All queues (timer, transfer, visibility) share the same limit. Please note that deletion Tasks have lower priority when consuming rate limit tokens and do not count against the per-Namespace limit.
• The default value for the preceding two configurations is 0, which means they will fall back to and use the value specified by history.persistenceMaxQPS and history.persistenceNamespaceMaxQPS.

Metrics

• task_latency_processing_nouserlatency, task_latency_userlatency, task_latency_nouserlatency, and task_latency_queue_nouserlatency are removed.
• New task_latency_user metric, which captures the latency for acquiring Workflow lock in a single Task Execution attempt.

History Scavenger

A retention validation on workflow has been added in the history scavenger. This feature is enabled by fault with worker.historyScannerVerifyRetention. A default grace period on this retention validation is 90 days + namespace retention time. The grace period is configurable by worker.executionDataDurationBuffer.

All changes

2022-09-19 - 9703d33 - Post-release: bump version and upgrade dependencies (#3408)
2022-09-19 - 91c31f1 - Fix merge map of payload (#3412)
2022-09-19 - 1088382 - Fix task reschedule policy (#3413)
2022-09-20 - c672c37 - Adds a retryable error for when we try to delete open executions (#3405)
2022-09-20 - d890a2f - Propagate CloseVisibilityTaskId to DeleteExecutionVisibilityTask (#3404)
2022-09-20 - be727b3 - Retry attempts to delete open visibility records (#3402)
2022-09-21 - 8f24d1f - Revert "Retry attempts to delete open visibility records" (#3420)
2022-09-21 - 9dfdf75 - Rename parameters of MergeMapOfPayload (#3418)
2022-09-22 - ff40b89 - Retry attempts to delete open visibility records (#3402) (#3421)
2022-09-23 - c6fb6b8 - Fixes an issue where last-heartbeat time was set to the first event's timestamp (#3361)
2022-09-26 - f4af2d5 - Fix list batch operation to include division (#3431)
2022-09-27 - b4b61ff - Reorder grpc interceptors (#3423)
2022-09-27 - 85f400e - Add postgres es development script (#3429)
2022-09-27 - e3e1cce - Retry attempts to delete open workflow executions (#3411)
2022-09-27 - 769b865 - Add cluster ID into ringpop app (#3415)
2022-09-27 - 6530847 - Use original branch token instead of deserializing as branch info and then re-serializing (#3384)
2022-09-27 - d27ea89 - Fix action metrics (#3434)
2022-09-28 - 8929def - Namespace replication for failover history (#3439)
2022-09-28 - c150474 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - e0bbabe - Per-namespace workers should only run on active cluster (#3426)
2022-09-29 - b497033 - Ensure urfave/cli accepts flag values with comma (#3440)
2022-09-29 - 363d4c0 - Log warning only when there is an error in SA size validation (#3443)
2022-09-29 - 97d0ec3 - Supply optionally configured workflow settings as hints (#3442)
2022-09-29 - 47bb155 - Update dependencies and pin otel (#3444)
2022-09-29 - 3815915 - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-09-30 - 3099274 - Fix reset workflow in replication reapply (#3449)
2022-10-03 - 175b916 - Use safer default TIMESTAMP for MySQL. (#3424)
2022-10-04 - a86d455 - Index history size when workflow closes (#3451)
2022-10-05 - fc37cd4 - Update ns version history in handover state (#3456)
2022-10-06 - 0ae4cc5 - Add config filter by task type (#3455)
2022-10-06 - 0df719d - Update replication use branch token (#3447)
2022-10-10 - fa51f5d - Move record child workflow completed to api package (#3350)
2022-10-10 - 62cd143 - Move verify child workflow completion recorded to api package (#3351)
2022-10-10 - 71a1de3 - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - bc451db - Compare task type enum (#3464)
2022-10-10 - c6b472e - Fix exclude tags with withTags method (#3466)
2022-10-10 - 158737a - Remove old logic for checking workflow deletion task dependencies from delete_manager (#3427)
2022-10-10 - d33559a - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-11 - 0e70cf7 - Move get / poll mutable state to api package (#3467)
2022-10-11 - 7278168 - Move describe workflow to api package (#3469)
2022-10-11 - 0dddd3c - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-11 - 429c0af - Move replication APIs to api package (#3472)
2022-10-11 - d90f3ab - Move NDC logic to ndc package, move workflow reset to api package (#3465)
2022-10-11 - 740c7a3 - Move reapply events to api package (#3476)
2022-10-11 - d252656 - Move remove signal from mutable state to api package (#3475)
2022-10-11 - b292c22 - Move delete workflow to api package (#3473)
2022-10-11 - 32ea0bf - Move refresh workflow to api package (#3477)
2022-10-12 - 16184da - Fix scheduled queue max read level update for single processor (#3474)
2022-10-12 - ceae241 - Handle sync workflow state task in replication dlq handler (#3482)
2022-10-13 - bf8e1f1 - Bump UI to v2.7.0 (#3480)
2022-10-13 - 46dcb4f - Turns on the history scavenger for SQL backends (#3462)
2022-10-13 - 86fde8e - Move query workflow to api package (#3486)
2022-10-13 - ef21405 - Fix sanitize mutable state after replication (#3479)
2022-10-13 - 457eb05 - Clean up duplicate empty task id (#3490)
2022-10-13 - a0ea177 - Add execution scavenger for retention (#3457)
2022-10-15 - a4fe3a7 - Remove UI v1 from development environment (#3485)
2022-10-17 - dc84e87 - Update replication timestamp with no task (#3487)
2022-10-17 - 8e5ccb5 - schama -> schema (#3501)
2022-10-17 - b8d46a5 - Do not add version 0 to failover history (#3483)
2022-10-17 - cebbde9 - Sync proto API 1.12.0 (#3500)
2022-10-17 - 968c576 - Create visibility GetWorkflowExecution API (#3488)
2022-10-18 - ecde543 - Remove now parameter from task generator interface (#3478)
2022-10-18 - 9cb646b - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 8a7538a - Warning log on new event during set workflow (#3508)
2022-10-19 - 8d645e8 - Increase visibility RPS and burst dynamic configs for integration tests (#3221)
2022-10-19 - 4b4493b - Bump Golang version to 1.19 (#3507)
2022-10-19 - 3aff85e - Fix scanner start dep (#3513)
2022-10-19 - c7713b1 - Support arbitrary history task category in SQL persistence (#3489)
2022-10-19 - d5edb0e - Plumb in timeouts instead of referencing the uninitialized execution info during workflow creation (#3512)
2022-10-20 - 6ee5ac5 - Check shard ownership by assertion (#3416)
2022-10-20 - c21ffad - initialize otelMetricsHandler logger (#3518)
2022-10-20 - af18756 - Simplify interface & remove unused vars (#3516)
2022-10-22 - 39ee372 - Fix otel gauge metric type (#3525)
2022-10-24 - 827a59f - Support batch operation for delete workflow (#3497)
2022-10-25 - 42fca45 - add metricstest package (#3528)
2022-10-26 - 5fac4d4 - Add ArchiveExecutionTask to our set of tasks (#3524)
2022-10-26 - b42bc85 - Add task version mismatch metrics (#3491)
2022-10-26 - 119478a - Change execution scavenger to call admin delete (#3526)
2022-10-26 - 275b62b - Fix replication task nil case (#3531)
2022-10-27 - 3a0d879 - Remove SA and memo after close visibility task is completed (#3504)
2022-10-27 - ddc33bc - Skip archival if it was already done before deleting history data (#3523)
2022-10-28 - d4d7684 - Add service name tag to metrics emitted using metrics handler (#3537)
2022-10-31 - 4eec379 - Add retention validation in history scavenger (#3541)
2022-10-31 - 6d9c595 - Increase timeouts in debug mode (#3542)
2022-10-31 - 88d3411 - Take rpc address from config for local cluster (#3546)
2022-10-31 - 1f69721 - Add deadlock detector (#3492)
2022-11-01 - 176cd42 - Add a new archiver client which doesn't retry (#3544)
2022-11-03 - 4a9d6ab - Queue processor handle shard ownership lost (#3553)
2022-11-03 - e2ef735 - Add a CanSkipVisibilityArchival flag to the CloseExecutionTask (#3529)
2022-11-03 - 7860ec0 - Fix shard info serialization (#3555)
2022-11-04 - 53000fc - Fix namespace handover log (#3559)
2022-11-04 - c2d5496 - Generate ArchiveExecutionTask instead of DeleteHistoryEventTask whenever an execution is closed (#3533)
2022-11-04 - 6721ace - Improve integration test framework (#3547)
2022-11-06 - b300010 - Add some config values for the archival queue (#3554)
2022-11-07 - 02b8cca - Make shard more resilient to append history timeout (#3564)
2022-11-07 - 9fa1a90 - Replace metrics client/scope with MetricsHandler (#3561)
2022-11-08 - bccf128 - Handle namespace not found in replication (#3557)
2022-11-08 - bbd386e - Add jitter for retention timer (#3569)
2022-11-09 - 8683398 - Fix task serialization in persistence tests (#3576)
2022-11-10 - 44154d3 - bump ui to v2.8.3 (#3573)
2022-11-11 - 5e466c3 - Fix metrics tag after metrics refactoring (#3579)
2022-11-14 - 574aee6 - Move queue metrics to metrics package (#3587)
2022-11-14 - 50936e1 - Add a tool to limit the number of pending child workflows (#3575)
2022-11-14 - ef05ac8 - Fix scheduled queue lookahead (#3563)
2022-11-14 - 3a08eed - Limit Cassandra memory appetite to 1Gb in development configuration (#3581)
2022-11-14 - 76b58a7 - Fix resilient shard (#3584)
2022-11-14 - b74b0e3 - Upgrade dev grafana version used in docker (#3593)
2022-11-14 - dde2afb - Update api and sdk references (#3595)
2022-11-14 - 4f2b98e - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-14 - e7f99d0 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - c82d43c - Return an error whenever we exceed the number of pending child executions (#3586)
2022-11-15 - c14f643 - Fix visibility queue log message and level (#3600)
2022-11-15 - 25e4a8f - Don't log a warning when the number of child workflows is high (#3598)
2022-11-15 - 02b913f - Clean up task latency metrics (#3510)
2022-11-16 - 3d32ebc - Add tdbg to goreleaser (#3609)
2022-11-16 - e96c17a - Initialize new branch token based on replication target (#3604)
2022-11-17 - be30bab - Disable eager activity dispatch for buggy clients (#3610)
2022-11-17 - ee1abe1 - Fix scheduled task precision (#3591)
2022-11-18 - d349987 - Add request validation on batch APIs (#3621)
2022-11-18 - be7daec - Add limits on other workflow fields (#3599)
2022-11-18 - e26ea38 - Retry CreateIndex in integration tests request upon failure (#3618)
2022-11-18 - e4d1b98 - Add methods to check workflow size violations for several other fields (#3602)
2022-11-18 - 81b79a7 - Change max namespace count limit to per poller type (#3608)
2022-11-18 - 66db3ae - Remember retention timer task state in memory for retry (#3620)
2022-11-18 - 21118ed - Enforce the per-workflow pending activity constraint (#3611)
2022-11-18 - b13ed16 - Add flag to enable workflow closed clean up (#3603)
2022-11-18 - 4483f10 - Handle unknown cluster during replication (#3619)
2022-11-18 - 0c735e4 - Fix function name typos (#3623)
2022-11-18 - df160db - Enforce the pending cancel request limit (#3615)
2022-11-18 - 32212ea - Task scheduler rate limiter (#3606)
2022-11-18 - 91a67f4 - Enforce the per-workflow pending signal limit (#3617)
2022-11-18 - 9541374 - Keep multi cursor default reader always running (#3626)
2022-11-18 - b43ca63 - Fix immediate task fire time equality check (#3628)
2022-11-21 - 86631e9 - Fix typo in dynamic config for VisibilityProcessorEnsureCloseBeforeDelete (#3632)
2022-11-21 - ded2f67 - Improve test log format (#3634)
2022-11-22 - 3520803 - Extract duplicated getTemporalPackageDir into common function (#3637)
2022-11-22 - b0647ae - Randomization and configuration for scheduler throttle duration (#3636)
2022-11-22 - 13fa5f7 - Handle unknown cluster in replication (#3639)
2022-11-22 - 473d487 - Revert change 'De-duplicate reapply event' (#3640)
2022-11-22 - e40a1df - Enable history scavenger to scan expired workflow (#3643)
2022-11-22 - 13047df - Fix persistence GetHistoryTask (#3645)
2022-11-22 - 17b121a - Fix fail workflow batch id (#3646)
2022-11-22 - a4002ab - Fix pending tasks metric (#3648)
2022-11-22 - cd18725 - Fix bug in GetRepoRootDirectory code (#3644)
2022-11-23 - d11074d - Make replication task reading logic more resilient (#3596)
2022-11-23 - d54cd1d - Simplify ServerOptions applyFunc (#3649)
2022-11-23 - 788b2b7 - Use fx group for deadlock detector (#3650)
2022-11-23 - 5658d5e - Disable visibility verify close before delete (#3651)
2022-11-23 - 9bbaa27 - Handle unknown cluster with a unique string (#3652)
2022-11-23 - 5dd8cdd - Log when force replicate failed (#3653)
2022-11-23 - f46f2a4 - Fix one metrics name (#3654)
2022-11-23 - 306797b - Update persistence error metrics (#3655)
2022-11-24 - b717bd5 - Enforce min interval for continue as new runs (#3659)
2022-11-25 - 7a2fab5 - Fix a flaky persistence integration test (#3664)
2022-11-25 - 53f4615 - Enable host level pool and multi-cursor by default (#3660)
2022-11-28 - 001ec0e - Consolidate service name constants (#3656)
2022-11-28 - c2f21fe - Fix grpc listener initialization (#3657)
2022-11-28 - 0547245 - Set default catchup window eagerly (#3658)
2022-11-28 - 82cbe64 - make some s3 utility functions public (#3668)
2022-11-28 - 997434c - Fix replication task execution latency (#3671)
2022-11-28 - b381293 - Do not panic if passive processor is missing (#3672)
2022-11-28 - 362987f - Return an error when we fail to instantiate the server (#3642)
2022-11-28 - 937c38f - Update workflow state validator (#3673)
2022-11-28 - ee4648e - Set limit on DescribeSchedule query/signal loop (#3669)
2022-11-28 - 318749b - Don't wake up scheduler workflow when paused (#3670)
2022-11-29 - f389611 - Fix defer not able to capture return error issue (#3675)
2022-11-29 - f6a340d - Fix slice pending task count (#3676)
2022-11-30 - 0101924 - Improve execution scavenger (#3674)
2022-12-01 - ff3f95c - Remove empty visibility tag (#3681)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools