Release Highlights

Worker Versioning (preview)

Links to docs:

• https://docs.temporal.io/workers#worker-versioning
• https://docs.temporal.io/dev-guide/java/versioning#worker-versioning
• https://docs.temporal.io/dev-guide/golang/versioning#worker-versioning

Description: Worker Versioning simplifies the process of deploying changes to Workflow Definitions.

⚠️ This feature requires Advanced Visibility

In the SDK, a user can identify a Worker with a Build ID, which should be associated at build time, and configure that Worker to use Versioning.

An operator can use the UpdateWorkerBuildIdCompatibility gRPC API (and the high-level APIs in the SDKs) to define which builds are compatible with each other and control task routing for existing and new Workflows based on the compatibility constraints.

Additionally, operators and automated scripts can use the GetWorkerTaskReachability API to query for build IDs that are no longer reachable and whose associated workers can be retired.

⚠️ To enable worker versioning, turn on the frontend.workerVersioningDataAPIs, frontend.workerVersioningWorkflowAPIs, and worker.buildIdScavengerEnabled dynamic config flags (disabled by default in this release).

This feature introduces a couple of schema changes:

1. A new BuildIds Search Attribute that is used to track reachability. This new Search Attribute requires a Visibility schema update. See the schema upgrade documentation for the required commands for performing the upgrade.
2. A new task_queue_user_data table, which stores the Versioning data for the various Task Queues and is replicated using the Namespace replication queue. See the schema upgrade documentation for the required commands for performing the upgrade.

The Matching Service will now load Task Queue user data from persistence for the Task Queue root partition, and non-root partitions will try to obtain and maintain a fresh view of the user data by long-polling other Task Queue partitions for the same Task Queue.

If user data loading is suspected to be causing processing overhead, it can be completely disabled by using the matching.loadUserData dynamic config flag.

Note that disabling the loading of user data will essentially disable the Versioning feature and will cause tasks for versioned Workflows to be dropped by the Matching Service, causing versioned Workflows to be stuck. This approach was chosen over dispatching versioned Workflows to arbitrary Workers and breaking the user-defined compatibility semantics. Manual intervention will be required to regenerate Tasks for stuck versioned Workflows.

As part of this work, the UpdateWorkerBuildIdCompatiblity, UpdateNamespace, and RegisterNamespace APIs now have stricter rate limiting because they can produce replication messages for Global Namespaces. The Namespace replication queue is a critical resource that is not partitioned and serves as a mechanism for delivering failover messages between Clusters.

A few more dynamic configuration options were added for this feature:

• limit.workerBuildIdSize is the byte length limit for a Worker Build ID as used in the RPC methods for updating the version sets for a Task Queue. Do not set this option to a value greater than 255 for Clusters that use SQL-based persistence due to predefined VARCHAR column width.
• limit.versionCompatibleSetLimitPerQueue is the maximum number of compatible sets allowed in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.versionBuildIdLimitPerQueue is the maximum number of Build IDs allowed to be defined in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.taskQueuesPerBuildId limits the number of Task Queue names that can be mapped to a single Build ID.
• limit.reachabilityTaskQueueScan limits the number of Task Queues to scan when responding to a GetWorkerTaskReachability query.
• limit.reachabilityQueryBuildIds limits the number of Build IDs that can be requested in a single call to the GetWorkerTaskReachability API.
• worker.removableBuildIdDurationSinceDefault is the minimum duration since a Build ID was last set as default in its containing set for it to be considered for removal. The value is used by the Build ID scavenger.
• frontend.rps.namespaceReplicationInducingAPIs limits the requests-per-second (RPS) rate for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceRPS.namespaceReplicationInducingAPIs is a per-host and per-Namespace RPS limit for Namespace replication-including APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceBurst.namespaceReplicationInducingAPIs is a per-host and per-Namespace burst limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.globalNamespaceRPS.namespaceReplicationInducingAPIs is a Cluster global, per-Namespace RPS limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). The limit is evenly distributed among available Frontend Service instances. If this option is set, it overwrites the per-instance limit configured with frontend.namespaceRPS.namespaceReplicationInducingAPIs. This config is EXPERIMENTAL and might be changed or removed in a later release.

Synchronous Update (preview)

Synchronous Workflow Update enables a gRPC client of a Workflow Execution to issue requests to that Workflow Execution and receive a response. These requests are delivered to and processed by a client-specified Workflow Execution. Updates are differentiated from Queries in that the processing of an Update is allowed to modify the state of a Workflow Execution. Updates are different from Signals in that an Update returns a response.

Updates can be handled by Workflows written using the latest Go and Java SDKs, with other languages to follow.

Any gRPC client can invoke Updates via the WorkflowService.UpdateWorkflowExecution API, which returns the Update outcome inline. Additionally, past Update outcomes can be observed via the WorkflowService.PollWorkflowExecutionUpdate API.

Update functionality is disabled by default but can be enabled per Namespace by setting the frontend.enableUpdateWorkflowExecution flag to true for that Namespace in dynamic config.

In this release, Update outcomes are stored for the same period as the Workflow history itself and can be observed for as long as that history exists. The execution and retention of Updates is configured via two optional dynamic configuration values:

• history.maxTotalUpdates controls the total number of Updates that a single Workflow Execution can support. The default is 2000.
• history.maxInFlightUpdates controls the number of Updates that can be “in-flight” (that is, concurrently executing, not having completed) for a given Workflow Execution. The default is 10.

Batch Reset via BatchOperation API

Description: The server now supports resetting a batch of Workflow Executions. StartBatchOperationRequest now supports the Operation type of ResetOperation.

This operation depends on two additional parameters:

• ResetReapplyType: Determines whether to reapply Signals after resetting the Workflow.
• ResetType: Determines which Event ID to choose when resetting the batch of Workflows.
  • RESET_TYPE_FIRST_WORKFLOW_TASK: Resets to the Event of the first Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.
  • RESET_TYPE_LAST_WORKFLOW_TASK : Resets to the Event of the last Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.

Dynamic rate limiter

The dynamic rate limiter is an advanced system-protection mechanism. By default, it’s disabled. When enabled, the per-host persistence rate limiter automatically adjust based on the average number of errors returned from persistence and the average latency of persistence requests.

The rate limit is adjusted using a multiplier between 0.1 and 1.0 (default 1.0) on the host-level persistence rate limiter. If either the ratio of persistence calls that result in an error or the average latency of persistence requests exceeds their respective thresholds, the multiplier is reduced. If the multiplier is less than its maximum (1.0) and neither threshold is exceeded, the multiplier is increased.

Several new dynamic config options are available to tune these adjustments. They are grouped in a map for each service, with the key <service>.persistenceDynamicRateLimitingParams, but currently only apply to the History Service.

• enabled: Whether dynamic rate limiting is enabled for this service (default false).
• refreshInterval: How often the rate limit multiplier and dynamic config options are updated (default 10s).
• latencyThreshold: The limit for average latency that should trigger a rate limit reduction when exceeded (default 0.0; not used).
• errorThreshold: The ratio of errors to total requests that should trigger a rate limit reduction when exceeded (should be between 0 and 1; default 0.0; not used).
• rateBackoffStepSize: The amount the rate limiter multiplier is reduced when either limit is exceeded (should be between 0 and 1; default 0.3).
• rateIncreaseStepSize: The amount the rate limiter multiplier is increased when neither threshold is exceeded and the multiplier is less than its maximum (should be between 0 and 1; default 0.1).

Updated defaultAuthorizer and defaultJWTClaimMapper

Important: If you use defaultAuthorizer or defaultJWTClaimMapper, this is a breaking change. You need to update your JWT generator to produce permission for the system-level role as “temporal-system:role” instead of “system:role”, where role is read, write, admin, or worker. For safe rollout, you can add “temporal-system:role” to the list of permissions before rollout and remove “system:role” after rollout is done.

Related PRs: #4227 #4244

Note that defaultAuthorizer and defaultJWTClaimMapper are not used by default. You must explicitly configure it to use them.

• Remove access to AdminService and OperatorService APIs from System RoleWriter.
• Remove the Namespace lowercase change. Admins of Namespace “Foo” will not have access to namespace “foo”.
• Previously, the defaultJWTClaimMapper treated “system” as a special Namespace, and granted system-level access to the admin of the Namespace named “system”. This is now changed to use “temporal-system”, which is a real system Namespace. Roles on the “temporal-system” Namespace will have system-level access.

Visibility

Deprecation heads up

• Standard Visibility is entering a deprecation process. We are marking it deprecated with this version (1.21), and we expect to remove it in v1.23.0. If you are using Cassandra as your Visibility store, check the deprecation warning below. If you use Standard Visibility with a SQL database, see the v1.20.0 release highlights for details about upgrading to Advanced Visibility. If you want to migrate to a different Visibility store, see Visibility store.
• Cassandra, as a Visibility store, will be deprecated together with Standard Visibility. Please migrate away from using Cassandra as your Visibility store. For migration information, see Visibility store.
• Docs links:
  • Concept - Visibility
  • Cluster Deployment - Visibility store configuration

Configs

Some Visibility configs have changed. The following lists all the changes, and all the old keys will be removed in the next release.

For the Server config

• Configure Elasticsearch as a Visibility store
  • New key: visibilityStore
  • Old key: advancedVisibilityStore

For dynamic configs

• Enable reading from secondary Visibility
  • New key: system.enableReadFromSecondaryVisibility
  • Old keys: system.enableReadVisibilityFromES / system.enableReadFromSecondaryAdvancedVisibility
• Secondary Visibility writing mode
  • New key: system.secondaryVisibilityWritingMode
  • Old keys: system.enableWriteToSecondaryAdvancedVisibility / system.advancedVisibilityWritingMode
• Visibility maximum read QPS
  • New key: system.visibilityPersistenceMaxReadQPS
  • Old keys: system.standardVisibilityPersistenceMaxReadQPS / system.advancedVisibilityPersistenceMaxReadQPS
• Visibility maximum write QPS
  • New key: system.visibilityPersistenceMaxWriteQPS
  • Old keys: system.standardVisibilityPersistenceMaxWriteQPS / system.advancedVisibilityPersistenceMaxWriteQPS

List and Scan APIs

The List and Scan APIs have the following changes.

• For List and Scan API, the default maximum page size is reduced from 10000 to 1000. If you call with a larger number, it will cap at 1000, and it will not throw an error. As for pagination, as long as you get a non-null token, there might be a next page: a page with less than 1000 does not necessarily mean that it’s the last page. The maximum page size can be changed using the dynamic config frontend.visibilityMaxPageSize; the dynamic config frontend.esIndexMaxResultWindow has been removed.
• The Scan API, even though its documentation says that the result set is not guaranteed to be sorted, is used to support the ORDER BY clause, and it had a default order. With this release, the ORDER BY clause is no longer supported. For Elasticsearch, it has been optimized for faster scanning without guaranteeing order.
• When using Elasticsearch, the Scan API is used to do a scroll search or open a point in time search if available. Now, it supports only point in time searches, as supported by Elasticsearch 7.10 and later. Currently, we do not support OpenSearch, and the Scan API won’t work with OpenSearch.
• Support for the ORDER BY clause is now disabled by default. You can enable it by setting the dynamic config system.visibilityDisableOrderByClause to false, and it can be set per Namespace. Although you can still enable it, this feature will likely be removed in a future release.

Frontend Service shutdown behavior

Description: The shutdown procedure for the Frontend Service has been changed slightly to work better with load balancers.

• First, the Frontend Service returns NOT_SERVING to gRPC health checks for the duration of the dynamic config frontend.shutdownFailHealthCheckDuration (default 0 seconds).
• Second, the Frontend Service tells gRPC to stop accepting new connections and RPCs. Currently running RPCs (including long polls) will not be interrupted yet.
• After the value of the dynamic config frontend.shutdownDrainDuration (default 1 second), currently running RPCs will be canceled.
• The sum of the two durations is limited to 300s. After that time, the process exits.

The default maximum long-poll duration is 60s. To avoid any client errors, we recommend setting frontend.shutdownDrainDuration to at least the 60s and setting frontend.shutdownFailHealthCheckDuration to at least as long as it takes your health checking system to notice the failed health check and stop sending traffic to that Frontend Service.

Temporal SDKs expect errors and retry intelligently, so these settings are not required for the correct execution of Workflow.

(The previous behavior had incoming RPCs during the drain interval return an application-level “Shutting down” error instead of a gRPC-level error. Also, the sum of the two durations was limited to 120s.)

Dynamic config validation

Description: For dynamic config values that are interpreted as durations, previously, an integer value (that is, with no unit specified) would be treated as an error, and the default value would be used instead. Now integers are treated as seconds. (Floats are still not allowed.)

It’s highly recommended to use values with units for clarity. If using the default YAML-based dynamic config, use strings like "30s" or "5m" instead of 30 and 300. If using a custom dynamic config implementation, ensure that values are returned with the Go type time.Duration.

Additionally, dynamic config values that fail to parse are logged at warning level. (After some amount of them, they’re logged only sporadically).

Per workflow limit

Related PR: #4326

Description: The default values for various per-Workflow limits have been reduced or tightened. More specifically:

• limit.numPendingChildExecutions.error, limit.numPendingActivities.error, limit.numPendingSignals.error, and limit.numPendingCancelRequests.error have been reduced to 2000. If exceeded, Workflow Tasks that generate new Child Workflows or Activities, signal external Workflows, or request cancellation of external Workflow Commands will fail. Be aware that there’s no partial acceptance of Workflow Commands. As a result, if Commands in a single Workflow Task response exceed those limits, the Workflow Task will keep failing.
• history.maximumSignalsPerExecution is now 10,000. If exceeded, new Signal requests sent to the Workflow are rejected and return an InvalidArgument error.

All changes

2023-06-23 - 521641f - Update updateRegistry store on mutable state reload (#4532)
2023-06-22 - b8feb95 - Perform task range completion on shard rangeID update (#4528)
2023-06-21 - 8699916 - Prevent scavenger from removing build ids that were recently default for a set (#4527)
2023-06-21 - 5b4e2e3 - Add per build id timestamp for when it was last made set default (#4526)
2023-06-21 - cdf158e - Add per version set timestamp for when it was last made default in queue (#4524)
2023-06-21 - 2cd1579 - Simplify XDC test logic (#4522)
2023-06-21 - f2fb098 - Add string dynamic config filter by namespace id (#4523)
2023-06-21 - 198f0e4 - remove monitor WhoAmI (#4506)
2023-06-20 - a62450a - Bring back branch token for branch deletion (#4521)
2023-06-20 - 7f064ea - Wrap setting futures in taskQueueManager (#4494)
2023-06-20 - 35ad0a5 - Validate startedEventId when close WorkflowTask (#4519)
2023-06-20 - 36c60c3 - Update duplicate WorkflowExecutionInfo CLI proto (#4520)
2023-06-16 - bdd5801 - Use search attribute size limit for BuildIds truncation (#4516)
2023-06-16 - 20baa97 - Spool tasks with "default" versioning directive along with unversioned tasks (#4512)
2023-06-16 - 7f3bb1a - Apply stricter rate limiting for the RegisterNamespace API (#4513)
2023-06-16 - 3392a7a - Fix wrong format specifier in client_gen (#4511)
2023-06-16 - bddbc74 - Cluster not found (#4501)
2023-06-16 - 9330f52 - Record ES bulk took time after err check (#4508)
2023-06-15 - 187f699 - Add InvalidStateTransitionWorkflowExecutionUpdateCounter metric (#4503)
2023-06-15 - 0a109d5 - Add missing reset type params (#4505)
2023-06-15 - 6a9d9d8 - Fix typo in batch reset (#4504)
2023-06-15 - 1644435 - Check if WorkflowTaskFailed event is nil when workflow completes (#4500)
2023-06-15 - ae2d232 - Refactor matching operations when user data disabled (#4493)
2023-06-14 - 73189b7 - Return NonRetryableApplicationError if input is invalid in ForceReplicationWorkflow and NamespaceHandoverWorkflow (#4498)
2023-06-14 - c712b57 - Fix error checking (#4496)
2023-06-14 - 021a84e - Functional tests: Set ReturnNewWorkflowTask for task poller always to true (#4476)
2023-06-14 - c5efe93 - Remove usages of AcceptedRequest field (#4489)
2023-06-14 - 2db855d - Schedule a new workflow task after force flush buffer events (#4490)
2023-06-14 - 8795c6c - Add namespace migration support for replication stream (#4488)
2023-06-14 - 741edb5 - Test update not found due to shard reload case (#4495)
2023-06-13 - 363e92c - Disable revive.unexported-return linter (#4466)
2023-06-13 - a24557b - Log errors instead of crashing in OTel metrics handler (#4485)
2023-06-13 - 598b0c3 - Fix some typos and comments in metrics package (#4478)
2023-06-13 - 03ca8e8 - Refactor task queue user data loading (#4487)
2023-06-13 - 71748a0 - Only replicate user data for multi cluster setups (#4484)
2023-06-12 - ea9ad78 - Reuse unloadTaskQueue helper in versioning_test (#4482)
2023-06-12 - a6ae651 - Fix some invalid metrics definitions (#4465)
2023-06-12 - fc28119 - Increase test timeout to 30 minutes (#4480)
2023-06-12 - 75ffab3 - Add TestDescribeTaskQueue to versioning tests (#4474)
2023-06-12 - 6e43222 - Add TestDescribeWorkflowExecution to versioning tests (#4473)
2023-06-12 - 30c9be5 - Add worker identity to TestUpdateWorkflow tests (#4475)
2023-06-10 - eeec6e0 - Get rid of MetricName type (#4479)
2023-06-09 - eeee3a0 - Update poller info on base tqm (#4472)
2023-06-09 - 06d2ce8 - Add dynamic config to disable loading task queue user data (#4477)
2023-06-09 - d44aa1e - Add special rate limiter for namespace replication inducing APIs (#4455)
2023-06-09 - cee5ba3 - Move poller map to matchingEngine (#4471)
2023-06-09 - 65300d5 - Remove duplicate keys from dynamic config (#4470)
2023-06-09 - e100485 - Block versioned cross-tq activities, child workflows, CAN (#4459)
2023-06-09 - bef83ae - Create normal workflow task if unseen events exist (#4469)
2023-06-09 - 54abd99 - Merge history tree insert with history node append (#4464)
2023-06-09 - 30be627 - Implement build id scavenger (#4430)
2023-06-08 - d8f70e1 - Remove unused metrics.ServiceIdx (#4454)
2023-06-08 - 912b889 - Remove unused MetricType concept (#4452)
2023-06-08 - 137886a - Implement merge sets operation for Build IDs (#4447)
2023-06-08 - dd4c7de - Enable workflow update is server development configs (#4461)
2023-06-08 - ac1944f - Enforce uniqueness of replication stream (#4448)
2023-06-08 - a4bc991 - Add more asserts on history events attributes (#4456)
2023-06-08 - 27d0a1f - Use task queue base name for dynamic config (#4445)
2023-06-08 - e5d58dc - More functional tests for worker versioning (#4446)
2023-06-08 - f8453ce - Hide BuildIds search attribute for schedules (#4458)
2023-06-07 - 7608a0c - Add retryable client for persistence task manager (#4457)
2023-06-07 - a51abcf - Comments and event attributes in expected history in tests (#4426)
2023-06-07 - b61c3f3 - Drop task when namespace is not on the cluster (#4444)
2023-06-06 - 79e212d - Add functional test to deduplicate updates by ID (#4438)
2023-06-06 - da83b0a - Add the event batch ID to update CompletionInfo (#4441)
2023-06-06 - c107bfd - Take acceptedEventID write out of commit callback (#4443)
2023-06-06 - 25c7342 - use existing acquire shard test setup (#4442)
2023-06-05 - 81c6197 - Remove unneeded FlushAndCreateNewBatch call from AddWorkflowTaskScheduleToStartTimeoutEvent (#4439)
2023-06-05 - d2ec969 - Add functional test for shard reload during update (#4437)
2023-06-02 - 6691620 - Write accepted event ID in update completion event (#4436)
2023-06-02 - 426f6d1 - Improve flexibility of UpdateInfo proto message (#4433)
2023-06-02 - de7f679 - Dynamic rate limiter (#4390)
2023-06-02 - 3bee7f7 - Upgrade OTEL module to the latest version (#4423)
2023-06-02 - c4c6d6d - Limit max number of update for workflow execution (#4413)
2023-06-02 - 2c90feb - Add test for heartbeat from speculative workflow task (#4425)
2023-06-02 - ce47340 - Fix task inMemoryNoUserLatency 2.0 (#4431)
2023-06-02 - b34d21a - Fix handling of batch signal input payloads (#4374)
2023-06-01 - d619d79 - Limit build IDs per queue to 100 (#4415)
2023-06-01 - d909a96 - Fix task inMemoryNoUserLatency (#4429)
2023-06-01 - 5bc1a72 - Add update counts to mutable state statistics (#4422)
2023-05-31 - e8c2a19 - Fix TestDefaultPublishRetryPolicy (#4421)
2023-05-31 - 35e7696 - Remove speculative workflow task restoration code (#4424)
2023-05-31 - 10a14e3 - Inline UpdateInfo Storage (#4420)
2023-05-31 - 5e0459e - Add functional test for update with existing scheduled workflow task (#4414)
2023-05-30 - 2a4aec0 - Make server fx init more extendable (#4419)
2023-05-30 - 8aab8ab - Improve task latency calculation and retry behavior (#4408)
2023-05-30 - 35708d9 - Fix delete visibility task notification (#4416)
2023-05-26 - d2f2516 - Refactor ES query converter (#4382)
2023-05-26 - 81eaa77 - Allow workflow to complete if there are updates in the registry (#4412)
2023-05-26 - 7e64d1d - Final update of api/sdk for worker versioning
2023-05-26 - a950528 - Allow GetTaskQueueUserData and matching long polls to use a 5m timeout (#4407)
2023-05-26 - 2de41e5 - Handle worker versioning for sticky queues (#4397)
2023-05-26 - b717268 - Address comments from #4346 review (#4405)
2023-05-26 - 188a7ec - Update API and SDK to latest master (#4402)
2023-05-26 - 209e250 - Worker versioning leftovers (#4400)
2023-05-26 - 8a325f8 - Implement GetWorkerTaskReachability API (#4346)
2023-05-26 - afe0304 - Add dynamic config to disable worker versioning progress APIs (#4398)
2023-05-26 - 453f54a - Server updates for use_compatible_version (#4394)
2023-05-26 - 773aefd - Add a few functional tests for worker versioning (#4375)
2023-05-26 - 8ffe902 - Redirect spooled tasks using versioning directive (#4357)
2023-05-26 - 3805cfd - Implement use_latest_build_id for activities, child workflows, continue-as-new (#4347)
2023-05-26 - 42e766e - Propagate task queue user data with long-poll requests (#4334)
2023-05-26 - bc58ba1 - Fix some versioning redirection logic (#4286)
2023-05-26 - 53a4a3a - Worker versioning - add BuildIDs search attribute (#4284)
2023-05-26 - 6b4e3cc - Redirect to versioned queues in matching (#4241)
2023-05-26 - ff6e6af - Minor fixes
2023-05-26 - 2d2f94b - Implement task queue user data seed (forced replication) (#4259)
2023-05-26 - 9e9f9ca - Rename imports to match convention (#4261)
2023-05-26 - 6e7917b - Implement task queue user data replication (#4237)
2023-05-26 - 1a3e277 - Copy build id to binary checksum when present (#4230)
2023-05-26 - 931adff - Thread version stamps through from history to matching (#4220)
2023-05-26 - 35dfeed - Misc versioning tasks (#4221)
2023-05-26 - 219f674 - Pass worker version capabilities through poller info (#4219)
2023-05-26 - 283931a - Refactor to use OwnsUserData (#4218)
2023-05-26 - 84437cf - Versioning data merge algo and hybrid logical clock utils (#4205)
2023-05-26 - 0a9785a - Refactor to use new protos and persistence (#4194)
2023-05-26 - c2fdc9f - Proto and persistence definitions for replicated versioning (#4172)
2023-05-26 - 47d3fe5 - Hot shard metrics (#4365)
2023-05-26 - 759aa2c - Reject signal if workflow attempted to close (#4395)
2023-05-26 - 01034e4 - Add ES point in time to Scan API (#4380)
2023-05-26 - d1e1243 - Rewrite some workflow history size calculation (#4258)
2023-05-26 - 15bc10d - Fix mutable state get event error handling (#4396)
2023-05-26 - 6132e58 - Improve retry util (#4403)
2023-05-26 - df9162b - Bump CI builder base version (#4406)
2023-05-26 - eb2845e - Fix prometheus metrics (#4393)
2023-05-25 - 0059b83 - Use WTFailed.EventId as completion_event_batch_id when workflow is terminated or timed out (#4392)
2023-05-25 - 9aa4ade - Fix merge issues from PR 4317 (#4404)
2023-05-25 - 378e1e6 - Add function to get custom search attributes mapper in namespace registry (#4317)
2023-05-24 - fea13ad - Do not generate scheduled task with timestamp in the past (#4367)
2023-05-24 - c2aee40 - Refactor UpdateWorkflowExecution functional tests (#4388)
2023-05-24 - 2fce7a5 - Enforce mutable state size limit (#4350)
2023-05-24 - df69684 - Bugfix: backfill replication queue state read ID calculation (#4391)
2023-05-24 - ca28eb0 - Fix partial namespace creation (2.0) (#4330)
2023-05-23 - 68856fe - Do not let API call timeout if workflow can't be locked (#4341)
2023-05-23 - b191203 - Avoid making a subscope in tallyMetricsHandler if possible (#4363)
2023-05-23 - e720b55 - fix: create btree_gin extension only if not exists (#4371)
2023-05-23 - 6be18a1 - Add functional test with OpenSearch 2 (#4124)
2023-05-23 - 6b600dc - Add metric to track ES bulk response took field (#4355)
2023-05-19 - 981d22b - Remove unused toDC config (#4370)
2023-05-19 - 09fc4c6 - Fail workflow execution update fast if workflow task constantly fails (#4376)
2023-05-19 - 944b893 - Add functional test for speculative workflow task in sticky task queue timeout (#4369)
2023-05-19 - d73481a - Add functional test to cover conversion of scheduled speculative workflow task to normal (#4364)
2023-05-19 - c719eac - Capture UpdateInfos in MutableState mutation (#4372)
2023-05-19 - ed49604 - Return # of state transition when generating last replication tasks (#4352)
2023-05-19 - f723450 - Bugfix: do not leak gRPC stream (#4368)
2023-05-18 - e61de47 - Add functional test to terminate workflow with pending speculative workflow task (#4366)
2023-05-18 - 680e4e1 - Fail query fast if workflow task attempt greater or equals 3 (#4359)
2023-05-18 - 9a6edae - Fix: failWorkflowTask returns wrong nextEventBatchID when failing speculative workflow tasks (#4354)
2023-05-18 - 3bfbb81 - Don't return workflow task if it is speculative and doesn't have any messages (empty) (#4356)
2023-05-17 - 007606d - Fix: use normal task queue if sticky worker unavailable during update workflow API call (#4353)
2023-05-16 - ab2f6b9 - Ensure workflow key is fully specified (#4351)
2023-05-16 - b9a9029 - Use the correct event version for task refresh (#4349)
2023-05-16 - f74966f - Increase default number of Elasticsearch bulk processor workers to 2 (#3738)
2023-05-16 - c42e7dd - Per-shard per-namespace rate limiter (#4302)
2023-05-16 - 5e9f623 - Add a config flag for enabling async updates (#4342)
2023-05-15 - 1240df2 - Fix mutable state access after workflow lock is released (#4333)
2023-05-15 - d5f0264 - Tag metric tracking custom order by with namespace (#4344)
2023-05-15 - d201491 - Bugfix: ExecutableTaskTracker LowWatermark funcion (#4343)
2023-05-15 - 9f9b828 - Fix misc. context value propagation issues (#4279)
2023-05-15 - 9712711 - Handle ProtocolMessageCommands in WFT completion (#4328)
2023-05-15 - 1a479ad - Use sequential task processor for replication (#4339)
2023-05-15 - 2607d61 - Add some documentation to the maximumInterval field of ExponentialRet… (#4316)
2023-05-15 - d4af860 - Add some documentation to the maximumInterval field of ExponentialRet… (#4318)
2023-05-15 - 467ea4c - Update Go SDK to v1.22.2 (#4327)
2023-05-15 - 11f4e68 - Add linters goerr113, paralleltest, and tparallel (#4239)
2023-05-15 - 0d140a3 - Fix caller info for migration activities (#4340)
2023-05-15 - 9d7545b - Truncate activity failure in mutable state (#4338)
2023-05-14 - a05d21c - Fix: set nonRetryable field of server failure to "true" when failure size exceeds limit (#4335)
2023-05-14 - d6ab8c5 - Add additional metrics for stream replication (#4323)
2023-05-12 - a136de7 - Make sure shard info always stays in a cleanup state (#4336)
2023-05-12 - aca0d91 - Change schedules default catchup window to 1 year (#4332)
2023-05-12 - f84a3c3 - Handful of minor changes to the update package (#4331)
2023-05-12 - f07921b - Update api-go (#4320)
2023-05-12 - 62611e2 - Refactor task queue related methods of mutable state (#4325)
2023-05-11 - a343bea - Revert "Fix partial namespace creation (#4267)" (#4322)
2023-05-11 - cd8821c - Enforce stricter per workflow limit (#4326)
2023-05-11 - 40e7627 - Remove unneeded duration/seconds conversion (#4321)
2023-05-11 - 6e7c669 - Create dynamic config to enable manual pagination for Elasticsearch (#4314)
2023-05-11 - a3ae602 - Add replication dlq metrics (#3876)
2023-05-11 - 3cbab42 - Fix partial namespace creation (#4267)
2023-05-11 - 58e247b - Prevent workflow close when updates are undelivered (#4313)
2023-05-11 - a6861bf - Async Updates and the Poll API (#4309)
2023-05-11 - a1fe0e6 - Manually build ES pagination query for default sorter (#4271)
2023-05-11 - d040cdf - Improve Update.WaitAccepted - followup (#4312)
2023-05-11 - 3bfac8a - Improve Update.WaitAccepted (#4311)
2023-05-10 - f34d5b7 - Default disable order by clause support (#4305)
2023-05-10 - 8ed296c - Use executable task tracker to dedup tasks when stream is re-established (#4310)
2023-05-10 - 25b7cbc - Add a per-workflow limit to in-flight updates (#4307)
2023-05-10 - 2339531 - Fix adding multiple search attributes of same type at once (#4273)
2023-05-10 - 1c7e895 - Refactor the Update type to a state machine (#4297)
2023-05-10 - dc7f28d - Skip over entire time range if paused and batch and cache time queries (#4215)
2023-05-10 - 807b791 - Add automaxprocs. (#4301)
2023-05-09 - 46caef3 - Refactor matching handlerContext (#4303)
2023-05-09 - 1597451 - Simplify task queue liveness (#4290)
2023-05-09 - 9f66510 - Use search attribute type map in visibility archival (#4304)
2023-05-09 - d0007b0 - In memory timeout timer queue for speculative workflow task (#4254)
2023-05-09 - 8bc74cb - Add server stream interceptor (#4300)
2023-05-08 - 5af117e - Bring PollWorkflowExecutionUpdate to HistoryEngine (#4298)
2023-05-08 - 8e6203a - Event buffer size limit (#4296)
2023-05-08 - 8fa4d3f - Correct spelling: updat_id -> update_id (#4299)
2023-05-08 - 89f707b - Small refactor to matching signalFatalProblem (#4292)
2023-05-08 - a43d451 - Enable Elasticsearch body compression with gzip (#4289)
2023-05-05 - 392e769 - Bump gocql to v1.4 (#4294)
2023-05-05 - 9fd0e26 - Handle namespace not found in history scan (#4287)
2023-05-05 - 08dcc70 - Expose better logs for dynamic config conversion errors (#4282)
2023-05-05 - c243c08 - Verify WFTs with large event buffers fail (#4272)
2023-05-05 - 9fdd08c - Make shard ownership more resilient when shard is busy (#4283)
2023-05-04 - 0bdbbfc - Allow plain int seconds for duration in dynamic config (#4280)
2023-05-04 - bb7d4af - Introduce update.Storage (#4277)
2023-05-04 - ea4946c - Use GracefulStop in frontend (#4212)
2023-05-04 - e6e2372 - Add Reset to Batch API (#4157)
2023-05-03 - 1d1ef27 - Remove force task refresh in passive processing logic (#4278)
2023-05-03 - 6e43d54 - Fix task attempt reset (#4276)
2023-05-03 - b757d8b - Fix metric tag for delete execution task (#4275)
2023-05-03 - 5783e78 - Adds update state indexing to MutableState (#4224)
2023-05-01 - d4cd05e - Remove support for ES PIT and scroll in ES client (#4223)
2023-05-01 - b6525f1 - Remove unused matching DB layer impl (#4141)
2023-05-01 - 2763a0f - Remove usages of ES PIT and Scroll (#4249)
2023-05-01 - 661d6d4 - Refactor fx Start/Stop usage (#4248)
2023-05-01 - 80687d7 - Remove unused functions in shard interface (#4255)
2023-05-01 - 2aa5481 - Remove unused dynamicconfig.Collection in top level fx container (#4253)
2023-04-28 - d6d7a0b - Add more logging (#4252)
2023-04-28 - 625f6d2 - Increase shutdown timeout to 5 minutes (#4195)
2023-04-28 - d94444d - Change dynamic config for worker versioning to be specific to data APIs (#4238)
2023-04-28 - 10c20a6 - Update dynamic config related to visibility (#4250)
2023-04-28 - 8740a20 - Support signals that do not immediately generate a workflow task (#4091)
2023-04-27 - 7a24d8e - defaultJWTClaimMapper treat namespace as case sensitive (#4244)
2023-04-27 - 62c35aa - Remove noisy queue action log (#4245)
2023-04-27 - 1ea69e0 - Turn autofix for golangci-lint and add goimports make target (#3958)
2023-04-27 - 4d27487 - Run no-op claim mapper even without auth info (#4197)
2023-04-27 - 23147c3 - Attempt to unify queue state (#4190)
2023-04-26 - ccbfa7d - Add histogram support in metricstest (#4177)
2023-04-26 - b1caaa0 - Code style simplification and improved readability for returning errors (#4236)
2023-04-26 - cb0a9c7 - Code style fix (#4235)
2023-04-26 - c0029aa - [WIP] Fix error return bug and some minor code style (#4234)
2023-04-25 - d8d9289 - Fix test race condition (#4232)
2023-04-25 - ec3ff8b - Block non admin role to access system services (#4227)
2023-04-25 - 5797648 - Fix broken tests (#4231)
2023-04-25 - 8eb42e3 - Bugfix shard ID mapping util & stream receiver monitor (#4229)
2023-04-25 - 7c4f3fd - Rename some var names in replication stream logic (#4211)
2023-04-25 - 2c2fb71 - Support ES secondary_visibility index in docker config template (#4213)
2023-04-25 - aceb878 - Use sqlite by default for functional test (#4210)
2023-04-25 - 683064c - Add sequential task scheduler (#4130)
2023-04-24 - e5372ec - Add dynamic config flag to disable worker versioning (#4222)
2023-04-24 - 0953f9b - Call List API from Scan API for SQL visibility (#4217)
2023-04-24 - 37cebd0 - Update vis store dev configs (#4208)
2023-04-24 - 6715e38 - Revert "Enable Scan API in SQL visibility store" (#4214)
2023-04-24 - 12655c4 - Refactor configs for visibility (#4095)
2023-04-24 - b97be78 - Wire replication stream caller side (#4178)
2023-04-21 - a52bb73 - Test sync update on sticky task queue (#4200)
2023-04-21 - ba09219 - Fix cron for schedules that do not match any time (#4206)
2023-04-21 - d6c298b - Use int64 as type for queue readerID (#4176)
2023-04-21 - feb938e - Add flush method to flush buffer (#3521)
2023-04-20 - 6c39401 - Validate messages against pending updates (#4188)
2023-04-20 - 0ca9e83 - Convert speculative workflow task to normal when it is persisted (#4069)
2023-04-19 - 501fdae - Implement batch history replication API (#4167)
2023-04-19 - bdbec13 - Rename UpdateQueueState to SetQueueState (#4187)
2023-04-19 - becc2f3 - Remove DefaultVisibilityIndexName from history config (#4189)
2023-04-19 - 8d3836a - Don't log error for ownership lost in matching task reader (#4183)
2023-04-19 - 2ff8d37 - Use correct API for transfer queue (#4186)
2023-04-19 - 8b7e3f7 - Add visibility manager dependency to delete manager (#4138)
2023-04-18 - af066c3 - Enable Scan API in SQL visibility store (#4161)
2023-04-17 - e792a68 - Add util to map shard IDs (#4175)
2023-04-17 - 0cf0365 - Handle errors when registering new queue readers (#4055)
2023-04-17 - 74d3f15 - Enforce visibility max page size (#4164)
2023-04-17 - 3cf1eb8 - Abort a task if schedule shuts down (#4125)
2023-04-17 - 2d0272c - Remove unused exporter from openTelemetryProviderImpl (#4169)
2023-04-14 - 52b0365 - Remove version_set_id from CompatibleVersionSet (#4170)
2023-04-14 - c1ec2bb - Switch from opaque branch token to branch info (#4168)
2023-04-13 - cdd1a21 - Split history tree insert from history node append (#4147)
2023-04-13 - 87dbab0 - Update Go SDK to v1.22.1 (#4166)
2023-04-12 - b8aab10 - Update Go SDK to v1.22.0 (#4163)
2023-04-12 - 1e2ecc4 - Fix schedule catchup window metric calculation while paused (#4152)
2023-04-12 - 04729dc - Add a hermetic PipeListener for testing net code (#4098)
2023-04-12 - cb70d58 - Use fx parameter object pattern for ringpop factory (#4115)
2023-04-11 - 0e18e19 - Validation API should use low priority lock (#4140)
2023-04-07 - 3ce9a17 - Fix error handling add search attributes (#4148)
2023-04-05 - 0d6e92f - Addressing comments in PR 4134 (#4144)
2023-04-01 - 6f1f022 - Change Scan API to have no Order(ES) (#4134)
2023-03-31 - ca1d906 - Do not generate workflow timeout task if workflow is closed (#4135)
2023-03-31 - 2ef3afd - Properly handle replication task lifecycle (#4126)
2023-03-31 - 7244681 - Reconnect on a few more gocql errors (#4132)
2023-03-31 - c97c973 - Use correct time for initial schedule memo (#4128)
2023-03-31 - fec04d1 - Reuse timer in schedule workflow (#4129)
2023-03-31 - 8955ebf - Cleanup redundant input args (#4096)
2023-03-29 - 0ca5360 - Auto clear stickiness (#4104)
2023-03-29 - d4b64e8 - Reduce the # of resend events if workflow is a reset workflow (#4123)
2023-03-29 - 9bc7edd - Remove now as required var for workflow mutation (#4122)
2023-03-29 - 3a56652 - Enforce concurrent count limit to all long poll calls (#4120)
2023-03-29 - d4e3251 - Fix data race in test (#4121)
2023-03-29 - 7ce28f8 - Log warnings for retryable ringpop errors (#4102)
2023-03-29 - 8858547 - Inject service name (#4119)
2023-03-29 - 8bd067a - Add poison pill support for replication stream (#4116)
2023-03-29 - 7228219 - Don't include stack traces for warnings during tests (#4101)
2023-03-29 - 8f030db - Wire replication stream E2E (#4097)
2023-03-29 - fca3662 - Add history size bytes to SQL visibility (#4090)
2023-03-29 - f4c1888 - Addressing PR-4099 comments (#4113)
2023-03-29 - f1b88e8 - Bump UI version (#4047)
2023-03-28 - bccdaaa - Add base workflow info to workflow replication task (#4107)
2023-03-28 - e4a49b2 - Add flag to enable execution scanner event Id validator (#4114)
2023-03-28 - 47424c0 - Better error message in ES query converter (#4112)
2023-03-28 - 939eefa - Fix missing import bug (#4111)
2023-03-28 - a20faeb - Use dedicated proto definition for replication stream (#4106)
2023-03-28 - afc5ac7 - Remove common.Daemon from Monitor interface (#4084)
2023-03-28 - 4431446 - Add replication stream receiver logic (#4093)
2023-03-26 - aef7c0b - Fix some minor linter warnings in the membership package (#4100)
2023-03-24 - f1ef4db - Change update namespace to upsert custom search attributes (#4080)
2023-03-24 - 8bdfccc - Refresh per-namespace worker periodically (#4103)
2023-03-24 - 3d6ac72 - Fail QueryWorkflow if last workflow task failed (#4099)
2023-03-24 - d05b15f - Support empty queries when listing archived workflows (#4027)
2023-03-24 - 63b22ca - Add support for workflow start delay (#3984)
2023-03-23 - 11cef80 - Flush buffer event when failing workflow task (#4010)
2023-03-23 - 19a30b9 - Fix build id compatibility spelling (#4086)
2023-03-23 - cd15735 - Hide ringpop config validation in ringpop package (#4085)
2023-03-23 - aee7c5f - Stub out new PollWorkflowExecutionUpdate API (#4081)
2023-03-23 - 5210ad4 - Add gRPC replication stream (#4057)
2023-03-23 - e67a1ce - Account for local activity run time in schedule sleep (#4079)
2023-03-22 - 832f4e6 - Break replication ack manager into smaller component (#4088)
2023-03-22 - cc2eb80 - Fix test failure from merge conflict (#4087)
2023-03-22 - 22522bf - Use more reliable workflow mutation check (#4076)
2023-03-22 - 6974ed3 - Fix namespace handover replication queue notification (#4082)
2023-03-22 - 66352ed - Track and emit metrics for entities in a single workflow (#4065)
2023-03-22 - 1439b87 - Handle max signal count limit for signal transfer task (#4051)
2023-03-22 - b3bc602 - Remove Lookup method from Monitor interface (#4072)
2023-03-21 - 19b465b - Remove unused methods from Monitor interface (#4071)
2023-03-21 - e8be006 - Lock current execution on start workflow (#4066)
2023-03-21 - d53f1b6 - Use preemptable caller type for low priority history tasks (#3993)
2023-03-21 - 78fdba2 - Add deadlines to schedule local activities (#4078)
2023-03-21 - 2af5bf4 - Fix SQL visibility page token deserializer (#4074)
2023-03-21 - fc56eea - Rename visibility writing mode consts (#4068)
2023-03-21 - dc018cd - Fixing saving custom search attribute after upgrading SQL DB (#4062)
2023-03-21 - e7b4276 - Add counter to track custom order by usage in Elasticsearch (#4064)
2023-03-21 - 4033f21 - Rename variables in timer package (#4067)
2023-03-21 - 411c65b - Set default values in returned calendar specs (#3969)
2023-03-20 - 0d0cad4 - Change disable order by dynamic config to per namespace (#4063)
2023-03-16 - 129f2f8 - Pass in namespace ID to initialize new history branch (#4058)
2023-03-15 - 07ffab5 - Use new API's simpler representation of versioning data (#3432)
2023-03-14 - 086f5ec - Rename inflight WT to started WT (#4043)
2023-03-14 - ddd0911 - Add gRPC stream error interceptor (#4019)
2023-03-13 - ed47a16 - Fix UnhandledCommand error for workflow task completion in CancelTimer (#4050)
2023-03-13 - b520a5b - Improve message sequence validator (#4037)
2023-03-10 - 122bb36 - Add more test for XDC (#4025)
2023-03-09 - 7b6ce25 - Unique owner for each shard context object (#4008)
2023-03-09 - 4c47aec - Convert query converter error to invalid argument (#4036)
2023-03-09 - 7f848d1 - Move update registry from mutable state to workflow context (#4032)
2023-03-09 - 85aa550 - Extract cluster config logic into ConfigureCassandraCluster() (#4034)
2023-03-09 - 75facb1 - Remove unused HostsUpdated field from ChangedEvent (#4015)
2023-03-09 - 818cbb1 - Refactor gocql.NewSession to allow passing in cluster config factory method (#4033)
2023-03-08 - 010917c - Flush buffer event with single cluster namespace (#4031)
2023-03-08 - d1159dc - Add support for namespace readthrough to persistence (#3908)
2023-03-06 - 6ea38d7 - Update history task reader progress (#4004)
2023-03-06 - e7cdd71 - Register history queue readers (#3995)
2023-03-06 - dc69278 - Restructure replication task implementation & UT (#3992)
2023-03-06 - e4162e8 - Remove unused persistence GetHistoryTask API (#4003)
2023-03-06 - 94f5824 - Add more test cases for XDC (#4009)
2023-03-06 - c8ad74a - Return raw replication task info from read dlq api (#4007)
2023-03-05 - c172708 - Hide remaining exported symbols in ringpop package (#4012)
2023-03-05 - 9740e7f - Move RingPop config validation to membership package (#4011)
2023-03-05 - 8c7e79a - Fix comment above NewServerFxImpl (#4013)
2023-03-05 - 061593b - Move ringpop code from membership to ringpop package (#3999)
2023-03-05 - abd0297 - Extract interface from membership.HostInfo (#3996)
2023-03-02 - 701b775 - Specify readerID on loading history tasks (#3994)
2023-03-02 - 715fe3f - History task reader persistence hint API (#3986)
2023-03-02 - 2977c72 - Upgrade sqlite to v1.21.0 to address security vulnerability (#4001)
2023-03-01 - 7086b94 - Make types not used outside of membership unexported (#3998)
2023-03-01 - 13e3170 - Use snake case for file names in common/membership (#3997)
2023-02-28 - eea242d - Refactor history branch manipulation logic into its own utility (#3946)
2023-02-27 - 35dcba8 - Improve scheduled queue lookahead logic (#3982)
2023-02-24 - edd92bb - Streamline localstack s3 development (#3976)
2023-02-22 - 3b98258 - Add SQL query converter tests (#3972)
2023-02-21 - bc2d025 - Refactor SQL query converter (#3971)
2023-02-21 - f522ff3 - Upgrade dependency for next release (#3974)
2023-02-21 - 3b71393 - Fix passing pointer of slice element in SQL query converter (#3970)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.21.0)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools