Release highlights

Elasticsearch schema changes

Elasticsearch schema v2 has been added.

The Elasticsearch schema v2 supports new built-in Search Attributes. This schema only adds new mappings, it is not a full reindex.

Use the v2 schema upgrade script schema/elasticsearch/visibility/versioned/v2/upgrade.sh. This can be done before or after upgrading to this release, but it must be done before using the experimental Scheduled Workflow feature.

Metrics

Multiple interfaces in the metrics package are deprecated. The following interfaces are scheduled to be removed in v1.18.0:

• common/metrics/interfaces.go
  • UserScope
  • Scope
  • Client
  • Reporter

Initial implementation to replace these interfaces is now:

• common/metrics/metrics.go
  • MetricsHandler

Workflow Execution deletion

Operator Service now exposes the new DeleteWorkflowExecution API, which accepts the name of a Namespace and a Workflow Execution. Both running and completed Workflows can be deleted. Running Workflows are terminated first. The API is exposed via tctl:

$ tctl config set version next
$ tctl workflow delete --workflow-id my_workflow

Namespace deletion

Operator Service now exposes the new DeleteNamespace API, which accepts the name of a Namespace to delete. Be very careful with this API: it can’t be undone. The API is exposed via tctl:

$ tctl config set version next
$ tctl namespace delete --name my_namespace_name

Namespace deletion is implemented using a system Workflow that is run by a system Worker. It is an asynchronous process, but the Namespace name can be reused as soon as API returns:

1. The Namespace is marked as deleted and renamed to a temporary name. After the DeleteNamespace API returns, the name can be reused (that is, a Namespace with the same name can be created).
2. All Workflow Executions in the deleted Namespace are deleted asynchronously from the database using the DeleteWorkflowExecution API described earlier.
3. The Namespace is deleted from the database.

Please note: this feature is still under active testing.

Host-level priority Task processing

• New Task processing framework in History service for multi-tenancy. This feature is disabled by default in v1.17 and will be enabled in a future release.
• Improved Task retry mechanism to avoid Tasks occupying a Worker goroutine for an extended period of time and blocking the entire queue.
• Configuration for enabling host-level Task Worker pool (history.timer/transfer/visibilityProcessorEnablePriorityTaskScheduler).
• Configuration for Worker pool size (history.timer/transfer/visibilityProcessorSchedulerWorkerCount) for controlling total traffic sent downstream.
• Configuration for adjusting high-priority Task quota allowed for each Namespace (history.timer/transfer/visibilityTaskHighPriorityRPS).
• Configuration for Task processing round-robin weights (history.timer/transfer/visibilityProcessorSchedulerRoundRobinWeights) among various priorities.
• New task_priority tag for Task processing metrics.

Scheduled Workflows (experimental)

Scheduled Workflows are a new feature that's more flexible than the existing CronSchedule. They're disabled by default in this release, but you can enable them with dynamic config settings. See more docs [here] (docs not written yet, add link later).

SQLite persistence

The SQLite module was replaced with a purego implementation. CGO is no longer required to use SQLite persistence. This means that you can use binaries attached to this release to run local server for development. Just download archive corresponding to your OS and architecture (Mac M1 works!), unzip it, and run server locally without docker-compose and any other dependencies with simple command:

$ ./temporal-server --env development-sqlite start

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag v1.17.0)

Server
Server With Auto-Setup
what is Auto-Setup?
Admin-Tools

What's Changed

• 2022-06-08 - f0f1793 - Fix error details bug in deletenamespace workflow (#2954)
• 2022-06-08 - d16cc46 - Adding serveroption for metrics.MetricHandler (#2978)
• 2022-06-08 - 9de6b45 - Fix getRemoteClusterInfo race condition (#2971)
• 2022-06-08 - 8114a4f - Remove unindexed fields from index schema template (#2976)
• 2022-06-08 - 5135a68 - check for nil config and return noop (#2979)
• 2022-06-08 - 49b57b7 - Don't index search attribute used only for passing data through visibility (#2972)
• 2022-06-07 - d3dc17e - Remove caching objects to prevent leak (#2966)
• 2022-06-07 - 8610945 - Update SDK to v1.15.0 (#2964)
• 2022-06-06 - e5b4e14 - Update docker-compose.yml (#2952)
• 2022-06-06 - e4918d9 - Adding reporter to shard context for access to UserScope (#2961)
• 2022-06-06 - e16e6e8 - Multi-cursor queue components: task key and range (#2956)
• 2022-06-06 - bef2790 - Add schedule-related search attributes (#2953)
• 2022-06-06 - ba8ce65 - Multi-cursor queue components: task predicates (#2957)
• 2022-06-06 - b4f2ff5 - also lookup scopedef by common if not found by service (#2959)
• 2022-06-06 - 96a0bcd - Tune host level task processing performance (#2955)
• 2022-06-06 - 81e7a13 - moving handler instantiation to allow noop reporter for autosetup (#2960)
• 2022-06-06 - 48cc8d1 - Revert "Refactor authorization interceptor by extracting utilities (#2945)" (#2958)
• 2022-06-06 - 14556dd - Fill in fields in ListSchedulesResponse (#2946)
• 2022-06-04 - fa2e06c - Update UI readme to v2 (#2936)
• 2022-06-04 - 8444848 - Fix host level task scheduler start/stop (#2948)
• 2022-06-04 - 732d43a - Fix task parallel processor test (#2951)
• 2022-06-03 - ebe2262 - Updates to metrics interface, road to deprecate metrics.Reporter (#2935)
• 2022-06-03 - e71a51f - Refactor authorization interceptor by extracting utilities (#2945)
• 2022-06-03 - e3a5e78 - Resetting open child workflow should not sent Terminated Failure to the parent #725 (#2913)
• 2022-06-03 - dfbcdce - Fix task rescheduler metric scope (#2943)
• 2022-06-03 - d1f73ad - Ensure timer max query level always above min level (#2944)
• 2022-06-03 - c3d172d - Clean up reset reapply default value (#2938)
• 2022-06-03 - bd5e722 - Updating dependency due to redaction (#2947)
• 2022-06-03 - 8de7333 - Handle deleted namespaces in perNamespaceWorkerManager (#2939)
• 2022-06-02 - a347c29 - Add gomock mocks for SDK Client and Worker (#2931)
• 2022-06-02 - 9e69052 - Add unit tests for perNamespaceWorkerManager (#2933)
• 2022-06-02 - 803276b - Improve deletenamespace logging (#2934)
• 2022-06-02 - 58d2964 - Fix cluster metadata remove (#2940)
• 2022-06-02 - 4c11b96 - Update replication processor/executor init logic (#2926)
• 2022-06-01 - 9ad9e72 - Add priority tag for task processing metrics (#2921)
• 2022-06-01 - 556b60d - Update shard persistence test (#2928)
• 2022-06-01 - 4c62f56 - Handle visibility task timeout in bulk operation (#2895)
• 2022-05-31 - f5ba752 - Add dynamic config to disable frontend schedule rpcs (#2908)
• 2022-05-31 - e790578 - Limit max number of shards used by tests (#2925)
• 2022-05-31 - 9679676 - Dynamic config for scheduler server worker (#2904)
• 2022-05-31 - 7af6cda - Delete global namespace (#2867)
• 2022-05-31 - 78beb81 - History queue processor task loading host RPS limit (#2920)
• 2022-05-31 - 775d1fb - Use MutableSideEffect for scheduler workflow tweakables (#2906)
• 2022-05-31 - 5d88844 - Fill in SupportsSchedules field (#2922)
• 2022-05-31 - 118aae5 - Generate replication task when updating cluster list (#2924)
• 2022-05-31 - 0f34935 - Fix npe for dlq retry policy (#2923)
• 2022-05-31 - 00394e9 - Add close time in mutable state (#2917)
• 2022-05-27 - df4d519 - Make task processing worker count config truly dynamic (#2911)
• 2022-05-27 - a7e3520 - metric interface refactor, consumer and producer decoupled (#2883)
• 2022-05-27 - 4e70983 - Improve deletenamespace workflow errors (#2909)
• 2022-05-27 - 3dbabab - Use namespace Id for replication APIs (#2914)
• 2022-05-27 - 3855a36 - moving fork to temporal (#2916)
• 2022-05-26 - 9bfadcd - Properly initialize task key fire time (#2907)
• 2022-05-26 - 9b8cadc - Fix timer task completion (#2910)
• 2022-05-26 - 7ab7ef2 - Add scheduler server worker (#2856)
• 2022-05-26 - 738b36f - Add namespace_id field to events with namespace field (#2903)
• 2022-05-26 - 3beaf63 - Add schedule rpc handlers (#2857)
• 2022-05-25 - 8a2dcec - Update vscode launch scripts (#2898)
• 2022-05-25 - 89c0ef3 - Rename legacy history replication task (#2901)
• 2022-05-25 - 2209c4d - Update comments for RequestCancelWorkflow (#2889)
• 2022-05-24 - df78a57 - Add sync workflow state task (#2853)
• 2022-05-24 - c39cfb7 - Misc. improvements for priority task processing (#2897)
• 2022-05-24 - 9e07a34 - Use workflow consistency checker for all APIs (#2873)
• 2022-05-24 - 4ac6e3f - Handle NDC active -> passive transition with transient workflow & no buffered event (#2845)
• 2022-05-23 - 6b2b3f2 - Remove deprecated shard info ack level fields (#2884)
• 2022-05-20 - faa215f - Use --env instead of --zone to start the server in development environment (#2878)
• 2022-05-20 - df43fba - Changed RegisterNamespace to initialize new Namespaces in REPLICATION_STATE_NORMAL (#2887)
• 2022-05-20 - d3f3a29 - add ContinueAsNew as action command (#2876)
• 2022-05-20 - b94ddc1 - Update base-ci-builder to 1.5.0 (#2880)
• 2022-05-20 - b530a35 - Add String method to ContextImpl to fix a race (#2879)
• 2022-05-20 - ad573c0 - Update tctl details in contributing.md (#2881)
• 2022-05-20 - a40214d - Guarantee history task execution (#2864)
• 2022-05-20 - 98f4525 - Clean up dynamic configs (#2877)
• 2022-05-20 - 5635101 - Fix bug and simplify dynamicconfig (#2875)
• 2022-05-20 - 548dd6c - Check visibility ack level in standby cluster for DeleteWorkflowExecution (#2870)
• 2022-05-20 - 2758e29 - Allow dynamic config filter by namespace or task queue name only (#2858)
• 2022-05-20 - 04f2094 - Fix typo (#2888)
• 2022-05-19 - d36291f - Refactor ndc history resender to handle multiple remote clusters (#2866)
• 2022-05-19 - b618a94 - Remove lock on current workflow ID (#2872)
• 2022-05-19 - 65ea107 - Return error when cluster info is not found (#2868)
• 2022-05-19 - 5eda907 - Add task count metric (#2869)
• 2022-05-19 - 1eed3a7 - Disable cgo by default (#2874)
• 2022-05-19 - 150516a - Replicate workflow state (#2852)
• 2022-05-19 - 148630e - Add cluster ID to clock (#2871)
• 2022-05-18 - b9c335e - Admin workflow deletion API (#2841)
• 2022-05-18 - b8b58b1 - Remove custom retries for ReclaimResourcesWorkflow (#2865)
• 2022-05-18 - 9edda59 - Add metering action metrics (#2863)
• 2022-05-18 - 29de7f9 - Bypass ack level check for delete workflow execution in standby cluster (#2859)
• 2022-05-17 - b7aa833 - Fix failover queue creation after shard reload (#2862)
• 2022-05-17 - 08de0ff - Guarantee shard shutdown (#2861)
• 2022-05-16 - ec73b48 - Add skeleton for Schedule rpcs (#2846)
• 2022-05-16 - e562b44 - Delete workflow executions in global namespace (#2855)
• 2022-05-16 - b93f878 - Development environment for multi cluster setup (#2848)
• 2022-05-16 - b385075 - Fix transfer standby queue processor locking (#2854)
• 2022-05-16 - 7d42654 - Add namespace argument to PerNSWorkerComponent.Register (#2847)
• 2022-05-16 - 4a47b81 - Do not load stiky queue for add task (#2850)
• 2022-05-13 - 5acbdd7 - Fix auth plugin configuration to match the new SQLAuthPlugin config block (#2844)
• 2022-05-12 - f959451 - Drop namespace replication task if it does not live in current cluster (#2842)
• 2022-05-12 - d95f94d - Add first_execution_run_id to Get/PollMutableStateResponse (#2836)
• 2022-05-12 - aba6e1f - Add scheduler time logic (#2666)
• 2022-05-12 - ab6e338 - Add per-namespace worker manager (#2799)
• 2022-05-12 - 87c3beb - Remove TerminateWorkflowExecution call when deleting namespace (#2837)
• 2022-05-12 - 85978d5 - Misc changes to migration workflow (#2838)
• 2022-05-12 - 7361f8b - Check workflow task after reapply events (#2840)
• 2022-05-12 - 690ad54 - Add RDS IAM auth plugin for SQL drivers (#2830)
• 2022-05-12 - 3c50e83 - Minor tweak to migration workflow wait replication check (#2839)
• 2022-05-12 - 208391f - Remove fossa from buildkite (#2843)
• 2022-05-11 - cf4153c - Delete running workflow executions (#2819)
• 2022-05-11 - a089ae5 - Fix WorkflowTaskAttempt metric definition (#2835)
• 2022-05-11 - 2b86009 - Resend history for pending standby activity workflow task (#2796)
• 2022-05-10 - d91d164 - Enforce timeout when updating shard (#2833)
• 2022-05-10 - 6ef79ea - Remove global StickyTTL (#2832)
• 2022-05-10 - 668c851 - Fix query (#2826)
• 2022-05-10 - 19972ae - Fix scheduler NPE for timer failover processor (#2831)
• 2022-05-10 - 153fb24 - Update docker-compose.yml (#2828)
• 2022-05-09 - c118f6b - Verify child first workflow task scheduled (#2822)
• 2022-05-09 - bffb755 - Fix potential deadlock in shard addTask (#2823)
• 2022-05-09 - be024bf - Use public WorkflowNotReady (#2825)
• 2022-05-09 - 9c9b4c9 - Fix sporadic duplicate key errors in mysql queue implementation (#2802)
• 2022-05-09 - 99fdfb6 - Allow refreshing closed or standby workflow tasks (#2809)
• 2022-05-09 - 5da9b57 - Refactor replication related components (#2816)
• 2022-05-09 - 3dc5143 - Deprecate StatsTypeTag (#2820)
• 2022-05-09 - 2f43b88 - Fix mutable state stale check for recordChildExecutionCompleted (#2821)
• 2022-05-09 - 256560c - Fix build (#2827)
• 2022-05-09 - 0c75418 - Add appendRawHistoryNodes api (#2797)
• 2022-05-08 - aa2c98e - Verify child completion recorded (#2806)
• 2022-05-07 - f68d9f7 - update slack link (#2818)
• 2022-05-07 - e3e48c9 - Rename clockpb to clocksbp import alias (#2815)
• 2022-05-07 - b31e512 - Remove obsolete nil CloseTime check (#2813)
• 2022-05-07 - 4767cd4 - Fix error handling in processParentClosePolicy (#2814)
• 2022-05-06 - 95dfa07 - Fix 5s delay on sticky queue (#2811)
• 2022-05-05 - f62095d - Remove task version and visibility timestamp override (#2789)
• 2022-05-05 - f1fb635 - Expose namespace data (#2804)
• 2022-05-05 - f1dbddc - Use part of namespace ID for delete namespace name (#2803)
• 2022-05-05 - ddb29dc - Expose namespace data (#2804)
• 2022-05-05 - dc39bc7 - Prepare 1.16.2 patch (#2808)
• 2022-05-05 - ae84776 - Always schedule first workflow task for started abandoned child (#2414)
• 2022-05-05 - aa566f7 - Add dynamic config for namespace refresh interval (#2766)
• 2022-05-05 - a8a4d27 - Remove UNSPECIFIED failed cause (#2807)
• 2022-05-05 - 91b099b - Ensure queue processor cluster ack level is below failover ack level (#2794)
• 2022-05-05 - 705cbe9 - Do not wake up workflow in retry backoff upon signal (#2771)
• 2022-05-05 - 3912a3e - Always schedule first workflow task for started abandoned child (#2414)
• 2022-05-05 - 1f44265 - Return NamespaceNotFound error from RespondWorkflowTaskCompleted handler (#2810)
• 2022-05-04 - aa257cd - Pass child initiated event version to child workflow (#2801)
• 2022-05-03 - eca0a0f - Remove unused method from MutableState (#2798)
• 2022-05-03 - 6614ea6 - Remove legacy execution table tests (#2793)
• 2022-05-02 - b7064c3 - Lazy init task executable logger (#2795)
• 2022-05-02 - 08a0226 - Ensure queue processor cluster ack level is below failover ack level (#2794)
• 2022-05-01 - e69d639 - Simplify SignalWithStart API (#2791)
• 2022-04-30 - 6035304 - Add NamespaceInvalidState and NamespaceNotFound errors (#2785)
• 2022-04-29 - f84d6d0 - Update proto linters (#2773)
• 2022-04-29 - 92ff8c8 - Fix flaky TestClusterMembershipReadFiltersCorrectly (#2790)
• 2022-04-29 - 723b7a5 - Remove task version and visibility timestamp override (#2789)
• 2022-04-29 - 4faf1f4 - Use same root tags for system scope and user scope (#2784)
• 2022-04-29 - 37dcdea - Move history API logic into its own package (#2783)
• 2022-04-29 - 2a938b8 - Limit number of delete workflow executions retries (#2768)
• 2022-04-28 - fcf7d19 - Limit task processing time (#2770)
• 2022-04-28 - c302cb3 - Move history API logic into its own package (#2780)
• 2022-04-28 - 130d36e - Move history API logic into its own package (#2778)
• 2022-04-28 - 08d0232 - Wire up host level task scheduler (#2779)
• 2022-04-27 - b747b17 - Update docker-compose.yml (#2769)
• 2022-04-27 - 8f98374 - Deprecate history task processor (#2764)
• 2022-04-27 - 724ce2a - Expose DeleteWorkflowExecution API on operatorservice (#2761)
• 2022-04-27 - 6ff2f59 - Support UI as a client in version checker (#2751)
• 2022-04-27 - 5540505 - Fix set default per unit histogram boundaries (#2775)
• 2022-04-27 - 07d8a41 - Fix conditional variable test (#2774)
• 2022-04-26 - f7efbad - Do not wake up workflow in retry backoff upon signal (#2771)
• 2022-04-26 - ca586dd - Slow down workflow task retry (#2765)
• 2022-04-26 - 8e3fb97 - Update mutable state consistency check logic (#2747)
• 2022-04-26 - 631e27d - Add dynamic config for namespace refresh interval (#2766)
• 2022-04-26 - 02542f9 - Don't fail ContinueAsNew command if parent namespace doesn't exist (#2767)
• 2022-04-24 - cd88c61 - Improve history client retry logic (#2762)
• 2022-04-22 - fa39d14 - Utilize shard clock (#2746)
• 2022-04-22 - 8951ead - Make shard accessible to history handler (#2745)
• 2022-04-22 - 5ce1a63 - Use metrics unit in user scope (#2759)
• 2022-04-22 - 58103d7 - Remove namespace field from ScheduleActivityTaskCommandAttributes message (#2753)
• 2022-04-22 - 4b44fd2 - Add failure event if target namespace is missed during transfer task processing (#2752)
• 2022-04-22 - 18399f2 - Update docker-compose.yml (#2758)
• 2022-04-22 - 10aa1c4 - Disable cgo by default (#2760)
• 2022-04-21 - 8426ac5 - Update upper histogram boundary (#2756)
• 2022-04-21 - 58456a4 - Utilize shard clock for workflow / activity task (#2744)
• 2022-04-21 - 041eaa3 - Task executable scheduler implementation (#2750)
• 2022-04-20 - ddf8abe - Use better retry logic in reclaimresources workflow (#2689)
• 2022-04-20 - dd4a427 - Task priority assigner implementation (#2740)
• 2022-04-20 - d84a19c - Task executable implementation (#2738)
• 2022-04-20 - a65e0ca - Fail workflow task with BadSearchAttributes cause if search attributes are wrong (#2742)
• 2022-04-20 - 7d46dc6 - Remove duplicated require section from go.mod (#2754)
• 2022-04-20 - 1336d46 - Task rescheduler implementation (#2739)
• 2022-04-20 - 04ac068 - Adding error type tag to peristence errors (#2748)
• 2022-04-20 - 0458cad - Upgrade OpenTelemetry to v0.29.0 (#2717)
• 2022-04-19 - d98c7f0 - Revert back to using file URI for sqlite dsn (#2731)
• 2022-04-19 - c6ae43d - Host level worker pool components interface (#2736)
• 2022-04-19 - a8d47d0 - Add shard clock proto definition (#2743)
• 2022-04-19 - 8c78ad8 - Prepare 1.16.1 patch (#2737)
• 2022-04-19 - 762bf91 - Always report service error as additional metric with error-type tag (#2741)
• 2022-04-19 - 6deeed5 - Use caller object as callback listener ID (#2734)
• 2022-04-19 - 254a381 - Use caller object as callback listener ID (#2734)
• 2022-04-19 - 0b27212 - Use uuid as namespace notification ID (#2733)
• 2022-04-18 - 78defe1 - Use uuid as namespace notification ID (#2733)
• 2022-04-15 - ca6b7ac - Do not drop workflow task if it failed due to unhandled events (#2720)
• 2022-04-15 - 0d0fc87 - Log NotFound error for transfer task processing (#2723)
• 2022-04-14 - 86c9da4 - Remove CLI version information (#2715)
• 2022-04-14 - 67e34a3 - Simplify history engine init & wait logic (#2725)
• 2022-04-13 - ea27233 - Do not drop workflow task if it failed due to unhandled events (#2720)
• 2022-04-13 - d541fc0 - Log NotFound error for transfer task processing (#2723)
• 2022-04-13 - ca4fa9e - Persistence context part 9: elasticsearch visibility store (#2713)
• 2022-04-13 - b8b9eea - replace sqlite3 with modernc (#2704)
• 2022-04-13 - 8fae0c7 - Update future to use generics (#2722)
• 2022-04-13 - 6409090 - Simplify history engine task read ID logic (#2724)
• 2022-04-13 - 57aa72c - Add broadcast shutdown channel (#2647)
• 2022-04-13 - 49a9ebd - refactor ringpop interfaces (#2680)
• 2022-04-13 - 490d18f - Cancel context earlier (#2721)
• 2022-04-13 - 2e55dd3 - Persistence context Part 7: SQL store implementation (#2711)
• 2022-04-13 - 25e2cd7 - Improve mem efficiency when mutating workflow (#2706)
• 2022-04-13 - 1686461 - Persistence context part 8: cassandra store (#2712)
• 2022-04-12 - 23480fc - Update dependencies (#2716)

New Contributors

• @Alex-Tideman made their first contribution in #2758
• @nagl-temporal made their first contribution in #2802
• @alexandrevilain made their first contribution in #2835
• @gnz00 made their first contribution in #2830
• @aaronjheng made their first contribution in #2888
• @arnesenfamily made their first contribution in #2892
• @feihuang made their first contribution in #2945

Full Changelog: v1.16.0...v1.17.0