Schema changes

This release contains a bug that could potentially lose tasks in failure scenario when history pods are not stable and keep bouncing between multiple nodes. Please upgrade to latest patch release 1.14.3 or later.

Before upgrading to your Temporal Cluster to this release 1.14, you must upgrade your storage schema version to the following:

• MySQL schema version 1.17
• PostgreSQL schema version 1.17
• Cassandra schema version 1.6

Use the Schema Upgrade Tool to upgrade your schema version.
How to use the Schema Upgrade Tool: https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Note that schema version is NOT the same as the storage type (database) version, see “Latest SQL database support” below.

tctl

• tctl batch - Added a flag to limit amount of goroutines: --concurrency.
• tctl admin - Removed direct database connection and switched to Admin API. Removed flags related to database connection.
• Added a cluster command to upsert or remove remote cluster. This feature is still in development and may change in future release.
• Renamed --min/max_read_level flag to --min/max_task_id when listing cluster tasks.
• tctl namespace - Added a flag to promote a local namespace to a global namespace: --promote-namespace.
• Removed commands related to Elasticsearch.
• Removed cluster metadata command.

Metrics

• Added option to configure histogram buckets per unit type (milliseconds, bytes, dimensionless). Existing metrics now have assigned relevant unit types. Example.
• Added config option to exclude reported metric tags. Example.
• Upgraded uber-go/Tally to v4.0.1.
• Upgraded opentelemetry version to v1.1.0.

Latest SQL database support

• Latest version of MySQL and PostgreSQL were tested. Temporal now officially supports MySQL 8 starting from version 8.0.19 and PostgreSQL 13 starting from version 13.5.

Multi-Cluster

• Cluster metadata definition is migrating from application static config file to a new DB table. Cluster metadata will be initialized once when the service first starts up. Future remote cluster metadata modifications need to be done via tctl admin cluster tool. Example.

Other

• Server will fail to start if dynamic config is malformed. Previously, default config was used if errors occurred.
• Upgraded golang to 1.17.
• Added safety check for empty JWT source URIs.
• Added Handover ReplicationState for graceful failover. This feature is still in development and will change in a future release.

Improvements/Fixes

• Fix unexpected matching GetTasks call growth on low-traffic task queues.
• Reduced overhead due to unloading history shards during short database unavailability.
• Limited Elasticsearch document _id to 512 bytes (#2112).

Internal refactoring

• Adopted uber/fx for server initialization and dependency injection.

All changes

2021-12-13 - 074d8ebb1 - Fix rollup metrics aggregation (#2298)
2021-12-10 - 29e81571b - New future implementation (#2265)
2021-12-09 - f684386fe - Rename simpleStore to standardStore (#2278)
2021-12-09 - 7feb6b3a5 - Replication handover state (#2280)
2021-12-09 - 2eaadeaad - Move ackLevel past gaps in task ids (#2281)
2021-12-08 - c4976c9ea - Only update maxReadLevel after successful re-acquire of shard (#2272)
2021-12-08 - 0e844bc3b - Add additional UTF-8 validation (#2276)
2021-12-07 - 88c6eb5d7 - patch tally histogram (#2270)
2021-12-06 - f430f8573 - Stop server start up on malformed dynamic config (#2268)
2021-12-06 - be0df0d3e - Use correct histogram type for dimensionless (#2264)
2021-12-06 - 93de50aa4 - Add custom buckets to rollup metrics (#2266)
2021-12-06 - 3f4de5243 - Fix Elasticsearch bulk processor metrics units (#2267)
2021-12-03 - beb9a5a15 - Remove namespace APIs from adminservice (#2261)
2021-12-03 - 8a409cd25 - Add option to configure per metric unit histogram buckets. (#2251)
2021-12-03 - 827c92225 - Fix flaky TestVisibility (#2259)
2021-12-03 - 3e19fff5c - Fix batcher (#2260)
2021-12-02 - f338d10d1 - Remove DB record version config (#2253)
2021-12-02 - afc775a7b - Fix onebox worker shutdown (#2255)
2021-12-02 - 822eed7f5 - handover workflow (#2258)
2021-12-02 - 725730697 - Use history event as signal payload data (#2254)
2021-12-02 - 68498e122 - Remove --state from namespace update command (#2257)
2021-12-02 - 27c97af32 - Add shard lifetime context (#2218)
2021-12-02 - 06bd292b5 - Add UpdateNamespace tests for DELETED namespace state (#2185)
2021-12-01 - c9315c3e4 - Add dedicated persistence execution mutable state integration tests (#2244)
2021-12-01 - c099c7963 - Support limited list workflow queries for standard  visibility (#2197)
2021-12-01 - aa17a4aba - Matching cleanup (#2246)
2021-12-01 - 873f28085 - Add handover namespaces to GetReplicationStatus response (#2248)
2021-12-01 - 815bfd7d0 - Fix error messages (#2247)
2021-12-01 - 799a3c1e3 - Inserted retries to avoid flaky persistence (#2232)
2021-12-01 - 4f28e32b4 - Add dedicated persistence execution mutable state integration tests (#2245)
2021-12-01 - 2a47cdf0e - Replace ldflags with go:embed (#2085)
2021-12-01 - 0d9fcd3eb - Deprecate static cluster metadata (#2231)
2021-11-30 - d49000975 - Upgrade golang to 1.17.3 (#2240)
2021-11-30 - be7482878 - Add dedicated persistence execution mutable state integration tests (#2241)
2021-11-30 - b5e343cea - Remove resource.Resource from temporal services (#2229)
2021-11-30 - 4a4cc58b4 - Add Makefile targets for arm64 (#2201)
2021-11-30 - 1edb0a6b1 - Add dedicated persistence execution mutable state integration tests (#2242)
2021-11-30 - 171d228fc - Remove duplicated history size from execution info (#2239)
2021-11-29 - f888c2840 - Namespace Handover (#2228)
2021-11-29 - a1a237022 - Fix forward / backward compatibility issue (#2237)
2021-11-29 - 6daf3b6fb - Remove unnecessary CreateWorkflowModeContinueAsNew mode (#2235)
2021-11-29 - 6c139012d - Fix ParentClosePolicy not respected cross-namespace (#2224)
2021-11-29 - 1c52e90b6 - Remove unnecessary start version from current execution table (#2236)
2021-11-24 - e741533d4 - Add testhelper package to create temp file and dir (#2214)
2021-11-24 - a0e896524 - Add interleaved weighted round robin scheduler (#2209)
2021-11-24 - 876f75ce7 - Introduce dynamic cluster metadata component (#2208)
2021-11-24 - 182195552 - Namespace state update fix (#2227)
2021-11-24 - 0f2da592b - Add enable remote connection tctl (#2213)
2021-11-24 - 0baca26c8 - Use proper logger for worker service (#2226)
2021-11-23 - fc4024331 - Switch to Admin API in tctl list task queue tasks (#2222)
2021-11-23 - c3fdcb813 - Extract common struct for service provider args (#2223)
2021-11-23 - 90e89dd7c - Create admin API to list shard tasks (#2200)
2021-11-23 - 78acd4d7b - Use context for shard acquisition timeout (#2219)
2021-11-23 - 769b69f41 - Switch to admin API in tctl list shard tasks by category (#2217)
2021-11-23 - 6c86649b9 - Create admin API to get task queue tasks (#2221)
2021-11-23 - 1bbc3fbd7 - Refactor tctl admin show workflow raw history, switch to admin API (#2225)
2021-11-22 - 69959bc5e - Allow namespace state update (#2220)
2021-11-22 - 17b9d4cae - Pin MySQL version to 8.0.19 (#2210)
2021-11-19 - c9579c2ef - Handle missing metrics config (#2205)
2021-11-19 - c24eaa4be - Add List Cluster Metadata API (#2206)
2021-11-19 - 67da8a3b9 - Use gopkg.in/yaml.v3 instead of v2 (#2203)
2021-11-19 - 21531bab5 - Cleanup serverOptions depency and add server metrics client (#2187)
2021-11-19 - 1dd76d3f7 - Wait replications (#2202)
2021-11-18 - e5740fa22 - Use t.Setenv function to set env in unit tests (#2196)
2021-11-18 - 66780893b - Update development environment to MySQL 8.0 (#2193)
2021-11-18 - 6257ac319 - Change shard persistence interface to get-or-create (#2199)
2021-11-18 - 590d9dc98 - Implement GetReplicationStatus (#2195)
2021-11-18 - 39bd317c7 - Update development environment to PostgreSQL 13.5 (#2194)
2021-11-18 - 2eb40fe73 - Add fan out to history client for GetReplicationStatus (#2198)
2021-11-18 - 00cb5194c - Add tiered storage queue processor (#2189)
2021-11-17 - ca556868a - Add GetReplicationStatus to history service (#2186)
2021-11-17 - 9a4982999 - Remove Elasticsearch admin commands (#2181)
2021-11-17 - 14fe930b3 - Switch to admin API in tctl register namespace (#2176)
2021-11-16 - e6757b9ed - Add try lock APIs to priority mutex (#2183)
2021-11-16 - de83ad266 - Add NamespaceValidator interceptor (#2169)
2021-11-16 - 407184d92 - Add IsLocked to priority mutex (#2180)
2021-11-16 - 196b57afc - Create admin API to update namespace (#2182)
2021-11-15 - b9b0615a5 - Create admin API to register namespace (#2175)
2021-11-15 - 58e380a9e - Make cluster metadata schema backward compatible (#2177)
2021-11-15 - 58af5da4d - Header support for signals and queries (#2149)
2021-11-15 - 43ac47ffb - Report acked replication task timestamp (#2173)
2021-11-15 - 170bde6ac - Use admin API for tctl list cluster members (#2162)
2021-11-14 - 6bfe955d7 - Support concurrency flag in the tctl batch command (#2168)
2021-11-12 - f1cbac2de - Disable fx logs (#2091)
2021-11-12 - b9edf16e6 - Create admin API to list namespaces (#2170)
2021-11-12 - a439a679e - Wire up cluster metadata APIs (#2174)
2021-11-12 - 5d6bb62b1 - Update opentelemetry version and add gauge cache. (#2161)
2021-11-12 - 59c571530 - Use admin API for tctl list namespaces (#2171)
2021-11-12 - 519bffb7d - Move DeleteWorkflowExecution into shard context (#2138)
2021-11-12 - 1198962e9 - Create admin API to list cluster members (#2159)
2021-11-11 - f141698a0 - Initialize SignalRequestedIDs correctly (#2167)
2021-11-11 - dc60fffe3 - Fix test behavior (#2166)
2021-11-11 - 725a4de90 - Add new admin APIs to manage remote cluster info (#2163)
2021-11-11 - 60fb3cd65 - Add protection for empty JWT key source URIs (#2142)
2021-11-11 - 3f1eb310e - Set VisibilityTimestamp on ReplicationTask (#2165)
2021-11-11 - 13082f322 - Migrate cluster metadata store to use cluster metadata info table (#2155)
2021-11-11 - 0f70d6192 - Matching GetTasks call reduction (#2164)
2021-11-10 - 8c308a0d7 - Add custom grpc inspectors (#2156)
2021-11-10 - 62ce8c584 - Move namespace name validation to RegisterNamespace (#2160)
2021-11-10 - 590b6eaeb - Add GetAllKeys to keyResolver (#2152)
2021-11-09 - e25f5dc7b - Remove command tctl admin cluster metadata (#2151)
2021-11-08 - f35895c9b - Add RequestID validation for ResetWorkflow (#2141)
2021-11-08 - e9f8078d3 - Extend admin Describe Cluster API response with cluster metadata (#2150)
2021-11-08 - c5baafa11 - Add new table for cluster metadata info and update cluster metadata struct (#2147)
2021-11-08 - 5561f2a48 - Add support exclusion of tags from metrics (#2137)
2021-11-08 - 31a576749 - Fix potential nil pointer use in shard context (#2148)
2021-11-07 - 4fb8100a9 - fix flaky TestActivityHeartbeatFailover (#2145)
2021-11-06 - f8fc78efd - Refactor worker for addsearchattributes using fx (#2144)
2021-11-05 - 62e85b420 - Update error type (#2135)
2021-11-05 - 1e161ebc9 - Add gen-replication workflow (#2139)
2021-11-04 - 6ad2edaf0 - Fix GenerateLastHistoryReplicationTask to use shard.AddTasks() (#2136)
2021-11-04 - 203ccb085 - Refactor shard context lifecycle (#2102)
2021-11-04 - 0917c302b - Remove sampled retention (#2133)
2021-11-03 - f463f0dbc - Move generation of replication task logic to task generator (#2127)
2021-11-03 - 851aaec89 - Use correct absolute paths to scripts in dockerfile (#2123)
2021-11-03 - 67b606425 - Add generic task processor (#2120)
2021-11-03 - 60a52ee1d - update cli usage (#2124)
2021-11-03 - 5acdf8654 - Add GenerateLastHistoryReplicationTasks to history service (#2129)
2021-11-03 - 577de1ad8 - Add worker manager to worker service (#2126)
2021-11-01 - ea45be2d7 - Move timer gate to common package (#2121)
2021-11-01 - c28665dd1 - Promote local namespace to global namespace (#2105)
2021-11-01 - 7e8b9650d - Specify the entrypoint absolutely (#2118)
2021-10-29 - f253ddfc0 - Patch tally import (#2115)
2021-10-29 - ed0743d90 - Remove dependency on resource from worker (#2098)
2021-10-29 - ae959113d - Add http status metric tag to bulk processor errors (#2113)
2021-10-29 - 744f26d28 - Use strong types for namespace Name and Id (#2082)
2021-10-29 - 48f78b781 - Add metrics for loaded task queue count as gauge (#2111)
2021-10-29 - 181ad20ae - Bump Web to v1.13.0 (#2110)
2021-10-28 - fc422185f - Limit Elasticsearch document _id to 512 bytes (#2112)
2021-10-28 - d2405acd2 - Make SQL CLI tool more idempotent (#2104)
2021-10-28 - 3f6253069 - Extract SearchSource object in Elasticsearch client (#2106)
2021-10-28 - 28db61ac2 - Added ability to specify dataconverter port via flag (#2093)
2021-10-28 - 0e67cd904 - Use PPROF_PORT env to configure pprof port in config_template.yaml (#2109)
2021-10-27 - e93f4b23c - Finish task serialization / deserialization (#2083)
2021-10-27 - 88473687e - Update submodules and go.temporal.io/api (#2103)
2021-10-27 - 1503e924e - Upgrade SDK and Tally (#2100)
2021-10-26 - ede54d90f - Add in flight poll activity / workflow task queue metrics (#2095)
2021-10-26 - cb9462ab2 - Shard context cleanups: private eventsCache, remove SetEngine (#2086)
2021-10-26 - af181b404 - Add per namespace burst limit control (#2067)
2021-10-26 - 9c64e2e8d - Remove deprecated task ID field from replication task (#2078)
2021-10-26 - 98814a00e - Remove half done task processing restructure from Cadence (#2074)
2021-10-26 - 8b4936a24 - Remove unnecessary method on history tasks (#2072)
2021-10-26 - 3ce382a90 - Shard context cleanups: [R]Lock/Unlock should be private (#2087)
2021-10-26 - 03fde05be - Move history tasks definition to history package (#2070)
2021-10-26 - 00388db7f - Rename workflow identifier to workflow key (#2071)
2021-10-25 - d83e601c0 - Return deserialization tasks as is from persistence layer (#2042)
2021-10-25 - d7a2d5609 - Post-release tasks (#2092)
2021-10-25 - d6771efef - Pass exit signals to temporal-server (#2069)
2021-10-25 - bdeff26cf - Move from io/ioutil to io and os package (#2062)
2021-10-25 - b3731a43b - Shard context cleanups: split two remove shard paths, don't hold engine in workflow context (#2088)
2021-10-25 - 9a3c57741 - Admin API GetShard (#2075)
2021-10-25 - 92a42adb8 - Allow port forwarding to the tctl web dataconverter (#2089)
2021-10-25 - 779f64591 - Init server via uber/fx (#2034)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.14.0)

Server
Server With Auto Setup
Admin-Tools