IMPORTANT

This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.

Release Highlights

Schema Changes

This release includes minor schema changes for MySQL/PostgreSQL and no schema changes for Cassandra.
Before upgrading your Temporal Server cluster, run the schema upgrade tool to support version 1.6.

Instructions for the schema update tool are here:
https://docs.temporal.io/docs/server/versions-and-dependencies/#upgrade-your-version-of-temporal.

Notable Changes

Deprecating services.<service>.metrics config

We currently have two ways to configure metrics for Temporal Server: one with a global.metrics section, another in services.<service>.metrics. The second option (in services.<service>.metrics) is deprecated and will be removed in v1.13.0.

Case-insensitive dynamic config

Dynamic config is now case insensitive.

Enhanced Visibility

Important: If you use the Enhanced Visibility feature backed by Elasticsearch, manual steps are required before upgrading to version 1.11. This release includes changes to the schema of the Elasticsearch index. Follow the steps below to change your schema prior to upgrading.

If you do not use the Enhanced Visibility feature, you can skip this section.

What you should know

Version 1.11 requires a breaking change to the Elasticsearch schema used by our experimental Enhanced Visibility feature. Manual migration is required and should take between 1 and 8 hours for most users with minor service degradation.

• What migration is being done? A new schema template and associated index has to be created and all documents need to be copied from the old index to the new one.
• Why was this needed? We did not make this breaking change lightly. Enhanced Visibility is an experimental feature that does not fall under our versioning policy. This was an internal schema refactor necessary in advance of promoting this to an officially supported, production ready feature in the next Temporal release.
• Minor degradation of service. During reindexing, the visibility feature is "frozen" — no updates will be shown when queried.
• Rare upgrade. We don't plan any breaking changes to the Elasticsearch schema in the next year.
• Duration. This migration will take some time — the actual speed depends on your setup. We estimate that most migrations should take between 1 and 8 hours.
• Rollbacks. We offer a recovery strategy in case you need to roll back the upgrade.
• Optional recommendations
  • Scale up Elasticsearch. The migration will be faster if you scale up your Elasticsearch cluster before migrating.
  • Practice run. You can run on a test cluster with one of our Docker images.

How to change your Elasticsearch schema

1. Before you upgrade your server, you need some files from this 1.11.0 release handy. Also you need direct access to Elasticsearch server. If you can't access it from your local machine, the admin-tools container is a good place to run all the steps. The container has all the needed tools, and you need to copy only these files (with kubectl cp, for example).

2. Create an index template (tune it if you use more replicas or shards):

   $ curl -X PUT "http://elasticsearch:9200/_template/temporal_visibility_v1_template" -H "Content-Type: application/json" --data-binary @index_template_v7.json --write-out "\n"
   {"acknowledged":true}

   Use index_template_v6.json if you are still on Elasticsearch 6.

3. Create an index. You can choose any name but it must start with temporal_visibility_v1. The default is temporal_visibility_v1_dev:

   $ curl -X PUT "http://elasticsearch:9200/temporal_visibility_v1_dev" --write-out "\n"
   {"acknowledged":true,"shards_acknowledged":true,"index":"temporal_visibility_v1_dev"}

4. Before you start reindexing, you need to block internal visibility queue processor to prevent new documents being created or deleted during reindexing. You can achieve this with a dynamic config setting:

   history.visibilityTaskWorkerCount:
     - value: 0
       constraints: {}

   If you added custom search attributes, you need to also add them to your dynamic config:

   frontend.validSearchAttributes:
     - value:
         OrderAmount: "Double"
         ProductId: "Keyword"

   You can check your custom search attributes by using the following command:

   $ tctl admin cluster get-search-attributes
   Custom search attributes:
   +-------------+----------+
   |    NAME     |   TYPE   |
   +-------------+----------+
   | OrderAmount | Double   |
   | ProductId   | Keyword  |
   +-------------+----------+
   ...

   Please note, full server restart (all nodes) is required for this setting to apply. After this step, visibility updates will stop getting written.

5. Run the reindex.sh script to reindex the old index to the new one. The script has input parameters with reasonable defaults. Please review the Input parameters section in reindex.sh. For example, if you are not using default index names, you need to pass a name. If you are on Elasticsearch 6, pass ES_VERSION=v6.

   If your index has a big number of shards (such as 100) you should specify SLICES_COUNT parameter with some reasonable value (such as 10), especially if you use AWS Elasticsearch to prevent 429 errors during reindexing.

   If you added custom search attributes, you need to pass their names to reindex.sh in a JSON array format. The default is 6 Custom*Field attributes. If you don't use them, pass an empty JSON array ('[]').

   $ CUSTOM_SEARCH_ATTRIBUTES='["ProductId", "OrderAmount"]' ES_SERVER=elasticsearch ES_VIS_INDEX_V0=temporal-visibility-prod ES_VIS_INDEX_V1=temporal_visibility_v1_prod ./reindex.sh

   The reindex.sh script has two steps: create custom search attribute fields in the new index, and start the reindexing process itself. Both steps show you the generated JSON and ask for confirmation. These steps are idempotent; if, for some reason, reindexing fails, you can run it again.

   Reindexing is an asynchronous operation that might take a while depending on your index size, index configuration, Elasticsearch load and configuration. Average speed is approximately 500K documents per minute. Expected output should be:

   Started reindex task 8z3GRTmSRyCtGPAKmO2RRA:155347. Check status with:
       curl http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347
   Query subtasks with:
       curl http://127.0.0.1:9200/_tasks?parent_task_id=8z3GRTmSRyCtGPAKmO2RRA:155347
   Cancel with:
       curl -X POST http://127.0.0.1:9200/_tasks/8z3GRTmSRyCtGPAKmO2RRA:155347/_cancel
   
   Waiting for reindex to complete (it is safe to Ctrl+C now).
   false
   false
   false
   true
   Reindex complete:
   {
     "total": 1234567,
     "updated": 0,
     "created": 1234567,
     "version_conflicts": 0
   }
   Source index temporal-visibility-dev document count:
   1234567
   Destination index temporal_visibility_v1_dev document count:
   1234567

   If, for some reason, you want to start a new reindexing process, you need to cancel the previous task if it is still running. The cancel request command is in a script output. You can read more about Elasticsearch task management here. The version_conflicts value for the second run should be equal to the created value of the first run.

6. When reindexing is done, the number of documents in the source and destination indexes should match. It might be a good idea to review the new index manually:

   $ curl "http://elasticsearch:9200/temporal_visibility_v1_dev/_search?pretty"

   A few things to notice:

   • Attr should not appear; instead, all custom search attributes should be at the document level.
   • All *Time fields should be in human-readable time format.
   • ExecutionStatusshould be a string.
   • The new field ExecutionDuration is added.

7. Upgrade Temporal Version to 1.11. As part of this upgrade, change the visibility index in the static config to the new index name. These two changes need to happen together.

   persistence:
     datastores:
       es-visibility:
         elasticsearch:
           indices:
             visibility: temporal_visibility_v1_dev

8. Wait for the upgrade to complete and make sure that all nodes run the new version. Remove the history.visibilityTaskWorkerCount setting from dynamic config and restart the server one more time.

9. Verify successful migration. Visibility updates created after Step 4 were buffered and are applied after the visibility queue is unblocked (step 8).

10. It is safe to delete the old index now. (You will need old index only if you decide to roll back.)

Roll back schema changes

If something goes wrong, there is a rollback path.

If the new Temporal Server version (1.11) hasn't written anything to the new index (for example, it crashes during startup), it is safe to just switch to the old version, which will continue to use the old index. (Reindexing doesn't modify the old index.)

If, for some reason, you need to roll back after new records were written to the new index, you need to run steps 4 through 8 using rollback.sh instead of reindex.sh (assuming you have not yet deleted the old index).

Miscellaneous

• Some Elasticsearch-related metrics were renamed to follow the general pattern: #1701.
• A new tctl command (tctl admin cluster remove-search-attributes) was added to unregister custom search attributes. It won't modify Elasticsearch schema, though, because that would require reindexing.
• You should never prefix your custom search attributes with Temporal because we might add some system search attributes in the future and would use this prefix.

All changes

2021-07-17 - 1135e13 - Remove docker push step from Buildkite build (#1740)
2021-07-16 - f16c5f5 - Add SLICES_COUNT param and failure report to reindex.sh (#1737)
2021-07-16 - cc47a1d - Disable Elasticsearch index auto creation (#1738)
2021-07-16 - 53c3354 - Elasticsearch template tuning (#1732)
2021-07-15 - b36dbeb - Ignore caller context cancellation error (#1734)
2021-07-15 - b123ce4 - Remove internode keep alive pings (#1735)
2021-07-14 - 746b75e - Add rollback.sh for reindex.sh (#1728)
2021-07-13 - fa94abb - Add ES_VERSION parameter to reindex.sh (#1725)
2021-07-13 - 21aa499 - Handle mutable state version migration edge case (#1727)
2021-07-09 - ef4ee12 - Split memo size checks from payload; move to history service (#1691)
2021-07-09 - 1a50d49 - Fix cassandra error handling edge case (#1721)
2021-07-09 - 02f7e40 - Increase default bulk actions count from 200 to 500 (#1722)
2021-07-08 - dcc4452 - Wire Cassandra layer using new error extraction util (#1712)
2021-07-08 - a87bf44 - Add support for authorization of read-only frontend calls (#1711)
2021-07-08 - a49fa1c - Adds configuration flag to set DisableInitialHostLookup on gocql client driver
2021-07-07 - 9e30a3b - Reimplement Cassandra error extraction util (#1710)
2021-07-06 - c4fd77a - Check shard status before workflow modification (#1707)
2021-07-05 - 28de844 - Use an SDK client to start workflows (#1694)
2021-07-02 - cd10b35 - Refactor Elasticsearch metrics (#1701)
2021-07-02 - 8913cca - Use mysql.Config for mysql connection string management (#1705)
2021-07-02 - 3d68d49 - Add hard minimum for retention duration (#1680)
2021-07-02 - 1aa343c - Update sdk to 1.8.0 (#1704)
2021-07-01 - 4d64518 - Separate liveness of task queue into a dedicated entity (#1699)
2021-06-30 - 5f1f1c3 - Enforce daemon coroutine at most once start / stop logic within matching service (#1697)
2021-06-30 - 3325b7d - Add metrics logging for forward error (#1695)
2021-06-29 - 226d330 - option to use customized client factory (#1692)
2021-06-25 - 7a39051 - Modify add-search-attributes tctl command to add only missing search attributes (#1676)
2021-06-24 - db8a165 - Fix SDK cache workflow task start to close timeout behavior (#1668)
2021-06-24 - c8efea8 - Rename LastProcessedEvent to LastWorkflowTaskCompleteId (#1677)
2021-06-24 - b8c83da - Fix to use the configured token. (#1687)
2021-06-24 - 279ba49 - Tune ES bulk processor dynamic config defaults (#1686)
2021-06-24 - 1b482d5 - Fix error prone retry time implementation (#1681)
2021-06-24 - 114c139 - Lower-case namespace name in default authorizer (#1671)
2021-06-24 - 004cae0 - typo: uknown (#1683)
2021-06-23 - f8cc50a - Fix enabled flag for TLS-FrontendClientConfig (#1678)
2021-06-23 - df5d49e - Add user metric scope (#1649)
2021-06-23 - dd13a0d - Refactor AddSearchAttributes to return an empty message (#1674)
2021-06-23 - a89ff40 - Add RemoveSearchAttributes API to the admin service (#1660)
2021-06-23 - a86f44e - Fix unable to reset workflow issue (#1673)
2021-06-23 - 09edffc - Persistence PayloadSerializer cleanup (#1682)
2021-06-22 - 7f4b257 - Fix typo in encryption algorithm name ec256 -> es256 (#1664)
2021-06-22 - 56b464e - Add optional args for DB names to install schema (#1667)
2021-06-22 - 41a953f - Add options to not reapply events after workflow reset (#1355)
2021-06-22 - 2661a96 - Update gocql (#1663)
2021-06-21 - 7c89d5f - Symlink current Elasticsearch index templates to v1 directory (#1659)
2021-06-21 - 36cd514 - Refactor search attributes GetType method (#1657)
2021-06-20 - 6b2727d - nil check reporters when stopping (#1656)
2021-06-17 - fc46c37 - Rename *Timestamp fields to *Time to better reflect its type (#1651)
2021-06-17 - f1352be - Sets Reconnection policy on gocql.ClusterObject (#1652)
2021-06-17 - b40d8f4 - Fix search attributes parsing logic (#1647)
2021-06-17 - 6459b90 - Autocreate custom search attributes fields in reindex script (#1653)
2021-06-17 - 5fb3478 - Make dynamic config case insensitive (#1635)
2021-06-17 - 3c427e6 - Add support for passing JWT audience to defaultJWTClaimMapper (#1648)
2021-06-17 - 31de6a4 - Add StateTransitionCount field to Elasticsearch index (#1650)
2021-06-17 - 2666ad9 - Restructure history service mutable state & related logics (#1641)
2021-06-17 - 0087afd - Adds keep-alive to gRPC connections between the frontend and history/matching services #1654
2021-06-16 - f65e5b2 - Bump Web to v1.10.0 (#1609)
2021-06-16 - 9a5e906 - Add ExecutionDuration ES field (#1645)
2021-06-16 - 7b6bcf2 - Set CGO_ENABLED to 1 for darwin and windows (#1588)
2021-06-15 - a879f7f - Rename Elasticsearch system search attributes (#1639)
2021-06-15 - a2aa44e - Refactor Elasticsearch PutMapping API (#1634)
2021-06-15 - 663c3ee - Use arbitrary Duration for retention (#1615)
2021-06-15 - 56e4631 - Refactor Queue for namespace replication (#1625)
2021-06-15 - 3bcfe16 - Use scaled_float instead of double for Double search attributes (#1633)
2021-06-15 - 26ac39b - Add mocks for data store interfaces (#1640)
2021-06-15 - 0eb0c37 - Convert Elasticsearch ExecutionStatus field to keyword (#1637)
2021-06-14 - fbb2ac2 - Fix auto-setup.sh for PostgreSQL (#1629)
2021-06-14 - f55d0bb - Convert Elasticsearch time fields to date type (#1624)
2021-06-14 - 9fc20ca - Refactor Dockerfiles to better support ARM (#1575)
2021-06-14 - 9f36e82 - hookup custom data store factory (#1630)
2021-06-14 - 8e34c29 - Add minimum date validation (#1631)
2021-06-14 - 3481ed6 - Fix Elasticsearch metrics and memo encoding (#1632)
2021-06-14 - 305c18c - Make sync match wait time configurable (#1627)
2021-06-14 - 15666ff - Emit dedicated metrics for context cancelled (#1628)
2021-06-11 - c41ddbd - Add ExecutionTime to WorkflowExecutionInfo (#1618)
2021-06-11 - 799ee02 - Emits warning/error logs when Temporal tries/fails to establish a new GoCQL session (#1623)
2021-06-10 - c78ea3e - Refactor ExecutionStore / ExecutionManager (#1614)
2021-06-10 - 79a537e - Add logger interceptor to frontend (#1616)
2021-06-10 - 7402bd7 - Enable mutable state DB version CAS by default (#1621)
2021-06-10 - 283bb1e - Update SDK to 1.7.0 (#1620)
2021-06-09 - 8043f68 - Use priority mutex for mutable state cache lock (#1612)
2021-06-09 - 202e981 - Report error for invalid retention days (#1610)
2021-06-08 - e3e7d4a - Introduce priority mutex & condition variable (#1611)
2021-06-08 - 80f04b5 - Remove unnecessary metrics on hisory cache (#1606)
2021-06-08 - 3311e67 - Fix error type if query task alrady gone (#1608)
2021-06-07 - ff8b201 - Allow tctl to start workflows with infinite timeout (#1602)
2021-06-07 - 8f63537 - Remove Attr prefix from Elasticsearch schema (#1586)
2021-06-07 - 804ed60 - Task manager impl continued (#1604)
2021-06-07 - 6b0921b - Adding TaskManagerImpl (#1600)
2021-06-07 - 27456ec - Fix opentelemetry metrics listen address (#1603)
2021-06-04 - f4bbcbc - Upgrade dgrijalva/jwt-go to v4.0.0-preview1 (#1597)
2021-06-04 - ea339fd - Remove CLI flags for security tokens (#1599)
2021-06-04 - 026a75e - Rename files for history manager and store (#1596)
2021-06-03 - f1c6935 - Refactor ShardManager (#1594)
2021-06-03 - d5f478b - Remove spew dependency (#1591)
2021-06-03 - 4951b7e - Fix DeleteHistoryBranch (#1590)
2021-06-03 - 2eb8fac - Include namespace in API error log (#1593)
2021-06-02 - d36837a - Use time.Time in internal visibility APIs (#1576)
2021-06-02 - 16584ef - Refactor persistence for HistoryStore (#1580)

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.11.0)

Server
Server With Auto Setup
CLI (tctl)
Admin-Tools