Overview of Changes
[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.2.12. There are two ways to work around this problem:
- If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.12 and create a new cluster.
- If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.12, create a new cluster and then load the previously saved data.
YSQL Changes
- [YSQL] Fix scan operator when range key is not fully-specified.#1576
- [YSQL] Fix ORDER BY range column of a secondary index.#1591
- [YSQL] Add YSQL RPCs to tserver utilz page.#1446
System Improvements
- Resolve ABA problem in LockFreeStackResolve ABA problem in LockFreeStack.#1561
- Fix recursive locking in Batcher.#1584
- Use boost::atomic for 16-bytes structures.#1189
- Fix integer overflow during Raft log read path.#1606
- Fix concurrent access to primary table info in RaftGroupMetadata.#1631
- A new priority-aware thread pool implementation for compactions and flushes.#1314
- Create mem trackers that reflects TC Malloc stats.#1633
- Make sure we don't read an invalid replicate index from the footer.#1625
Performance Benchmarks
This section gives an overview of the performance of this release. The benchmarks were done on the following setup:
- 3 nodes with 16-vCPUs
- Replication factor = 3
- Each key-value is ~64 bytes (combined)
- See setup details for more info
Summary
| API | Workload | Type | Throughput | Latency | Avg CPU on DB node |
|---|---|---|---|---|---|
| YCQL | KeyValue | 100% Reads | 218K | 1.17ms | 48% |
| YCQL | KeyValue | 100% Writes | 73K | 3.5ms | 44% |
| YCQL | Batched Writes | 100% Writes | 247K | 26ms | 52% |
| YCQL | Secondary Index | 100% Reads | 187K | 1.4ms | 44% |
| YCQL | Secondary Index | 100% Writes | 5K | 12.9ms | 51% |
Details
YCQL KeyValue
Workload CassandraKeyValue with 256 writers and 0 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Read latency, ms/op | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Write throughput, ops/sec | 73037.05 | 68884.95 | 74102.43 | 1211.67 | 73480.55 | 68884.95 | 68884.95 | 71993.35 | 74051.57 | 74102.43 | 74102.43 |
| Write latency, ms/op | 3.50 | 3.45 | 3.72 | 0.06 | 3.48 | 3.45 | 3.45 | 3.45 | 3.55 | 3.72 | 3.72 |
| Load tester CPU, user, % | 16.91 | 14.90 | 18.10 | 0.88 | 17.30 | 14.90 | 14.91 | 15.18 | 17.70 | 18.08 | 18.10 |
| Load tester CPU, system, % | 9.20 | 7.60 | 10.10 | 0.83 | 9.65 | 7.60 | 7.60 | 7.72 | 9.90 | 10.09 | 10.10 |
| Cluster node CPU, user, % | 32.93 | 23.80 | 38 | 3.45 | 32.45 | 23.80 | 24.31 | 30.13 | 37.20 | 37.49 | 38 |
| Cluster node CPU, system, % | 15.43 | 12.70 | 16.60 | 0.82 | 15.40 | 12.70 | 13.22 | 14.43 | 16.30 | 16.50 | 16.60 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraKeyValue with 192 writers and 16 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 18493.93 | 17156.62 | 19190.41 | 587.15 | 18606.10 | 17156.62 | 17156.62 | 17440.30 | 19151.33 | 19190.41 | 19190.41 |
| Read latency, ms/op | 0.87 | 0.83 | 0.93 | 0.03 | 0.86 | 0.83 | 0.83 | 0.83 | 0.92 | 0.93 | 0.93 |
| Write throughput, ops/sec | 60571.82 | 58544.09 | 62509.31 | 983.23 | 60550.72 | 58544.09 | 58544.09 | 59035.93 | 62380.73 | 62509.31 | 62509.31 |
| Write latency, ms/op | 3.17 | 3.07 | 3.28 | 0.05 | 3.17 | 3.07 | 3.07 | 3.08 | 3.25 | 3.28 | 3.28 |
| Load tester CPU, user, % | 17.68 | 16 | 20.70 | 0.87 | 17.55 | 16 | 16.06 | 17.20 | 18.39 | 20.58 | 20.70 |
| Load tester CPU, system, % | 8.73 | 8.10 | 10.40 | 0.54 | 8.50 | 8.10 | 8.11 | 8.22 | 9.58 | 10.36 | 10.40 |
| Cluster node CPU, user, % | 35.62 | 32.90 | 38.80 | 1.71 | 34.95 | 32.90 | 33.41 | 33.72 | 38.10 | 38.39 | 38.80 |
| Cluster node CPU, system, % | 15.38 | 14.90 | 15.80 | 0.19 | 15.40 | 14.90 | 15.01 | 15.10 | 15.60 | 15.70 | 15.80 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraKeyValue with 128 writers and 64 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 71301.76 | 64468.86 | 78149.80 | 6074.55 | 69364.13 | 64468.86 | 64468.86 | 64563.72 | 78080.94 | 78149.80 | 78149.80 |
| Read latency, ms/op | 0.90 | 0.82 | 0.99 | 0.08 | 0.92 | 0.82 | 0.82 | 0.82 | 0.99 | 0.99 | 0.99 |
| Write throughput, ops/sec | 42974.61 | 40464.12 | 45473.25 | 1883.46 | 42860 | 40464.12 | 40464.12 | 40927.76 | 45053.88 | 45473.25 | 45473.25 |
| Write latency, ms/op | 2.98 | 2.82 | 3.16 | 0.13 | 2.98 | 2.82 | 2.82 | 2.84 | 3.13 | 3.16 | 3.16 |
| Load tester CPU, user, % | 21.46 | 19.10 | 23.30 | 1.28 | 21.60 | 19.10 | 19.15 | 20 | 23.08 | 23.29 | 23.30 |
| Load tester CPU, system, % | 10.17 | 9.20 | 10.90 | 0.56 | 10.30 | 9.20 | 9.21 | 9.31 | 10.80 | 10.89 | 10.90 |
| Cluster node CPU, user, % | 38.31 | 28.30 | 41.60 | 3.01 | 39.80 | 28.30 | 34.10 | 34.34 | 41.09 | 41.20 | 41.60 |
| Cluster node CPU, system, % | 13.11 | 11.10 | 14 | 0.51 | 13.15 | 11.10 | 11.66 | 12.80 | 13.50 | 13.60 | 14 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraKeyValue with 64 writers and 128 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 113115.50 | 111032.59 | 114690.37 | 1211.91 | 113593.04 | 111032.59 | 111032.59 | 111280.88 | 114639.78 | 114690.37 | 114690.37 |
| Read latency, ms/op | 1.13 | 1.11 | 1.15 | 0.01 | 1.13 | 1.11 | 1.11 | 1.12 | 1.15 | 1.15 | 1.15 |
| Write throughput, ops/sec | 21349.23 | 20717.49 | 21753.92 | 264.27 | 21335.10 | 20717.49 | 20717.49 | 20915.45 | 21699.52 | 21753.92 | 21753.92 |
| Write latency, ms/op | 3.00 | 2.94 | 3.09 | 0.04 | 3 | 2.94 | 2.94 | 2.95 | 3.06 | 3.09 | 3.09 |
| Load tester CPU, user, % | 23.54 | 21.80 | 27.90 | 1.24 | 23.50 | 21.80 | 21.81 | 22.03 | 24.36 | 27.72 | 27.90 |
| Load tester CPU, system, % | 10.43 | 9.60 | 12.10 | 0.53 | 10.45 | 9.60 | 9.61 | 9.71 | 10.79 | 12.03 | 12.10 |
| Cluster node CPU, user, % | 38.33 | 34.30 | 42.90 | 3.03 | 37.40 | 34.30 | 34.62 | 35.21 | 42.60 | 42.89 | 42.90 |
| Cluster node CPU, system, % | 11.41 | 10.30 | 12.40 | 0.52 | 11.30 | 10.30 | 10.60 | 10.80 | 12.10 | 12.20 | 12.40 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraKeyValue with 0 writers and 256 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 218803.58 | 209206.20 | 225392 | 4068.91 | 218849.97 | 209206.20 | 209206.20 | 213317.10 | 224702.67 | 225392 | 225392 |
| Read latency, ms/op | 1.17 | 1.14 | 1.22 | 0.02 | 1.17 | 1.14 | 1.14 | 1.14 | 1.20 | 1.22 | 1.22 |
| Write throughput, ops/sec | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Write latency, ms/op | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Load tester CPU, user, % | 31.19 | 13 | 34.70 | 4.52 | 32.20 | 13 | 13 | 30.10 | 33.10 | 34.70 | 34.70 |
| Load tester CPU, system, % | 13.48 | 5.70 | 16.20 | 2.08 | 13.70 | 5.70 | 5.70 | 12.80 | 15.70 | 16.20 | 16.20 |
| Cluster node CPU, user, % | 36.82 | 17.10 | 46.40 | 6.92 | 34.90 | 17.10 | 20.36 | 31.92 | 45.70 | 45.90 | 46.40 |
| Cluster node CPU, system, % | 7.30 | 5.60 | 9.70 | 1.54 | 6.50 | 5.60 | 5.70 | 5.70 | 9.50 | 9.50 | 9.70 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
YCQL Secondary Index
Workload CassandraSecondaryIndex with 64 writers and 0 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Read latency, ms/op | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Write throughput, ops/sec | 4957.63 | 4654.57 | 5466.63 | 190.85 | 4935.75 | 4654.57 | 4654.57 | 4701.88 | 5270.11 | 5466.63 | 5466.63 |
| Write latency, ms/op | 12.92 | 11.71 | 13.75 | 0.48 | 12.96 | 11.71 | 11.71 | 12.15 | 13.61 | 13.75 | 13.75 |
| Load tester CPU, user, % | 2.23 | 2 | 3.20 | 0.28 | 2.10 | 2 | 2 | 2 | 2.64 | 3.20 | 3.20 |
| Load tester CPU, system, % | 1.55 | 1.40 | 1.80 | 0.11 | 1.50 | 1.40 | 1.40 | 1.40 | 1.80 | 1.80 | 1.80 |
| Cluster node CPU, user, % | 32.02 | 29.60 | 33.70 | 0.96 | 32.20 | 29.60 | 29.70 | 30.62 | 33.26 | 33.38 | 33.70 |
| Cluster node CPU, system, % | 19.25 | 17.80 | 20 | 0.52 | 19.30 | 17.80 | 18.06 | 18.42 | 19.88 | 19.94 | 20 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraSecondaryIndex with 32 writers and 64 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 59031.00 | 57817.73 | 62446.65 | 1109.89 | 58745.42 | 57817.73 | 57817.73 | 58043.71 | 60768.14 | 62446.65 | 62446.65 |
| Read latency, ms/op | 1.08 | 1.02 | 1.11 | 0.02 | 1.09 | 1.02 | 1.02 | 1.06 | 1.10 | 1.11 | 1.11 |
| Write throughput, ops/sec | 3131.73 | 3076.68 | 3206.08 | 31.21 | 3127.82 | 3076.68 | 3076.68 | 3089.73 | 3176.39 | 3206.08 | 3206.08 |
| Write latency, ms/op | 10.22 | 9.98 | 10.40 | 0.10 | 10.23 | 9.98 | 9.98 | 10.08 | 10.36 | 10.40 | 10.40 |
| Load tester CPU, user, % | 14.97 | 14.20 | 15.70 | 0.41 | 15 | 14.20 | 14.20 | 14.52 | 15.62 | 15.70 | 15.70 |
| Load tester CPU, system, % | 8.25 | 7.20 | 8.60 | 0.34 | 8.30 | 7.20 | 7.20 | 7.60 | 8.60 | 8.60 | 8.60 |
| Cluster node CPU, user, % | 37.01 | 34.70 | 40.10 | 1.55 | 36.50 | 34.70 | 34.92 | 35.22 | 39.18 | 39.32 | 40.10 |
| Cluster node CPU, system, % | 14.41 | 13.80 | 14.80 | 0.26 | 14.50 | 13.80 | 13.90 | 14.02 | 14.80 | 14.80 | 14.80 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraSecondaryIndex with 16 writers and 128 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 104821.67 | 99481.21 | 120241.63 | 7320.25 | 101330.40 | 99481.21 | 99481.21 | 99546.21 | 119939.73 | 120241.63 | 120241.63 |
| Read latency, ms/op | 1.23 | 1.06 | 1.29 | 0.08 | 1.26 | 1.06 | 1.06 | 1.07 | 1.29 | 1.29 | 1.29 |
| Write throughput, ops/sec | 1949.49 | 1929.03 | 1990.20 | 17.51 | 1943.61 | 1929.03 | 1929.03 | 1931.10 | 1978.04 | 1990.20 | 1990.20 |
| Write latency, ms/op | 8.21 | 8.04 | 8.29 | 0.07 | 8.23 | 8.04 | 8.04 | 8.09 | 8.28 | 8.29 | 8.29 |
| Load tester CPU, user, % | 19.35 | 18.10 | 22.90 | 1.18 | 19 | 18.10 | 18.10 | 18.26 | 21.06 | 22.90 | 22.90 |
| Load tester CPU, system, % | 8.89 | 8.40 | 10.30 | 0.59 | 8.60 | 8.40 | 8.40 | 8.40 | 10.06 | 10.30 | 10.30 |
| Cluster node CPU, user, % | 40.96 | 36.50 | 44.20 | 2.31 | 41.90 | 36.50 | 36.72 | 37.20 | 43.18 | 43.62 | 44.20 |
| Cluster node CPU, system, % | 11.82 | 11.10 | 12.30 | 0.33 | 11.90 | 11.10 | 11.10 | 11.20 | 12.20 | 12.24 | 12.30 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
Workload CassandraSecondaryIndex with 0 writers and 256 readers:
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric | mean | min | max | std.dev | median | 1% | 5% | 10% | 90% | 95% | 99% |
| Read throughput, ops/sec | 187239.42 | 185195.56 | 188133.96 | 804.53 | 187287.10 | 185195.56 | 185195.56 | 185805.85 | 188109.85 | 188133.96 | 188133.96 |
| Read latency, ms/op | 1.37 | 1.36 | 1.38 | 0.01 | 1.37 | 1.36 | 1.36 | 1.36 | 1.38 | 1.38 | 1.38 |
| Write throughput, ops/sec | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Write latency, ms/op | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Load tester CPU, user, % | 19.78 | 0.90 | 29.40 | 11.76 | 26.90 | 0.90 | 0.99 | 1.50 | 27.87 | 29.17 | 29.40 |
| Load tester CPU, system, % | 8.55 | 0.70 | 14 | 4.88 | 11.45 | 0.70 | 0.74 | 1 | 11.80 | 13.67 | 14 |
| Cluster node CPU, user, % | 34.88 | 12.30 | 48.50 | 9.85 | 36.65 | 12.30 | 16.62 | 22.21 | 48.20 | 48.36 | 48.50 |
| Cluster node CPU, system, % | 8.70 | 4.30 | 17.20 | 4.21 | 7.70 | 4.30 | 4.50 | 4.67 | 16.56 | 16.93 | 17.20 |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
YCSB
+--------------+----------------------+--------------------+--------------------+--------------------+
| Workload | Throughput (ops/sec) | Op Type | Avg Latency (us) | 99th% Latency (us) |
| workloada | 75054.10 | READ | 2527.53 | 20895 |
| | | UPDATE | 4032.19 | 25391 |
| | | | | |
| workloadb | 89653.56 | READ | 2711.76 | 7967 |
| | | UPDATE | 3391.83 | 8983 |
| | | | | |
| workloadc | 89456.49 | READ | 2753.47 | 7939 |
| | | | | |
| workloadd | 87712.48 | READ | 2765.26 | 8327 |
| | | INSERT | 3598.85 | 9671 |
| | | | | |
| workloade | 16485.62 | INSERT | 14684.71 | 70847 |
| | | SCAN | 14963.57 | 70271 |
| | | | | |
| workloadf | 49282.21 | READ | 2956.77 | 20815 |
| | | READ-MODIFY-WRITE | 7039.35 | 29743 |
| | | UPDATE | 4074.65 | 23871 |
| | | | | |
+--------------+----------------------+--------------------+--------------------+--------------------+