yugabyte/yugabyte-db v2.3.0.0

v2.3.0.0 (Released September 8, 2020)

on GitHub

YSQL

• Fix OOM when running large COPY TO statements by creating a new memory context for the loop over retrieved rows and resetting it after processing each row. #5205
• Fix OOM on file-sourced YB relations in the COPY FROM statement. Reset memory context regularly, per row, when memory is resettable and when rows are read from a file (not stdin). #5561
• Support transactional batch size for COPY FROM command with OOM fix. Batch sizes can be passed in with ROWS_PER_TRANSACTION in the COPY OPTION syntax. #2855 #5453
• For index backfill flags, use better default values. Set index_backfill_rpc_timeout_ms default from 60000 to 30000 and change backfill_index_timeout_grace_margin_ms default from 50 to 500. #5494
• Remove spurious error message "0A000: Alter table is not yet supported" from CREATE OR REPLACE VIEW. #5071
• Prevent consistency violations when a partitioned table has foreign key constraints due to erroneous classification as a single-row transaction. #5387
• Fix restore from a distributed backup fails for tables using SPLIT INTO without a primary key. #4993
• Fix the wrong result by avoiding pushdown of UPDATE statement with the RETURNING clause. #5366
• Improve error message when UPDATE changes partition so its clear what went wrong, fixed yb_pg_foreign_key pg_regress test for semantic merge conflict when updating primary keys and row-level partitioning. #659 #5179 #5310
• Allow the UPDATE statement to change primary key columns. #659
• Support row-level partitioning. #5179
• Block usage of TABLEGROUP with the SPLIT clause. For CREATE TABLE, usage is blocked in the grammar. For CREATE INDEX, if NO TABLEGROUP was provided, then presplitting for the index is allowed; otherwise, an error is issued. #5352
• Buffered operations may share single RPC with read operation. Reducing the number of RPC calls speeds up the CREATE TABLE statement. For example, the total number of RPC calls is dropped from 66 to 58 for a simple table like CREATE TABLE t(k INT PRIMARY KEY). #5177
• Add new ysqlsh describe metacommands for tablegroups: \dgr[+] [grpname] to describe tablegroups, \dgrt[+] [grpname] lists all tables/indexes within the specified tablegroup (or within all tablegroups if grpname is not specified), and \d <table_name> is modified to include tablegroup information in the footer, if any. #5088
• Add ALTER TABLEGROUP statements to support ALTER TABLEGROUP tablegroup_name RENAME TO ... and ALTER TABLEGROUP tablegroup_name OWNER TO .... Also changes pg_tablegroup entry corresponding to tablegroup_name to properly reflect new grpname and grpowner (if the user has proper permissions or ownership to issue the ALTER TABLEGROUP statement. #5249
• Add support for indexes to opt out of tablegroups (NO TABLEGROUPS) or select their own tablegroup (TABLEGROUP group_name). #5293
• For ysql_dump, enable serializable-deferrable mode by default. Add new --no-serializable-deferrable flag to disable the default mode. #5128
• Support ALTER COLUMN type for variants that don't require on-disk changes. Specifically only allowing ALTER COLUMN type for varch(n) and varbit(n). For example, changing column type from varchar(50) to varchar(255). #4424
• Enable USING clause in DELETE statement and FROM clause in UPDATE statement. #738 #5262
• Support GRANT/REVOKE/ALTER DEFAULT PRIVILEGES for tablegroups. #5087 #5160
• Implement tablegroup query layer changes. #4525
• Replace "Foreign Scan" with "Seq Scan" for EXPLAIN statement. #2076
• Fix initdb when index backfill is enabled. #5027
• Properly handle empty delete with backfill by sending appropriate messages to the client, including setting it skipped and rows_affected_count to 0. #5015
• DROP INDEX statement invalidates table cache entry for the index table but should invalidate the table cache entry for the indexed table. #4974
• yb-admin create_database_snapshot command should not require ysql. prefix for database name. #4991
• For non-prepared statements, optimize pg_statistic system table lookups and update debugging utilities. #5051
• Correctly show beta feature warnings by default. #5322

YCQL

• For WHERE clause in CREATE INDEX statement, return a Not supported error. #5363
• Fix TSAN issue in partition-aware policy for C++ driver 2.9.0-yb-8 (yugabyte/cassandra-cpp-driver). #1837
• Support YCQL backup for indexes based on JSON-attribute. #5198
• Correctly set release_version for system.peers queries. #5407
• Reject TRUNCATE operations when ycql_require_drop_privs_for_truncate flag is enabled. When enabled, DROP TABLE permission is required to truncate a table. Default is false. #5443
• Fix missing return statement in error case of the SetPagingState method in statement_params.cc. #5441
• Enable backfilling of transactional tables by default. #4708
• Fix ycqlsh should return failure when known that the create (unique) index has failed. #5161

Core database

• Fix core dump related to DNS resolution from cache for Kubernetes universes. #5561
• Fix yb-master fails to restart after errors on the first run. #5276
• Show better error message when using yugabyted and yb-master fails to start. #5304
• Disable ignoring deleted tablets on load by default. #5122
• [CDC] Improve CDC idle throttling logic to reduce high CPU utilization in clusters without workloads running. #5472
• For server_broadcast_addresses flag, provide default port if not specified. #2540
• [CDC] Fix CDC TSAN destructor warning. #4258
• Add API endpoint to download the root certificate file. #4957
• Do not load deleted tables and tablets into memory on startup. #5122
• Set the follower lag for leaders to 0 by resetting timestamp to the maximum value when a peer becomes a leader and when the peer loses leadership. #5502
• Flow keyspace information from yb-master to yb-tserver. #3020
• Implement meta cache lookups throttling to reduce unnecessary thrashing. #5434
• Fix rpcz/statements links in yb-tserver Web UI when pgsql_proxy_bind_address and cql_proxy_bind_address are 0.0.0.0. #4963
• DumpReplayStateToStrings should handle too many WAL entries scenario. Also, log lines only display fields critical for debugging and do not show customer-sensitive information. #5345
• Find the difference between the replica map and consensus state more quickly using map lookup. #5435
• Improve failover to a new master leader in the case of a network partition or a dead yb-tserver by registering tablet servers present in Raft quorums if they don't register themselves. Also, mark replicas that are in BOOTSTRAPPING/NOT_STARTED state with its true consensus information instead of marking it as a NON_PARTICIPANT. #4691
• Add YB-Master UI changes to support tablegroups and colocation. #5086
  • Adds a column to the user and index tables to display the YSQL OID information about the colocation parent table (if any).
  • For tablegroups, it displays the OID of the table's tablegroup (same as the OID present in pg_tablegroup as well as the reltablegroup column of pg_class for that relation).
  • For colocated databases, it displays the OID of the database.
  • Also, creates a table to display information about the parent tables. Wrap the names and uuid of the parent tables. Display YSQL OIDs as explained above.
• Fix MetaCache::TAbleData::stale is not getting reset back to false. #5245
• Do not crash yb-tserver when the op_id of the last WAL file is less than the committed op_id. #1560
• Add CREATE TABLEGROUP and DROP TABLEGROUP flow. #4525
• Add yb-admin list_snapshots SHOW_DELETED flag to show deleted snapshots that are still retained in memory. #5332
• Move read path functions to a separate file. #4944
• Wait for yb_client_admin_operation_timeout_sec for table creation in yb-admin import_snapshot task. #5295
• Create a cache for system-wide queries. #5043
• Annotate access to enable_collect_cdc_metrics flag and prevent TSAN warnings. #5303
• Raise yb-master and yb-tserver process ulimits to hard limits when appropriate. #4943
• Add a pre-flush step, triggered by yb-master, to snapshot operations to minimize the amount of data being flushed in sync. #4718
• Use Strand to remove intents and avoid wasting threads. #5265
• Add metric severity level [PORT] and server-side level-based filtering for metrics and prometheus-metrics. #4686
• Decrease logging frequency for no local transaction status tablet from on every warning to once per second. #5035
• Fix MonoTime subtraction implementation to return result of the operation instead of lhs. #5020
• When dropping tables, delete snapshot directories for deleted tables. #4756
• Set up a global leader balance threshold while allowing progress across tables. Add load_balancer_max_concurrent_moves_per_table and load_balancer_max_concurrent_moves to improve performance of leader moves. And properly update state for leader stepdowns to prevent check failures. #5021 #5181
• Reduce default value of load_balancer_max_concurrent_moves from 10 to 2. #5461
• Improve bootstrap logic for resolving transaction statuses. #5215
• Avoid duplicate DNS lookup requests while handling system.partitions table requests. #5225
• Handle write operation failures during tablet bootstrap. #5224
• Ensure strict_capacity_limit is set when SetCapacity is called to prevent ASAN failures. #5222
• Drop index upon failed backfill. For YCQL, generate an error to CREATE INDEX call if the index backfill fails before the call is completed. #5144 #5161
• Allow overflow of single-touch cache if the multi-touch cache is not consuming space. #4495
• Don't flush RocksDB before restoration. #5184
• Fix yugabyted fails to start UI due to class binding failure. #5069
• On yb-tserver restart, prioritize bootstrapping transaction status tablets. #4926
• For yb-admin commands, clarify syntax for namespace. #5482
• Improve for-loops to avoid unnecessary copies and remove range-loop-analysis warnings. #5458
• Apply a large transaction in a separate thread if the transaction will not fit into the limit of key-value pairs. The running transaction is removed from memory after all its intents are applied and removed. #1923
• Fix SEGV in Master UI when registering YB-TServer from Raft. #5501
• Heartbeats process only new consensus information unless master_ignore_stale_cstate flag is disabled. #5301
• Remove KernelStackWatchdog and use LongOperationTracker to waste fewer resources per thread. #5226
• Change default value of metric_node_name flag from DEFAULT_NODE_NAME to hostname:port. #4859

Yugabyte Platform

• For S3 backups, install s3cmd required for encrypted backup and restore flows. #5593
• When creating an on-premises provider, remove YB_HOME_DIR if not set. #5592
• Update to use templatized values for setting TLS flags in Helm Charts. #5424
• For YCQL client connectivity, allow downloading root certificate and key from the Certificates menu. #4957
• Bypass CSRF checks when registering and logging in due to bug with CSRF token not being set. #5533
• Set node certificate field to subject of CA instead of the issuer of CA. #5377
• Add option in the universe creation form to control whether node_exporter is installed on provisioned nodes (including creating prometheus user + group). #5421
• Add Microsoft Azure provider UI and related changes to Create Universe form. #5378
• Create default network resources for multi-region deployments in Microsoft Azure if they don't exist and store for use during universe creation. #5388
• For Microsoft Azure, use preexisting network resources. #5389
• Set Assign Public IP correctly using the UI. #5463
• Fix AreLeadersOnPreferredOnly times out when setting preferred zones in Edit Universe. #5406
• Fix yb_backup.py script to remove the sudo requirement and correctly change the user. #5440
• Add CSRF tokens to forms. #5419
• Allow configuration of OIDC scope (for SSO use) as well as email attribute field. #5465
• If Username and Password are left empty in the Custom SMTP Configuration, remove smtpUsername and smtpPassword from the payload. #5439
• Add UTC label to indicate cron expression must be relative to UTC. Also, add help text indicating when the next scheduled job will run. #4709
• Move the source of truth for communication ports to the universe level. #5353
• Use the appropriate range for the time period (over the entire step window rather than the last minute) when querying Prometheus for metrics. #4203
• Support backing up encrypted at rest universes. #3118
• Only display "Rolling Upgrade Delay" if the upgrade type is "Rolling". #5362
• Add guarding against never-ending loop when running yb_backup.py script and yb-server doesn't have any data directories. #5358
• Fix metric panel displaying incorrect time interval when reloading after changing graph filter. #5294
• Fix logging on health check script failure. #5340
• Rename label for enabling server-side encryption in S3 backups to Encrypt Backup for both backups and scheduled backup modals. #5273
• Add a VM backup script that includes Prometheus backups. #5120
• Add BackupAdmin role to authorize users to create a backup and restore tasks on universes, but not given other administrative privileges. #4694
• Modify yb_backup script to add "round-robin" behavior to thread allocations to better spread loads across nodes in parallel during snapshot upload or download. #4987
• Fix invalid yb_backup.py script behavior if --keyspace flag is not specified. #5199
• Add subtasks to check the number of pods deployed during upgrades. #5174
• Make use of parallelism argument for yb_backup.py script. Explicitly pass the number of threads to concurrently run backup script through UI. The form defaults to the backup script default value of 8 threads if no value is provided. #5283
• The Remove Node action should be available if the Add Node action fails. #5123
• Add the ability for user to specify the S3 backup target host that their bucket is located on. #3684
• Check if the on-premise node instance has a universe that is using it and make a link to the Universe Overview page. #4758
• Fix Cannot read property 'displayName' of undefined error. #5267
• Use Host Names setting is not displaying correctly in Provider Configuration page. #5252
• Disable glob before running the cleanup of old log files. #5169
• Schedule job for setUniverseKey should handle errors on a per-customer basis. #3142
• After the Yugabyte Platform (Yugaware) restart, any in-progress tasks should be canceled and marked as failed. #5045
• Redirect from login back to index if the session is already validated. Logout should redirect to the login endpoint. #5019
• Fix updating of Universes list after creating a new universe. #4784
• Fix restore payload when renaming table to include keyspace. And check for keyspace if tableName is defined and return invalid request code when keyspace is missing. #5178
• Add the ability to override communication port default values during universe creation. #5354

Note:

Prior to version 2.0, YSQL was still in beta. As a result, the 2.0 release included a backward-incompatible file format change for YSQL. If you have an existing cluster running releases earlier than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from your existing cluster and then import it into a new cluster (v2.0 or later) to use existing data.