risingwavelabs/risingwave v2.0.0

on GitHub

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• Supports Azure Blob file sources. #18295
• Supports glue catalog for iceberg sink and source. #17477
• Adds jdbc.query.timeout for JDBC sinks to set the timeout for queries. #18430
• Changes default Kafka sink message timeout from five seconds to five minutes. #18304
• Adds new parameters retry_on_conflict, batch_size_kb, batch_num_messages, and concurrent_requests for ElasticSearch sink.#17867
• Supports parameter bigquery.retry_times for BigQuery sink. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Breaking change: Adds a default soft and hard limit on actor count per worker parallelism. When the hard limit is reached, streaming workloads will fail. #18383
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. #17232

Full Changelog: v1.10.1...v2.0.0