risingwavelabs/risingwave v2.8.0 on GitHub

What's Changed

test(cdc): migrate mongodb-cdc tests to inline style with all data types by @xiangjinwu in #23753
fix: change placeholder of username in ldap filter config by @yezizp2012 in #23782
feat(vector): support stream vector index lookup by @wenym1 in #23288
refactor: improve cluster kill-and-recovery tests with precise recovery tracking per database by @shanicky in #23728
feat(iceberg): support s3 and glue iam role arn by @chenzl25 in #23775
chore(deps): Bump mach2 from 0.5.0 to 0.6.0 by @dependabot[bot] in #23784
feat(streaming): add functionality to override config when building actors by @BugenZhao in #23735
chore(ci): add maven cache for build-other.sh step by @BugenZhao in #23781
feat(metrics): add relation busy rate to dashboard by @kwannoel in #23730
chore: rename iceberg common fields to better names. by @chenzl25 in #23790
feat(iceberg): support assume role for s3 and glue catalog together with enable_config_load by @chenzl25 in #23788
feat(iceberg): change iceberg list interval sec to 10 sec by @chenzl25 in #23791
fix(iceberg): rebuild stream when errors occur for iceberg list executor by @chenzl25 in #23793
chore: replace number_prefix with unit-prefix by @Standing-Man in #23801
feat(mysql-cdc): monitor the progress of binlog offsets in the upstream and downstream processes by @wcy-fdu in #23692
fix(iceberg): fix iceberg engine with compaction extra config by @chenzl25 in #23809
refactor(meta): refine inflight info ownership and avoid unnecessary clone by @wenym1 in #23754
fix(iceberg): fix batch iceberg list executor by @chenzl25 in #23811
fix(dashboard): fix some foyer metrics name followed by OpenMetrics by @MrCroxx in #23818
feat: add periodic refresh table jobs & refactor ProgressTracker by @tabVersion in #23737
fix(iceberg): fix batch iceberg list executor by @chenzl25 in #23824
fix: support AdaptiveParallelismStrategy as string format in config by @shanicky in #23817
chore: add back grafana json by @tabVersion in #23831
chore(iceberg): better logging if mis-config enable_config_load on Cloud by @tabVersion in #23822
feat: migrate JoinEncodingType from session config to per-job config override by @BugenZhao in #23768
fix(ci): use larger runner with more disk space for workflows that build RW by @BugenZhao in #23842
fix(streaming): fix locality backfill state -- write upstream chunk to the state by @chenzl25 in #23838
fix: update Minio client initialization to use keyword arguments by @tabVersion in #23836
fix(ci): bump and pin versions of mdbook and related tools by @BugenZhao in #23840
fix(ci): update connector node integration test pipeline to java 17 & 21 by @tabVersion in #23854
feat(meta): include view definition in diagnostic report by @zwang28 in #23853
fix: Supports config checks within compaction group scheduling. by @Li0k in #23841
feat: implement watermark derivation for AsOf joins by @yuhao-su in #23803
fix(sink): use downstream pk to compact when stream key is its subset by @BugenZhao in #23859
perf(expr): support writer-style #[function] for array type by @silver-ymz in #23767
feat: add last_success_time tracking to refresh jobs by @tabVersion in #23834
fix(streaming): deduplicate locality backfill ordering edge casued by share sub-plan by @chenzl25 in #23869
fix(optimizer): fix gap fill column pruning by @chenzl25 in #23867
fix(test): delay mongodb-cdc test insertion by @xiangjinwu in #23866
fix(iceberg): add snapshot id to iceberg scan by @chenzl25 in #23882
feat(frontend): add handler for ALTER .. [SET | RESET] CONFIG by @BugenZhao in #23748
refactor(meta): introduce strong typed subscriber id by @wenym1 in #23883
fix(optimizer): use shard_by_exact_key instead of shard_by_key for locality backfill by @chenzl25 in #23888
feat(optimizer): improve index selection for topn by @chenzl25 in #23894
refactor(foyer): upgrade foyer to v0.21.0-dev by @MrCroxx in #23750
refactor(config): remove unnecessary prefix in configs by @BugenZhao in #23889
chore: upgrade parking_lot to 0.12.5 by @Li0k in #23902
chore: replace sleep + query pattern to retry X backoff 1s by @tabVersion in #23855
feat(cdc): allow user alter debezium.max.queue.size to avoid JVM OOM by @wcy-fdu in #23886
fix: invalid schema type for LogicalVectorSearchLookupJoin by @silver-ymz in #23893
fix(test): Use tokio time pausing in ut to avoid unstable clocks for periodic barriers by @Chiro11 in #23905
fix(ci): fix several failures in main-cron by @BugenZhao in #23900
test(backfill): add more tests for locality backfill by @chenzl25 in #23906
fix: use on_conflict_do_nothing for mysql backend by @tabVersion in #23915
fix(optimizer): fix agg to dedup optimization by @chenzl25 in #23922
refactor: improve stream parallelism resolution and error handling by @shanicky in #23921
feat(meta): add service impl for ALTER .. SET CONFIG by @BugenZhao in #23864
fix(mongo-cdc): add connection check for mongo cdc by @wcy-fdu in #23850
feat(sink): exactly once sink on coordination process by @Chiro11 in #23657
feat(frontend): add system table rw_streaming_job_config by @BugenZhao in #23914
feat: use datafusion engine for Iceberg batch query by @silver-ymz in #23860
feat: migrate configs for sync log store and over-window executor from session config to per-job config override by @BugenZhao in #23792
fix(sink): fix redshift timestamptz error by @xxhZs in #23812
feat: persist compactor iceberg flag across meta and diagnostics by @Li0k in #23884
chore: allow altering owner of iceberg table by @yezizp2012 in #23944
chore(deps): Bump uuid from 1.18.1 to 1.19.0 by @dependabot[bot] in #23951
feat: Expose license status, limits, and features in diagnose output by @Li0k in #23885
refactor(hummock): refine fetch_block impl, restore old style by @MrCroxx in #23957
feat(stream,dashboard): aggregate tokio actor metrics by fragment by @kwannoel in #23747
refactor(streaming): move MaterializeCache to a separate module by @BugenZhao in #23961
feat: support setting pause_on_next_bootstrap_offline by @EdwinaZhu in #23910
chore(streaming): add explicit target for expression error log (stream_expr_error) by @kwannoel in #23964
fix(cdc): fix snapshot schema by @zwang28 in #23970
feat(meta): add streaming job table in diagnose output by @BugenZhao in #23966
fix(metric): fix error metrics in dashboard by @zwang28 in #23974
fix(doc): missing GNU Parallel installation by @Standing-Man in #23851
feat: support created_at and initialized_at for rw_views by @Standing-Man in #23774
refactor(file_sink): do not use executor id for file sink by @wcy-fdu in #23981
feat(tokio): bump tokio to 1.47 by @chenzl25 in #23985
perf(streaming): remove passed_actors in barrier to reduce encoded size by @BugenZhao in #23987
chore: bump thiserror-ext to avoid dep on thiserror & bump thiserror to 2.0.17 by @BugenZhao in #23999
fix: fix auto cdc schema change with schema binding and memory leak in fe by @yezizp2012 in #23993
chore(deps): Bump github.com/sirupsen/logrus from 1.8.1 to 1.8.3 in /integration_tests/datagen by @dependabot[bot] in #24007
refactor(streaming): reuse code via generics between Merge and Receiver executors by @BugenZhao in #23984
refactor: Reuse and manage Nats client connections by @tabVersion in #23963
fix(test): fix mysql_partition test in main cron by @wcy-fdu in #23965
refactor(sink): Backup new exactly once sink table & Remove some legacy code by @Chiro11 in #24001
test(sink): Enable integration test for exactly once sink by @Chiro11 in #23833
refactor(sink): Delay the initialization of IcebergSinkWriter after LogReader alignment by @Chiro11 in #23949
fix: notify local subscribers when streaming job backfill finishes by @shanicky in #24021
chore(deps): Bump net.snowflake:snowflake-jdbc from 3.23.0 to 3.23.1 by @hzxa21 in #23813
feat(dashboard): inspect streaming job info & config override by @BugenZhao in #24037
fix(sink): Missing field update in sink auto schema change by @Chiro11 in #24039
refactor(error): use provide functionality from thiserror-ext to avoid patching thiserror by @BugenZhao in #24008
chore(deps): Bump criterion from 0.7.0 to 0.8.1 by @dependabot[bot] in #24043
refactor(mysql-cdc): cache latest schema history file records to reduce object storage I/O by @wcy-fdu in #23976
feat: support sink payload as bytes into downstream mq by @tabVersion in #22925
fix: pass file cache recover mode config correctly by @hzxa21 in #24031
refactor: refactor write-mode and compaction.type to enum by @Li0k in #23942
feat: support risingwave scalar function in datafusion by @silver-ymz in #24002
fix: add retry delay and improve error handling in observer and fragment modules by @shanicky in #24057
refactor: split batch and streaming exchange service by @BugenZhao in #24050
refactor(foyer): upgrade foyer to released version v0.21.0 by @MrCroxx in #24047
feat(frontend): improve notifications for slow DDL (CREATE TABLE and ALTER RATE LIMIT/PARALLELISM) by @EdwinaZhu in #24016
refactor(watermark): use clean_watermark_indices to replace clean_watermark_index_in_pk by @yuhao-su in #23877
fix: fix sstable serde to avoid losing bloom filter by @hzxa21 in #24025
fix: add retry mechanism to mqtt_nested_source query in e2e test by @Copilot in #24063
feat: optimize streaming job count query and improve diagnostics by @shanicky in #24068
feat(frontend): support backfill order control for sinks by @Copilot in #23958
chore(ci): check duplicated lines in logs & fix verbose logs by @BugenZhao in #24045
feat: unify the managment of privileges for the iceberg sink, source and table by @yezizp2012 in #24038
chore(pgwire): add start of execution logs for all queries by @kwannoel in #24079
fix(test): fix flaky time travel e2e tests by @BugenZhao in #24087
chore: increase log filter levels for some external crates by @BugenZhao in #24085
feat(streaming): use actor-local config for exchange by @BugenZhao in #24036
feat: add vnode-aligned compaction config for large single-table group by @Li0k in #24053
feat(sink): Support schema change in decouple sink by @Chiro11 in #24048
feat(pulsar): add support for read_compacted option in consumer properties by @tabVersion in #23582
feat(iceberg): support DeleteScan for datafusion engine by @silver-ymz in #24065
feat(sink): Support schema change in two-phase commit coordination by @Chiro11 in #24069
fix: better handle iceberg error when refreshing by @tabVersion in #23847
fix(frontend): support view sink internal state tables by @kwannoel in #24051
refactor(meta): separate per job cdc backfill tracker by @wenym1 in #23795
refactor(compute): avoid locking metrics in exchange loop each time we yield a message by @kwannoel in #24099
fix(sink): Parameters lost when sink-decouple by @Chiro11 in #24114
feat(iceberg): bump iceberg rust to 20251111 version by @chenzl25 in #24004
refactor(iceberg): simplify scan_task_to_chunk by using the sdk to handle delete by @chenzl25 in #24089
feat(meta): trigger barrier immediately on force checkpoint by @wenym1 in #24141
fix(frontend): reject defining watermarks on upsert source by @BugenZhao in #24113
fix(meta): avoid global recovery on single database reload failure by @wenym1 in #24140
feat(iceberg): do not commit large statistical to avoid persistent metadata becoming too large by @wcy-fdu in #24054
chore: update package version in Cargo.toml by @tabVersion in #24147
feat(deltalake): bump deltalake to 0.29 and remove arrow 55 by @chenzl25 in #24145
refactor: improve error handling for datafusion engine by @silver-ymz in #24097
refactor(meta): avoid self-generate command in barrier worker by @wenym1 in #23858
fix(iceberg): fix iceberg_source_streaming flaky test by @chenzl25 in #24150
feat(risedev): add ADBC Snowflake driver support by @tabVersion in #23986
fix(frontend): forbid temporal join with snapshot backfill by @wenym1 in #24149
fix(iceberg): change iceberg partition field id from 0 to 1000 by @chenzl25 in #24139
fix(meta): fill original snapshot backfill epoch in alter mv by @wenym1 in #24153
feat(iceberg): support explain datafusion plan by @chenzl25 in #24157
fix(ci): fix package version check script when there are branches like release-2.7-suffix by @BugenZhao in #24160
feat: backup refresh jobs in meta snapshot by @tabVersion in #24158
feat: support ALTER CONNECTION CONNECTOR WITH by @tabVersion in #22563
refactor(streaming): remove watermark cache in state table by @BugenZhao in #24148
chore(ci): bump main branch package version after creating new release branches by @cyliu0 in #24165
fix(ci): correct bump main PR body by @cyliu0 in #24173
feat(streaming): add metrics for oom and other exit statuses on kernel dashboard by @kwannoel in #24175
feat(streaming): always enable merge barrier align metrics by @kwannoel in #24163
fix: fix hummock metrics bucket by @Li0k in #24143
feat: improve type coercion and validation for datafusion execution by @silver-ymz in #24120
chore(pgwire): add sql to the log span by @kwannoel in #24171
chore: update error message by @wcy-fdu in #24181
fix: simplify alter connection test by @tabVersion in #24185
refactor(streaming): keep & validate row id if specified regardless of op in RowIdGen by @BugenZhao in #24164
fix(ci): check in dhparam.pem for ldap test by @BugenZhao in #24180
feat: include more useful info in diagnose by @hzxa21 in #23992
refactor: strong typed executor and operator id by @wenym1 in #24204
feat: snowflake source support by @tabVersion in #23988
fix(streaming): wrap snapshot backfill executor by @wenym1 in #24205
feat(stream, observability): add job-level cpu usage profiling for streaming job by @EdwinaZhu in #24151
fix: support async javascript UDTF by @hzxa21 in #24197
feat(dml): add an option for emitting upsert records by @BugenZhao in #24152
refactor(pgwire): use concrete error type in Session by @BugenZhao in #24201
chore(ci): always upload docker compose logs by @BugenZhao in #24217
chore: update rules for risedev usage in build-run-test.mdc by @BugenZhao in #24224
feat(hummock): make block based XOR16 filter threshold configurable by @Li0k in #24208
fix(iceberg): fix row id gen in iceberg fetch executor by @chenzl25 in #24231
feat(expr): support casting serial to timestamptz to extract time from row id by @BugenZhao in #24233
chore: ignore .cargo/ (for trailing space check and maybe more) by @xiangjinwu in #24244
refactor: separate fragment metadata loading and actor assignment rendering by @shanicky in #24155
fix(snapshot-backfill): fix snapshot backfill recovery panic by @wenym1 in #24245
feat(meta): create new actors for finished snapshot backfill jobs by @wenym1 in #24189
feat(iceberg): support schema change(add column) for non exactly once iceberg sink by @wcy-fdu in #24246
fix(iceberg): stop coordinator when drop iceberg engine table & improve retry by @chenzl25 in #24259
test(streaming): add a test case demonstrating scan.startup.mode = 'latest' with shared source by @BugenZhao in #24253
feat: support aggregate function for datafusion by @silver-ymz in #24206
fix: XorFilterReader estimate_size by @Li0k in #24262
chore(deps): Bump org.apache.logging.log4j:log4j-core from 2.25.1 to 2.25.3 in /java by @dependabot[bot] in #24210
refactor(storage): use clean_watermark_index in StateTable by @BugenZhao in #24232
chore(deps): Bump org.bouncycastle:bcpkix-jdk18on from 1.78 to 1.79 in /java/connector-node/risingwave-sink-jdbc by @dependabot[bot] in #23666
chore(deps): Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0 in /integration_tests/datagen by @dependabot[bot] in #23771
fix: count star may panic in datafusion by @silver-ymz in #24267
feat(frontend): support planning table with watermark-ttl by @BugenZhao in #24121
feat: improve default compaction config by @Li0k in #24209
feat: support order by in datafusion by @silver-ymz in #24275
fix: fix iceberg compaction queue duplicated id by @Li0k in #24279
feat(sink): Sink schema change from barrier to persistent by @Chiro11 in #24271
feat: support sql state 3D000 & add slt tests by @BugenZhao in #24216
fix(dashboard): download heap dump collapsed analyzed result by @BugenZhao in #24287
chore(ci): add semver tag for images build on non-release branches by @cyliu0 in #24283
fix(sink): fix jdbc sink setTransactionIsolation to prevent RDS Proxy session pinning by @xxhZs in #24270
chore(deps): Bump redis from 0.32.7 to 1.0.2 by @dependabot[bot] in #24265
feat: add backfill-specific parallelism control for streaming jobs by @shanicky in #24026
refactor(dashboards): rewrite to reduce nested string quotes by @EdwinaZhu in #24238
feat(binder): reuse existing expressions when binding extra columns in ORDER BY by @BugenZhao in #24298
feat(risectl, dashboard): support profiling nodes other than compute node by @BugenZhao in #24278
fix(test): increase recovery wait time in backfill parallelism test by @Copilot in #24309
feat: make decimal as datafusion native type by @silver-ymz in #24284
feat(expr): allow jsonb null values to be cast to scalar types as null by @howenyap in #23608
chore(ci): use file creation date as copyright year for license header check by @BugenZhao in #24317
fix(expr): resolve PG incompatibility with NULL handling in array con… by @Standing-Man in #23399
fix(optimizer): improve table scan better locality index selection by @chenzl25 in #24318
feat(iceberg): improve iceberg decimal handling: adjust ICEBERG_DECIMAL_PRECISION = 38 by @chenzl25 in #24316
fix(iceberg): Remove some limit between sink-decouple and exactly-once by @Chiro11 in #24290
chore: fix iceberg schema change test in main cron by @wcy-fdu in #24324
refactor(meta): throttle jobs by fragment by @wenym1 in #24301
fix(meta): include cdc metadata in backup by @zwang28 in #24327
feat(iceberg): support schema change for exactly once iceberg sink by @Chiro11 in #24273
feat(expr): add array_reverse by @howenyap in #23478
fix: fix split huge group condition by @Li0k in #24243
feat: support window function in datafusion by @silver-ymz in #24297
fix(meta): fix race condition during metadata backup by @zwang28 in #24269
fix(meta): Fix loop control in alter mv graph matching by @Chiro11 in #24326
perf(storage): avoid redundant key encoding in block builder by @Li0k in #24228
refactor(streaming): remove cleaned_by_watermark field in table by @BugenZhao in #24266
refactor(sink): replace ForceAppendOnly with a separate ignore_delete field by @BugenZhao in #24310
feat(cdc): introduce configurable and adjustable max.queue.size.in.bytes by @wcy-fdu in #23897
fix(iceberg): relax meta store check for iceberg connection by @chenzl25 in #24260
feat: support LogicalUnion and LogicalDedup in datafusion by @silver-ymz in #24300
perf: Reduce block-meta copies in the compactor . by @Li0k in #24256
feat: shorten_block_smallest_key by @Li0k in #24274
feat: export iceberg stats to datafusion by @silver-ymz in #24334
refactor: preload recovery context for upstream sink union recovery by @shanicky in #24255
fix(optimizer): fix topn index selection for scan with output col idx by @chenzl25 in #24242
chore(ci): refine hints on license header checks by @BugenZhao in #24343
refactor(sink): Split sink schema change to use two-phase commit by @Chiro11 in #24288
feat: maintain per vnode min/max in state table for pruning by @hzxa21 in #24214
chore(deps): Bump serial_test from 3.2.0 to 3.3.1 by @dependabot[bot] in #24347
feat(iceberg): support explain datafusion physical plan by @chenzl25 in #24345
feat(optimizer): convert lateral table-function apply to project-set by @BugenZhao in #24254
fix(privilege): Use super user for dummy session to avoid permission checks by @Chiro11 in #24352
feat(stream): remove dispatcher on error to avoid recovery by @wenym1 in #24289
fix(metrics): join lookup miss metric in take_state_opt by @yuhao-su in #24363
feat(cdc): introduce reset source command by @wcy-fdu in #23945
fix(sink): fix force-append-only sink panicked at inconsistent changes when specified pk does not guarantee uniqueness by @BugenZhao in #24368
ci: enable datafusion feature for docker and release by @silver-ymz in #24379
test: add tests for metadata backup sync with meta model by @tabVersion in #24362
feat(expr): postgres_query (aka PostgreSQL batch source) SSL support by @xiangjinwu in #24340
fix(planner): reject impure expressions on retract stream on to_proto by @BugenZhao in #24353
chore: refine the temporal join error message by @hzxa21 in #24391
fix(cdc): fix connector failure due to OpenLineage by @zwang28 in #24382
chore: fix unit test for state table vnode pruning by @hzxa21 in #24390
chore: bump futures-async-stream for correct code coverage report by @BugenZhao in #24385
fix(streaming): some executors should always clear the cache on scaling regardless of cache_may_stale by @BugenZhao in #24384
feat(frontend): auto disable snapshot backfill in some cases by @wenym1 in #24360
fix: make catalog queries transaction-aware in recovery context by @shanicky in #24407
fix(test): avoid cleaning up reducer input log file by @wenym1 in #24406
fix(test): only check SST subset in meta backup sst test by @wenym1 in #24405
fix(optimizer): fix unify first last value rule by @chenzl25 in #24398
fix: initialize notification infra before first ticker tick by @shanicky in #24049
fix: fix the non-exit problem after seek rewind. by @Li0k in #24416
fix(test): force stable scale for numeric matching in two_phase_approx_percentile_merge_stateless_agg.slt by @BugenZhao in #24420
ci: trigger homebrew bump on release by @BugenZhao in #24422
feat: add reschedule support for backfilling jobs by @shanicky in #23998
feat(backup): validate object store parameter by @zwang28 in #24426
doc: refine README messaging by @yingjunwu in #24433
feat(meta): avoid sending inject barrier request to all workers by @wenym1 in #24361
feat(diagnose): add hostname column to diagnosis output by @BugenZhao in #24436
perf: improve l0 seclection by @Li0k in #24417
chore(test): enable coverage for madsim tests by @BugenZhao in #24389
refactor(meta): include upstream partial graph id in exchange request by @wenym1 in #24342
feat: enhance DataFusionExecuteChecker to align with DataFusionPlanConverter by @silver-ymz in #24341
test: resolve massive log when prepare sql server data by @tabVersion in #24408
feat(meta): include backfill_type and is_serverless in rw_ddl_progress by @wenym1 in #24401
ci(kafka): replace bitnami with redpanda for kafka sasl tests by @xiangjinwu in #24437
feat(streaming): executor part for watermark TTL by @BugenZhao in #24186
feat(streaming): add dev config materialize_force_overwrite_on_no_check and rename unsafe_enable_strict_consistency to unsafe_disable by @BugenZhao in #24443
chore(iceberg): upgrade iceberg rust by @chenzl25 in #24372
fix(optimizer): preserve eq key during rewrite in physical join plan nodes by @BugenZhao in #24442
chore(deps): Bump num-bigint-dig from 0.8.4 to 0.8.6 by @xiangjinwu in #24454
fix(docker): update docker image version and fix prometheus to scrape iceberg compactor by @chenzl25 in #24460
fix(streaming): fix some bugs for tables with watermark TTL & add tests for append-only table by @BugenZhao in #24459
fix(sqlparser): also quote keyword identifiers by @BugenZhao in #24461
feat(grafana): fix relative memory utilization panel by @kwannoel in #24445
fix(test): relax nexmark watermark lower bound in madsim scaling test by @BugenZhao in #24473
feat: make materialized view status visible by @Standing-Man in #24329
fix(stream): fix inconsistent global approx percentile initial state by @wenym1 in #24480
feat: add global shared object cache for iceberg table by @yezizp2012 in #24230
refactor(grafana): organize metrics by component by @kwannoel in #24490
fix(optimizer): align watermark propagation in StreamGroupTopN with executor by @BugenZhao in #24481
fix(optimizer): revert 23180 of dynamic filter watermark derivation by @chenzl25 in #24484
fix(storage): read version globally visible after initialized by @wenym1 in #24492
fix(stream): fix inconsistent global approx percentile initial state again by @wenym1 in #24500
feat(storage): handle uploader events in batch by @wenym1 in #24498
feat(license): skip rwu limit check for single-node deployments by @BugenZhao in #24495
feat(storage): support state clean for watermark in either pk or value column by @zwang28 in #24041
refactor: only handle partial graphs in local barrier worker by @wenym1 in #24239
chore(license): polish the license error message by @hzxa21 in #24479
chore: refine license check by @hzxa21 in #24514
feat(frontend): update IndexCatalog to support background ddl by @Standing-Man in #24349
refactor(grafana): move alerts and errors into separate section by @kwannoel in #24493
refactor(grafana): move relation and actor level metrics from cluster_essential to streaming section by @kwannoel in #24494
feat(pg-cdc): support geometry type for postgres cdc by @wcy-fdu in #24491
feat: batch insert fragments and fragment relations to minimize stream job preparation time by @yezizp2012 in #24475
feat(meta): persist is_serverless_backfill field in meta store table by @wenym1 in #24520
refactor(streaming): fix and refactor the watermark based state clean for band join by @yuhao-su in #24134
ci: upload redis cluster logs on failure & add sleep before running slt by @BugenZhao in #24496
refactor: introduce LogicalIcebergIntermediateScan to reduce iceberg list count by @silver-ymz in #24474
refactor(grafana): separate barrier metrics from relation metrics by @kwannoel in #24499
fix(cdc): don't commit empty offset by @zwang28 in #24529
fix(frontend): improve kill process by @chenzl25 in #24531
refactor(grafana): remove under-utilized metrics from cluster-essentials section by @kwannoel in #24501
fix(meta): allow handling drop command for empty database by @wenym1 in #24528
fix(frontend): fix incorrect row_number value with non-zero offset by @wenym1 in #24527
refactor(debezium): make debezium emit initial heartbeat by @zwang28 in #24523
refactor(grafana): simplify streaming panels by @kwannoel in #24502
fix(debezium): fix schema history record comparator by @zwang28 in #24482
fix: check connection existence for iceberg_engine_connection when setting by @yezizp2012 in #24521
chore: remove useless recursive cte code by @chenzl25 in #24519
feat(meta): support drop/cancel creating snapshot backfill job by @wenym1 in #24286
refactor(pg-cdc): change the automatically created publication to per source by @wcy-fdu in #24421
fix(test): fix flaky test introduced by postgres geometry type by @wcy-fdu in #24532
chore: generate grafana dashboard and add github action to check by @wenym1 in #24536
fix(meta): update cluster resource on exits of all worker node types by @BugenZhao in #24534
chore(deps): Bump rust_decimal from 1.39.0 to 1.40.0 by @dependabot[bot] in #24463
chore(deps): Bump async-nats from 0.45.0 to 0.46.0 by @dependabot[bot] in #24542
feat: Make VALIDATE clause optional when creating webhook connection by @cnsky1103 in #24109
fix(compaction): enable value column state cleaning for fast compaction by @zwang28 in #24543
feat(sink): support specifying ignore_delete for upsert sink type by @BugenZhao in #24311
refactor(cdc): refactor cdc source creation logic by @zwang28 in #24515
chore(deps): Bump ldap3 from 0.11.5 to 0.12.1 by @dependabot[bot] in #24130
refactor: extract clients and server impls of MonitorService & impl for frontend and compactors by @BugenZhao in #24276
fix: use the metadata of the streaming job table to construct consistent notification info by @yezizp2012 in #24525
feat(frontend): support sink backfill rate limit by @kwannoel in #24075
feat(meta): support recoverable backfill order control by @yezizp2012 in #24539
chore(deps): Bump rust-embed from 8.8.0 to 8.9.0 by @dependabot[bot] in #23644
feat: clean up table stats when truncate table by @Li0k in #24550
chore(deps): Bump the aws group across 1 directory with 5 updates by @dependabot[bot] in #24464
feat(storage): tracker per table imm memory by @wenym1 in #24538
feat(ci): generate grafana artifacts for cloud by @kwannoel in #24545
chore(deps): Bump tempfile from 3.19.1 to 3.20.0 by @dependabot[bot] in #21834
refactor(test): add tests for watermark based state cleaning by @zwang28 in #24549
chore(deps): Bump rust-embed from 8.8.0 to 8.11.0 by @dependabot[bot] in #24465
feat(snapshot-backfill): support alter rate limit for snapshot backfill by @wenym1 in #24302
chore(deps): Bump notify from 8.0.0 to 8.2.0 by @dependabot[bot] in #24129
test(cdc): wait for and verify data before schema change by @xiangjinwu in #24562
chore(deps): Bump bytes from 1.10.1 to 1.11.0 by @dependabot[bot] in #23770
chore(deps): Bump duration-str from 0.17.0 to 0.18.0 by @dependabot[bot] in #23968
feat(storage): enable watermark read for all WatermarkSerdeType by @zwang28 in #24563
chore(deps): Bump fancy-regex from 0.16.1 to 0.17.0 by @dependabot[bot] in #24126
chore(deps): Bump arc-swap from 1.7.1 to 1.8.0 by @dependabot[bot] in #24251
chore(deps): Bump crepe from 0.1.8 to 0.2.0 by @dependabot[bot] in #24132
chore(deps): Bump fs-err from 3.1.0 to 3.2.2 by @dependabot[bot] in #24257
chore(deps): Bump http from 1.3.1 to 1.4.0 by @dependabot[bot] in #23828
chore(deps): Bump rustls-pki-types from 1.13.0 to 1.14.0 by @dependabot[bot] in #24510
chore(deps): Bump quick-xml from 0.38.3 to 0.39.0 by @dependabot[bot] in #24432
chore(deps): Cleanup unused conditional deps (e.g. fiemap) by @dependabot[bot] in #23405
chore(deps): Bump pbjson from 0.8.0 to 0.9.0 by @dependabot[bot] in #24058
chore(deps): Bump pbjson-build from 0.8.0 to 0.9.0 by @dependabot[bot] in #24059
chore(deps): Bump lru from 0.16.2 to 0.16.3 by @dependabot[bot] in #24374
chore(deps): Bump aws-sdk-dynamodb from 1.96.0 to 1.98.0 by @dependabot[bot] in #24394
chore(deps): Bump typify from 0.4.3 to 0.5.0 by @dependabot[bot] in #23618
chore(deps): Bump console-subscriber from 0.4.1 to 0.5.0 by @dependabot[bot] in #24005
chore(deps): Bump const-str from 0.7.0 to 1.0.0 by @dependabot[bot] in #24314
chore(deps): Bump the aws group with 3 updates by @dependabot[bot] in #24570
chore(deps): Bump nix from 0.30.1 to 0.31.0 by @dependabot[bot] in #24572
chore(deps): Bump jsonwebtoken from 10.1.0 to 10.2.0 by @dependabot[bot] in #23718
chore: release binary without debug info to reduce size by @BugenZhao in #24565
feat: support alter function owner to by @Standing-Man in #24554
fix(mssql): correctly handle cases where database_name is omitted in create table by @wcy-fdu in #23739
fix(meta): handle post collect snapshot backfill job on collecting first job barrier by @wenym1 in #24359
fix(metrics): fix hummock meta miss metrics by @MrCroxx in #24435
fix(foyer): upgrade foyer verstion to fix hybrid miss metrics by @MrCroxx in #24577
fix(storage): fix incorrect min_epoch derived from MAX_EPOCH and ttl by @wenym1 in #24540
feat(storage): refine imm memory limiter management by @wenym1 in #24555
feat(meta): enable profiling in MonitorService for meta node by @BugenZhao in #24566
chore(deps): Bump adbc_snowflake from 0.21.0 to 0.22.0 by @dependabot[bot] in #24410
feat(storage): support state table iter respecting watermark by @zwang28 in #24567
feat: support ALTER SECRET OWNER TO by @Standing-Man in #24548
fix(stream): refine locality backfill progress by @chenzl25 in #24556
chore: remove trailing whitespace from filenames by @Standing-Man in #24585
feat(expr): implement gamma/lgamma function by @Standing-Man in #23556
feat(iceberg): add Google authentication support for Iceberg REST catalog by @tuantran0910 in #24457
fix(sink): fix snowfalke starrocks redshift executor_id to uuid by @xxhZs in #24213
fix(iceberg): Iceberg JDBC Catalog using aws assume role for s3 access by @BaudoinWR in #24573
feat(sink): add intermediate.schema.name for redshift sink by @xxhZs in #24223
feat(source): allow subset of struct fields for file source of parquet encoding by @wcy-fdu in #24530
fix(meta): tolerate partial await_tree failures by @BugenZhao in #24587
chore: avoid unnecessary returning of entities when performing insert and update operations on the metastore by @yezizp2012 in #24557
fix: remove non-deterministic background enabling for ddl tests by @yezizp2012 in #24588
feat(meta): allow handling barrier manager request during recovery by @wenym1 in #24592
fix(meta): support recoverable backfill order control for snapshot backfill by @wenym1 in #24590
fix: correctly handle vis map in hash dispatcher after noop elimination by @hzxa21 in #24580
fix: union all will panic because a hidden column by @yihong0618 in #24448
chore(deps): Bump async-openai from 0.30.1 to 0.32.3 by @dependabot[bot] in #24430
ci(license): reduce hawkeye false positive due to implicit merge by @xiangjinwu in #24589
fix(compute): avoid panic in GetStreamingStats by @BugenZhao in #24604
feat(meta): pause time travel vacuum when having snapshot backfilling jobs by @wenym1 in #24602
refactor(config): ser/de None value as empty string by @BugenZhao in #23931
feat: add adaptive parallelism strategies for streaming jobs by @shanicky in #24250
chore(deps): Bump jsonwebtoken from 10.2.0 to 10.3.0 by @dependabot[bot] in #24619
fix(frontend): increase default query log truncate len from 1024 to 65536 by @chenzl25 in #24622
refactor: refactor iceberg compaction trigger condition and fix snapshot count by @Li0k in #24594
feat: introduce force resume backfill node/job interface in risectl by @yezizp2012 in #24551
chore: more robust postgres geometry type test by @wcy-fdu in #24595
feat(metric): add relation type to relation infos by @yezizp2012 in #24625
feat(meta): add conifguration option and improve observability for time travel. by @zwang28 in #24626
feat(iceberg): enforce merge-on-read for append-only tables by @Copilot in #24487
feat(grafana): add overview tables for relation time breakdowns by @kwannoel in #24544
fix(ci): replace legacy codecov uploader with new cli by @BugenZhao in #24593
fix(optimizer): avoid panic rewriting streaming limit on empty schema by @BugenZhao in #24631
fix(session): require explicit unit when setting statement_timeout by @xiangjinwu in #24633
fix: ensure creating job info is cleaned after creation or cancellation by @yezizp2012 in #24609
fix(metrics): fix sink commit metrics time unit by @wcy-fdu in #24632
feat(storage): periodically rebuild snapshot iter stream by @wenym1 in #24608
refactor(sink): Flush pending epochs when sink schema change by @Chiro11 in #24277
feat: enable snapshot backfill by default by @wenym1 in #24116
fix(source): pad NULL on type mismatch by @BugenZhao in #24607
perf: avoid fetching stream nodes in system catalogs by @yezizp2012 in #24641
feat(meta): render after reload runtime info and isolate database render failure by @wenym1 in #24637
fix: invalid iceberg scan statistic in datafusion by @silver-ymz in #24650
fix: Java Integration test cannot allocate memory in static TLS block by @tabVersion in #24652
fix: remove adjacent update check by @chenzl25 in #24645
fix: role check for admin database access by @cnsky1103 in #24383
feat(metrics): add backfill metadata metrics by @kwannoel in #24576
feat(iceberg): support iceberg v3 delete vector (read and write) by @chenzl25 in #24470
chore: chore works for release-2.8 by @wenym1 in #24653
fix(stream): fix eliminate no op by @chenzl25 in #24659
refactor(meta): do not persist backfill order of locality provider by @wenym1 in #24661
feat: mapping more datafusion native functions by @silver-ymz in #24638
fix: extract ProfileServiceImpl to heap_profile crate to avoid jemalloc TLS in JNI by @BugenZhao in #24658
fix(ci): avoid shaded gRPC classpath conflict in connector test by @BugenZhao in #24623
feat(sink): support copy into wihout barrier for redshift by @xxhZs in #23086
feat: iceberg cleanup expired files by @Li0k in #24660
chore: update version to v2.8.0-rc.1 by @cyliu0 in #24725
fix(snapshot-backfill): fix deadlock caused by next_epoch (#24693) by @risingwave-ci in #24738
feat(frontend): support enable_serverless_backfill (#24686) by @risingwave-ci in #24747
feat(optimizer): improve locality backfill for agg (#24710) by @risingwave-ci in #24779
feat: add a config to control whether state table vnode prunning is enabled (#24703) by @risingwave-ci in #24799
feat(refill): support meta cache refill concurrency limit (#24787) by @risingwave-ci in #24789
feat(connector): add configurable write-parquet-properties for Iceberg sink and compaction (#24612) by @risingwave-ci in #24862
fix(optimizer): move locality enforcement to logical_rewrite_for_stream (#24850) by @risingwave-ci in #24874
fix(iceberg): fix staging pk in iceberg refreshable table (#24872) by @risingwave-ci in #24876
fix(iceberg): upgrade iceberg dependencies (#24873) by @risingwave-ci in #24879
feat(license): maintain rwu limit check for single-node deployments (#24796) by @risingwave-ci in #24845
feat(sink): support alter jdbc sink url , username and password (#24846) by @risingwave-ci in #24865
fix(meta): fix compaction group merge scheduling efficiency (#24757) by @risingwave-ci in #24803
refactor: improve compaction schedule (#24733) by @risingwave-ci in #24807
feat: jdbc sink alter-on-fly fields (#24846) (#24861) by @tabVersion in #24884
fix(iceberg): fix flaky test t-refresh-composite-pk (#24888) by @risingwave-ci in #24892
fix: update broken documentation URLs (#24897) by @risingwave-ci in #24902
feat(sink): add keeplive with option for pg rust sink and jdbc sink (#24388) by @risingwave-ci in #24905
fix(storage): fix last_serde for NonPkPrefixSkipWatermarkState and ValueSkipWatermarkState (#24890) by @risingwave-ci in #24894
fix(meta): fence drop vs split updates and skip stale fragment splits (#24887) by @risingwave-ci in #24910
chore: update version to v2.8.0 by @cyliu0 in #24916

New Contributors

@cnsky1103 made their first contribution in #24109
@tuantran0910 made their first contribution in #24457
@BaudoinWR made their first contribution in #24573

Full Changelog: v2.7.2...v2.8.0