risingwavelabs/risingwave v0.1.13 on GitHub

For installation and running instructions, see Get started.

Main changes

SQL features

SQL commands:
- Improves the formatting of response messages of EXPLAIN statements. #5541
SQL functions:
- to_char() now supports specifying output format in lowercase. #5032
  to_char(timestamp '2006-01-02 15:04:05', 'yyyy-mm-dd hh24:mi:ss') → 2006-01-02 15:04:05
- generate_series now supports negative steps. #5231
```
SELECT * FROM generate_series(5,1,-2);
generate_series 
-----------------
               5
               3
               1
(3 rows)
```
- Adds support for sum/min/max functions over interval-type data. #5105, #5549
- Adds support for array concatenation. #5060, #5345
- Adds support for specifying empty arrays. #5402
- Casting from array to varchar is now supported. #5081
  array[1,2]::varchar → {1,2}
- Casting from varchar to integer allows leading and trailing spaces. #5452
  ' 1 '::int → 1
Adds new system catalog and psql meta-commands. #5127, #5742
- \d: Lists all relations in the current database. (Materialized) sources are not supported yet.
- \dt: Lists all tables in the current database.
- \dm: Lists all materialized views in the current database.
- \di: Lists all indexes in the current database.
- pg_catalog.pg_index: Contains information about indexes.

Connectors

Nested columns are now supported for the datagen connector. #5550

What's Changed

chore(docs): update the binary version with new tag by @huangjw806 in #4297
feat(frontend): support user alter by @cnissnzg in #4261
feat(meta): pause concurrent checkpoint by @BugenZhao in #4269
build(toolchain): bump to 20220729 by @TennyZhuang in #4292
fix: add retry in wait_epoch by @hzxa21 in #4287
feat(frontend): Add pg_catalog.pg_class by @Graphcalibur in #4293
feat(sqlsmith): ensure panicked queries are reproducible by @kwannoel in #4228
fix(binder): Fix ambiguous column name error message by @Graphcalibur in #4282
chore(planner test): upgrade serde_yaml to 0.9 & remove yq by @xxchan in #4307
fix(frontend): fix display pk of stream table scan by @chenzl25 in #4279
refactor(meta): provide SST id to worker id mapping in commit_epoch. by @zwang28 in #4285
feat(batch): support delimiter argument in batch string_agg by @richardchien in #4298
feat(object store): add list method by @zwang28 in #4257
chore(streaming): remove table id in arrange node by @wcy-fdu in #4281
fix: refine error prompting by @neverchanje in #4227
feat(S3_bench): Add s3_bench by @xxhZs in #4299
feat(metrics): add metrics for barrier latency at each stage by @xxhZs in #3965
feat(sqlsmith): enable sub-query generation by @marvenlee2486 in #4216
fix: Messages should be skipped after parsing failure by @shanicky in #4317
feat(stream): support delim argument in streaming string_agg by @richardchien in #4321
feat(sqlsmith): print e2e errors by @kwannoel in #4310
fix(optimizer): ApplyValues. by @likg227 in #4323
feat(frontend): expose Vnode function as __rw_vnode for easy debugging by @richardchien in #4327
fix(frontend): only schedule dml to cn which contain table source writer by @yezizp2012 in #4329
feat(risedev): support 3 compute nodes in playground by @BugenZhao in #4332
fix(compaction): Skip compaction task assignment check for trivial move tasks by @hzxa21 in #4333
refactor(batch): refactor hash join by @wzzzzd in #4017
fix(risedev): cannot d after adding shared in-memory hummock support by @BugenZhao in #4353
refactor(frontend): rename __rw_vnode to rw_vnode by @richardchien in #4342
feat:binary format serializer by @ZENOTME in #4284
chore: add metric for sink executor by @tabVersion in #4337
fix: use correct column indices for chain & batch lookup join by @BugenZhao in #4346
refactor(storage): get new SST id in batch by @zwang28 in #4308
feat(compaction): optimize iterator and sstable builder for compaction speed by @Little-Wallace in #4034
feat(sqlsmith): test stream queries by @kwannoel in #4361
feat(frontend): support acl column in pg_namespace by @cnissnzg in #4326
feat(planner): Use explain verbose for all planner tests by @jon-chuang in #4359
feat(batch): apply row-based encoding for materialize executor and batch mode by @wcy-fdu in #4335
fix(batch): Stop polling after data stream returns None by @liurenjie1024 in #4371
refactor(streaming): apply row-based encoding for append_only top_n by @wcy-fdu in #4373
refactor(batch): unify scan type in batch scan by @wcy-fdu in #4375
fix(storage): fix manual_compaction for l0 sub_compaction by @Li0k in #4334
fix(common): use smaller upperbound for generators to avoid range overflow by @wzzzzd in #4377
feat(meta): support manual compaction for specific sst ids from CN by @soundOfDestiny in #4338
feat(storage): provide prefix_iter for prefix_bloom_filter by @Li0k in #4316
feat(storage): introduce filter_key_extractor_manager by @Li0k in #4347
refactor(frontend): simplify ConditionDisplay by @xxchan in #4382
feat(compaction): collect remote io time for every compaction task by @xinchengxx in #4320
refactor(batch): remove append_chunk in nested loop join and hash join by @wzzzzd in #4381
feat(planner): Bind Natural Join by @jon-chuang in #4349
feat(optimizer): support 2-phase streaming agg by @richardchien in #4341
fix(parser): parse negative numbers by @kwannoel in #4356
fix: add pair number check in startup msg by @yezizp2012 in #4393
feat: generator row id using vnode in source executor by @yezizp2012 in #4362
chore(developer-docs): add note on compilation bottlenecks by @kwannoel in #4395
refactor(lru): use on_release event listener to capture owned key/value when entries evicted or erased by @MrCroxx in #4396
feat(optimizer): Implement ProjectJoin rule by @jon-chuang in #4385
refactor(meta): eliminate usages of RwError by @HuaHuaY in #4364
feat: support current_schema and session_user by @neverchanje in #4358
fix: not report error when encounter UnexpectedEof by @ZENOTME in #4378
feat(compaction): support sub-level in L0 by @Little-Wallace in #3111
feat(planner): general USING/NATURAL column reference disambiguation by @jon-chuang in #4391
feat: kafka sink e2e test by @tabVersion in #4402
refactor(expression): make building expression from prost consistent in style by @Gun9niR in #4411
chore(sqlsmith): use updated binder interface by @kwannoel in #4408
feat(sqlsmith): generate explicit type castings by @kwannoel in #4419
feat(sqlsmith): Enable gen_agg but workaround distinct agg by @marvenlee2486 in #4421
fix(binder): allow Clone for expr::Subquery as part of CTE by @xiangjinwu in #4424
fix(frontend): make WITH optional for user create and alter by @yezizp2012 in #4414
fix(frontend): fix table option for internal table by @Li0k in #4416
fix: use SomeShard as the distribution of batch scan by @xxchan in #4420
feat: support explain sink by @tabVersion in #4430
chore: update comments for dispatcher by @xxchan in #4413
chore(ci): remove repeated ci steps by @huangjw806 in #4423
chore(risedev): unify clippy between risedev c and pre-unit-test.sh by @Li0k in #4436
fix: fix observer version check and add sync in compute/compactor node by @yezizp2012 in #4439
feat(test): make simulation test deterministic! by @wangrunji0408 in #4336
feat(cache): introduce tiered cache abstraction by @MrCroxx in #4406
chore: support more query statement in extended query mode by @ZENOTME in #4441
fix(ci): fix main ci build timeout by @huangjw806 in #4444
refactor(common): add rpc client pool by @zwang28 in #4410
fix(cache): clean pending request immediately when future cancel by @Little-Wallace in #4422
fix: build on non-linux target by @MrCroxx in #4448
feat(meta): piggyback extra info in heartbeat RPC. by @zwang28 in #4435
fix(planner): Pullup predicates in LogicalScan into BatchLookupJoin by @jon-chuang in #4453
feat(meta): add source info and stream source split info in get_clust… by @KeXiangWang in #4277
fix(planner): DynamicFilter's LHS should follow upstream distribution by @jon-chuang in #4452
feat(meta): meta push initial hummock version into CN by @soundOfDestiny in #4459
feat(frontend): add CREATEUSER and NOCREATEUSER option for user create/alter by @cnissnzg in #4447
feat(optimizer): support stateless 2-phase agg optimization for append-only min/max by @richardchien in #4433
fix(meta): the ddl lock is not really locked by @KeXiangWang in #4461
refactor(row-serde): make row-serde directory clearer and update comments by @wcy-fdu in #4443
fix(planner): Tumble can accept CTE as input by @jon-chuang in #4450
feat(sqlsmith): generate extreme values by @kwannoel in #4345
refactor(sink): rename sink properties by @tabVersion in #4465
fix(planner): Disallow select * with no tables by @jon-chuang in #4451
fix(worker-node-mgr): worker node manager should return error if no workers found by @BowenXiao1999 in #4466
fix(test): fix or ignore several unit tests on madsim by @wangrunji0408 in #4460
fix(planner): Correctly Alias CTE and columns by @jon-chuang in #4432
feat(frontend): push the calculation of inputs of join's condition to its two sides. by @likg227 in #4276
fix: ignore epoch check with stop mutation, this might break some scenarios in failover by @yezizp2012 in #4446
chore(risedev): avoid using ambiguous "playground" term with risedev d by @BugenZhao in #4475
chore(frontend): tweak visibility & remove dead code & fix clippy by @xxchan in #4472
fix(frontend): display InputRef index when column name is empty in explain strings by @richardchien in #4445
feat(sqllogictest): sync changes from 0.6 by @skyzh in #4478
doc: fix markdown toc by @wanglei4687 in #4477
feat(storage): track GC watermark SST id. by @zwang28 in #4369
fix(risedev): check clippy should report warning by @skyzh in #4479
fix(storage): batch qurry with_table_option by @Li0k in #4464
fix(optimizer): push Apply down scalar agg. by @likg227 in #4386
feat(storage): GC orphan SST in object store. by @zwang28 in #4370
feat(test): make deterministic test ready by @wangrunji0408 in #4476
chore(grafana): clean up metric by @zwang28 in #4482
fix(optimizer): fix wrong project when output indices having dup keys by @st1page in #4481
refactor(frontend,meta): refactor index catalog by @chenzl25 in #4340
refactor(meta): remove force rewrite of join -> delta by @skyzh in #4480
feat(batch): add more e2e tests for batch join executors by @wzzzzd in #4300
fix(binder): reject unsupported not null rather than ignoring silently by @xiangjinwu in #4487
feat(batch-task): init task status management by @BowenXiao1999 in #4437
test(batch): support generating data of more types in test_utils by @wzzzzd in #4490
fix(test): fix pg_class e2e test by @zwang28 in #4488
feat(fronted): implement __rw_table() to support query on internal table by @cnissnzg in #4412
feat(common): introduce async stack trace by @BugenZhao in #4383
fix: allow create source with row format debezium_json by @tabVersion in #4494
fix(streaming): no extra buffer for remote exchange by @BugenZhao in #4495
feat(stream): support cache for group topN by @shmiwy in #4471
refactor(meta): split HummockManager impl into several files. by @zwang28 in #4497
feat(optimizer): add functional dependency support by @Enter-tainer in #4201
chore(ci): use customized profile for pull requests by @BugenZhao in #4502
refactor(meta): add StreamClient struct by @HuaHuaY in #4485
feat(expr): cast time to interval for cross-type comparison by @xiangjinwu in #4291
fix(meta): fix internal_table notification for mv and snapshot by @Li0k in #4467
feat(storage): integrate tiered cache system by @MrCroxx in #4463
fix(batch): consume dml status and ensure task id always found by @BowenXiao1999 in #4516
fix: drop internal tables when drop table/index by @yezizp2012 in #4499
fix: set http2 connection windows size to unlimited by @yuhao-su in #4505
feat(CI):add extended_query e2e test in CI by @ZENOTME in #4491
refactor(encoding): remove cell-based encoding by @wcy-fdu in #4523
refactor(test): remove stream proto test case by @ice1000 in #4519
fix(meta): fix stop all actors step of recovery by @cnissnzg in #4525
feat(sink): add debezium update in KafkaSink by @cnissnzg in #4486
feat(common): support serializing struct and list with array index to memcomparable format by @wzzzzd in #4524
fix(compactor): fix estimate memory size for compression by @Little-Wallace in #4484
feat(storage): Modify the locks on shared buffer and local version by @xxhZs in #4536
feat: support show sink by @tabVersion in #4531
feat(expr): support assignment casts for temporal types by @xiangjinwu in #4526
feat(ci): add sqlsmith e2e to ci by @kwannoel in #4319
chore(sqlsmith): use single-thread tokio runtime by @skyzh in #4348
fix(planner): handle invalid hop arguments by @kwannoel in #4537
refactor(fragmenter): delete unneeded self. in fragmenter by @ice1000 in #4544
fix(meta): LSM/SST metrics is stale when there is no alive compactor by @KivenChen in #4449
fix(storage): remove timeout when get compaction group id by @zwang28 in #4542
feat(stream key): rename logical_pk && move add count for stream agg in to stream by @st1page in #4539
fix(meta): fix catalog notification by @Li0k in #4530
feat(connector): better names for connector properties by @TennyZhuang in #4503
feat(expr): support casting temporal types to string by @xiangjinwu in #4538
fix: file cache append index out of range by @MrCroxx in #4552
feat(compactor): split code to more files to make code more readable by @Little-Wallace in #4551
refactor(meta): merge UserManager, CatalogManager and FragmentManager(Part 1) by @HuaHuaY in #4541
chore: remove debug context and allow to check current context with async stack trace by @BugenZhao in #4535
fix(frontend): fix lookup join predicate by @chenzl25 in #4563
fix(test): fix deterministic test by @wangrunji0408 in #4550
refactor: propose the API semantics of abort_task by @BowenXiao1999 in #4560
feat(meta): migration by default when recovery enabled in meta by @cnissnzg in #4568
fix(expr): reject flag for regexp_match(es) by @xxchan in #4546
refactor(storage): refactor row_serde and storage table to match row-based encoding by @wcy-fdu in #4528
fix: stack unwind when profiling by @fuyufjh in #4574
feat(optimizer): covering index selection by @chenzl25 in #4520
feat(sqlsmith): generate ternary and variadic functions by @kwannoel in #4572
feat(compactor): separate compactor options for flush and compact by @Little-Wallace in #4570
refactor(stream): extract iter_state_table function by @richardchien in #4583
feat(common): support all data types in sort_util by @richardchien in #4582
refactor(stream): remove SourceError by @guzzit in #3781
refactor(stream,agg): add StateTableColumnMapping to convert column indices by @richardchien in #4584
fix(common): fix wrong memcomparable encoding for struct/list/interval by @wzzzzd in #4575
feat(optimizer): support non-covering index selection by @chenzl25 in #4587
feat(batch): abort task when failed by @BowenXiao1999 in #4578
refactor(stream): remove useless state_table_to_upstream method by @richardchien in #4596
fix: use UpstreamHashShard distribution for table scan by @xxchan in #4534
feat(runtime): replace tokio-tracing with minitrace by @skyzh in #4576
fix(sink): resolve schema mismatch due to row_id column by @nanderstabel in #4304
fix(test): change name in Sink with properties by @nanderstabel in #4604
refactor(expr): support string-to-interval casting by @xiangjinwu in #4595
chore(runtime): remove jaeger support by @skyzh in #4606
refactor(batch): rewrite OrderByExecutor by @wzzzzd in #4500
fix(common): justify intervals before hashing by @wzzzzd in #4600
refactor(connector): refine code style & use seperate threads for data generation sources by @BugenZhao in #4564
fix(ci): make cron e2e script executable by @kwannoel in #4608
chore(metrics): add input_buffer_blocking_duration metrics by @KeXiangWang in #4585
fix(index): create index name to lowercase by @st1page in #4613
feat: decouple vnode mapping and catalog by @yezizp2012 in #4594
fix(common): fix memcomparable encoding for list by @wzzzzd in #4599
refactor(stream,agg): refactor GenericExtremeState to get rid of hardcoded assumption on column arrangements by @richardchien in #4592
fix(dashboard): fix outdated proto & add new operators by @xxchan in #4602
feat(optimizer): pk minimization for join and hop window using functional dependency by @Enter-tainer in #4607
refactor(state_store): use binary-search for get prune sst by @xinchengxx in #4573
fix(ci): add ci-dev profile flags to main-cron config by @kwannoel in #4624
feat(stream): infer group topn's state table catalog in frontend. by @shmiwy in #4597
fix(parser): fix parsing right angle brackets by @xxchan in #4627
fix: bitmap inconsistent encoding by @xxchan in #4603
refactor: simplify the task status and stage state by @BowenXiao1999 in #4629
fix(handler): handle quoted names in drop user by @xiangjinwu in #4628
refactor(streaming): encode row in hash join operator cache by @yuhao-su in #4630
feat(storage): introduce filter_key_extractor for sst builder by @Li0k in #4394
chore(pgwire):remove unnecessary infer by @ZENOTME in #4632
fix(streaming): fix topn cache for recovery by @chenzl25 in #4631
feat: async stack trace middleware for compute-node gRPC by @BugenZhao in #4581
refactor(meta): refactor compaction task sending and cancellation. by @zwang28 in #4620
feat(object_store): implement streaming upload for s3 and in mem object store by @Gun9niR in #4589
fix(config): fix datestyle parameter by @xxchan in #4506
feat(state_store): use binary-search for get prune sst by @xinchengxx in #4622
fix(optimizer): fix logical scan distribution key by @chenzl25 in #4636
test(optimizer):add test for distributed plan by @st1page in #4638
fix(meta): Stale SST Metrics without Alive Compactors #4380 by @KivenChen in #4571
fix(meta): limit compactor's concurrent task number by @zwang28 in #4633
refactor: use macro to iterate fields in struct or elements in list by @neverchanje in #4649
refactor: use ellipsis to skip details of Struct by @neverchanje in #4648
fix(binder): cte name assert use the real name by @st1page in #4655
feat(sstable_store): implement streaming upload for sstable store by @Gun9niR in #4644
doc: improve comments of index_selection.rs by @fuyufjh in #4653
refactor(meta): use oneof for pin_version response by @wenym1 in #4656
test(nexmark): add nexmark q17 e2e test by @TennyZhuang in #4660
refactor(frontend,meta): don't infer and create state tables for (stateless) local simple agg by @richardchien in #4661
feat(planner): Support nested joins by @jon-chuang in #4401
feat(frontend): meta push max_committed_epoch to frontend by @Gun9niR in #4651
fix(stream,agg): fix bug of cache maintenance of extreme state by @richardchien in #4659
refactor(batch): use memcomparable encoding to accelerate top-n executor by @wzzzzd in #4642
feat(batch): support lookup join running in distributed mode by @chenzl25 in #4647
fix(protobuf): store offset and limit in uint64 in TopNNode and LimitNode by @wzzzzd in #4666
refactor: value encoding error by @fuyufjh in #4671
feat(storage): introduce streaming_table write and point-get by @wcy-fdu in #4588
refactor(storage): remove common stuff in streaming table by @wcy-fdu in #4679
refactor(expr): impl Copy for AggKind to get rid of some clone by @richardchien in #4680
feat(batch):add metrics for batch exchange executor by @ZENOTME in #4577
feat(storage): move sync_uncommitted tasks and ssts from the shared buffer to the local version by @xxhZs in #4492
fix: minstant elapsed do not panic on time goes backward by @BugenZhao in #4668
fix(binder): is_correlated for intermediate subquery by @xiangjinwu in #4677
feat(compaction): limit memory usage for compaction read by @Little-Wallace in #4590
feat(test): parallel e2e test under deterministic simulation by @KveinAxel in #4675
feat(streaming): vnode bitmap updates for state table by @BugenZhao in #4621
test: add tests for list by @neverchanje in #4689
refactor: refine error message of type casting failure by @neverchanje in #4690
refactor(stream,agg): pass all columns from stream chunk to managed state by @richardchien in #4676
feat(risedev): add slt-batch/streaming/all by @xxchan in #4683
fix(optimizer): fix some tests for distributed plan by @xxchan in #4686
chore(storage): improve metrics for bloom filter by @Li0k in #4664
fix(binder): correct the table name and use alias for table function by @richardchien in #4698
feat: use risectl to get flamegraph of compute node by @yuhao-su in #4695
feat(optimizer): not use 2 phase for stream agg with group key by @st1page in #4684
feat(ctl): add sst gc command by @zwang28 in #4674
feat(stream): tolerate compute error by default by @xxchan in #4652
chore: replace ensure then remove with try_into by @BugenZhao in #4702
refactor: replace Epoch in storage with HummockEpoch by @Gun9niR in #4699
chore: fix dashboard worker node info by @yezizp2012 in #4711
chore(risedev): add a message for saved profiling result by @xxchan in #4712
feat(meta): Enable deterministic unit test manager/cluster/tests/test_heartbeat via #[cfg(madsim)] by @jon-chuang in #4697
feat(meta): meta push version deltas to CN by @soundOfDestiny in #4473
fix(binder): disallow subquery in BoundValues. by @likg227 in #4579
refactor(stream): move actor_id into ctx by @xxchan in #4707
feat(expr): directly evaluate on row for macro-generated expressions by @BugenZhao in #4667
feat(batch): batch dml support index lookup by @chenzl25 in #4701
feat(frontend): add support for array_agg in frontend by @richardchien in #4717
refactor(encoding): row deserialize only handle value by @wcy-fdu in #4704
fix(frontend): reject invalid string_agg calls in frontend by @richardchien in #4724
feat(sqlsmith): support generating string_agg calls by @richardchien in #4725
test: add test for Date type by @nanderstabel in #4727
fix(meta): fix trigger_manual_compaction incorrect table_id by @Li0k in #4714
feat: update row-id generator when vnode mapping changed by @yezizp2012 in #4716
feat(storage): filter out SST by watermark in GC. by @zwang28 in #4512
fix(metrics): fix iterator metrics data report duplication and lost by @MrCroxx in #4723
refactor(stream): remove unnecessary Arc<Mutex> in rearranged_chain by @TennyZhuang in #4731
feat(binder): resolve primary key constraint and pass it in StreamSourceInfo.pk_column_ids by @xiangjinwu in #4722
refactor(stream_key): rename pk_indices in stream plan to stream_key by @st1page in #4733
refactor(table): rename pk into stream key by @st1page in #4732
fix(opt): dist of batch expand by @st1page in #4718
ci(test): add deterministic parallel e2e test by @wangrunji0408 in #4738
feat(storage): Upload all data to shared storage as a sstable by @xxhZs in #4442
test(batch): add more e2e tests for OrderBy/TopN/Limit by @wzzzzd in #4744
chore(deps): bump dependencies by @BugenZhao in #4749
feat: set the field names in DataType::Struct by @neverchanje in #4669
feat(compaction): support compact several files for overlap-strategy by @Little-Wallace in #4619
chore: migrate logging from log to tracing by @wangrunji0408 in #4745
fix(storage): fix incorrect lock order by @zwang28 in #4753
chore(parser): remove todo for parsing nested negative expressions by @kwannoel in #4758
fix(binder): fix panic when index on non composite type by @nanderstabel in #4763
feat(stream): new boxed group_top_n executor from proto by @shmiwy in #4682
feat(batch): prepare for union operator by @chenzl25 in #4761
chore: fix typos & add typo checker to ci by @xxchan in #4767
refactor(meta): indicate worker expircy in status code. by @zwang28 in #4739
chore: improve some error message display by @xxchan in #4770
fix: a potential conflict of task output channel in lookup join by @BowenXiao1999 in #4746
chore: remove unnecessary macro for hummock matrics by @MrCroxx in #4772
feat(meta): release privileges when dropping catalog by @HuaHuaY in #4742
chore(deps): bump pulsar-rs to remove prost 0.10 dep by @BugenZhao in #4771
feat(metrics): show compaction result by compactor-node by @Little-Wallace in #4775
refactor(storage): remove shared buffer when an epoch is being synced by @wenym1 in #4773
fix(risedev): replace shell with bash by @yuhao-su in #4783
feat(tracing): print slow request with minitrace by @TennyZhuang in #4778
chore(streaming): use separate runtime for actors by @BugenZhao in #4756
refactor(storage): relax assumption in sst id tracker by @zwang28 in #4548
feat: enforce chain parallelism and fill proper upstream with same vnode range by @yezizp2012 in #4740
refactor(batch): Make local mode default by @liurenjie1024 in #4789
fix(dashboard): add expand for visualization by @TennyZhuang in #4791
fix: cargo run playground by @fuyufjh in #4792
test: add nexmark q15 into e2e tests by @fuyufjh in #4793
fix(optimizer): fix logical scan clone with predicate by @chenzl25 in #4796
feat(ci): randomly pick from multiple frontends in parallel e2e by @BugenZhao in #4795
fix(compactor): fix drop mv stuck by @Li0k in #4798
chore(source): add more field in nexmark by @KeXiangWang in #4802
feat(risedev): add a command to open rust docs by @xxchan in #4794
fix(frontend): Change the column name in the table pg_class to be compatible with PostgreSQL by @HuaHuaY in #4809
feat(frontend): support casting between arrays by @wzzzzd in #4741
feat(plan-test): suggest run-planner-test instead of using panic hook by @xxchan in #4800
chore: extract max cache size as const by @fuyufjh in #4812
fix(sqlsmith): use distributed execution mode to avoid overflows by @kwannoel in #4806
refactor(batch,agg): get rid of two slightly different output methods by @richardchien in #4816
fix(meta): fix source notification by @Li0k in #4818
chore(risedev): use command line argument for async stack trace by @BugenZhao in #4808
feat(metrics): introduce tiered cache metrics by @MrCroxx in #4786
fix(local_version): fix deadlock bug by @Little-Wallace in #4810
feat(storage): introduce streaming table iter by @wcy-fdu in #4787
test: add nexmark q20 by @fuyufjh in #4804
fix: make debezium work again by @tabVersion in #4781
refactor(frontend): check the validity of ApproxCountDistinct and Count in frontend by @richardchien in #4825
feat(optimizer): support index merge optimization by @chenzl25 in #4820
feat(metrics): add media type label to object store latency metric by @StrikeW in #4713
feat(storage): Initial interfaces for shared buffer redesign by @hzxa21 in #4780
refactor(batch,agg): improve creation of agg states by @richardchien in #4824
fix(optimizer): fix the bug of selective distinct aggregates. by @likg227 in #4673
fix(metrics): fix tiered cache metrics p99 tag by @MrCroxx in #4846
fix(binder): depth for has_correlated_input_ref is on Query rather than Select by @xiangjinwu in #4840
feat(storage): add sanity check in streaming table and fix range of MemTable by @wcy-fdu in #4832
feat(compactor): cancel compact task when no idle compactor by @Little-Wallace in #4830
chore(planner, sqlsmith): bump libtest-mimic by @kwannoel in #4837
chore: migrate issue template to issue forms by @xxchan in #4859
refactor(storage): apply streaming table for all executors and remove origin StateTable and MemTable by @wcy-fdu in #4854
chore(ci): introduce ci-release profile for main CI by @BugenZhao in #4863
refactor(frontend): refine the doc & impl for batch traits by @xxchan in #4826
feat(planner_test): print better debugging message for panics by @kwannoel in #4867
feat(batch): add task level metrics delete by @ZENOTME in #4757
fix(sqlsmith): disable single value by @kwannoel in #4865
refactor(binder): consolidate expr iterators on Select and Values by @xiangjinwu in #4853
feat(test): test fault-recovery on madsim by @wangrunji0408 in #4843
fix(storage): fix GC watermark id by @zwang28 in #4839
test: add e2e test for debezium cdc source by @tabVersion in #4835
fix(risedev): provide client-address for compactor by @zwang28 in #4872
fix(barrier): Fix bug about barrier collect by @xxhZs in #4813
feat(frontend): support primary key of tables by @wzzzzd in #4852
feat(batch,bench): add microbench for batch hash agg by @richardchien in #4838
chore: rename source ci following new naming by @tabVersion in #4874
feat: support heap profiling from risedev by @fuyufjh in #4871
refactor(streaming): refactor look up executor to use streaming table by @wcy-fdu in #4879
fix: use offsets in ListRef::value_at() by @nanderstabel in #4831
feat(compaction): refactor compactor executor to reduce thread context switch by @Little-Wallace in #4750
feat(test): support sync point for integration test by @zwang28 in #4855
feat(batch,agg): support array_agg for batch mode by @richardchien in #4862
feat(batch): monitor task pending and execution time by @wzzzzd in #4799
feat(optimizer): improve index range cost by @chenzl25 in #4877
fix(compactor): fix version get for trivial_move by @Li0k in #4890
feat: add metrics of stream join cached rows & entries by @fuyufjh in #4891
chore(expr): cleanup all RwErrors by @BugenZhao in #4873
feat(stream,agg): support array_agg for streaming mode by @richardchien in #4895
feat(parser): parse explain by @st1page in #4730
feat(optimizer): support or condition for scan range by @chenzl25 in #4851
fix(sstable): use accurate uncompressed size to avoid grow up vector by @Little-Wallace in #4878
chore: print warning message when client address is not specified by @Gun9niR in #4876
feat(storage): add current_epoch to support decoupling by @xxhZs in #4760
chore: bump madsim to support coarse clock by @BugenZhao in #4901
chore(ci): use one session for each file in serial mode e2e tests by @xxchan in #4902
fix(opt): fix rewrite_provided_distribution by @st1page in #4904
feat(optimizer): simplify expand by @likg227 in #4591
fix(source): introduce state_table_id as state store prefix by @tabVersion in #4897
feat: Make tpch distributed explicitly by @liurenjie1024 in #4905
feat: Storage: Support batch delete in object store by @ALeitert in #4823
refactor(frontend): generate meaningful identify for PlanNode by @ZENOTME in #4889
refactor: simplify error handling of sink by @neverchanje in #4848
refactor: rename DebugExecutor to WrapperExecutor by @fuyufjh in #4899
feat(test): use correct primary key for tpch & nexmark ddl by @wzzzzd in #4882
feat: do not update downstream removed actors' merge node in case of cascading migration by @shanicky in #4925
fix(frontend): fix subquery depth matching by @chenzl25 in #4888
fix: array access for multidimensional array by @nanderstabel in #4928
fix(streaming): rewrite update op to insert/delete for ProjectSet by @xxchan in #4931
fix(optimizer): fix index scan output schema by @chenzl25 in #4926
feat(test): add compactor to deterministic test by @wangrunji0408 in #4917
fix(sqlparser): keyword INOUT was mis-fixed to INPUT as typo by @xiangjinwu in #4936
test: add nexmark sql21 by @KeXiangWang in #4940
chore: increase vacuum batch size by @zwang28 in #4913
fix(meta): use correct context id in report_compact_task_impl by @zwang28 in #4932
chore(ci): automate tpch benchmark with cloud by @huangjw806 in #4906
fix(risedev): revert accidental change to risedev.yml by @richardchien in #4943
chore: add barrier passed actor trace for debugging by @MrCroxx in #4911
refactor(fragmentor): some refactoring done when I tried to understand the code by @ice1000 in #4946
feat(frontend): derive order property from primary keys by @GeorgKreuzmayr in #4719
fix(pgwire): reduce crash in pgwire by @ZENOTME in #4900
feat(metrics): add metrics for node number by @Gun9niR in #4933
feat(compose): support tiered cache in compose profile by @MrCroxx in #4957
refactor(meta): use apply_version_delta to replace apply_compact_result by @Li0k in #4952
feat(compaction): support streaming upload of sst as an option for compactor by @Gun9niR in #4685
feat(optimizer): support ApplyJoinRule one side push down optimization by @chenzl25 in #4945
fix(stream,agg): ignore NULL values in streaming min/max by @richardchien in #4960
feat(compute): let compute node handle deltas pushed from meta by @soundOfDestiny in #4721
feat(storage): add optional sanity check for committed SSTs by @zwang28 in #4910
refactor(binder): refactor is not distinct from by @chenzl25 in #4962
feat(state_store): add metrics for skip keys by @Little-Wallace in #4909
feat(storage): apply new batch table by @wcy-fdu in #4883
feat(compaction): move the metric: comapct_frequency to meta node by @Gun9niR in #4938
fix(source): do not panic on "no available readers" by @BugenZhao in #4967
feat(test): deterministic e2e fuzzing by @wangrunji0408 in #4950
doc: fix broken links in markdown by @xxchan in #4880
chore: fix feature request link by @xxchan in #4980
fix(ci): change the auto benchmark region config by @huangjw806 in #4974
chore(source): check schema on table source writing chunks by @BugenZhao in #4970
refactor(storage): remove row_serde trait by @wcy-fdu in #4973
chore: update all org links by @TennyZhuang in #4981
refactor(compaction): remove duplicate compaction config in CompactStatus by @zwang28 in #4963
feat(stream): generate 2-phase top n rule. by @shmiwy in #4971
refactor(batch): refactor batch string_agg by @richardchien in #4949
chore: remove cell-based field in column mapping by @wcy-fdu in #4985
fix(stream,agg): fix behavior of streaming string_agg to skip NULL value by @richardchien in #4959
refactor(meta): refactor ddl service and some code clean by @yezizp2012 in #4678
fix(compaction): cancel previously assigned tasks when compactor (re)connecting by @zwang28 in #4968
test: add nexmark sql22 by @KeXiangWang in #4986
test: add endless NexMark e2e test (based on NexMark source) by @fuyufjh in #4828
feat(optimizer): Equal join predicate support null safe equal type by @chenzl25 in #4983
refactor(meta): simplify error handling by @neverchanje in #4989
refactor: simplify the scheduler logic & remove duplicate & meaningle… by @BowenXiao1999 in #4924
feat: notify and store fragment vnode mapping in frontend by @HuaHuaY in #4929
feat(batch): support null-safe in hash join executor by @wzzzzd in #4994
chore: github workflow: auto assign milestone to issues by @fuyufjh in #4992
feat(compaction): limit write-amplification for TierCompactionPicker by @Little-Wallace in #4908
test: add test for batch null safe hash join by @chenzl25 in #4999
refactor(batch,agg): refactor batch selective agg by @richardchien in #4972
fix(planner): reject subquery/agg/table in agg call by @xiangjinwu in #4893
style(frontend): unnecessary ColumnId get and new by @xiangjinwu in #5003
feat(optimizer): support two phase limit and topn for local mode by @chenzl25 in #4920
fix(strage): fix bugs caused by sync_uncommitted_data out of order by @xxhZs in #4996
fix: fix migration in recovery by @yezizp2012 in #5006
refactor(meta): simplify error handling (part2) by @neverchanje in #4995
feat(frontend): Derive order property from input by @GeorgKreuzmayr in #4903
refactor(common): clean up toml config methods by @zwang28 in #4984
fix(optimizer): fix apply scan rule with null reject by @chenzl25 in #5007
chore: add issues to GitHub Project automatically by @fuyufjh in #5018
refactor(storage): avoid state store clone when using start write batch by @wenym1 in #5008
refactor(stream): refactor 2-phase topN to reduce the code duplication by @shmiwy in #5017
fix(storage): incorrect row op of "insert then update" by @BugenZhao in #5020
refactor(doc): update doc for new relational table and encoding by @wcy-fdu in #5002
feat(frontend): serialize stage graph by @Flpha0830 in #4805
chore: fix workflow of adding issues to project by @fuyufjh in #5026
feat(test): support killing all kinds of node in deterministic e2e test by @wangrunji0408 in #4969
feat(binder): bind SQL OVER window function by @xxchan in #4979
feat: Increase default promethues scrape interval. by @liurenjie1024 in #5031
chore: fix codecov badge in README by @fuyufjh in #5030
feat(batch): use compute client pool for grpc exchange. by @wzzzzd in #5023
feat(storage): maintain local related sst index by @wenym1 in #4845
chore: temporarily disable file cache before bug fix by @MrCroxx in #5036
refactor(iterator): remove useless loop by @Little-Wallace in #4991
refactor(storage): clean up duplicate code in relational table by @wcy-fdu in #5038
fix(risedev): specify config-path for frontend in compose by @zwang28 in #5040
feat(storage): add support for S3 multi prefixes by @StrikeW in #4965
chore: remove unused ErrorCode variants by @xxchan in #5047
chore(grafana): minor tweaks by @BugenZhao in #5052
chore(ci): do not soft fail on parallel e2e release by @BugenZhao in #5053
refactor(storage): refactor get_from_table by @Li0k in #4993
fix(filecache): fix file cache get read guard by @MrCroxx in #5039
fix: fix privileges check by @HuaHuaY in #5055
feat: add retry etcd client for meta store by @yezizp2012 in #5056
fix(optimizer): Use null safe equal for subquery optimization by @chenzl25 in #5021
feat(batch): support is null for scan range by @chenzl25 in #5054
refactor(frontend): refactor expr visitor and plan visitor by @chenzl25 in #5058
fix(optimizer): small fix for distinct aggregates. by @likg227 in #4948
refactor(binder): extract special type infer rules from FunctionCall:new by @xiangjinwu in #5062
chore: enable tiered cache on CI and 3 node compose by @MrCroxx in #5061
fix(optimizer): report error when Apply can't be unnested. by @likg227 in #5041
refactor: maintenaning vnode mapping notify in fragment manager by @yezizp2012 in #5075
refactor(fragmenter): remove unneeded struct, prepare for rewrite by @ice1000 in #5064
feat(streaming): estimated cache size for hash join by @BugenZhao in #5057
feat(table): add an optional vnode columns in table catalog by @st1page in #5079
fix: casting Varchar to multidimensional List by @nanderstabel in #5063
fix: delete not null of first query demo in README.md by @ziyi-yan in #5085
feat(frontend): Add and Use Sort Aggregation Plan Node by @GeorgKreuzmayr in #5046
feat(streaming): support configuration change on simple dispatcher by @BugenZhao in #5083
feat(binder): bind FETCH by @xxchan in #5092
feat(batch): support null safe lookup join by @chenzl25 in #5076
refactor(dashboard): ui redesign by @xxchan in #5087
fix(Storage): Fix the reverse order problem of barrier_manager's sst by @xxhZs in #5065
chore(deps): bump lz4-sys from 1.9.3 to 1.9.4 by @dependabot in #5088
doc: configure system variables by @lmatz in #5093
feat(batch):create executor level metrics. by @ZENOTME in #4964
refactor: try to use select in async stream function by @BowenXiao1999 in #5022
feat(expr): support array_cat function and || operator for arrays by @richardchien in #5060
refactor(sqlparser): remove numeric literal postfix by @xxchan in #5091
feat(compaction): refactor sstable writer by @Little-Wallace in #4982
fix(metrics): fix iterator local metrics lost and enable iterator local metrics report checker in debug mode by @MrCroxx in #5074
feat(stream): support stream null safe hash join by @chenzl25 in #5095
feat(stream): support sum/min/max over interval type in streaming query by @TennyZhuang in #5105
fix(storage): assert vnode existence in state table by @BugenZhao in #5011
refactor(stream): refactor append only top n to use one state table by @shmiwy in #5049
refactor(meta): refine barrier injection to improve readability by @StrikeW in #5099
refactor(fragmenter): assign table_id while translating stream nodes to prost by @ice1000 in #5086
feat(dashboard-srv): support list sources by @xxchan in #5098
feat(hummock/meta): Reliable task cancellation via heartbeats + external task cancellation gRPC by @jon-chuang in #4496
feat(frontend): support explain distsql by @Flpha0830 in #5048
fix(storage): sync epoch error handling by @StrikeW in #5112
fix(batch): better error handling for single value by @chenzl25 in #5113
fix(compaction): cancel unassigned/unsent task in manual compaction. by @zwang28 in #5104
chore(test): add primary key constraints for nexmark sources & planner tests by @BugenZhao in #5124
fix(compaction): periodically trigger compaction for all compaction groups by @zwang28 in #5102
fix(notification): fix lock order issue by @soundOfDestiny in #5126
feat(test): add etcd simulator by @wangrunji0408 in #5084
feat: create source with avro row format by @tabVersion in #5059
perf(expr): optimize to_char function with constant template by @TennyZhuang in #5130
refactor(compaction): simplify cancellation by @zwang28 in #5131
refactor: always insert a exchange singleton on top dml by @BowenXiao1999 in #4752
feat: actor migration with reschedule semantics by @shanicky in #4956
doc: update architecture design doc and graph by @hzxa21 in #5115
fix(storage): fix s3 client config by @StrikeW in #5134
refactor(compaction): add unit tests for manual compaction. by @zwang28 in #5090
refactor: refactor definition of serialize_datum by @st1page in #5140
refactor(compaction): recycle meta file to reduce memory usage by @Little-Wallace in #5123
refactor(test): simplify sync point by @wangrunji0408 in #5034
refactor: unify dml and dql handler path by @BowenXiao1999 in #5144
feat(frontend): support \d, \dt and \dm by @yezizp2012 in #5127
chore(meta): add some forgotten code and refactor notification manager a little by @HuaHuaY in #5132
feat(dashboard): add plan overview based on d3-dag by @xxchan in #5118
fix(Batch): sortAgg should not ouput NULL when no inputs by @st1page in #5150
feat(stream): use memcomparable row as Cache key in GenericExtremeState by @Gun9niR in #5094
chore: rename frontend/test_runner -> planner_test by @xxchan in #5152
fix(sqlsmith): match on NumericOutOfRange string representation by @kwannoel in #5149
refactor(meta): execute revert funcs in blocking manner by @zwang28 in #5148
ci(test): add coverage and split build and run for deterministic test by @wangrunji0408 in #5077
refactor(frontend): cleanup duplicate and elusive codes for WITH OPTIONS by @BugenZhao in #5142
feat: health check RPC for meta node by @yinfredyue in #5154
chore(compaction): remove consistent hash overlap strategy. by @zwang28 in #5166
style(connector): avoid Arc by @TennyZhuang in #5146
feat(planner&optimizer): plan OverAgg & convert it to group TopN by @xxchan in #5116
feat(storage): Implement BlockStream, SstableStore::get_block_stream(), ObjectStore::streaming_read(), and SstableStreamIterator. by @jon-chuang in #4111
feat(expr): support lowercase format in to_char function by @TennyZhuang in #5032
fix: fix deterministic create mv ut fail by @yezizp2012 in #5179
feat(streaming): support configuration change on receiver executor by @BugenZhao in #5174
refactor: cancel running batch query of ctrl-c by @BowenXiao1999 in #5033
feat: reschedule interface in risedev by @shanicky in #5185
feat(meta): set min_pinned_id instead of 0 when subscribe by @soundOfDestiny in #5042
fix(storage): fix reversed sst order when collecting barrier from CN by @wenym1 in #5169
feat: Skip checking duplicates when explaining "create mv" by @yinfredyue in #5159
feat(stateTable): write interface with stream Chunk by @st1page in #5151
feat(dashboard): support visualization of actor box by @xxchan in #5172
refactor(stream dispatcher): rename hash value to vnode by @st1page in #5195
refactor(meta): extract barrier scheduler from barrier manager by @BugenZhao in #5187
feat(dashboard): add layout DAG algorithm by @mikechesterwang in #5180
fix: meta push split assignments down when recovery by @tabVersion in #5160
feat: Set tcp no delay for compute nodes by @liurenjie1024 in #5201
fix(docs): fix the minor error in comments by @xinchengxx in #5183
feat(batch): refactor the micro benchmark by @likg227 in #5158
fix(ctl): use given epoch when list_kv by @zwang28 in #5203
refactor(stream TopN): code clean up by @st1page in #5202
refactor(batch): do not consume data chunk in handler by @BowenXiao1999 in #5173
refactor(metrics): refactor tiered cache metrics by @MrCroxx in #5178
fix: kafka read dulpliction when recovery by @tabVersion in #5199
feat(storage): use a single object to store meta and data file by @Little-Wallace in #5135
feat(batch): add micro benchmark for expand. by @likg227 in #5204
fix: a potential bug in state transition by @BowenXiao1999 in #5211
fix(meta): expired RiseCtl worker causes meta panics by @chenzl25 in #5212
feat(meta): let pin_snapshot be intra-meta method by @soundOfDestiny in #5188
fix: source take snapshot after mutation by @tabVersion in #5168
fix: fix force stop actor in recovery by @yezizp2012 in #5218
refactor: use strum discriminants to generate the code for DataTypeName by @neverchanje in #5184
feat(risectl): add notification client for risectl. by @zwang28 in #5222
test: enable tpch q15 by @fuyufjh in #5223
feat(dashboard): render source dependency graph by @xxchan in #5215
feat(optimizer): Only accept scalar subquery can guarantee at most one row at compile time by @chenzl25 in #5147
refactor(storage): move current_epoch out of hummock version by @xxhZs in #5082
chore(meta): remove unused notify_all by @Li0k in #5239
refactor(streaming): refine degree usage by @yuhao-su in #5162
refactor(source): let SourceParser construct StreamChunk directly by @TennyZhuang in #5216
refactor(storage): remove shared_buffer replicated_batch by @Li0k in #5234
refactor(compaction): unify task cancellation by @zwang28 in #5240
fix(parser/binder): remove/reject not supported stmt by @xxchan in #5232
fix: incorrect naming of tx and rx by @BowenXiao1999 in #5243
feat(stream): use key encoding for pk in JoinEntryState by @Gun9niR in #5137
feat(dashboard): new stream plan overview by @xxchan in #5242
feat: process metric for macos by @tabVersion in #5219
test: add a test case for window group top n by @xxchan in #5253
feat: Skip name duplicate checking when explaining by @yinfredyue in #5200
fix(compaction): check sst upload result by @Gun9niR in #5261
chore: rename TableSourceV2 to TableSource by @fuyufjh in #5268
feat: support rescheduling table source & handle corner cases by @BugenZhao in #5226
feat(frontend): Use SortAgg when output requires sort by @GeorgKreuzmayr in #5120
fix(test): data should be from upstream in tpch_upstream.slt by @BugenZhao in #5276
fix(storage): sync a bounded epoch range in each sync call by @wenym1 in #5236
feat: improve memtable conflicts error message by @fuyufjh in #5275
fix(frontend): mix START_TRANSACTION and BEGIN by @ZENOTME in #5267
feat(frontend): Use index scan if it satisfies required order by @GeorgKreuzmayr in #5280
fix(stream): fix wrong usage of chunk builder in topn by @xxchan in #5286
refactor: Row::serialize() does not need to return Result by @fuyufjh in #5281
fix: handle zero and negative step for generate_series by @xxchan in #5231
feat(catalog): add value indices in table catalog by @wcy-fdu in #5296
feat(risectl): risectl hummock: add list-version-deltas by @StrikeW in #5288
fix(compaction): fix in-correct epoch watermark by @Little-Wallace in #5207
test: add streaming query e2e tests: tpch q16 & q21 by @fuyufjh in #5292
feat(test): support killing node during execution by @wangrunji0408 in #5300
feat(frontend): add frontend metrics by @ZENOTME in #5138
refactor(test): switch sync_point by feature flag by @zwang28 in #5298
fix(frontend): remove pg_get_userbyid limit operator by @chenzl25 in #5279
feat: explain dist stream plan by @st1page in #5295
feat(compaction): Split sub-compaction by index-block by @xinchengxx in #5139
test: move datagen tests to e2e_test/source by @xxchan in #5318
ci: add stale message to stale action by @xxchan in #5308
refactor: ArrayBuilder::finish doesn't need to return Result by @fuyufjh in #5315
fix(optimizer): refine index cost by using primary table row size to estimate non-covered index cost by @chenzl25 in #5317
fix(streaming): only use receiver executor for no-shuffle exchange by @BugenZhao in #5311
fix: handle mem-table error by @fuyufjh in #5323
feat(object store): remove fallback by @Gun9niR in #5266
chore(storage): add metrics for compaction write amplification by @Li0k in #5078
fix: fix update/remove vnode mapping assert in frontend by @yezizp2012 in #5301
feat(optimizer): support deduplication by setting overagg rank = 1 by @chenzl25 in #5282
fix(metrics): collect node num metrics at process level by @Gun9niR in #5287
refactor(storage): manage version_ids_in_use in unpin_worker instead of LocalVersion by @odysa in #5153
feat: draft pr for execute root fragment at FE instead of CN by @BowenXiao1999 in #4688
test: add test for javascript pg driver by @neverchanje in #5294
feat(sqlsmith): add primary key to tpch tables by @kwannoel in #5332
refactor(storage_table): remove deprecated OrderedColumnDesc by @kwannoel in #5334
refactor(compaction): avoid duplicate write path in trivial move. by @zwang28 in #5314
feat(risedev): introduce DocSlt tool to extract SQL examples written in SLT syntax to SLT files by @richardchien in #5246
feat(compaction): support compaction task scheduling policy based on score by @Gun9niR in #5044
fix: force user specify at least one pk when create debezium source by @tabVersion in #5307
fix(pgwire): return no_data in describe message when encounter non-query statement by @ZENOTME in #5319
refactor(sqlparser): improve parsing for subquery relation by @xxchan in #5252
feat(filecache): call MADV_WILLNEED after meta file grow by @MrCroxx in #5309
refactor(compaction): add more tests by @zwang28 in #5208
doc(test): add developer guide for deterministic test by @wangrunji0408 in #5342
feat(filecache): replace Vec::resize with Vec::reserve and unsafe Vec::set_len to eliminate memset by @MrCroxx in #5313
fix(dashboard): fix compactor address in compact frequency by @Gun9niR in #5333
style: refine the naming of acronyms by @TennyZhuang in #5347
feat(compaction group): add new compaction groups in need by @soundOfDestiny in #5265
feat(storage): batched iterator & e2e tests with in-memory state store by @BugenZhao in #5322
feat(storage): streaming table serialize row with value_indices by @wcy-fdu in #5305
feat(bench): introduce read write rate limiter for file cache bench for accuracy by @MrCroxx in #5343
test: disable TPC-H Q15 due to #5277 by @fuyufjh in #5362
refactor: ArrayBuilder::append doesn't need to return Result by @fuyufjh in #5357
chore(storage): add more TaskStatus for metrics report by @Li0k in #5340
fix(meta): some fixes to CompactorManager by @Gun9niR in #5355
fix(meta): recovery use wrong epoch by @xxhZs in #5364
fix: fix notification about source internal tables after reboot by @yezizp2012 in #5370
refactor(test): test CompactorManager by @zwang28 in #5373
fix(opt): order derive of batch scan by @st1page in #5368
doc: fix broken link by @xxchan in #5375
chore(deps): bump axum-core from 0.2.7 to 0.2.8 by @dependabot in #5379
feat(object store): expose failure count metric by @Gun9niR in #5348
refactor(config): turn const into configuration parameters by @lmatz in #5361
fix:disable collecting executor metrics in local execution mode by @ZENOTME in #5372
refactor(storage): move epoch shared buffer to sync_uncommitted_epoch when sync start by @wenym1 in #5289
feat(runtime): allow using tokio runtime in simulation by @wangrunji0408 in #5369
fix(bench): fix bench_multi_builder by @Gun9niR in #5263
chore: replace lazy_static with std::lazy::LazyLock by @wangrunji0408 in #5385
feat: set sstable capacity for flush job by @Little-Wallace in #5237
fix(source): error handling in SourceParser::parse by @TennyZhuang in #5260
refactor(stream): improve readability of TopN cache by @xxchan in #5376
fix: improve error for time window function by @xxchan in #5230
fix(sqlparser): distinguish table and table function by @xxchan in #5229
feat(streaming): hash join degree table only write and read value by @wcy-fdu in #5365
fix(binder): castability check between array and string by @xiangjinwu in #5081
chore(storage): sstable item avg size metrics by @Li0k in #5377
fix: fix deterministic test doc by @BowenXiao1999 in #5389
feat(batch): use separated runtime for batch. by @likg227 in #5363
fix(compaction): fix level compaction picker. by @zwang28 in #5381
feat(meta): round-robin scheduler & minimal scheduling mode by @BugenZhao in #5387
fix(sqlparser): reject reserved words as column name by @TennyZhuang in #5125
feat(RPC): support a simple connection pool for RPC client. by @chenzl25 in #5384
refactor(metric): refactor grafana dashboard and remove unused metrics by @hzxa21 in #5404
refactor(compaction): stall compaction scheduler when there is no available compactor. by @zwang28 in #5405
chore: refine source manager by @shanicky in #5391
refactor: misc. minor improvements to state_table by @fuyufjh in #5398
doc(notification): clearify which workers will trigger pin_version … by @soundOfDestiny in #5351
feat(frontend):ignore COMMIT and ROLLBACK by @ZENOTME in #5397
fix(frontend): refine error msg for drop internal table by @HuaHuaY in #5412
feat(dashboard): support explain distributed plan by @Flpha0830 in #5198
feat(ci): build with earlier version of GLIBC by @huangjw806 in #5414
chore(streaming): clean-up and disallow RwError in stream crate by @BugenZhao in #5407
fix(meta): report worker num in the cluster fully from meta node by @Gun9niR in #5401
refactor(streaming): remove epoch on state table read interface by @st1page in #5380
fix(pgwire): remove all write usage by @BowenXiao1999 in #5235
fix(sst): capacity estimate by @Little-Wallace in #5356
fix(streaming): add vnode check for state table write_chunk by @BugenZhao in #5417
feat(storage): decouple sync epoch into seal and await sync epoch by @wenym1 in #5329
fix: test build warning by @TennyZhuang in #5422
refactor: use abort() for query execution by @BowenXiao1999 in #5352
refactor(config): reduce connector_message_buffer_size and make it configurable by @lmatz in #5416
style: refine all constructor namings by @TennyZhuang in #5425
feat: scaling with reschedule semantics by @shanicky in #5358
feat(metrics): metrics for aggregation by @lmatz in #5427
feat(binder): support type alias int2 to unblock regress tests by @xiangjinwu in #5428
feat(frontend): support creating empty arrays by @xiangjinwu in #5402
chore: modify default value for no_available_compactor_stall_sec by @zwang28 in #5435
feat(planner test): add stream fragment plan test by @st1page in #5426
refactor(storage): add value indices assert in batch mode by @wcy-fdu in #5424
feat(stream): add join key in logical pk by @st1page in #5344
fix(storage): correctly handle future deadlock by @zwang28 in #5432
chore(proto): remove generic type by @Li0k in #5438
feat(pgwire):support more param type by @ZENOTME in #5431
chore(frontend): disable no-shuffle exchange optimization by @BugenZhao in #5423
feat(storage): call seal epoch when collecting a barrier to ensure seal order by @wenym1 in #5339
feat(batch): optimize small range scan by @chenzl25 in #5436
fix(risedev): bump kafka version by @BugenZhao in #5453
fix: fix overflow when generate migration plan in recovery by @yezizp2012 in #5450
feat(compaction group): lifecycle of new created compaction groups by @soundOfDestiny in #5419
chore(proto): enable ENUM_ZERO_VALUE_SUFFIX lint by @xxchan in #5458
fix(stream): should NOT panic when update is spitted in dispatch by @st1page in #5338
doc(prost): update doc for prost-helper by @xxchan in #5459
feat(explain): explain table in explain stream distsql by @st1page in #5454
fix(rpc_client): enables gRPC keepalive ping by @zwang28 in #5445
feat(frontend): top_n catalog support reteintion_seconds by @Li0k in #5442
feat(BTreeMapTransaction): delete data both in staging and original `… by @soundOfDestiny in #5463
test(regress): enable more bool tests except ERROR/IS UNKNOWN by @xiangjinwu in #5448
feat(object store): lazily create multipart upload by @Gun9niR in #5406
fix(compaction): fix skip key count by @Little-Wallace in #5443
feat(meta): async recursively drop database by @HuaHuaY in #5374
chore(grafana): remove unused dashboard and meaningless metrics by @wcy-fdu in #5468
fix(optimizer): fix outer join predicate push down by @st1page in #5473
refactor(frontend): rename order_key to pk by @wcy-fdu in #5475
feat(source): Optimize JsonParser in source connector by @jon-chuang in #5367
fix(streaming): corners cases for scaling (Part 1) by @BugenZhao in #5470
feat(explain): add explain distribution key in state table by @st1page in #5484
refactor(config): eliminate hardcoded cache size by @lmatz in #5410
fix(state table): fix vnode index in stream hash agg value state by @st1page in #5467
feat(batch): remove lookup join backtrace by @chenzl25 in #5483
chore(deps): Bump aws sdk version to latest by @hzxa21 in #5466
feat(storage): support clearer way to gracefully shutdown buffer worker by @wenym1 in #5469
refactor(optimizer): some refactor on stream topn by @st1page in #5489
refactor(EpochPair):some refactor epoch pair by @st1page in #5488
refactor(frontend): remove is_index_on field in table catalog by @wcy-fdu in #5480
refactor(pgwire): support complete prepared statement by @ZENOTME in #5485
refactor(stream): some refactor on stream agg by @st1page in #5491
feat(object store): retriable streamed request body by @Gun9niR in #5493
fix(json): M1 uses Neon SIMD instructions by @lmatz in #5495
feat(pgwire):support flush message by @ZENOTME in #5496
refactor(storage): refactor function for reuse by @Li0k in #5494
fix: make actor abortable and refine force stop all actors when recovery by @yezizp2012 in #5444
feat(pgwire):support all text format param by @ZENOTME in #5498
feat(streaming): introduce subtasks in actor to improve I/O concurrency by @BugenZhao in #5449
feat(stream): implement TopN with ties by @xxchan in #5486
chore(error): remove unnecessary Clone and use Box in RwError by @TennyZhuang in #5506
chore: use rustflags for lints by @xxchan in #5508
style: introduce some lints by @xxchan in #5511
chore: update mergify failure message by @xxchan in #5509
refactor(expr): use macro by @ice1000 in #5513
fix(batch): fix local execute query stuck by @chenzl25 in #5502
fix(notification service): call pin_snapshot only when worker is fr… by @soundOfDestiny in #5515
feat(expr): limit timestamp datagen to microseconds, improve timestamp cast error msg by @jon-chuang in #5514
feat(storage): iter skip metrics by @Li0k in #5492
feat(batch): buffer enough build side data of lookup join before lookup by @chenzl25 in #5501
feat: add upstream_fragment_ids to Fragment by @shanicky in #5451
fix(frontend): release hummock snapshot in local execution mode by @hzxa21 in #5531
style: allow clippy for breaking changes by @xxchan in #5530
fix(state_table): call commit_no_data_expected on state table even with nothing to flush by @hzxa21 in #5533
feat(planner): improve explain formatting by @jon-chuang in #5541
feat(storage): handle sync in flush controller by @wenym1 in #5430
fix(frontend): fix unpin_snapshot_before in HummockSnapshotManager by @hzxa21 in #5522
feat(ctl): list pinned hummock versions and snapshots by @zwang28 in #5532
chore(grafana): add metric for hummock version and snapshot by @zwang28 in #5528
feat(pgwire):change PgResponse to use stream mode in transfer row by @ZENOTME in #5535
fix(batch): support sum(interval) by @TennyZhuang in #5549
chore: unifying the code for migration and scaling by @shanicky in #5545
feat(connector): Datagen struct type by @jon-chuang in #5550
refactor(storage): refactor storage table get with wait epoch by @chenzl25 in #5525
feat(storage): add shared buffer batch id by @wenym1 in #5557
feat(streaming): use EpochPair in state table and executors do not save epoch by @wcy-fdu in #5548
feat: added the steps to run flink bench via pipeline by @sumittal in #5471
feat(batch): support hash based lookup join by @chenzl25 in #5554
refactor: remove unnecessary Result by @fuyufjh in #5558
refactor(single_value): remove obsolete single_value by @kwannoel in #5562
test: introduce ch-benchmark by @nanderstabel in #5155
fix(stream): fix TopN's usage of arguments by @xxchan in #5551
feat(dashboard): add plan node detail, add quick search by @xxchan in #5539
feat(object store): configure bucket lifecycle to clean up incomplete multipart upload by @Gun9niR in #5546
feat(compaction): record stale key count in sstableinfo by @Little-Wallace in #5505
refactor(storage): remove order_index of staging by @Li0k in #5569
feat(state_table): add upsert method for StateTable to make the semantics clean by @richardchien in #5565
feat(test): add kafka simulator by @wangrunji0408 in #5536
chore: speed up prepare kafka by run in parallel by @tabVersion in #5572
refactor(batch): rename lookup join inputs name by @chenzl25 in #5566
refactor(array): don't return Result in Array::compact by @TennyZhuang in #5575
fix(source): cast_f64 for simd_json Value and regression test by @jon-chuang in #5577
fix(object store): correctly handle get lifecycle response by @Gun9niR in #5582
feat(optimizer): support join commute rule to convert right join type to left join type by @chenzl25 in #5578
refactor: use state table for state store in source by @tabVersion in #5433
refactor: use parse-display crate to simplify code by @wangrunji0408 in #5571
feat(meta): add checkpoint_frequency and support decoupling by @xxhZs in #4966
feat(array): support parsing struct value in StreamChunk from pretty-printed chunks (for test) by @wangrunji0408 in #5547
chore: add max_heartbeat_interval config for risedev by @yezizp2012 in #5581
refactor(array): use Column::from to construct a Column by @TennyZhuang in #5585
refactor(fe): refactor table catalog builder by @st1page in #5584
chore(storage): update the method of HummockReadVersion by @wenym1 in #5591
chore: set unused_must_use's lint level to Forbid by @xxchan in #5593
doc: add docs for batch micro benchmarks by @xxchan in #5595
feat(compaction): notification based compaction trigger by @Gun9niR in #5500
feat: introduce LruManager by @yuhao-su in #5395
refactor: add assertion for PrecomputedHasher by @fuyufjh in #5574
test(regress): enable int2 by @xiangjinwu in #5452
refactor(storage): refactor code related to LocalVersion to make the code cleaner by @wenym1 in #5568
fix(streaming): wait epoch before scaling (Part 2) by @BugenZhao in #5517
refactor(source): merge all inner reader into one stream by @waruto210 in #5611
chore(planner test): add name field and improve error handling by @wangrunji0408 in #5602
feat(frontend):change query_handle to return data stream by @ZENOTME in #5556
refactor(stream): simplify pause stream by @wangrunji0408 in #5627
refactor(streaming): use key encoding in SerializedKey by @Gun9niR in #5596
refactor(connector): remove redundant clone by @waruto210 in #5625
fix(meta): pin & unpin snapshot for mview creation with correct order by @BugenZhao in #5606
chore: exclude generated & vendored files in linguist by @BugenZhao in #5629
refactor: simplify HashKeyDispatcher by @xxchan in #5626
chore: clean up some dead code in keyspace by @wcy-fdu in #5638
fix(stream): fix stream approximate count distinct estimation algorithm. by @chenzl25 in #5622
refactor(storage): remove the Option of CompactionGroupId in function levels by @Li0k in #5633
feat: add definition column for pg_matviews_info by @yezizp2012 in #5637
fix(source): fix parse avro datetime by @waruto210 in #5641
refactor(compaction): reduce the usage of mutex in task progress tracking by @Gun9niR in #5631
feat(frontend): support TopN with ties by @xxchan in #5538
feat(storage): add basic interface for HummockStorage base on HummockReadVesrion by @Li0k in #5620
refactor(storage): user iterator accept generic hummock iterator by @wenym1 in #5583
feat(frontend): Allow concatenating arrays with compatible types by @CAJan93 in #5345
refactor: simplify for_all macros by @xxchan in #5651
fix(optimizer): improve index selection for large in query and better display for scan ranges. by @chenzl25 in #5649
feat(batch): optimize take data task for local execution by @chenzl25 in #5639
refactor(plan): extract partially the data structures in logical by @ice1000 in #5630
chore(ci): remove fink benchmark by @huangjw806 in #5653
refactor(stream): re-organize TopN code by @xxchan in #5658
refactor: improve memory stats perfomance by @yuhao-su in #5659
refactor(array): remove duplicate code in {DataChunk, StreamChunk}::from_pretty by @wangrunji0408 in #5663
fix(dashboard): actor not show multiple dispatchers by @jon-chuang in #5666
fix(binder): ExprVisitor and ExprMutator should visit agg_call order_by and filter by @xiangjinwu in #5664
fix: fix clippy options (rustflags) by @fuyufjh in #5667
fix(expr): replace unimplemented panic with Err by @xiangjinwu in #5672
chore(risedev): disable tier-cached by @zwang28 in #5671
style: enable clippy upper-case-acronyms-aggressive by @TennyZhuang in #5681
style: remove unused imports while test feature is not enabled by @waruto210 in #5680
chore(dashboard): gen proto by @kwannoel in #5603
chore(grafana): add metrics description in grafana panels by @wcy-fdu in #5662
fix: return err for stream top n with limit 0 by @xxchan in #5687
doc: index page for crate docs by @xxchan in #5685
feat(batch): support batch Group TopN by @xxchan in #5688
refactor(config): make BOUNDED_BUFFER_SIZE configurable by @lmatz in #5682
ci: publish crate docs to gh-pages by @xxchan in #5686
fix: array subscript doesn't have correct precedence by @CAJan93 in #5673
refactor(storage): replace RwError with dedicated error types in StateTable by @wcy-fdu in #5695
refactor: improve sanity check error message by @fuyufjh in #5694
refactor: remove batch_deserialize in stata table by @fuyufjh in #5696
refactor(streamAgg): refactor the state of streamAgg by @st1page in #5598
refactor: remove create_source/sync_source rpc in CN(#5269) by @PanQL in #5654
refactor(plan_node): continue the extraction in #5630 by @ice1000 in #5699
refactor(plan_node): Last part on rewriting logical nodes by @ice1000 in #5700
chore: remove temporary compaction group switch by @zwang28 in #5701
feat(risedev): add typos check for risedev check by @MrCroxx in #5643
chore(deps): bump tracing-subscriber and use nu-ansi-term by @TennyZhuang in #5703
refactor(source): all in async-stream by @wangrunji0408 in #5647
refactor(meta): remove ddl lock for scaling by @BugenZhao in #5706
feat(pgwire): replace BoxStream in PgResult with static type by @Gun9niR in #5677
refactor(notification): introduce subscribe type and avoid passing addr and worker type by @wenym1 in #5702
feat(source): support parsing nested record/array in avro parser and fix bug when load schema from s3 by @waruto210 in #5679
fix(test): reset pg client after connection broken by @wangrunji0408 in #5710
refactor(storage): add dedicated error type for StorageTable by @kwannoel in #5618
feat: introduce source actor scaling by @shanicky in #5648
refactor(steam,agg): use insert or update according to previous state instead of upsert by @richardchien in #5708
test(encoding): bench key/value encoding by @Gun9niR in #5165
feat(storage): use timeout for filter_extractor acquire by @Li0k in #5689
refactor(stream,agg): enable state table sanity check for streaming agg by @richardchien in #5709
test(scale): introduce deterministic scaling tests by @BugenZhao in #5657
refactor(storage): state table uses StorageError instead of a dedicated error type by @wcy-fdu in #5720
fix(storage): fix SST GC watermark by @zwang28 in #5715
refactor(streaming): stream agg only read and write value column by @wcy-fdu in #5411
fix: fix recovery hang and in-consistency for create/drop streaming jobs by @yezizp2012 in #5586
feat(storage): start notification manager when hummock start by @wenym1 in #5716
refactor(compaction group): change use case because vartrx fixed by @soundOfDestiny in #5727
feat(source): enable SASL for Kafka connector by @xx01cyx in #5693
feat(source): support parsing nested message/list in protobuf parser and fix bug in prepare data for kafka by @waruto210 in #5717
fix(build): fix CI doc test by adding dependency by @xx01cyx in #5737
chore: clean up compaction group register of source id by @yezizp2012 in #5733
chore: improve streaming debug info by @yuhao-su in #5735
fix(streaming): correct row concat and enable tpch q21 by @yuhao-su in #5736
feat(id generator): support 64bit id generation by @soundOfDestiny in #5731
fix(meta): ban scaling of the mviews being created by @BugenZhao in #5732
chore: collect static log features to a separate feature by @MrCroxx in #5725
refactor(streaming): merge OrderedRowSerializer and OrderedRowDeserializer into one by @wcy-fdu in #5734
refactor(storage): HummockEvent support VersionUpdate by @Li0k in #5650
refactor: report task execution fail msg via str by @BowenXiao1999 in #5544
feat: add pg_catalog.pg_index and support \di by @yezizp2012 in #5742
feat(bench): introduce OpenTelemetry tracing and eBPF tracing for file cache bench by @MrCroxx in #5640
feat(frontend): support create index with if not exists syntax by @chenzl25 in #5747
refactor(fe): refactor handle query by @st1page in #5744
chore(meta): remove log caused by TrivialMove by @Li0k in #5752
fix(ctl): scan table by @zwang28 in #5707
feat(planner): trace "logical rewrite for stream" when explain & minor fixes for scalar subquery by @BugenZhao in #5758
feat(planner test): support EXPLAIN stmt by @xxchan in #5760
chore(build): statically link openssl by @BugenZhao in #5762
refactor(plan_node): Second last part on rewriting logical nodes (for real this time) by @ice1000 in #5765
feat(state_store): refactor write stall to make code simple by @Little-Wallace in #5748
refactor(test): move duckdb tests to a dedicated folder by @lmatz in #5757
refactor: Remove unnecessary to_secs method by @liurenjie1024 in #5768
feat(batch): improve the implementation of BatchTaskMetrics by @Gun9niR in #5770
fix: reset file cache when apply compose deploy and enable tiered cache by default by @MrCroxx in #5772
fix(meta): resolve chain node with dynamic filter in the same fragment by @BugenZhao in #5779
chore: remove Arc from Registry by @Gun9niR in #5778
feat(source): improve impl of parsing list by @waruto210 in #5777
feat(storage): compaction deterministic test tool by @StrikeW in #5441
fix: refuse to create source if no meta info by @tabVersion in #5788
fix(storage): fix ConcatSstableIterator seek. by @zwang28 in #5746
fix(compaction group): check existence of given compaction group id b… by @soundOfDestiny in #5462
refactor(stream,agg): remove row count column assert by @richardchien in #5793
refactor(plan_node): Last part on rewriting logical nodes by @ice1000 in #5766
refactor(plan_node): Introduce GenericPlanRef by @ice1000 in #5797
feat(optimizer): refine the conversion from Join to Dynamic Filter by @BugenZhao in #5789
feat(storage): support iter for new state store by @wenym1 in #5601
refactor(config):make DEFAULT_CHUNK_SIZE configurable by @lmatz in #5784
feat(storage): add StateTable::iter_with_pk_range for DynamicFilter by @jon-chuang in #5774
feat(frontend): support create index with distributed by syntax by @chenzl25 in #5745
chore: use version 0.2.0-alpha for main branch by @fuyufjh in #5801
refactor(handler): extract RowSet formatting from scheduler to handler by @xiangjinwu in #5802

Full Changelog: v0.1.12...v0.1.13