risingwavelabs/risingwave v0.1.3

on GitHub

Features

• feat(bench): support various workload types (#5 by @Sunt-ing)
• feat(frontend): introduce planner (#44 by @xiangjinwu)
• feat(bench): support getseq (#47 by @Sunt-ing)
• feat(meta): support parallel materialize (#58 by @TennyZhuang)
• feat(bench): add prepocess and mimic RocksDB db_bench (#85 by @Sunt-ing)
• feat(streaming): add & test cache clearing for more executors (#55 by @BugenZhao)
• feat(connector): kinesisSplitReader (#57 by @tabVersion)
• feat(mview): support barrier from meta (#88 by @MrCroxx)
• feat(expr): add NegExpr (#63 by @likg227)
• feat(optimizer): use fixedbitset instead of bit-set (#114 by @st1page)
• feat(batch): add aggregation with distinct (#98 by @lmatz)
• feat(mview): mv on mv support parallel upstreams (#117 by @MrCroxx)
• feat(planner): add batch scan plan node definition (#81 by @BowenXiao1999)
• feat(fragmenter): introduce batch plan fragmenter (#65 by @BowenXiao1999)
• feat(optimizer): get inputs col index (#163 by @st1page)
• feat(dashboard): resolve chain node of mv on mv (#162 by @BugenZhao)
• feat: introduce config in batch env and use chunk_size in config (#157 by @fuyufjh)
• feat(stream): add stateless LocalSimpleAggExecutor in CN (#164 by @TennyZhuang)
• feat(Optimizer): logical topn and limit plannode (#167 by @st1page)
• feat: add metrics for hummock get latency (#166 by @wcy-fdu)
• feat(storage): add block cache (#168 by @skyzh)
• feat(type): remove prec and scale in Decimal (#175 by @likg227)
• feat: add lower expr (#171 by @xx01cyx)
• feat(frontend): Add drop_table in handler (#180 by @mczhuang)
• feat(mview): support epoch-aware snapshot read for chain node (#197 by @MrCroxx)
• feat(frontend): implement prune_col for Limit (#205 by @TennyZhuang)
• feat: use TABLE_V2 by default (#123 by @BugenZhao)
• feat: add ascii expr (#172 by @xx01cyx)
• feat(stream): hash dispatcher: support multi columns (#222 by @fuyufjh)
• feat(binder): derive column types for values. (#186 by @likg227)
• feat(optimizer): add property satisfy method (#226 by @st1page)
• feat: impl hummock compaction task RPCs. (#246 by @zwang28)
• feat: implement scan/ingest_batch stat for in-mem storage (#236 by @Sunt-ing)
• feat: introduce globally unique id for fragment and display on dashboard (#252 by @yezizp2012)
• feat: add MockHummockMetaClient that uses HummockManager. (#249 by @zwang28)
• feat(compaction): use meta client and local version manager instead o… (#225 by @soundOfDestiny)
• feat(common): add search expression (#235 by @lmatz)
• feat: Disable search operator (#259 by @liurenjie1024)
• feat: Batch add tpch q12 (#256 by @liurenjie1024)
• feat(stream): support parallel source (#232 by @fuyufjh)
• feat: add range scan count in grafana dashboards (#280 by @wcy-fdu)
• feat: Batch tpch q15 (#273 by @liurenjie1024)
• feat(dashboard): resolve chain node and fragment id (#283 by @mikechesterwang)
• feat(binder): bind & plan base table scan (#174 by @xiangjinwu)
• feat(dashboard): support multiple nodes (#299 by @skyzh)
• feat: release a worker node's hummock resources when it is deleted. (#292 by @zwang28)
• feat(connector): kinesis enumerator (#12 by @tabVersion)
• feat(connector): kinesis state support (#102 by @tabVersion)
• feat(dashboard): support mv query on dashboard (#310 by @yezizp2012)
• feat(ss-bench): support statistics flag (#128 by @Sunt-ing)
• feat: add more info to identity string of executors (#300 by @lmatz)
• feat(binder): bind & plan insert (#302 by @xiangjinwu)
• feat(meta): extract inner nodes from chain node (#307 by @MrCroxx)
• feat(frontend): get compute nodes's address from meta server when java frontend starts (#323 by @HuaHuaY)
• feat(scheduler): introduce augment (exchange source info, parallelism) phase (#243 by @BowenXiao1999)
• feat(catalog): add hidden row_id column in frontend (#309 by @BowenXiao1999)
• feat(optimizer): add plan root (#254 by @st1page)
• feat: add grpc metrics for hummock (#306 by @wcy-fdu)
• feat(optimizer): mapping rewrite method (#331 by @st1page)
• feat(batch): distributed table_v2 insertion & scanning (#334 by @BugenZhao)
• feat(dashboard): support multi-worker-node (#340 by @mikechesterwang)
• feat(streaming): inject more info into the identity string of stream executors (#344 by @lmatz)
• feat(batch): temporarily add min/max for bool required by current frontend for tpch (#362 by @xiangjinwu)
• feat(filesystem source): Define the Filesystem source interface. (#335 by @pangzhenzhou)
• feat: Add batch tpch q4 (#369 by @liurenjie1024)
• feat(streaming): remote channel building for mv on mv (#352 by @MrCroxx)
• feat(dashboard): filter mv & search and locate actor (#378 by @mikechesterwang)
• feat(risedev): print command in risedev log (#382 by @skyzh)
• feat(batch): Add tpch q22. (#388 by @liurenjie1024)
• feat(optimizer): condition && join predicate (#367 by @st1page)
• feat(meta): add reference count for tables, check when create/drop tables (#364 by @xx01cyx)
• feat(ci): enable 3-node streaming e2e (#405 by @MrCroxx)
• feat(hummock): add aws-s3 dashboards for hummock monitor (#413 by @wcy-fdu)
• feat(dashboard): clearer view for compute nodes (#420 by @mikechesterwang)
• feat(util): Support value encoding and memcomparable encoding for decimal (#419 by @likg227)
• feat(risedev): add frontend-node rust (#390 by @BowenXiao1999)
• feat: java frontend passes epoch to cn (#406 by @zwang28)
• feat: add in expression (#411 by @lmatz)
• feat(meta): add gRPC metrics (#436 by @skyzh)
• feat(streaming): support LRU for streaming hash join (#337 by @yuhao-su)
• feat: add meta grpc monitor in grafana dashboards (#448 by @wcy-fdu)
• feat(ss-bench): support displaying throughput (#449 by @Sunt-ing)
• feat: support parsing row type (#463 by @neverchanje)
• feat(ci): run risedev e2e on EC2 (#468 by @skyzh)
• feat(optimizer): project with mapping (#471 by @st1page)
• feat: zehao/modify rocksdb (#479 by @zehaowei)
• feat(ss-bench): Implement deleterandom operation for ss-bench (#482 by @Sunt-ing)
• feat(streaming): remove output for broadcast dispatcher (#499 by @BugenZhao)
• feat: introduce struct type (#485 by @neverchanje)
• feat(ss-bench): add random seed option and optimize algorithm (#500 by @Sunt-ing)
• feat: introduce NotificationService in meta. (#434 by @HuaHuaY)
• feat: drop mv on mv (#506 by @BugenZhao)
• feat: implement struct array's layout (#451 by @neverchanje)
• feat(meta): collect from creating actors (#512 by @BugenZhao)
• feat(ss-bench): support --scans option (#503 by @Sunt-ing)
• feat(expr): add builtin function TUMBLE_START (#507 by @TennyZhuang)
• feat: hummock wait epoch (#465 by @zwang28)
• feat(meta): add state field to worker node (#518 by @zwang28)
• feat(connector): kinesis support role arn and external id (#435 by @tabVersion)
• feat(meta): support instant flush in barrier manager (#524 by @BugenZhao)
• feat(proto): add hidden column in Column Desc (#530 by @st1page)

Fixes

• fix(streaming): hash join test with non-equi condition (#6 by @yuhao-su)
• fix(streaming): separate key and row data types in top n (#7 by @lmatz)
• fix(meta): incorrect range_lookup_key_range implementation (#46 by @zwang28)
• fix(riselab): add back tracing option (#49 by @skyzh)
• fix(streaming): correct vec capacity in get_hash_values (#52 by @yuhao-su)
• fix(batch): filter tricks itself into fetching child's input when there is none (#48 by @lmatz)
• fix(riselab): do not enable all components on --default (#83 by @BugenZhao)
• fix: add more tests for MetaStore (#84 by @zwang28)
• fix(riselab): minio not initialized (#97 by @skyzh)
• fix(batch): use shuffle aggregation for distinct aggregation (#125 by @lmatz)
• fix(frontend): use original string for BigDecimal instead of parseDouble (#129 by @lmatz)
• fix(batch): incorrect schema for RowSeqScan executor (#134 by @BugenZhao)
• fix(streaming): ignore notifier send error of local barrier manager (#152 by @BugenZhao)
• fix(streaming): shared executor keyspace (#156 by @yezizp2012)
• fix(mview): fix dispatcher updates when sharing upstream mview (#183 by @MrCroxx)
• fix(batch): COUNT(*) from mview table (#182 by @BugenZhao)
• fix(tracing): barrier aggregation tracing (#187 by @skyzh)
• fix(storage): mview scan will miss keys (#184 by @skyzh)
• fix: PG protocol frontend text serialization failed. (#219 by @liurenjie1024)
• fix: set reasonable intervals for state store stats (#229 by @Sunt-ing)
• fix(hummock): correct thread dependency implementation (#242 by @Sunt-ing)
• fix(storage): return error instead of panic in simple manager (#248 by @skyzh)
• fix(storage): do not open port for prometheus test (#250 by @skyzh)
• fix(streaming): allow only stateless aggregator to be parallelized (#241 by @TennyZhuang)
• fix: fix MockHummockMetaClient (#260 by @zwang28)
• fix(frontend): occasional table v2 create error (#267 by @BugenZhao)
• fix: disable source e2e tests temporarily (#287 by @fuyufjh)
• fix(dashboard): mv on mv with multiple nodes support (#305 by @BugenZhao)
• fix: ignore the failed case in source e2e test (#298 by @fuyufjh)
• fix: ignore the failed case in source e2e test (#313 by @MrCroxx)
• fix: fix uneeded async func def (#315 by @yezizp2012)
• fix(dashboard): fix performance issue (#317 by @skyzh)
• fix(meta): fix inconsistency when delete table fragments fail in metastore (#348 by @yezizp2012)
• fix(hummock): add full key to multi builder initially while disallowing split (#349 by @BugenZhao)
• fix(stream): LocalSimpleAgg should not output Chunk while no inputs (#354 by @TennyZhuang)
• fix: enable block cache for hummock+s3 state store (#355 by @wcy-fdu)
• fix(streaming): simple dispatcher should not use 0 proto (#356 by @skyzh)
• fix: MetaStore transaction should expose cf as parameter. (#358 by @zwang28)
• fix(dashboard): restore actor addr display (#368 by @MrCroxx)
• fix(doc): remove unused dependency (#370 by @BowenXiao1999)
• fix(storage): fix TiKV state store (#350 by @hzxa21)
• fix: fix tikv statestore scanning with None limit (#395 by @wcy-fdu)
• fix(dashboard): meta store trait & resolve mv on mv (#400 by @BugenZhao)
• fix: remove a dummy file, out (#452 by @lmatz)
• fix(streaming): regression caused by #444 (#460 by @skyzh)
• fix(metrics): set batch_write latency to a more realistic value (#475 by @Sunt-ing)
• fix(frontend): typo and index shifting in join (#487 by @lmatz)
• fix: replace rate with irate for grafana real-time monitoring (#505 by @wcy-fdu)
• fix(ci): use longer timeout for source e2e (#533 by @BugenZhao)
• fix(frontend): clippy warning (#550 by @skyzh)

Refactors

• refactor: use quantile to monitor latency on grafana dashboards (#4 by @wcy-fdu)
• refactor: implement datetime arrays (#11 by @HuaHuaY)
• refactor: refine meta transaction impl (#51 by @zwang28)
• refactor(frontend): simplify/rename BoundExpr to Expr (#74 by @xiangjinwu)
• refactor: simplify rand_array (#77 by @HuaHuaY)
• refactor(common): split agg into multiple files under new mod (#78 by @lmatz)
• refactor(streaming): move SharedContext to mod.rs (#89 by @wyhyhyhyh)
• refactor: rename DataTypeKind to DataType (#99 by @fuyufjh)
• refactor(riselab): rename to risedev (#101 by @skyzh)
• refactor: modeling stream fragment (#122 by @yezizp2012)
• refactor: remove unused proto msg (#131 by @yezizp2012)
• refactor: remove get_stats_ref in storage (#120 by @wcy-fdu)
• refactor(meta): remove unused cf config in config.rs (#139 by @yezizp2012)
• refactor(stream): rename node_id to operator_id (#153 by @TennyZhuang)
• refactor(catalog): remove primary keys and use Map instead of vector (#144 by @BowenXiao1999)
• refactor(meta): use type alias instead of primitive type for different ids (#158 by @BugenZhao)
• refactor: simplify HummockContext management (#136 by @zwang28)
• refactor: remove config field in MetaSrvEnv (#193 by @zwang28)
• refactor: fix grafana dashboards (#220 by @wcy-fdu)
• refactor(streaming): combine local channel pool and remote channel pool (#224 by @wyhyhyhyh)
• refactor(dashboard-ui): refactor dashboard ui using nextjs (#227 by @mikechesterwang)
• refactor: implement MetadataModel trait for HummockManager (#233 by @zwang28)
• refactor(frontend): use LocalSimpleAgg in TwoPhaseAgg (#216 by @TennyZhuang)
• refactor: channel pool retain impl (#238 by @MrCroxx)
• refactor(meta): add default implementation for Transactional trait (#240 by @zwang28)
• refactor(ci): remove start_cluster.sh (#244 by @skyzh)
• refactor(dashboard): add more comments and fix some bugs/typo (#247 by @mikechesterwang)
• refactor(stream): rename SimpleAgg to GlobalSimpleAgg (#258 by @TennyZhuang)
• refactor(frontend): create table v2 handler (#263 by @BugenZhao)
• refactor(expr): DRY in expression template (#269 by @TennyZhuang)
• refactor(frontend): pass database catalog snapshot to binder (#221 by @xiangjinwu)
• refactor(optimizer): change some declaration as reference (#271 by @st1page)
• refactor: capture variable in format macro (#281 by @skyzh)
• refactor: enable clippy await_holding_lock (#288 by @skyzh)
• refactor: make MetadataModel::select returns Option (#286 by @zwang28)
• refactor: add remote channel when building actors (#289 by @wyhyhyhyh)
• refactor(connector): kinesis credentials (#266 by @tabVersion)
• refactor(ss-bench): simplify code by macros (#314 by @wangrunji0408)
• refactor: remove unused types in storage crate. (#303 by @zwang28)
• refactor(meta): refine create mv and remove redundant codes (#325 by @MrCroxx)
• refactor(hummock): pass reference to sst builder (#347 by @skyzh)
• refactor(meta): prefer to use default implementation in MetaStore (#353 by @TennyZhuang)
• refactor(meta): remove MetaStore::put_batch[_cf] (#366 by @TennyZhuang)
• refactor(fragmenter): refine create mv ctx (#383 by @MrCroxx)
• refactor(meta): static dispatch MetaStore (#389 by @TennyZhuang)
• refactor(ss-bench): share data among operations (#371 by @Sunt-ing)
• refactor: remove table v1 from backend (#363 by @BugenZhao)
• refactor(source): remove SourceWriter from trait Source (#398 by @fuyufjh)
• refactor(meta): recursive visit in fragment building (#113 by @BugenZhao)
• refactor: unify TableScan and MaterializedViewScan (#417 by @BugenZhao)
• refactor: remove table_v1 in java frontend (#422 by @wyhyhyhyh)
• refactor(meta): refactor MetaStore and remove sled (#424 by @TennyZhuang)
• refactor(frontend): rename Expr::to_expr_impl (#425 by @st1page)
• refactor(ss-bench): make getseq use global data (#416 by @Sunt-ing)
• refactor(state): Use correct epoch value in executor for storage reads and writes (#399 by @hzxa21)
• refactor: clean stuff related to table source & several renames (#426 by @BugenZhao)
• refactor: Consolidate some server concepts. (#429 by @twocode)
• refactor(frontend): explain output as multiple rows (#445 by @xiangjinwu)
• refactor(streaming): remove total_count in JoinEntryState (#453 by @yuhao-su)
• refactor: remove unused fields in hummock RPCs (#433 by @zwang28)
• refactor(ss-bench): reduce code redundance in get_random and get_seq (#466 by @Sunt-ing)
• refactor(ss-bench): update batch_size related logics (#473 by @Sunt-ing)
• refactor: add 3-node monitor and gRPC meta monitor in grafana dashboards (#476 by @wcy-fdu)
• refactor(streaming): fix incorrect usage of ColumnDesc in MergeNode and ExchangeNode (#481 by @fuyufjh)
• refactor: OrderType: replace InputRef with column_idx & misc. (#483 by @fuyufjh)
• refactor(style): make column_at of Stream/DataChunk consistent (#488 by @fuyufjh)
• refactor: refactor table id (#504 by @st1page)
• refactor: introduce new type ColumnId (#509 by @fuyufjh)
• refactor: remove unused fields in ColumnDesc (#522 by @st1page)
• refactor: unify 2 OrderPairs (#527 by @fuyufjh)
• refactor(streaming): use column ids in MaterializeExecutor (#535 by @fuyufjh)
• refactor(hummock): decoupling upload and load SST (#538 by @MrCroxx)
• refactor(frontend): make LogicalScan explain output stable (#529 by @xiangjinwu)
• refactor: store worker id in meta client (#539 by @zwang28)
• refactor: serialize a single row in OrderedRowsSerializer (#544 by @lmatz)
• refactor: make all unit tests can be run concurrently (#520 by @TennyZhuang)

Tests

• test: add tpch q17 for batch and streaming (#8 by @lmatz)
• test: add tpch q16 for batch (#119 by @lmatz)
• test(batch): add tpch q18 for batch (#137 by @lmatz)
• test(streaming): add tpch q18 for streaming (#143 by @lmatz)
• test(batch): add tpch q2 for batch (#268 by @lmatz)
• test(batch): add tpch q19 for batch (#275 by @lmatz)
• test(batch): force table v1 for distributed batch e2e tests (#234 by @BugenZhao)
• test(compaction): test get_compaction on hummock_manager (#265 by @soundOfDestiny)
• test(batch): add tpch q20 for batch (#282 by @lmatz)
• test: add integration tests for compaction (#330 by @zwang28)
• test(streaming): add tpch q4 for streaming (#427 by @lmatz)
• test(streaming): add tpch q12 for streaming (#428 by @lmatz)
• test: support datadriven test for sqlparser (#438 by @neverchanje)
• test(streaming): add tpch q2 for streaming (#489 by @lmatz)
• test(streaming): add tpch q20 for streaming (#493 by @lmatz)

Others

• chore: fix cargo fmt (#1 by @neverchanje)
• chore: temporarily replace risingwave links (#2 by @Sunt-ing)
• chore: fix macro_rules indent to 4 space (#3 by @skyzh)
• chore: fix codecov after repo migration (#21 by @fuyufjh)
• chore(ci): add health check for kafka docker (#105 by @shanicky)
• style: fix clippy warnings 20220208 (#138 by @TennyZhuang)
• chore: use ghcr instead of dockerhub (#118 by @tabVersion)
• chore: bump dependencies (#161 by @skyzh)
• chore: reduce logs in release mode (#203 by @skyzh)
• chore(ci): run source 1-node test with hummock (#261 by @skyzh)
• style: enable clippy::no_effect_underscore_binding (#264 by @TennyZhuang)
• chore: rename some workflows (#255 by @TennyZhuang)
• chore: add doc label (#301 by @tabVersion)
• chore(ci): extend ci timeout (#322 by @skyzh)
• chore(risedev): remove compute node config in frontend (#329 by @skyzh)
• chore(meta): refactor stream schedule (#339 by @yezizp2012)
• chore: some renaming in streaming (#341 by @wyhyhyhyh)
• chore: clean comments in meta (#346 by @wyhyhyhyh)
• chore(ci): use much stricter timeout to prevent performance degradation (#357 by @BugenZhao)
• chore: remove fasthash (#376 by @tabVersion)
• chore: clean all unused deps (#377 by @TennyZhuang)
• chore: improve PR template (#380 by @fuyufjh)
• chore(stream): separate identity and logical operator info (#418 by @lmatz)
• chore: Update default settings in README (#401 by @twocode)
• chore: missing hash join identity and show op info in jaeger (#444 by @lmatz)
• chore(ci): download sqllotictest bin directly (#446 by @MrCroxx)
• style: enable let_chains to simplify some codes … (#470 by @TennyZhuang)
• chore(ci): build dashboard and push it to another branch (#486 by @mikechesterwang)
• chore: abort on panic with backtrace (#543 by @skyzh)
• chore(storage): migrate to aws-s3-sdk (#549 by @skyzh)