What's Changed
Highlights
- Metric Engine: handles a large number of small tables, making it particularly suitable for cloud-native monitoring.
- Region Migration: enhances the user experience and simplifies region migrations with straightforward SQL commands.
- Inverted Index: dramatically improves the efficiency of locating data segments relevant to user queries, significantly reducing the IO operations needed for scanning data files and thus accelerating the query process.
Features:
- feat: add modulo function by @etolbakov in #3147
- feat: support HTTP&gRPC&pg set timezone by @Taylor-lagrange in #3125
- feat(mito): enable inverted index by @zhongzc in #3158
- feat: upgrade pgwire to 0.19 by @sunng87 in #3157
- feat: let tables API return a stream by @fengjiachun in #3170
- feat: add tests-fuzz crate by @WenyXu in #3173
- feat: tables stream with CatalogManager by @fengjiachun in #3180
- feat: adds date_format function by @killme2008 in #3167
- feat: auto config cache size according to memory size by @QuenKar in #3165
- feat: precise filter for mito parquet reader by @waynexia in #3178
- feat: adds parse options for SQL parser by @killme2008 in #3193
- feat(tests-fuzz): add CreateTableExprGenerator & AlterTableExprGenerator by @WenyXu in #3182
- feat: make procedure able to return output by @WenyXu in #3201
- feat: copy database from by @v0y4g3r in #3164
- feat: introduce information schema provider cache by @WenyXu in #3208
- feat: lazy initialize vector builder on write by @waynexia in #3210
- feat: make query be aware of timezone setting by @killme2008 in #3175
- feat: add create alter table expr translator by @WenyXu in #3203
- feat: read column and region info from state cache by @waynexia in #3222
- feat: enable concurrent write by @WenyXu in #3214
- feat: add Arrow IPC output format for http rest api by @sunng87 in #3177
- feat: change Range Query’s default align behavior aware of timezone by @killme2008 in #3219
- feat: update dashboard to v0.4.7 by @ZonaHex in #3229
- feat: http sql api return schema on empty resultset by @sunng87 in #3237
- feat: add pg create alter table expr translator by @WenyXu in #3206
- feat: read metadata from write cache by @QuenKar in #3224
- feat: batch create ddl by @fengjiachun in #3194
- feat: add insert/select generator & translator by @WenyXu in #3240
- feat: return request outdated error on handling alter by @waynexia in #3239
- feat: don't map semantic type in metric engine by @waynexia in #3243
- feat: create tables in batch on prom write by @fengjiachun in #3246
- feat: Only allow inserts and deletes operations to be executed in parallel by @fengjiachun in #3257
- feat: basic types for Dataflow Framework by @discord9 in #3186
- feat: initial configuration for grafana dashboard by @waynexia in #3263
- feat: support fraction part in timestamp by @discord9 in #3272
- feat: use simple filter to prune memtable by @waynexia in #3269
- feat: Basic Definitions for Expression&Functions for Dataflow by @discord9 in #3267
- feat: support cache for batch_get in CachedMetaKvBackend by @fengys1996 in #3277
- feat: put all filter exprs in a filter plan separately by @waynexia in #3288
- feat: administration functions by @killme2008 in #3236
- feat(mito): adjust seg size of inverted index to finer granularity instead of row group level by @zhongzc in #3289
- feat(mito): add options to ignore building index for specific column ids by @zhongzc in #3295
- feat: organize tracing on query path by @waynexia in #3310
- feat: impl partitions and region_peers information schema by @killme2008 in #3278
- feat: batch get physical table routes by @fengjiachun in #3319
- feat: Defines structs in the merge tree memtable by @evenyag in #3326
- feat(metric-engine): set index options for data region by @zhongzc in #3330
- feat: data buffer and related structs by @v0y4g3r in #3329
- feat: Implement KeyDictBuilder for the merge tree memtable by @evenyag in #3334
- feat: replace pk index with pk_weight during freeze by @v0y4g3r in #3343
- feat: merge tree data parts by @v0y4g3r in #3346
- feat: Defines more structs and methods for a partitioned merge tree by @evenyag in #3348
- feat(flow): impl ScalarExpr&Scalar Function by @discord9 in #3283
- feat: impl migrate_region and procedure_state SQL function by @killme2008 in #3325
- feat: add isnull function by @KKould in #3360
- feat: skip filling NULL for put and delete requests by @waynexia in #3364
- feat: impl merge reader for DataParts by @v0y4g3r in #3361
- feat: Implement write and fork for the new memtable by @evenyag in #3357
- feat: distinguish between different read paths by @v0y4g3r in #3369
- feat: Add freeze and fork method to the memtable by @evenyag in #3374
- feat: merge tree dedup reader by @v0y4g3r in #3375
- feat: Implement iter for the new memtable by @evenyag in #3373
- feat: Implement dedup for the new memtable and expose the config by @evenyag in #3377
- feat: change how region id maps to region worker by @waynexia in #3384
- feat: make tls certificates/keys reloadable (part 1) by @sunng87 in #3335
- feat(grafana): enable shared tooltip, add raft engine throughput by @waynexia in #3387
- feat: Implement partition eviction and only add value size to write buffer size by @evenyag in #3393
- feat: enable zstd compression and encodings in merge tree data part by @v0y4g3r in #3380
- feat(flow): impl for MapFilterProject by @discord9 in #3359
- feat: flush or compact table and region functions by @killme2008 in #3363
- feat: Use a partition level map to look up pk index by @evenyag in #3400
- feat(index): measure memory usage in global instead of single-column and add metrics by @zhongzc in #3383
- feat: enable ArrowFlight compression by @tisonkun in #3403
- feat: zero copy on split rows by @fengjiachun in #3407
- feat: Support automatic DNS lookup for kafka bootstrap servers by @J0HN50N133 in #3379
- feat: add configuration for tls watch option by @sunng87 in #3395
- feat: employ sparse key encoding for shard lookup by @v0y4g3r in #3410
- feat: support
Create Table ... Likeby @KKould in #3372 - feat: tableref cache by @fengjiachun in #3420
- feat: add verbose support for tql explain/analyze by @etolbakov in #3390
- feat: reduce a clone of string by @fengjiachun in #3422
- feat: Correct server metrics and add more metrics for scan by @evenyag in #3426
- feat: support tracing rule sampler by @Taylor-lagrange in #3405
- feat: decode prom requests to grpc by @v0y4g3r in #3425
- feat: implement multi-dim partition rule by @waynexia in #3409
Breaking Changes
- feat!: switch prom remote write to metric engine by @waynexia in #3198
- refactor!: rename initialize_region_in_background to init_regions_in_background by @WenyXu in #3216
- feat!: new partition grammar - parser part by @waynexia in #3347
Fixes
- fix: change back
GREPTIME_DB_HEADER_NAMEheader key name by @shuiyisong in #3184 - fix(index): S3
EntityTooSmallerror by @zhongzc in #3192 - fix: remove name matcher from processed matcher list by @waynexia in #3213
- fix: fix default value cannot accept negative number by @WenyXu in #3217
- fix: only register region keeper while creating physical table by @WenyXu in #3223
- fix: fix MockInstance rebuild issue by @WenyXu in #3218
- fix: security update for shlex and h2 by @sunng87 in #3227
- fix: fix create table ddl return incorrect table id by @WenyXu in #3232
- fix: init parquet reader metrics twice by @evenyag in #3242
- fix: IntermediateWriter closes underlying writer twice by @waynexia in #3248
- fix: decouple columns in projection and prune by @waynexia in #3253
- fix(Copy From): fix incorrect type casts by @WenyXu in #3264
- fix: SQL insertion and column default constraint aware of timezone by @killme2008 in #3266
- fix: cli export database default value by @tisonkun in #3259
- fix: use
fe_optsaftersetup_frontend_pluginsin standalone by @shuiyisong in #3275 - fix: fix incorrect StatusCode parsing by @WenyXu in #3281
- fix(util): join_path function should not trim leading
/by @dalprahcd in #3280 - fix: correct the case sensitivity behavior for PromQL by @waynexia in #3296
- fix: $TARGET_BIN not found when docker run the image by @daviderli614 in #3297
- fix(index): sanitize S3 upload buffer size by @zhongzc in #3300
- fix: commit_short sqlness test case by @tisonkun in #3313
- fix: bump libgit2-sys from 0.16.1+1.7.1 to 0.16.2+1.7.2 by @dependabot in #3316
- fix: split write metadata request by @fengjiachun in #3311
- fix(index): encode string type to original data to enable fst regex to work by @zhongzc in #3324
- fix: disable ansi contorl char when stdout is redirected by @waynexia in #3332
- fix: typo in lint config by @waynexia in #3358
- fix: treat "0" and "1" as valid boolean values. by @MichaelScofield in #3370
- fix: remove unused imports in memtable_util.rs by @v0y4g3r in #3376
- fix: resets dict builder keys counter and avoid unnecessary pruning by @evenyag in #3386
- fix: throw errors instead of panic by @WenyXu in #3391
- fix: some read metrics by @v0y4g3r in #3404
- fix: partition region id by @fengjiachun in #3414
- fix: show table names not complete from information_schema by @killme2008 in #3417
- fix: mitigate memory spike during startup by @niebayes in #3418
- fix: complete interceptors for all frontend entry by @shuiyisong in #3428
New Contributors
- @dalprahcd made their first contribution in #3280
- @caicancai made their first contribution in #3299
- @SteveLauC made their first contribution in #3352
- @KKould made their first contribution in #3360
- @J0HN50N133 made their first contribution in #3379
Full Changelog: v0.6.0...v0.7.0