GreptimeTeam/greptimedb v0.3.0 on GitHub

What's Changed

Features

feat(datanode): iImplement the heartbeat response handler by @WenyXu in #1547
feat(mito): Combine the original and procedure's implementation by @evenyag in #1468
feat(servers): Add metrics based on axum's example by @evenyag in #1638
feat(servers): collect samples by metric by @evenyag in #1706
feat(tql): add initial implementation for explain & analyze by @etolbakov in #1427
feat: Add FlushPicker to flush regions periodically by @evenyag in #1559
feat: Add a global TTL option for all tables by @evenyag in #1679
feat: Add app version metric by @evenyag in #1685
feat: Add column supports at first or after the existing columns by @lizhemingi in #1621
feat: Add metrics for cache hit/miss for object store cache by @nearsyh in #1405
feat: Add request type and result code to grpc metrics by @evenyag in #1664
feat: Compress manifest and checkpoint by @Taylor-lagrange in #1497
feat: Drop table by procedure by @evenyag in #1401
feat: Log error in GreptimeRequestHandler by @evenyag in #1507
feat: Record process metrics by @evenyag in #1646
feat: Remove store from procedure config by @evenyag in #1489
feat: Switch to the procedure framework by @evenyag in #1448
feat: Trigger flush based on global write buffer size by @evenyag in #1585
feat: Use LocalScheduler framework to implement FlushScheduler by @evenyag in #1531
feat: acquire table_id from region_id by @WenyXu in #1656
feat: adapt window reader to order rules by @v0y4g3r in #1671
feat: add WindowedReader by @v0y4g3r in #1532
feat: add cache for catalog kv backend by @Fengys123 in #1592
feat: add catalog/schema/table count as catalog metrics by @sunng87 in #1499
feat: add delete WAL in drop_region by @DevilExileSu in #1577
feat: add information_schema as exception of cross schema check by @sunng87 in #1551
feat: add invalidate cache step by @WenyXu in #1658
feat: add invalidate table cache handler by @WenyXu in #1633
feat: add metrics for ingested row count by @sunng87 in #1645
feat: add metrics for protocol interfaces by @sunng87 in #1495
feat: add semantic_type to information_schema.columns by @sunng87 in #1530
feat: add support for information_schema.columns by @sunng87 in #1500
feat: add table id and engine to information_schema.TABLES by @haohuaijin in #1407
feat: adopt REPLACE interceptor and quit all processes on exit by @waynexia in #1478
feat: broadcast with mailbox by @fengjiachun in #1661
feat: bump opendal to 0.33 by @killme2008 in #1479
feat: change default selector in meta from "LeaseBased" to "LoadBased" by @Fengys123 in #1598
feat: clean procedure's state after it is done by @evenyag in #1543
feat: convert values to vector directly by @waynexia in #1704
feat: distributed planner basic by @waynexia in #1599
feat: enable tokio console in cluster mode by @MichaelScofield in #1512
feat: exists API for KVStore by @fengjiachun in #1695
feat: expose output_ordering on scan plan by @waynexia in #1425
feat: ignore mailbox message into stat by @fengjiachun in #1560
feat: impl instant query and add tests by @waynexia in #1452
feat: impl instant query interface by @waynexia in #1410
feat: impl literal only PromQL query by @waynexia in #1641
feat: implement /api/v1/labels for prometheus by @haohuaijin in #1580
feat: implement CloseRegionHandler by @WenyXu in #1569
feat: implement JsonOpener and CsvOpener by @WenyXu in #1367
feat: implement OpenTableHandler by @WenyXu in #1567
feat: implement ParquetFileReaderFactory by @WenyXu in #1423
feat: implement Prometheus-compatible API in gRPC by @waynexia in #1449
feat: implement COPY for cluster by @MichaelScofield in #1388
feat: implement parsing format from hashmap by @WenyXu in #1420
feat: improve and distinguish different errors for IllegalInsertData by @NiwakaDev in #1503
feat: improve metrics and log level by @sunng87 in #1470
feat: insert with stream by @fengjiachun in #1703
feat: make RepeatedTask invoke remove_outdated_meta method by @NiwakaDev in #1578
feat: make grpc insert requests in a batch by @MichaelScofield in #1687
feat: make log level and destination configurable from config files by @DevilExileSu in #1444
feat: manage multiple engine procedure in the engine manager by @evenyag in #1434
feat: memtable stats by @v0y4g3r in #1591
feat: merge scan for distributed execution by @waynexia in #1660
feat: meta metrics by @fengjiachun in #1538
feat: metasrv mailbox by @fengjiachun in #1481
feat: meter write request by @shuiyisong in #1447
feat: metrics for storage engine by @sunng87 in #1574
feat: migrate substrait to datafusion official implementation by @waynexia in #1238
feat: more configurable logging levels by @sunng87 in #1630
feat: open catalogs and schemas in parallel by @gitccl in #1527
feat: optimizer rule to pass expected output ordering hint by @waynexia in #1675
feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain by @DevilExileSu in #1501
feat: procedures for file table engine by @evenyag in #1417
feat: region failover procedure by @MichaelScofield in #1558
feat: serialize/deserialize support for PromQL plans by @waynexia in #1684
feat: show create table by @killme2008 in #1336
feat: sql dialect for different protocols by @killme2008 in #1631
feat: support "delete" in distributed mode by @MichaelScofield in #1441
feat: support /api/v1/label/<label_name>/values from Prometheus by @gitccl in #1604
feat: support azblob storage. by @QuenKar in #1659
feat: support creating the physical plan for JSON and CSV files by @WenyXu in #1424
feat: support frontend-meta heartbeat by @Fengys123 in #1555
feat: support to copy table from/to CSV and JSON format file by @WenyXu in #1475
feat: support to create external table by @WenyXu in #1372
feat: support to create parquet format external table by @WenyXu in #1463
feat: support to load missing region by @WenyXu in #1651
feat: support user config manifest compression by @Taylor-lagrange in #1579
feat: supports CTE query by @killme2008 in #1674
feat: table metrics by @discord9 in #1469
feat: time_zone variable for mysql connections by @sunng87 in #1607
feat: update dashboard to v0.2.2 by @ZonaHex in #1426
feat: update dashboard to v0.2.3 by @ZonaHex in #1430
feat: update dashboard to v0.2.4 by @ZonaHex in #1553
feat: update dashboard to v0.2.5 by @ZonaHex in #1563
feat: update dashboard to v0.2.6 by @ZonaHex in #1700
feat: update pgwire to 0.14 by @sunng87 in #1504
feat: update table metadata in lock by @MichaelScofield in #1634
feat: use server time if it's not specified by @waynexia in #1480
feat: window inferer by @v0y4g3r in #1648

Breaking Changes

feat!: reorganize the storage layout by @killme2008 in #1609

Fixes

fix(table-procedure): Open table in RegisterCatalog state by @evenyag in #1617
fix: unwrap() None in NULL value exist multi-field table during prometheus query_range by @haohuaijin in #1571
fix: Copy from must follow the order of table fields issue by @WenyXu in #1521
fix: Print source error in subprocedure failure message by @evenyag in #1683
fix: add log on leader stepdown by @fengjiachun in #1450
fix: add s3 region info by @WenyXu in #1486
fix: add s3 region info by @WenyXu in #1492
fix: allow cross-schema query on information_schema by @sunng87 in #1568
fix: avoid writing manifest and wal if no files are actually flushed by @v0y4g3r in #1698
fix: checkpoint GC task also deletes the file with the last version by @evenyag in #1491
fix: clippy warning by @waynexia in #1669
fix: correct schema/table count in catalog metrics by @sunng87 in #1565
fix: disable dashboard by @v0y4g3r in #1494
fix: do not alias relation before join by @waynexia in #1693
fix: do not change timestamp index column while planning aggr by @waynexia in #1688
fix: enable region failover test by @WenyXu in #1699
fix: execute sql query in another catalog by @Fengys123 in #1457
fix: failed to insert data with u8 by @Fengys123 in #1701
fix: fix datanode cannot start while failing to open tables by @WenyXu in #1601
fix: fix doesn't release closed regions issue by @WenyXu in #1596
fix: fix type casting issue by @WenyXu in #1652
fix: fix typo by @WenyXu in #1649
fix: frontend opt should respect http addr in config file when no com… by @v0y4g3r in #1456
fix: grpc client keepalive by @v0y4g3r in #1461
fix: incorrect show create table output by @NiwakaDev in #1514
fix: insert distributed table if partition column has default value by @NiwakaDev in #1498
fix: install python3-pip in Dockerfile by @yaoyinnan in #1644
fix: invalidate route cache on renaming table by @Fengys123 in #1691
fix: invalidate table route cache by @WenyXu in #1663
fix: meta admin API get catalog table name error by @paomian in #1603
fix: no active datanode when frontend start by @Fengys123 in #1533
fix: object store cache bug by @killme2008 in #1482
fix: object store caching bug, #1466 by @killme2008 in #1467
fix: prepare statement doesn't support insert clause by @killme2008 in #1680
fix: recompile&register scripts as UDF on reboot by @discord9 in #1421
fix: refreshing Dashboard returns 404 by @MichaelScofield in #1562
fix: remove region number validation by @WenyXu in #1593
fix: respect MySQL timestamp format by @v0y4g3r in #1510
fix: sequence out of range by @fengjiachun in #1597
fix: sqlness failed due to region failover by @MichaelScofield in #1690
fix: support restart sqlness in distributed mode by @waynexia in #1443
fix: table scan without projection by @waynexia in #1404
fix: the latest number of regions by @fengjiachun in #1546
fix: uses nextest in the Release CI by @waynexia in #1582
fix: vector and matrix in Prometheus use different field by @waynexia in #1520
fix: wal replay ignore manifest entries by @v0y4g3r in #1612
fix: wrong max_table_id log in remote catalog manager by @haohuaijin in #1516
fix: wrong next column in manifest by @lizhemingi in #1440

Others

build: Download assets to cargo output dir by @evenyag in #1476
build: change release CI to only run test on linux by @waynexia in #1548
chore(datasource): make CompressionType follow the style of the guide by @WenyXu in #1522
chore(deps): bump arrow/parquet to 40.0, datafuson to the latest HEAD by @waynexia in #1677
chore(deps): update substrait dep in client by @waynexia in #1453
chore(toolchain): update rust-toolchain to 2023-05-03 by @waynexia in #1524
chore: Bump OpenDAL to v0.36 by @Xuanwo in #1678
chore: add len() to Bytes and StringBytes by @shuiyisong in #1455
chore: add a uniform prefix to the metrics using the official recommendation of by @paomian in #1590
chore: add boxed error for custom error map by @shuiyisong in #1635
chore: add configurator to http server by @paomian in #1488
chore: add fmt for statement query by @paomian in #1588
chore: add grpc query interceptor by @shuiyisong in #1626
chore: add metrics for table route getting by @Fengys123 in #1636
chore: add some metrics for grpc client by @paomian in #1398
chore: don't render reproduce as shell in issue template by @waynexia in #1397
chore: fix code styling by @WenyXu in #1623
chore: fix some typo and add deriv to plan in promql by @haohuaijin in #1438
chore: ignore symbol link target file by @waynexia in #1696
chore: reduce the number of requests for meta by @Fengys123 in #1647
chore: reject table creation when partitions exceeds peer number by @Fengys123 in #1654
chore: release 0.2.0 by @v0y4g3r in #1413
chore: remove Release prefix from release name by @v0y4g3r in #1409
chore: remove info log by @WenyXu in #1483
chore: remove redundant code. by @DevilExileSu in #1502
chore: remove useless Option type in plugins by @paomian in #1544
chore: truncate route-table by @fengjiachun in #1619
chore: unify code styling by @WenyXu in #1523
chore: update RustPython depend by @discord9 in #1406
chore: upgrade promql-parser version by @yuanbohan in #1484
chore: use alicloud imagehub by @daviderli614 in #1418
ci: always upload sqlness log by @waynexia in #1692
ci: enable test on release by @waynexia in #1428
ci: set whether it is the latest release by using 'ncipollo/release-action and update install.sh by @zyy17 in #1400
ci: upgrade nightly release tag from v0.2.0 to v0.3.0 by @zyy17 in #1446
docs: Add docs about schema structs by @evenyag in #1373
docs: add play section by @sunng87 in #1528
docs: fix timestamp rendering in readme by @v0y4g3r in #1399
docs: rfc of distributed planner by @waynexia in #1554
docs: update readme by @killme2008 in #1549
refactor(frontend): adjust code structure by @waynexia in #1615
refactor: Holds histogram in the timer to avoid clone labels if possible by @evenyag in #1653
refactor: add "table engine" to datanode heartbeat. by @QuenKar in #1616
refactor: add scan_to_stream() to Table trait to postpone the stream generation by @waynexia in #1639
refactor: catalog by @v0y4g3r in #1454
refactor: change mailbox_messages to mailbox_message by @WenyXu in #1557
refactor: default catalog and schema are created at Metasrv by @Fengys123 in #1391
refactor: move code out of loop by @WenyXu in #1657
refactor: move rpc's commons to common-meta by @fengjiachun in #1625
refactor: move the common part of the heartbeat response handler to common by @WenyXu in #1627
refactor: parallelize open table by @shuiyisong in #1392
refactor: refactor BufferedWriter by @WenyXu in #1439
refactor: refactor stmt_options_to_table_options by @WenyXu in #1403
refactor: remove redundant plugins argument by @sunng87 in #1436
refactor: remove unnecessary async from RepeatedTask::start by @evenyag in #1545
refactor: remove useless error by @v0y4g3r in #1624
refactor: remove version column by @v0y4g3r in #1576
refactor: replace Copy Format with datasource Format by @WenyXu in #1435
refactor: set the filters for testing logs by @MichaelScofield in #1637
refactor: support layered configuration by @zyy17 in #1535
refactor: support parsing env list by @zyy17 in #1595
refactor: update proto and rename incorrect region_id fields by @evenyag in #1670
test: Fix s3 region in test by @evenyag in #1493
test: add SELECT ... LIMIT ... test cases for distributed mode by @MichaelScofield in #1419
test: add basic cases for distributed TQL by @waynexia in #1437
test: add integration case to check dashboard path by @waynexia in #1422
test: add tests for external table by @WenyXu in #1460
test: add write test for region failover by @WenyXu in #1673
test: fix sqlx compatibility and adds integration test for sqlx by @sunng87 in #1686
test: move instances tests to "tests-integration" by @MichaelScofield in #1573

New Contributors

@nearsyh made their first contribution in #1405
@daviderli614 made their first contribution in #1418
@DevilExileSu made their first contribution in #1444
@NiwakaDev made their first contribution in #1498
@gitccl made their first contribution in #1527
@Taylor-lagrange made their first contribution in #1497
@QuenKar made their first contribution in #1616
@yaoyinnan made their first contribution in #1644

Full Changelog: v0.2.0...v0.3.0