questdb/questdb 7.3

on GitHub

Overview

Data deduplication

This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.

Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.

To control data deduplication the following SQL syntax is available:

-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts) 
PARTITION BY DAY WAL 
DEDUPLICATE UPSERT KEYS(ts, id)

-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);

-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);

We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.

New IPV4 data type

This release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.

What is new

CORE

• data deduplication by @ideoma in #3524, #3566
• add ipv4 as supported column type by @allegraharris in #3523
• enable prepending jvm flags from docker run by @yifanstefanwang in #3567
• make wal apply more fair in case of skewed transaction frequences by @ideoma in #3592

SQL

• implement function to convert two longs to UUID by @jerrinot in #3559

What is fixed

CORE

• fix wal to non-wal table conversion leaving table non-writable by @ideoma in #3572
• fix physical rows written metric on partition split by @ideoma in #3577
• fix vacuum command deleting live table columns by @ideoma in #3589

SQL

• fix exception updating symbol column with char value by @ideoma in #3578
• unexpected NullPointerException on COUNT(SELECT ...) query by @puzpuzpuz in #3565
• fix sql bugs by @bziobrowski in #3479
• fix data loss causes by concurrent column drop transactions by @bluestreak01 in #3583
• fix race condition in ILP authentication by @jerrinot in #3599
• fix Invalid Column error with multiple JOIN by @bziobrowski in #3604
• limit reordering of left join-ed tables by @bziobrowski in #3606
• fix string and char comparison by @bziobrowski in #3607

UI

• fix incorrect rendering of last few columns in Web Console grid

New Contributors

• @auula made their first contribution in #3593
• @yifanstefanwang made their first contribution in #3567
• @allegraharris made their first contribution in #3523

Full Changelog: 7.2.1...7.3