Summary

This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.

The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9

Support Segment Merge and Roll-up

LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.

To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.

At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.

Major Changes:

• Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)
• Merge/Rollup task scheduler for offline tables. (#7178)
• Fix MergeRollupTask uploading segments not updating their metadata (#7289)
• MergeRollupTask integration tests (#7283)
• Add mergeRollupTask delay metrics (#7368)
• MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)
• Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)

UI Improvement

This release also sees improvements to Pinot’s query console UI.

• Cmd+Enter shortcut to run query in query console (#7359)
• Showing tooltip in SQL Editor (#7387)
• Make the SQL Editor box expandable (#7381)
• Fix tables ordering by number of segments (#7564)

SQL Improvements

There have also been improvements and additions to Pinot’s SQL implementation.

New functions:

• IN (#7542)
• LASTWITHTIME (#7584)
• ID_SET on MV columns (#7355)
• Raw results for Percentile TDigest and Est (#7226),
• Add timezone as argument in function toDateTime (#7552)

New predicates are supported:

• LIKE(#7214)
• REGEXP_EXTRACT(#7114)
• FILTER(#7566)

Query compatibility improvements:

• Infer data type for Literal (#7332)
• Support logical identifier in predicate (#7347)
• Support JSON queries with top-level array path expression. (#7511)
• Support configurable group by trim size to improve results accuracy (#7241)

Performance Improvements

This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:

• Reduce the disk usage for segment conversion task (#7193)
• Simplify association between Java Class and PinotDataType for faster mapping (#7402)
• Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)
• Replace MINUS with STRCMP (#7394)
• Bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)
• Use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)
• Add option to limit thread usage per query (#7492)
• Improved range queries (#7513)
• Faster bitmap scans (#7530)
• Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)
• Map bitmaps through a bounded window to avoid excessive disk pressure (#7535)
• Allow RLE compression of bitmaps for smaller file sizes (#7582)
• Support raw index properties for columns with JSON and RANGE indexes (#7615)
• Enhance BloomFilter rule to include IN predicate(#7444) (#7624)
• Introduce LZ4_WITH_LENGTH chunk compression type (#7655)
• Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)
• Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)
• During segment pruning, release the bloom filter after each segment is processed (#7668)
• Fix JSONPath cache inefficient issue (#7409)
• Optimize getUnpaddedString with SWAR padding search (#7708)
• Lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)
• Inline binary comparison ops to prevent function call overhead (#7709)
• Memoize literals in query context in order to deduplicate them (#7720)

Other Notable New Features and Changes

• Human Readable Controller Configs (#7173)
• Add the support of geoToH3 function (#7182)
• Add Apache Pulsar as Pinot Plugin (#7223) (#7247)
• Add dropwizard metrics plugin (#7263)
• Introduce OR Predicate Execution On Star Tree Index (#7184)
• Allow to extract values from array of objects with jsonPathArray (#7208)
• Add Realtime table metadata and indexes API. (#7169)
• Support array with mixing data types (#7234)
• Support force download segment in reload API (#7249)
• Show uncompressed znRecord from zk api (#7304)
• Add debug endpoint to get minion task status. (#7300)
• Validate CSV Header For Configured Delimiter (#7237)
• Add auth tokens and user/password support to ingestion job command (#7233)
• Add option to store the hash of the upsert primary key (#7246)
• Add null support for time column (#7269)
• Add mode aggregation function (#7318)
• Support disable swagger in Pinot servers (#7341)
• Delete metadata properly on table deletion (#7329)
• Add basic Obfuscator Support (#7407)
• Add AWS sts dependency to enable auth using web identity token. (#7017)(#7445)
• Mask credentials in debug endpoint /appconfigs (#7452)
• Fix /sql query endpoint now compatible with auth (#7230)
• Fix case sensitive issue in BasicAuthPrincipal permission check (#7354)
• Fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)
• Add segmentNameGeneratorType config to IndexingConfig (#7346)
• Support trigger PeriodicTask manually (#7174)
• Add endpoint to check minion task status for a single task. (#7353)
• Showing partial status of segment and counting CONSUMING state as good segment status (#7327)
• Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)
• Check schema backward-compatibility when updating schema through addSchema with override (#7374)
• Optimize IndexedTable (#7373)
• Support indices remove in V3 segment format (#7301)
• Optimize TableResizer (#7392)
• Introduce resultSize in IndexedTable (#7420)
• Offset based realtime consumption status checker (#7267)
• Add causes to stack trace return (#7460)
• Create controller resource packages config key (#7488)
• Enhance TableCache to support schema name different from table name (#7525)
• Add validation for realtimeToOffline task (#7523)
• Unify CombineOperator multi-threading logic (#7450)
• Support no downtime rebalance for table with 1 replica in TableRebalancer (#7532)
• Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)
• Adjust tuner api (#7553)
• Adding config for metrics library (#7551)
• Add geo type conversion scalar functions (#7573)
• Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)
• Add MV raw forward index and MV BYTES data type (#7595)
• Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)
• Improve get tenant API to differentiate offline and realtime tenants (#7548)
• Refactor query rewriter to interfaces and implementations to allow customization (#7576)
• In ServiceStartable, apply global cluster config in ZK to instance config (#7593)
• Make dimension tables creation bypass tenant validation (#7559)
• Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)
• Reject query with identifiers not in schema (#7590)
• Round Robin IP addresses when retry uploading/downloading segments (#7585)
• Support multi-value derived column in offline table reload (#7632)
• Support segmentNamePostfix in segment name (#7646)
• Add select segments API (#7651)
• Controller getTableInstance() call now returns the list of live brokers of a table. (#7556)
• Allow MV Field Support For Raw Columns in Text Indices (#7638)
• Allow override distinctCount to segmentPartitionedDistinctCount (#7664)
• Add a quick start with both UPSERT and JSON index (#7669)
• Add revertSegmentReplacement API (#7662)
• Smooth segment reloading with non blocking semantic (#7675)
• Clear the reused record in PartitionUpsertMetadataManager (#7676)
• Replace args4j with picocli (#7665)
• Handle datetime column consistently (#7645)(#7705)
• Allow to carry headers with query requests (#7696) (#7712)
• Allow adding JSON data type for dimension column types (#7718)
• Separate SegmentDirectoryLoader and tierBackend concepts (#7737)
• Implement size balanced V4 raw chunk format (#7661)
• Add presto-pinot-driver lib (#7384)

Major Bug fixes

• Fix null pointer exception for non-existed metric columns in schema for JDBC driver (#7175)
• Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)
• Fixed pinot java client to add zkClient close (#7196)
• Ignore query json parse errors (#7165)
• Fix shutdown hook for PinotServiceManager (#7251) (#7253)
• Make STRING to BOOLEAN data type change as backward compatible schema change (#7259)
• Replace gcp hardcoded values with generic annotations (#6985)
• Fix segment conversion executor for in-place conversion (#7265)
• Fix reporting consuming rate when the Kafka partition level consumer isn't stopped (#7322)
• Fix the issue with concurrent modification for segment lineage (#7343)
• Fix TableNotFound error message in PinotHelixResourceManager (#7340)
• Fix upload LLC segment endpoint truncated download URL (#7361)
• Fix task scheduling on table update (#7362)
• Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)
• Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema (#7366)
• Fix currentOffset volatility in consuming segment(#7365)
• Fix misleading error msg for missing URI (#7367)
• Fix the correctness of getColumnIndices method (#7370)
• Fix SegmentZKMetadta time handling (#7375)
• Fix retention for cleaning up segment lineage (#7424)
• Fix segment generator to not return illegal filenames (#7085)
• Fix missing LLC segments in segment store by adding controller periodic task to upload them (#6778)
• Fix parsing error messages returned to FileUploadDownloadClient (#7428)
• Fix manifest scan which drives /version endpoint (#7456)
• Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)
• Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)
• Fix pql double quote checker exception (#7485)
• Fix minion metrics exporter config (#7496)
• Fix segment unable to retry issue by catching timeout exception during segment replace (#7509)
• Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)
• Fix segment generation commands (#7527)
• Return non zero from main with exception (#7482)
• Fix parquet plugin shading error (#7570)
• Fix the lowest partition id is not 0 for LLC (#7066)
• Fix star-tree index map when column name contains '.' (#7623)
• Fix cluster manager URLs encoding issue(#7639)
• Fix fieldConfig nullable validation (#7648)
• Fix verifyHostname issue in FileUploadDownloadClient (#7703)
• Fix TableCache schema to include the built-in virtual columns (#7706)
• Fix DISTINCT with AS function (#7678)
• Fix SDF pattern in DataPreprocessingHelper (#7721)
• Fix fields missing issue in the source in ParquetNativeRecordReader (#7742)