apache/paimon release-2.1.0 on GitHub

Source releases

What's Changed

Use repository secrets for PyPI publishing by @JingsongLi in #8951
[core] Fix index source metadata compatibility by @JingsongLi in #8952
[core] Fix error classification for invalid bucket key by @XiaoHongbo-Hope in #8974
[python] Fix predicate pruning with incomplete statistics by @XiaoHongbo-Hope in #8978
Bump brace-expansion from 1.1.16 to 1.1.18 in /docs by @dependabot[bot] in #8984
[python] Fix benchmark package lint by @XiaoHongbo-Hope in #8986
[core] Fix forced L0 compaction level selection by @JingsongLi in #8993
[core][flink][spark] Introduce postpone fixed-bucket write builder by @JingsongLi in #8992
[flink] Speed up lookup join tests by @JingsongLi in #8997
[core] Avoid strict-mode conflicts for index-only compact by @LsomeYeah in #8996
[format] Fix reads and predicate pushdown when Parquet files differ from the declared schema by @sundapeng in #8995
[core] Fix chain table silent data loss for DV rows by @ArnavBalyan in #8990
[flink] Support SQL DELETE for data evolution tables by @XiaoHongbo-Hope in #8988
[orc] Support time type predicate push down for orc by @ArnavBalyan in #8983
[python] Parse overrides in ConfigResponse and fix merge priority by @wenhaozhao011-cmd in #8989
[format] Fix DELETE and UPDATE targeting wrong rows after Parquet predicate filtering by @XiaoHongbo-Hope in #8987
[docs] Fix procedure parameter names by @lilei1128 in #8975
[flink] Recover coordinator commit without intended failover by @ifndef-SleePy in #8970
[filesystem][s3] Fix multipart part upload missing SSE-C encryption parameters by @thswlsqls in #8969
[oss] Update bundled NOTICE for the aliyun-sdk-oss 3.17.4 upgrade by @thswlsqls in #8967
[spark] Fix mergedRecordSize truncation to zero by @cxzl25 in #8965
[core] Add conflict detection for concurrent merge-into & compaction by @steFaiz in #8964
[format][python] Support additional MAP BLOB key types by @QuakeWang in #8963
[obs] Add listFilesIterative implementation in HadoopCompliantFileIO by @thswlsqls in #8960
[filesystem][obs] Remove access key and secret from required options in OBSLoader by @thswlsqls in #8958
[docs] Add dls:// to the supported filesystems table by @thswlsqls in #8956
[fs] Fix JindoFileIO to return actual file access time and owner by @thswlsqls in #8955
[spark] Avoid scanning partition entries without partition_idle_time by @lilei1128 in #8932
[core] Add privilege checks for partition and consumer ops by @cxzl25 in #8929
[core] Limit index-only compact handling to data evolution by @LsomeYeah in #9001
[flink] Expose sorted index build stream by @leaves12138 in #8998
[core] Preserve index manifest compatibility by @JingsongLi in #9004
[format] Avoid lossy Parquet predicate pushdown by @JingsongLi in #8999
[vortex] Preserve physical row order in scans by @huwh in #9003
[python] Expand native Rust scan planning by @XiaoHongbo-Hope in #9000
[jindo] Disable directory timestamps for faster listings by @sundapeng in #9019
[core] Narrow format table partition listing by name pattern by @sundapeng in #9025
[spark] Make max_pt work on format tables by @sundapeng in #9024
Bump undici from 7.28.0 to 7.29.0 in /docs by @dependabot[bot] in #9028
[format] Reject incompatible annotated integers as BIGINT by @sundapeng in #9010
[core][flink][spark][python][vector] Support vector indexes on DV tables by @XiaoHongbo-Hope in #8930
[core] Support TopN pushdown for BTree global indexes by @discivigour in #8991
[parquet] Support boolean predicate push down for parquet by @ArnavBalyan in #8971
[core] Support full-text indexes on DV tables by @XiaoHongbo-Hope in #9031
Bump fast-uri from 3.1.4 to 3.1.5 in /docs by @dependabot[bot] in #9029
[python] Add Python 3.12, 3.13 to CI by @PyRSA in #9018
[python] Add CI coverage for native split planning by @XiaoHongbo-Hope in #9011
[format][python] Support TIME keys for MAP BLOB by @QuakeWang in #9009
[core] Fix dropping stats in data evolution scan by @leaves12138 in #8994
[python] Write postpone batches to fixed buckets by @XiaoHongbo-Hope in #8985
[python] Reuse manifest merge results on commit retry by @XiaoHongbo-Hope in #9040
[core] Support BLOB fields in partial-update tables. by @Stephen0421 in #8928
[core] Remove versioned serializer abstraction by @JingsongLi in #9039
[spark] Stage postpone fixed-bucket writes before rescale by @Zouxxyy in #9017
[flink] Clone Paimon action supports add additional table options by @yuzelin in #9042
[fs] Delegate tryToWriteAtomic through FileIO wrappers by @plusplusjiajia in #9034
[spark] Support skipping expired partitions during compaction by @lilei1128 in #9021
[core] Fix scan from tag on branch table. by @Stephen0421 in #9044
[spark] Recompute bucket number for postpone overwrite by @Zouxxyy in #9051
[core][python] Fix global index coverage for residual predicates by @XiaoHongbo-Hope in #9050
[flink] Fix Flink version detection for key-only deletes on non-release versions by @thswlsqls in #9053
[python] Plan data evolution row ranges with pypaimon-rust by @XiaoHongbo-Hope in #9038
[python] Group write rows by partition/bucket in Arrow to avoid GIL-bound per-row extraction by @yugan95 in #9047
[core][flink][spark] Refactor global index building and support incremental refresh by @leaves12138 in #8818
[core] Cache projected schemas for data evolution stats by @leaves12138 in #9046
[core] Optimize BTree negative predicate complement by @zhoulii in #8945
[core] Centralize negative predicate complements in sorted index readers by @JingsongLi in #9060
[spark] Cache blob field lookup in SparkInternalRow by @zhuxiangyi in #8943
[core] Fix file operation thread limit by @huangxiaopingRD in #9082
[hive] Support multiset type in hive writes by @thswlsqls in #9081
[hive] Unescape partition path values when building Hive split predicate by @thswlsqls in #9080
[hive] Fix ClassCastException when reading BLOB columns by @thswlsqls in #9079
[hive] Fix Hive read failure for CHAR/VARCHAR columns exceeding Hive length limits by @thswlsqls in #9075
[hive] Normalize column comments written to Hive metastore by @thswlsqls in #9073
[fs] Delegate listFilesIterative through RESTTokenFileIO by @tonymtu in #9071
[spark] Improve unresolved expression errors in MERGE INTO by @Zouxxyy in #9068
[core][flink][spark] Support default bucket number for postpone writes by @Zouxxyy in #9067
[avro] Fix ClassCastException when writing MULTISET with non-string element type by @thswlsqls in #9066
[format] Skip the CSV header only in the split that contains it by @thswlsqls in #9065
[docs] Fix full-text index option defaults and search query examples by @thswlsqls in #9062
[flink] Report which required argument is missing in action factories by @thswlsqls in #9059
[format][python] Support binary keys for MAP BLOB by @QuakeWang in #9057
[cdc] Fix NPE when metadata columns are used with debezium-bson format by @thswlsqls in #9055
[core] Tag diff query at postpone table should check bucket number by @yuzelin in #9048
[core] Validate data types for max/min aggregate functions by @jackylee-ch in #9013
[python] Align atomic write temporary file naming by @zhoulii in #9027
[docs] Align scalar index search docs with code by @huangxiaopingRD in #9084
[cdc] Support PostgreSQL jsonb type in postgres sync by @ArnavBalyan in #9020
[core] Abort single file writers on unchecked close failures by @wombatu-kun in #9016
[core] Fix wrong privilege type in NoPrivilegeException for assertCanCreateDatabase by @cxzl25 in #9015
[spark][flink] Fix vector-field column conversion in catalogs by @jackylee-ch in #9014
[python] Defer BLOB payload reads until after filtering by @XiaoHongbo-Hope in #8808
[core] Share JDBC connection pool across catalog instances in same JVM by @nickdelnano in #8323
[core] Fix SnapshotManager watermark binary searches with null watermarks by @u70b3 in #9037
[python] Inherit watermark in PyPaimon commits by @JingsongLi in #9090
[core][mosaic] Preserve projected raw reads by @jianguotian in #9086
[spark] Rebase data evolution merge updates after compaction by @JingsongLi in #9091
[spark] Remove upsert-key support by @JingsongLi in #9093
[spark][python] Preserve rows when rebasing stale updates by @JingsongLi in #9096
[core][flink][python] Support nullable primary keys by @JingsongLi in #9094
[core] Reduce memory for update index by @leaves12138 in #9092
[core] Support projected binary index manifest scans by @leaves12138 in #9095
[python] Support DLF token path authentication by @liujiayi771 in #9089
[python] Fix native-plan CI with updated paimon-rust by @XiaoHongbo-Hope in #9109
[python][ray] Add row-id range processing API by @JingsongLi in #9112
[python] Forward scan options to native planner by @XiaoHongbo-Hope in #9124
[python] Avoid an extra file-size I/O for Blob reads by @XiaoHongbo-Hope in #9125
Bump nanoid from 3.3.16 to 3.3.18 in /docs by @dependabot[bot] in #9110
[core][python] Prune data evolution groups by merged stats by @XiaoHongbo-Hope in #9114
[python] Align IN predicates with SQL null semantics by @JingsongLi in #9132
[format] Fix RLE skipping at sparse row-range boundaries by @CrownChu in #9127
[format] Fix cumulative DELTA_BINARY_PACKED read count by @leaves12138 in #9137
[core] Add file index system table by @lilei1128 in #9138
[core] Fix partition timestamp pattern replacement for prefix variable names by @juntaozhang in #9136
[core] Honor query-level search-mode overrides by @CrownChu in #9128
[core] Carry the row count and byte size a format table writer already counted by @sundapeng in #9121
[core] Spell out what a negative partition statistic means by @sundapeng in #9120
[flink] Fix nullable primary key test across Flink versions by @QuakeWang in #9118
[rest] Do not replay a POST the server cannot absorb twice by @sundapeng in #9122
[common] Fix concurrent remote stream leak in CachingSeekableInputStream by @wombatu-kun in #9098
[core] Fix file index filtering with deletion vectors by @huangxiaopingRD in #9088
[iceberg] Fix Iceberg metadata after a rollback by @plusplusjiajia in #9036
[flink] Fix flaky test duplicate nullable PK IT by @ArnavBalyan in #9117
Fix #9113 Exception when reading Paimon table via Hive proxy user in … by @tyoushinya in #9116
[common] Fix timestamp partition values being written but not readable by @ArnavBalyan in #9115
[format][python] Reject duplicate MAP BLOB keys by @QuakeWang in #9107
[core] Reject Variant type for primary key and partition key by @ArnavBalyan in #9105
[core] Fix stale default value in sort-compaction.range-strategy description by @jackylee-ch in #9104
[lance] Reject unsupported element types nested in ARRAY by @jackylee-ch in #9102
[python] Reuse streams for coalesced blob reads by @XiaoHongbo-Hope in #9133
[python] Keep native planning for Blob descriptor reads by @XiaoHongbo-Hope in #9144
[core] Remove configurable manifest format by @leaves12138 in #9154
[lance] Reject unsupported VARIANT and BLOB types in LanceFileFormat schema validation by @thswlsqls in #9150
[python] Fix row-id updates for Arrow columns over 2 GiB by @discivigour in #9146
[spark][test] Use explicit Paimon provider in feature tests by @Zouxxyy in #9145
[core] Add block-based manifest Avro reader by @JingsongLi in #9160
[spark] Fix MERGE INTO silently skipping WHEN NOT MATCHED BY SOURCE actions by @kerwin-zk in #9155
[python] Add efficient VARIANT path get and replace by @XiaoHongbo-Hope in #9147
[flink] Fix flaky Lumina data evolution index refresh IT by @ArnavBalyan in #9153
[python] Align exact VARIANT integer and decimal types by @XiaoHongbo-Hope in #9168
[spark] Respect blob-compaction.enabled in Spark compact procedure by @jackylee-ch in #9165
[format] Support delimiter as fallback key of csv.field-delimiter by @Zouxxyy in #9170
[core] Compact adjacent overlapping blob ranges together by @leaves12138 in #9175
[spark] Fix IndexOutOfBoundsException for vector_search with too few arguments by @jackylee-ch in #9164
[core] Release every reader in SortMergeReaderWithMinHeap.close() by @PDGGK in #9163
[core] Restore the interrupt status in the two JDBC catalog paths that drop it by @PDGGK in #9159
[iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath by @bube in #9158
[iceberg] Omit unknown bounds in Iceberg manifest partition summary by @thswlsqls in #9152
[core] Fix IS NAN predicate failing with file index by @ArnavBalyan in #9143
[python] Fix locale-sensitive Date header and nonce replay on retry by @sundapeng in #9173
[common] Finish the lookup-store close chain instead of stopping at the first failure by @PDGGK in #9172
[spark] Improve compaction logs in CompactProcedure by @Zouxxyy in #9169
[core] Unify manifest Avro readers by @leaves12138 in #9182
[core][flink] Remove file-based privilege system by @JingsongLi in #9186
[core] Fix level 1 inflation in universal compaction by @tsreaper in #9185
[core] Fix invalid length when reading spilled data by @tsreaper in #9183
[core] Parallelize primary-key full-text search by @QuakeWang in #9184
[flink] Fix CSV delimiter fallback for format tables by @ArnavBalyan in #9178
[core] Register every bucket in bucketList so overflow rows spread by @PDGGK in #9180
[spark] Add Format Table partition operation tests and clearer errors by @sundapeng in #9119
[core] Optimize data evolution compaction planning by @JingsongLi in #9177
[test] Fix URI-based temporary paths by @JingsongLi in #9191
[flink] Avoid overlapping sorted index key ranges by @leaves12138 in #9189
[spark] Introduce reassign_row_id procedure by @zhuxiangyi in #9192
[core] Add manifest Avro writer by @leaves12138 in #9194
[core] Refactor conflict detection by table type by @JingsongLi in #9196
[core] Allow first-row merge engine for primary-key managed BLOB tables. by @Stephen0421 in #9201
[core][format] Support Arrow bundle writes by @jianguotian in #9100
[spark] Retry data evolution compaction after merge conflicts by @JingsongLi in #9197
[python] Fix pyjindo advanced config setting in pypaimon by @timmyyao in #9206
[common] Fix like optimization for escaped patterns by @ArnavBalyan in #9205
[common] Fix incorrect date for casting pre epoch time by @ArnavBalyan in #9203
[docs] Clarify snapshot record count semantics by @wwj6591812 in #9202
[spark] Do not display an unknown partition statistic as a number by @sundapeng in #9200
[python] Support create_partitions in RESTCatalog by @plusplusjiajia in #9198
[core] Fix dynamic bucket allocation across partitions by @ArnavBalyan in #9193
[iceberg] Exclude NaN from manifest partition bounds by @ArnavBalyan in #9188
[core] avoid reading stale blobs during fallback read. by @steFaiz in #9208
[core] Avoid repeated sorted-index source lookups by @QuakeWang in #9187
[flink] Avoid redundant HTTP BLOB existence probes by @wwj6591812 in #9181
[core] Fix incorrect file indexes table range filtering by @ArnavBalyan in #9176
[core] Support latest snapshot delta scan in batch by @wwj6591812 in #9142
[common] Fix missing rows for null range bitmap indexes by @ArnavBalyan in #9140
[spark] Prevent table deletion for self referencing RTAS by @ArnavBalyan in #9139
[python] Avoid Arrow offset overflow when merging read batches by @XiaoHongbo-Hope in #9209
Bump org.apache.logging.log4j:log4j-api from 2.25.4 to 2.25.5 by @dependabot[bot] in #9215
[docs] Sync procedure documentation by @huangxiaopingRD in #9101
[core] Fix stale file index cleanup by @huangxiaopingRD in #9085
[common] Compare maps of different representations without throwing by @PDGGK in #9211
[flink][spark] Support manifest sort in compact_manifest by @discivigour in #9083
[core] Support arbitrary time granularity for chain table delta computation by @juntaozhang in #8185
[core] Optimize ordinary manifest merging with raw Avro blocks by @leaves12138 in #9213
[core] Speed up row-id manifest sorting by @leaves12138 in #9135
[core] Fix incremental diff scan missing first snapshot by @ArnavBalyan in #9234
[core] Parallelize primary-key sorted-index scan by @QuakeWang in #9231
[core] Support DATE, TIMESTAMP and TIMESTAMP_NTZ in variant shredding schema by @juntaozhang in #9230
[service] Fix KvQueryClient hang when query location lookup fails on retry by @thswlsqls in #9229
[core] Close every writer and shut both executors down when one close fails by @PDGGK in #9227
[flink] Fix scan partitions with partial partition spec by @ArnavBalyan in #9225
[spark] Fix missing data for delete hiding remaining records by @ArnavBalyan in #9224
[format] Validate blob file footer and index metadata by @Stephen0421 in #9223
[spark] Add option for Hive-style dynamic partition writes by @kerwin-zk in #9222
[core] Validate deletion vector mode when resetting table options by @ArnavBalyan in #9237
[python] Update paimon-rust to speed up native-plan CI by @XiaoHongbo-Hope in #9235
[python] Add tests for Data Evolution BLOB writes on branches by @XiaoHongbo-Hope in #9221
[spark][flink] Support bucket-level compaction for fixed-bucket tables by @lilei1128 in #9219
[python][ray] Fix update_by_row_id failure on empty datasets by @XiaoHongbo-Hope in #9217
[spark] Fix bucketed scan reporting for unsupported timestamp precision by @zhuxiangyi in #9216
[python] Fix GenericVariant UUID decoding for Java compatibility and support datetime/UUID by @juntaozhang in #9204
[python] Stream scalar update_by_predicate assignments by file group by @XiaoHongbo-Hope in #9240
[core] Support dynamic read batch sizing by @Gabriel39 in #9243
[python] Support callable assignments in predicate updates by @XiaoHongbo-Hope in #9239
[core] Refactor read batch sizing API by @JingsongLi in #9248
[python] Preserve manifest order in Data Evolution splits by @XiaoHongbo-Hope in #9242
[core] Support geospatial data types by @JingsongLi in #9251
[core] Reuse projected entries in manifest run merge by @leaves12138 in #9241
[ci][python] Fix lint-python native-plan CI failure by @XiaoHongbo-Hope in #9255
[core] Close the data writer even when the changelog writer fails by @PDGGK in #9254
[spark] Merge into query fails for is non null condition by @ArnavBalyan in #9252
[python] Avoid concurrent import failures during split deserialization by @XiaoHongbo-Hope in #9250
[core] Compare binary elements and keys by value in collect and merge_map by @PDGGK in #9249
[oss] Add fs.oss.cname.enabled so non-Aliyun endpoints sign the bucket by @PDGGK in #9238
[core] Take the stream position outside the finally so close always runs by @PDGGK in #9233
[common] Fix ClassCastException when computing row size for blob fields by @ArnavBalyan in #9087
[core] Fix file index pushdown for bitmap64 deletion vectors by @QuakeWang in #9141
[core] Fix snapshots table failing IN filter for unknown id by @ArnavBalyan in #9134
[core] Avoid external sort for fragmented manifests by @leaves12138 in #9257
[python] Tighten lazy import comments by @XiaoHongbo-Hope in #9260
[spark] Expose written columns for streaming micro-batches by @LsomeYeah in #9023
[python] Avoid deleting files during distributed commit failures by @JingsongLi in #9269
[python] Fix import deadlock: lazy-resolve package exports, drop custom locks by @XiaoHongbo-Hope in #9267
[iceberg] Emit row-lineage metadata fields for Iceberg format version 3 by @vbabenkoru in #9244
[flink] Fix unstable changelog compaction test by @QuakeWang in #9281
[java][python] Add VARIANT path upsert via variant_set by @XiaoHongbo-Hope in #9253
[core] Add Multivalue global index primitives by @JingsongLi in #9288
[core] Add Multivalue global index for array membership by @JingsongLi in #9282
[rest] Fix OpenAPI contract for client generation by @JingsongLi in #9289
[core] Support array set predicates in multivalue index by @JingsongLi in #9294
[core] Remove MultiValueIndexFileMeta by @JingsongLi in #9305
[python] Add vectorized splice fast path for variant_set root inserts by @XiaoHongbo-Hope in #9259
[spark] Push down array predicates to multivalue indexes by @JingsongLi in #9306
[common] Avoid temporary long[] allocations when compressing blob footer index by @Stephen0421 in #9307
[core] optimize column directives by @steFaiz in #9304
[core] Fix cascading re-merge in BinaryExternalSortBuffer spill merge by @zhoulii in #9303
[arrow] Fix nanosecond timestamp conversion for pre-epoch values by @jackylee-ch in #9299
[core][rest] Report partition statistics when registering partitions by @sundapeng in #9295
[core] Validate the carried-over default value when a column type changes by @PDGGK in #9292
[core] Keep tied nulls in range bitmap TopN for multi-key sorts by @jackylee-ch in #9280
[core] Match the lookup high level record by position, not by identity by @PDGGK in #9279
[format] Escape the escape character when writing CSV by @PDGGK in #9278
[core] Build expired partitions from their values instead of a joined string by @PDGGK in #9277
[format] Stop aliasing the output vector when tracking the previous delta value by @PDGGK in #9275
[python] Introduce hll_sketch aggregator function by @zhuxiangyi in #9274
[core][python] Record full build version in snapshots by @leaves12138 in #9283
[spark] Support consumer progress in streaming source by @LsomeYeah in #9273
[hive] Fix chain table HMS partition drop skipped by fallback scan. by @Stephen0421 in #9272
[api] Resume truncated HTTP GET response bodies by @wwj6591812 in #9271
[core] Preserve failures during manifest rewrite cleanup by @QuakeWang in #9270
[iceberg] Support custom table properties passthrough to Iceberg REST catalog by @JTaky in #9268
[core] Keep the deletion-vector meta cache when a table is copied by @PDGGK in #9266
[core] Read expire_tags older_than as a wall clock, not as an instant by @PDGGK in #9263
[flink] Surface a failed asynchronous lookup refresh instead of dropping it by @PDGGK in #9262
[format] Close the row-file input stream when the footer or index fails to parse by @PDGGK in #9258
[common] Fix NullPointerExceptions on null transform inputs by @plusplusjiajia in #9214
[iceberg] Assign manifest-level row lineage for Iceberg format version 3 by @vbabenkoru in #9245
[python] Group row tracking commits by partition by @YannByron in #9312
[flink] Fix literal conversion for BETWEEN predicates by @QuakeWang in #9310
[core] Encapsulate split serialization protocols by @leaves12138 in #9316
Bump org.eclipse.jetty:jetty-security from 12.1.8 to 12.1.10 in /paimon-iceberg by @dependabot[bot] in #9311
[core] Add an optional positional long[] of per-column maximum sequence numbers to DataFileMeta by @leaves12138 in #9313
[core] Avoid global index rebuilds for unrelated column compaction by @leaves12138 in #9069
[core] Report partition statistics from format table commits by @sundapeng in #9296
[spark] Fix timestamp precision in key dynamic writes by @Zouxxyy in #9322
[python][ray] Push down self-merge matched conditions by @XiaoHongbo-Hope in #9314
[core][spark] Support collecting format table partition statistics in MSCK REPAIR TABLE by @sundapeng in #9297
[python] Support callable assignments in Ray self-merge by @XiaoHongbo-Hope in #9327
[core] Optimize data evolution conflict scans by @leaves12138 in #9334
[spark]to DE insert into path,the target table should also be target splits by @weijietong in #9333
[core] Fix flaky column type update test by @ArnavBalyan in #9325
[python][ray] Avoid routing shuffle in self-merge updates by @XiaoHongbo-Hope in #9335
[python] Support target file row num for blob and vector writes by @YannByron in #9317
[flink] Fix asynchronous partition refresh generation race by @QuakeWang in #9318
[paimon-vector] Upgrade paimon-vector-index to 0.4.0 by @shyjsarah in #9341
[core][spark][flink] Support TRUNCATE TABLE on format tables by @sundapeng in #9330
[core][spark] Support ANALYZE TABLE on catalog-managed format tables by @sundapeng in #9298
[core][spark] Use format partition row counts in scan statistics by @JingsongLi in #9351
[core][spark] Fix format table INSERT OVERWRITE ignoring dynamic-partition-overwrite by @sundapeng in #9353
[python][ray] Rebase self-merge updates after compaction by @XiaoHongbo-Hope in #9339
[python][ray] Flatten large self-merge predicates by @XiaoHongbo-Hope in #9361
[Spark] Support reading nested null fields for Parquet struct by @ArnavBalyan in #9364
[format] Support numeric type widening when reading Avro files by @ArnavBalyan in #9362
[format] Introduce support for Parquet byte stream split encoding by @ArnavBalyan in #9359
[common] Fix wildcard matching for line terminators by @ArnavBalyan in #9358
[spark] Fix MERGE INTO delete clause corrupting partial-update tables by @zhuxiangyi in #9356
[spark] Forbid delete and update on first-row table (#6718) by @RunningDB in #9355
[core] Fix null handling in ArrayContains by @QuakeWang in #9354
[common] Correct local dependency for string conversion by @ArnavBalyan in #9346
[format] Fix corrupt output by overlapping custom line delimiters by @ArnavBalyan in #9344
[iceberg] Skip an already-deleted manifest list when expiring old metadata by @lucasgameiroborges in #9343
[Fix] Support KEY_DYNAMIC bucket mode in CompactorSinkBuilder by @zhang-arvin in #9338
[core] Fix incorrect substring result for unicode characters by @ArnavBalyan in #9328
[common] Fix substring transform query failure for non int type by @ArnavBalyan in #9326
[core] Fix scan version for fallback branch by @ArnavBalyan in #9323
[flink] Avoid having QueryFileMonitor always busy by @thswlsqls in #9291
[flink] Shut down KvQueryServer when QueryExecutorOperator closes by @thswlsqls in #9287
[core] Fix flaky concurrent duplicate file discard test by @ArnavBalyan in #9285
[flink] Support savepoint auto-tag on coordinator commit path by @ifndef-SleePy in #9309
[vector] Expose IVF search tuning options by @shyjsarah in #9367
[api] Support external Jackson for simple REST requests by @1fanwang in #9369
[core] Reduce Variant buffer copies across Arrow and Parquet paths by @JingsongLi in #9371
[api][spark] Treat an unreported partition statistic as unknown, not as zero by @sundapeng in #9379
[spark] Resolve primary-key and bucket-key properties like other identifiers by @zhuxiangyi in #9372
[core] Prune value columns when building lookup files for PersistPositionProcessor by @liangjie3138 in #9374
[core][python] Fix partial ranges when updating newly added Blob columns by @zhoulii in #9388
[core] Fix nullability parsing for nested types by @zhoulii in #9381
[core] Quiesce manifest rewrite workers before cleanup by @QuakeWang in #9376
[spark] Introduce CTAS support for partitioned tables by @ArnavBalyan in #9378
[spark] Fix conditional query failing on variant casts by @ArnavBalyan in #9385
[core][spark] Push down object table filters by @Stephen0421 in #9383
[common] Follow Spark's SUBSTRING position semantics and count characters in TRIM by @plusplusjiajia in #9329
[python] Fix quadratic cost of writing multiple batches by @yugan95 in #9375
[common][python] Support SUBSTRING and TRIM in column masking by @plusplusjiajia in #9212
[api] Add REST permission management by @JingsongLi in #9393
[flink] Avoid slot deadlock when sorting normalized index keys by @leaves12138 in #9394
[flink] Fix lookup output projection with internal key fields by @leaves12138 in #9392
[core] Manifest read pool in expiration use scanManifestParallelism by @yuzelin in #9395
[api] Add REST policy management by @JingsongLi in #9400
[format] Fix Parquet delta length reads across vector batches by @ArnavBalyan in #9366
[core][iceberg] Include table name in commit/Iceberg-commit exception by @JTaky in #9391
[dcos] Supplementary document description by @lilei1128 in #9404
[hive] Avoid binding to HiveConf.ConfVars constants renamed in Hive 4 by @leosanqing in #9403
[docs] Add REST management OpenAPI by @JingsongLi in #9406
[core] Add public binary serialization for BucketVectorSearchSplit by @JunRuiLee in #9386
[flink] Do not shuffle to a negative channel when the partition hash is Integer.MIN_VALUE by @PDGGK in #9408
[core] Expose bitmap index format APIs by @leaves12138 in #9414
[core] Snapshot expiration deletes data and changelog files in batches by @yuzelin in #9409
[core][test] Add REST management query authorization support by @JingsongLi in #9416
[core][spark] Add REST management procedures by @JingsongLi in #9410
[core][spark] Fix reading newly added BLOB columns after MERGE by @zhoulii in #9413
[python][torch] Add batch-first streaming datasets by @XiaoHongbo-Hope in #9365
[python][ray] Adapt default partitions to input size by @XiaoHongbo-Hope in #9357
[core] Compare IcebergRef by value by @PDGGK in #9405
[spark] Advance streaming source past empty full snapshots by @lilei1128 in #9401
[common] Release the BTree index input when construction or close fails by @PDGGK in #9407
[core] Fix shared decompression buffer in stable manifest blocks by @yuzelin in #9430
[core] Improve row ID reassignment under concurrent writes by @JingsongLi in #9439
[core] Add manifest file name in ManifestFileBlockMerger exception message for troubleshooting by @yuzelin in #9432
[core] Parallelize row ID reassignment manifest rewrites by @JingsongLi in #9441
[flink] Commit format table overwrites once by @sundapeng in #9433
[spark] Reject SET LOCATION for format tables by @sundapeng in #9435
[spark] Reject empty partition values for format tables by @sundapeng in #9436
[python] Add HDF5 DataSource ingestion by @YannByron in #9411
[core] Stream row ID reassignment manifest rewrites by @JingsongLi in #9443
[python] Support MAP<K, BLOB> in multimodal BLOB reads by @XiaoHongbo-Hope in #9428
[python] Default multimodal vector files to Parquet by @JingsongLi in #9444
[core] Add exact partitioned FM global index by @JingsongLi in #9442
[python] Fix BLOB CRC32 compatibility with Java by @liangjie3138 in #9426
[spark] Fix min and max query failure on metadata columns by @ArnavBalyan in #9431
[flink] Release both catalogs FlinkGenericCatalog wraps by @PDGGK in #9422
[core] Reject resetting ignore-delete and ignore-update-before from true by @PDGGK in #9421
[format] Fix reading Avro nullable unions when null value is in second branch by @ArnavBalyan in #9418
[flink][cdc] Fix bucket-aware state recovery for multi-table CDC sink by @Stephen0421 in #9417
[api][core] Parallelize format table overwrite commit by @sundapeng in #9397
[core] Support FM indexes for primary-key tables by @JingsongLi in #9447
[flink] Support unaligned checkpoints for coordinator commit by @JingsongLi in #9451
[core] Harden and slim FM index storage by @JingsongLi in #9450
[core][flink][spark] Treat non-positive row counts as unknown by @sundapeng in #9456
[python] Add DataFusion extra for merge conditions by @XiaoHongbo-Hope in #9463
[common] Release the input stream when FileIndexPredicate fails to open it by @PDGGK in #9462
[core]rename FileStoreCommitImpl var name by @weijietong in #9457
[iceberg] Refuse nanosecond timestamps while Iceberg metadata is enabled by @plusplusjiajia in #9455
[api] Stop reading a server error message as a format string by @plusplusjiajia in #9454
[format] Fix ORC zstd compression option inconsistent across writers by @ArnavBalyan in #9453
[api] Fix race condition causing queued tasks to execute after SequentialBatchIterator close by @juntaozhang in #9452
[metrics] Add totalFileCount and minAvgFileSize metrics to detect small file problem by @junmuz in #9449
[hive] Fix query failure when dynamic filters are unavailable by @ArnavBalyan in #9448
[python] Add RoboMIND AgileX HDF5 pipeline by @YannByron in #9445
[format][python] Add packed video frame storage by @JingsongLi in #9461
[flink][cdc] Fix missing primary keys for Debezium records by @ArnavBalyan in #9465
[Fix] MongoDB CDC: handle null fullDocument in transaction to prevent NullNode exception (#9332) by @zhang-arvin in #9336
[iceberg] Allow publishing VARIANT columns with Iceberg format version 3 by @vbabenkoru in #9246
[core][python] Reuse PK sorted indexes with retired source files by @wangyong9999 in #9460
[core] Fix column masking correctness in query-auth reads by @plusplusjiajia in #8570
[format][python] Refactor video storage internals by @JingsongLi in #9470
[flink] Fix decimal column analysis with null statistics by @ArnavBalyan in #9469
[core] Read the table branch schemas when rewriting file indexes by @plusplusjiajia in #9468
[iceberg] Refuse time precisions the mirror cannot publish by @plusplusjiajia in #9467
[python][torch] Support distributed iterable dataset sharding by @XiaoHongbo-Hope in #9429
[spark] Fix flaky row tracking concurrent compaction test by @JingsongLi in #9475
[core][python] Support multiple packed video fields by @XiaoHongbo-Hope in #9473
[multimodal] Support aligned multi-video file groups by @JingsongLi in #9480
[python] Add LeRobot Dataset v3 import by @XiaoHongbo-Hope in #9446
[spark] Clamp Format Table ANALYZE parallelism by @sundapeng in #9482
[format] Support nested variant column pruning for shredded parquet files by @juntaozhang in #9389
[core] Rename FM index identifier to fm by @JingsongLi in #9487
[python][torch] Make map-style reads lazy by @XiaoHongbo-Hope in #9486
[spark] Support parallel sorted index building across Paimon partitions by @liangjie3138 in #9491
[python] Reject overlapping row-id update batches by @YannByron in #9484
[spark] Support partition pruning for data evolution self-merge by @zhoulii in #9489
[spark] Support timestamp_ntz and variant in generic row access by @shyjsarah in #9493
[spark] Skip time-travel self-merge test on Spark 3.2 by @sundapeng in #9502
[spark] Show format table partition statistics by @sundapeng in #9501
[python] Add distributed RoboMIND action backfill by @YannByron in #9496
[iceberg] Leave the nanosecond refusal to the validator that knows the table by @plusplusjiajia in #9503
[python][ray] Cover conditional self-merge compaction rebase by @JingsongLi in #9516
[core] Support changelog-producer.ignore-update-before and changelog-producer.ignore-delete options by @junmuz in #9531
[python] Add ROSBag ingestion support by @YannByron in #9530
[common][spark] Fix z-order boolean FALSE colliding with the null sentinel by @LuciferYang in #9527
[common] Make BinaryRow.anyNull respect the row offset by @LuciferYang in #9525
[common] Grow from one byte in MemorySliceOutput when the segment is empty by @LuciferYang in #9523
[spark] Let MSCK see the null partition of a value-only format table by @sundapeng in #9521
[common] Fix double position advance in DataOutputSerializer.writeBytes by @LuciferYang in #9520
[common] Reject non-struct inner fields in variant shredding schema by @LuciferYang in #9518
[common][spark] Fix boolean hilbert value colliding with the null sentinel by @LuciferYang in #9515
[common] Fix partition getter indexing in InternalRowPartitionComputer by @LuciferYang in #9513
[common] Fix permit accounting of SemaphoredDelegatingExecutor under interruption and rejection by @LuciferYang in #9511
[test-util] Set accessible on @Parameters provider and @Parameter field by @thswlsqls in #9510
[pvfs] Set working directory to the catalog root in PaimonVirtualFileSystem by @thswlsqls in #9507
[docs] Fix stale paimon-vindex version in vector index documentation by @thswlsqls in #9505
[Bug] Fix concurrent LocalTableQuery lookups returning null across files by @zhang-arvin in #9500
[format] Introduce byte stream split for writing to parquet by @ArnavBalyan in #9499
Bump browserslist from 4.28.2 to 4.28.8 in /docs by @dependabot[bot] in #9542
[core]make stats model work while migrating by @weijietong in #9541
[docs] Fix misleading DROP COLUMN doc for hive catalog by @sd4324530 in #9538
[rest] Make RESTTokenFileIO cache maximum size configurable by @tonymtu in #9528
[Bug] Fix debezium-json cannot infer primary keys from standard Debezium message key by @zhang-arvin in #9495
[format] Preserve csv values matching the configured null literal by @ArnavBalyan in #9492
[spark] Fix analyze failure for timestamp ntz columns by @ArnavBalyan in #9485
[format] Prevent text tables from accepting unsupported schemas by @ArnavBalyan in #9479
[flink] Fix schema inference for multi partition Kafka topics by @ArnavBalyan in #9478
[format] Fix positional schema evolution for ORC reads by @ArnavBalyan in #9464
[python] Expose Arrow batch reader for multimodal scans by @XiaoHongbo-Hope in #9537
[spark] Support action predicate pruning for data evolution self-merge by @zhoulii in #9544
Bump fast-uri from 3.1.5 to 3.1.7 in /docs by @dependabot[bot] in #9554
[python] Cache decoded BLOB indexes by @YannByron in #9547
[python] Infer multimodal vector columns by query dimension by @wangzhigang1999 in #9553
[core][spark][flink] Support sub-field-level data evolution for nested columns by @zhuxiangyi in #8334
[format] Give each row-format reader its own projected-row wrapper by @LuciferYang in #9561
[python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload by @Stephen0421 in #9539
[core][spark][flink] Fix nested data evolution isolation by @JingsongLi in #9564
[format] Support sep as fallback key of csv.field-delimiter by @jackylee-ch in #9565
[format] Keep parsing remaining fields after a malformed one in PERMISSIVE CSV mode by @LuciferYang in #9556
[format] Read BINARY-encoded decimals at scratch index 0 in parquet updater by @LuciferYang in #9563
[vortex] Fix predicate pushdown literals for DATE and second-precision TIMESTAMP by @thswlsqls in #9558
[format] Compare array wrapper-group names case-insensitively in ParquetReaderUtil by @LuciferYang in #9568
[spark] Use the field index instead of the field id for TopN pushdown by @jackylee-ch in #9566
[core][python][spark] Optimize data evolution write column metadata by @JingsongLi in #9574
refactor(core): make SchemaManager an interface by @baiyangtx in #9174
[api][core][spark] Support custom partition locations by @sundapeng in #9540
[docs] Regenerate the configuration reference by @LuciferYang in #9592
[python] Make multimodal table creation race-safe by @XiaoHongbo-Hope in #9590
[python] Notify callbacks for successful retried commits by @XiaoHongbo-Hope in #9589
[parquet] Push down StartsWith as Parquet binary range by @lilei1128 in #9594
[python] Persist LeRobot dataset metadata in Paimon by @XiaoHongbo-Hope in #9529
[common] Fail on a short remote read instead of caching zero padding by @jackylee-ch in #9593
[api] Trim whitespace in the bucket-key option value by @jackylee-ch in #9591
[format] Report unresolvable JSON string casts instead of a bare NPE by @LuciferYang in #9588
[core] Clean up data file extra files by @zjw1111 in #9586
[format] Build fresh ORC writer options per created writer by @LuciferYang in #9583
[format] Close the text stream when line-reader construction fails by @LuciferYang in #9579
[common] Fix pre-epoch timestamp casts truncating toward zero by @jackylee-ch in #9577
[format] Close the ParquetFileReader when post-construction setup fails by @LuciferYang in #9576
[format] Close the Avro stats extractor's stream on a corrupt file by @LuciferYang in #9573
[format] Fail fast when skipping DELTA_LENGTH_BYTE_ARRAY data past end of page by @LuciferYang in #9571
[spark] Reject variant extraction paths containing the metadata delimiter by @jackylee-ch in #9569
[spark] Disable create table in default db by @junmuz in #9549
[spark] Fix batch delete for partial update sequence groups by @ArnavBalyan in #9546
[core] Avoid retaining rejected cache pages by @yuzelin in #9658
[vortex] Honor write.batch-memory in the writer factory by @thswlsqls in #9605
[docs] Document the VECTOR data type by @thswlsqls in #9603
[core] Fix stale option key in clustering.strategy description by @thswlsqls in #9602
[arrow] Fix second-precision timestamp conversion for pre-epoch values by @thswlsqls in #9600
[spark] Fix inconsistent snapshots for data evolution V1 UPDATE by @zhoulii in #9597
[Flink] Add configurable initial scan mode for streaming compaction by @lilei1128 in #9548
[core] Unify block cache ownership and avoid redundant SST reads by @JingsongLi in #9663
[core] Support reordered fields in manifest Avro files by @JingsongLi in #9683
[python] Simplify LeRobot metadata tables and training tags by @YannByron in #9676
[ci][python] Fix Python 3.6 CI after Bullseye EOL by @XiaoHongbo-Hope in #9684
Bump js-yaml from 4.1.1 to 4.3.1 in /docs by @dependabot[bot] in #9685
[python] Fix Arrow offset overflow in merge condition filtering by @XiaoHongbo-Hope in #9678
[flink] Honor snapshot limit in exactly-once monitor source by @jianguotian in #9677
[python][torch] Add lazy contiguous window dataset by @YannByron in #9580
[python] Invalidate commit retry cache after overwrite by @XiaoHongbo-Hope in #9688
[python] Add end-to-end test for row-id reassignment conflict by @XiaoHongbo-Hope in #9689
feat(core): Support schema management over REST in RESTCatalog by @baiyangtx in #9673
[python] Support snapshot properties in table commits by @YannByron in #9692
[python][ray] Fix count metadata for row-level authorization by @QuakeWang in #9681
Bump colord from 2.9.3 to 2.10.0 in /docs by @dependabot[bot] in #9698
Bump svgo from 3.3.4 to 3.3.5 in /docs by @dependabot[bot] in #9701
Bump js-yaml from 4.3.1 to 4.3.2 in /docs by @dependabot[bot] in #9699
Bump joi from 17.13.4 to 17.13.7 in /docs by @dependabot[bot] in #9700
[python] Add direct LeRobot capture writer by @YannByron in #9665
[python] Add paired ACT storage benchmark by @YannByron in #9466
[python] Remove duplicate BLOB validation during table creation by @wangzhigang1999 in #9704
[core] Fix default NOT IN composition in file index reader by @lilei1128 in #9612
[core] Fall back to renaming when multipart upload is unavailable by @Zouxxyy in #9610
[python] Add Paimon LeRobot map-style dataset by @XiaoHongbo-Hope in #9498
[core][spark] Let DROP PARTITION repair overlapping Format Table locations by @sundapeng in https://github.com/apache/paimon/pull/9712
[docs] Explain Data Evolution design and file contracts with diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9711
[docs] Organize maintenance guides by task by @JingsongLi in https://github.com/apache/paimon/pull/9713
[common] Default missing variant shredding fields to the unshredded index by @LuciferYang in https://github.com/apache/paimon/pull/9616
[common] Always accept the first entry in bitmap index block packing by @LuciferYang in https://github.com/apache/paimon/pull/9614
[common] Do not zero the HeapBytesVector buffer on reset by @LuciferYang in https://github.com/apache/paimon/pull/9644
[common] Stop nesting row-tracking vector wrappers per batch by @LuciferYang in https://github.com/apache/paimon/pull/9634
[common] Overflow-safe URI length check in VideoFrameDescriptor by @LuciferYang in https://github.com/apache/paimon/pull/9618
[common] Keep escaped characters in StringToMapCastRule entry splitting by @LuciferYang in https://github.com/apache/paimon/pull/9620
[common] Close the file-index stream when reader construction fails by @LuciferYang in https://github.com/apache/paimon/pull/9636
[common] Propagate checked IOException from RESTTokenFileIO#fileIO by @LuciferYang in https://github.com/apache/paimon/pull/9638
[format] Validate nested types for JSON tables by @LuciferYang in https://github.com/apache/paimon/pull/9585
[common] Run the z-order sign flip at the width of the bit pattern by @LuciferYang in https://github.com/apache/paimon/pull/9628
[core] Split large normal files in data evolution compaction by @JingsongLi in https://github.com/apache/paimon/pull/9709
[api][core] Reset Format Table partition locations on overwrite by @sundapeng in https://github.com/apache/paimon/pull/9710
[python] Preserve LeRobot recording metadata in direct writer by @YannByron in https://github.com/apache/paimon/pull/9708
[docs] Restructure primary-key table documentation and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9718
[docs] Reorganize append table guides and replace diagrams with SVGs by @JingsongLi in https://github.com/apache/paimon/pull/9719
[docs] Reorganize multimodal table guides and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9721
[docs] Restructure concepts documentation and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9722
[docs] Reorganize Spark guides and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9724
[docs] Reorganize Flink guides and add SVG diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9725
[docs] Restructure PyPaimon guides and add SVG diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9726
[python] Import LeRobot video datasets by @XiaoHongbo-Hope in https://github.com/apache/paimon/pull/9494
[core][python] Add option to skip manifest merge in write-only mode by @JingsongLi in https://github.com/apache/paimon/pull/9727
[filesystem] Isolate Jindo OSS presigning dependencies (#9856) by @jerry-024 in https://github.com/apache/paimon/pull/9887

New Contributors

@wenhaozhao011-cmd made their first contribution in #8989
@huwh made their first contribution in #9003
@u70b3 made their first contribution in #9037
@tyoushinya made their first contribution in #9116
@Gabriel39 made their first contribution in #9243
@vbabenkoru made their first contribution in #9244
@lucasgameiroborges made their first contribution in #9343
@1fanwang made their first contribution in #9369
@leosanqing made their first contribution in #9403
@wangyong9999 made their first contribution in #9460

Full Changelog: release-2.0.0...release-2.1.0