Addax 6.2.0

Addax 6.2.0 is a stabilization release: 38 fixes, features and refactors across the engine, the shared libraries, the plugins and the shipped scripts since 6.1.0, several of them closing paths that corrupted, dropped or silently truncated data, plus 23 dependency updates and a larger end-to-end suite (27 cases, up from 18). It also adds four capabilities — the server now runs every job in its own process, DuckDB gains a reader and a writer, the HTTP reader prefetches pages and the Elasticsearch writer keeps several bulk requests in flight. Nine changes are backward-incompatible, reaching the server API, both FTP plugins, the Excel reader, the HTTP reader, both InfluxDB 2 plugins, the Paimon writer and the Elasticsearch writer (which now requires Elasticsearch 7.0 or later) — read the migration notes before upgrading.

The version is 6.2.0 rather than 6.1.1 on purpose: this release contains new features and backward-incompatible changes, which is a minor bump under the versioning scheme.

⚠️ Breaking changes

1. The server runs every job in its own process

A review of the server module found that jobs ran inside the server JVM, where the engine keeps per-job state process-wide: jobs could see each other's settings (request parameters were written into the system properties), could not be cancelled, and took the server down with them when they crashed. A failed task reported only java.lang.reflect.InvocationTargetException; a malformed job was accepted and failed asynchronously; the finished-task table grew without a bound.

Every job now runs the way the command line tool runs one — a JVM of its own, its parameters as -D properties, its own log file, the process killed when the server stops. A failed task answers with the error line of its own log plus the exit code and the log file name, a malformed job is rejected with 400 before a task is created, and a task can end as CANCELLED via the new POST /api/cancel. New options: --task-timeout, --max-tasks (bounds the finished-task table), --bind, ADDAX_SERVER_PARALLEL; YAML jobs are accepted; request bodies above 8 MiB are refused; unknown methods answer 405 with an Allow header.

• Migration: clients must handle 400 on submit and the CANCELLED status. Query parameters are -D properties of the job again (?jobName= feeds the report payload as documented). Each job costs a JVM start of about 1-2 seconds.

2. The FTP writer was reworked (#1651)

The plugin had defects that broke SFTP writes and could lose data on FTP: mkDirRecursive ran mkdir twice per missing directory level, so the second call failed with SSH_FX_FAILURE and SFTP could only write into directories that already existed completely; getOutputStream appended instead of truncating, so a re-run doubled the rows and compressed output concatenated two archives into one file; a store never read its completion reply, so a transfer the server rejected went unnoticed; the data timeout was built with Duration.ofSeconds on a millisecond value; append mode did not keep file names unique; useKey=false still authenticated with the private key; connectPattern was documented but ignored.

• Migration: append mode re-runs now write a new file beside the previous one instead of appending into it — use the truncate mode when a fixed file name is required. A job that relied on the private key while useKey was false must set useKey to true. Compressed output with codecs other than zip, gzip and bzip2 now carries the codec's extension.
• SFTP writes create missing target directories, failed uploads fail the task instead of reporting success, and connectPattern (PORT and PASV) is honoured.

3. The FTP reader sets up the control connection before it is opened

commons-net builds the control reader and writer while connecting, from the control encoding set at that moment, so the encoding assigned after login() never reached them: commands stayed on ISO-8859-1 and every non-ASCII path went out as ?. The configured timeout was applied after connect() too and never bounded the control connection. The control encoding is now UTF-8 and the timeouts are set before connecting.

• Migration: paths and file names outside ASCII now work against UTF-8 servers for every command, listing and retrieval alike. A server that speaks another encoding on the wire needs a control encoding option that does not exist yet.

4. Excel: a blank cell is now a null

The Excel reader lost columns: a row with a cell left out in the middle shifted every following value one place to the left, silently corrupting the data. It also could not read large workbooks at all (WorkbookFactory was handed a stream POI cannot seek in; a 500k row workbook failed, and holding a sheet as XMLBeans objects cost 3.2GB of live heap for a 20MB file).

• Migration: a cell without a value is now a null instead of an empty string, and a formula without a stored result reads as null instead of being evaluated. A job that relied on empty strings for blank cells can map them back with the nullToDefault transformer; a workbook whose formulas were never calculated only needs to be opened and saved by Excel once, which stores the results the reader now reads.
• xlsx is read with the event model: 500k rows by 8 columns in 3.6s with 841MB peak RSS. Every record is padded to the widest row with nulls, so columns stay aligned. New options trim, sheetIndex and sheetName; a workbook with several sheets logs which one was read; a file that is no workbook is skipped with a warning instead of failing the whole job.

5. The HTTP reader no longer damages or drops data, and prefetches pages

A wildcard column list took the keys of the first record only, so every later record lost the fields it happened to add; request parameters were stringified ("NUM": 5 was sent as "5"); URL parameters were not percent-encoded and a query string configured in the URL was dropped for POST; a timeout, an empty body or a 3xx were read as "0 records" and the job reported success; an endpoint that ignores the paging parameters made the reader loop without a bound (75,000 requests in 12 seconds); the client-level Authenticator answered server 401 challenges with the proxy credentials.

• Migration: with column: ["*"] the records of a page may now hold more columns than before (the union of the keys of every page); a writer with a fixed schema needs an explicit column list, or a response whose records all share the same keys.
• A wrong resultKey, a 3xx or an empty body now fail the job. New options: maxPages (bounds the paging, with a duplicate-page check), sslVerify (turns the certificate and hostname check on), and prefetchPages — with the default single page in flight the round trip of the next page is spent while the current one is parsed and written (8.5s to 6.5s for a 21-page read against a 300ms endpoint with a writer that needs 100ms per page), prefetchPages=4 brings the same read to 2.3s.

6. The InfluxDB 2 writer fails on a write error

Points were handed to the async WriteApi and the result was never looked at, so a rejected write only reached the log while the job reported "Total N records, Error 0" and exited 0 — a wrong token (401), a missing bucket (404), points the retention policy drops (422) and a server that refused every attempt (503, which stored 0 of 20000 points) all looked like success. The timestamp conversion kept only the sub-second part of a microsecond or nanosecond timestamp, collapsing every point of a second onto 1970-01-01T00:00:00Z; a null boolean aborted the job with an NPE; a numeric tag value threw ClassCastException.

• Migration: a record whose column count differs from column + 1 now fails the job instead of writing a subset of the fields; a date or timestamp column outside the first one is written as an RFC3339 string instead of the local rendering; a failed write is no longer retried by the client, so the job has to be run again (a point write is idempotent, so a rerun is safe).
• The writer now writes through WriteApiBlocking (a failure fails the job and names the bucket and the number of points), batches by batchSize, turns on gzip (a 1000-point body shrinks 7.5x) and splits into one task per reader slice instead of failing when channel is above 1. The precision matrix (s, ms, us, ns) lands every record at the time it belongs to and keeps the microsecond part exactly. New tagColumns item writes the value of a named column as a tag, so a reader-to-writer round trip keeps the tags of the source. Verified against InfluxDB 2.8: 29 end-to-end jobs and 4 wire-level checks.

7. The InfluxDB 2 reader resolves columns from the index and streams the rows

Reading more than one measurement never worked (the generated flux query did not compile), the field probe described the first table only (the fields of a second measurement were dropped silently), an empty column probe left raw column strings in the slice config and killed the task with ClassCastException, and values were cast to the type recorded at probe time (a field that is long in one measurement and double in another threw).

• Migration: a downstream writer now receives _time as a timestamp and a boolean field as boolean, both were strings before. A consumer that needs the previous representation has to format the value itself, for example with a groovy transformer.
• Columns are resolved from the Influx index inside the task and validated against the union over all requested measurements; an unknown measurement or column is reported by name; a time range without data finishes with 0 records instead of failing. Reads are streamed: on a 200,000 row read (InfluxDB 2.8, -Xmx256m) the wall clock drops from 7.76s to 4.04s and the peak RSS from 944MB to about 628MB.

8. The Paimon writer leaves the bucket to Paimon (#1638)

The writer assigned the bucket itself, hashing the whole row. Paimon merges the keys of a primary key table only inside a bucket, so the versions of one key were spread over several buckets and the table kept duplicate primary keys. On top of that DATE was written as a Timestamp, TINYINT/SMALLINT as Integer and FLOAT as Double (Paimon keeps those as int, byte, short and float, so every write failed with a cast error), and every batchSize rows were committed on their own, costing one snapshot and one data file per thousand rows.

• Migration: a job that writes to a dynamic bucket table (bucket = -1, the Paimon 1.2 default for a primary key table) now fails at init and needs a table with a fixed bucket number. batchSize no longer has any effect (one writer, one commit per task). Records whose columns cannot be converted are no longer written (previously a half-filled row was).
• DATE is written as epoch days, TINYINT/SMALLINT/FLOAT at their own width, ARRAY/MAP as Paimon's own containers; an optional column list maps the reader columns to the table columns by name; an unknown writeMode is refused instead of appending.

9. The Elasticsearch writer requires Elasticsearch 7.0 or later

The writer built every request the way Elasticsearch 5 expected: mappings were put at /{index}/{type}/_mapping, the bulk request went to /{index}/{type}/_bulk, a geo_shape carried the quadtree parameters removed in 6. Against 8.11 the mapping request answers "no handler found" and the job failed at its first step; the plugin also never declared the httpclient that jest needs, so it could not build a connection at all.

• Migration: the writer speaks the typeless API introduced in Elasticsearch 7.0 and is verified against 7.0+ (8.11 in the test suite); Elasticsearch 5.x and 6.x are out of support for this plugin. A job that still carries type or actionType keeps working, the keys are ignored like any other unknown key. tree and precision on a geo_shape are ignored with a warning.
• The index is created with its settings and mappings in one request, a record whose primary key column is empty is written without an id (previously every such record landed on the literal id "null"), a byte field is written as the number it is, and a mapping Elasticsearch refuses fails the job instead of quietly degrading to its own.

🐛 Fixes

Data correctness

• elasticsearchreader: a filter whose evaluation fails — a keyword column compared with true, which Ognl reports as a plain NumberFormatException rather than an OgnlException — aborted the whole job with a raw stack trace, although the documented behavior is to keep the record and warn once. Every exception is caught now and the single warning names it (#1660).
• elasticsearchreader: a long below Integer.MIN_VALUE was truncated by the integer cast, and an integer beyond the range of a long aborted the task. A refused connection or wrong credentials are no longer reported as "index not exist". A scroll whose search body carries no size gets batchSize filled in: 200k documents are read in 6s instead of 42s, in 200 pages instead of 20,000.
• mongodbreader: a collection whose _id is a date, decimal128, uuid or binary read 0 documents and reported success — a slice bound lost its type in the JSON round trip of the slice configuration, and a type-sensitive comparison then matched nothing (#1620). Binary now reads as the BytesColumn type the documentation and the writer use (a binary cell used to arrive as BsonBinary{type=0, data=[1, 2, 3]}); a projection names only the configured leaves and excludes _id when it is not read (7.48MB instead of 10.89MB over 200,000 documents); a column list naming both nested and nested.a.b works instead of failing with "Path collision".
• mongodbwriter: an update-mode record whose update key is missing or null built the filter {key: null} — which MongoDB matches against every document where the field is absent — and its upsert overwrote an unrelated document while the job reported success; such records are collected as dirty now (#1621, #1622). A failed batch is written unordered so the server keeps every document it accepts and only the rejected positions are collected: on 2000 records with 4 rejects, failed records went 1000 → 4 and runtime 6s → 3s. The inverted preSql json/item branches are fixed (an item-only configuration built a null filter, a json-only one deleted by the item list), type: int stores int32 as documented, and a username is no longer required for clusters without access control (#1600).
• hivereader: a binary cell took an exception path on every row (getBytes always throws on Hive), and the fallback decode/re-encode turned unhex('00ff80') into six bytes instead of three; a NULL binary cell turned the whole row dirty. BINARY/VARBINARY/BLOB/LONGVARBINARY now read getObject, which hands back the exact bytes (#1615). The plugin follows the managed hive 3.1.3 client instead of a stripped 2.1.1 one: a Hive 3.1.0 server reads again (768 rows of both test.time_test_orc and test.time_test_parquet, previously 768 dirty records), timestamp-with-local-time-zone and interval columns decode, and both Kerberos entry points get all the way to the realm lookup.
• lib-rdbms: the reader asked every ResultSetMetaData for isSigned(), which is optional in JDBC and which Hive refuses for every column — reading 768 rows from HiveServer2 reported success with 0 rows delivered. The BIGINT UNSIGNED overflow is now decided from the value itself and the optional metadata accessors go through fallbacks (#1614).
• hdfsreader: parquet and orc files written by Spark, Hive or Arrow are readable — timestamps are decoded from the schema (INT96 and INT64 in millis, micros and nanos), decimals from int32, int64 or fixed byte arrays, the elements of a list and the values of a map from their own field, and an orc date as the calendar day it is (it was a day early west of UTC). A value the declared type cannot hold no longer abandons the rest of the parquet file, a missing optional field is a null, a column the file cannot serve fails at startup instead of collecting one dirty record per row, and the format is read from the file magic instead of opening every text file the job listed.
• s3reader: the wildcard pattern was turned into a regular expression without quoting the rest of it — the dot in data/*.csv matched any character and an unrelated data/foo1csv was read into the job, a plus sign became a quantifier and failed the job, a bracket did not compile at all. Literal parts are quoted now, the object list is deduplicated and sorted (an exact name next to a pattern covering it listed the same object twice, so its records came out twice), and an object that vanished after the listing fails the job instead of reporting an incomplete success. fileFormat other than csv and text fails with a message instead of being parsed as CSV.
• s3writer: the object name was split at the first dot, so report.2024.csv left report_<uuid>.2024 in the bucket — the suffix now comes from the last dot. The listing pages past the first 1000 keys, so truncate cleans the whole prefix and nonConflict sees conflicts beyond that page; deletes go out in batches of 1000.
• lib-storage: a bare pattern (path: ["*.csv"]) resolves against the working directory instead of failing with "Cannot find any file in path". When compress is not configured, the leading bytes of the stream are matched against the compression magics: a gzip file read without compress used to be handed to the CSV parser as text — one record of binary noise, reported as success. The txt, ftp, hdfs and s3 readers all benefit.
• excelwriter: a configured header was destroyed by the first data record; a string with a control character produced a package no conformant reader can parse; a literal _xHHHH_ code came back as a different value; DOUBLE and TIMESTAMP columns were written as text; the date style the date cells pointed at was never defined in the package. All fixed, plus a temp file moved into place and log4j-to-slf4j so POI logging reaches the addax log. A 200k x 8 export writes in ~0.41s instead of ~0.67s for a file about 4% larger.
• jsonfilereader: every column was re-resolved from the job configuration once per record, a SimpleDateFormat per date value, the channel flushed after every record — about three quarters of the run time on a 500k x 12 file (13.0s → 4.0s). The json-path options are applied for real (the result of addOptions() was discarded), nested objects and arrays keep their json text instead of silently becoming null, a missing path yields a null column instead of failing the job, and a path item ending with ] no longer crashes (#1601).
• ftpreader: the wildcard pattern was converted to a regex by hand and only ., * and ? were quoted, so /a+b*.csv matched aab.csv; /*.csv in the root never listed anything; a relative path crashed or silently reported an existing path as missing. Patterns are compiled with the JDK glob matcher, every configured path must be absolute, and a path that yields no file is reported at WARN level.
• txtfilewriter: writeMode: append could never work on an existing target — prepare() deliberately keeps the files and startWrite() then required createNewFile(), which returns false exactly for the file to append to. Append now opens the target with CREATE|APPEND (a plain open would silently truncate the rows already there). Appending to an existing compressed target is refused with an actionable config error, because each run emits one complete compressed stream (#1661).

Stability and performance

• elasticsearchwriter: the new parallelBulk option keeps several bulk requests in flight (default 1 is the previous loop): 20,000 documents in 200 batches took 7.5s at 1, 3.7s at 2 and 1.8s at 4, with every document delivered exactly once. Five tasks reached 71k records/s where one reached 33k.
• icebergwriter: the plugin was unusable — a jackson-annotations mismatch against the hadoop BOM made catalog.loadTable throw NoClassDefFoundError before a single row was read, and parquet/orc versions clashed with iceberg's own. Behind that: BINARY was handed over as byte[] where the writers take a ByteBuffer, TIME as LocalDateTime where they take LocalTime, list/map element types were ignored, a failed conversion still wrote a half-filled row, and every batchSize rows built its own writer and committed — 1000 rows with batchSize 100 left 10 files and 10 snapshots. One PartitionedFanoutWriter commits once at the end of the task, files roll at the table's write.target-file-size-bytes, and a task that fails writes nothing. A task holds one open file per partition for its whole life, so a table partitioned at a very high cardinality needs the task's data volume to be bounded accordingly.
• server: (see breaking change 1) plus escapeJson no longer leaves control characters raw, task snapshots are published as one immutable record instead of three unsynchronized writes, and the launch script passes the logback and log directory properties.
• mongodbreader: a task bounds its connection pool (one client per task with the driver default allowed 100 connections per task), the cursor is exempt from the server-side idle timeout a slow writer can trigger, and the password is decrypted into a local variable instead of being written back into the configuration the framework prints at debug level.
• hivereader: column metadata is resolved once per ResultSet instead of asking ResultSetMetaData#getColumnType per cell; a configured splitPk is called out with a warning, since on Hive it means one full scan per channel instead of a seek.
• hdfsreader: the column plan is resolved once per file instead of on every cell, and the path listing no longer opens every file to guess its format.
• lib-rdbms: unknown SQL dialects are parsed generically instead of aborting the job (#1595) — DuckDB is absent from Druid's DbType enum, and the pre-check ran for every reader job. DuckDB dialect support added: double-quoted identifiers, streaming reads, INSERT OR REPLACE / ON CONFLICT templates (#1596).

Scripts, packaging and CI

• install_plugins.sh accepts -o as an alias for --offline, as the usage text always said.
• The release workflow waits for VALIDATED instead of PUBLISHED (17m51s of the 21m45s deploy step in the 6.1.0 run); green now means "uploaded and validated", and Central publishes it minutes later.
• The commit-lint workflow is the only job in its file; the verify job computed exactly what the E2E build already does. The E2E job skips pull requests that cannot change it (sources, poms, assembly descriptors, plugin metadata, the suite itself), workflow_dispatch always runs it.
• Dependabot's Hive 4 proposal is held off: Hive 4 republished hive-standalone-metastore as a pom aggregator and moved its classes, so hivereader cannot resolve the artifacts it declares — moving to a Hive 4 client is a migration with a test against the target server (#1619).
• Documentation: the branding was unified behind one description and one tagline (MESSAGING.md is the source of truth), the READMEs gained the DuckDB logo, and CONTRIBUTING now asks for a local verification and its result instead of a unit test suite the project does not have.

🧪 Testing

The end-to-end suite grew from 18 to 27 cases:

• DuckDB (5 cases, #1596): the appender happy path and the reordered column list that must fall back (both assert the data lands in the right columns), one row per native type, a MySQL → DuckDB round trip, and a 200k row querySql read. The harness gains addax_duckdb_sql, which builds a DuckDB file through the shipped driver since there is no server to start and no fixture directory to seed.
• S3 reader (250_s3reader_objects): a seeded moto bucket — an object the pattern must not match, a pattern holding a regex metacharacter, a gzip object and a plain one read in the same job. setup.sh may now exit 77 to skip a case without failing it, and a selection that needs no database no longer requires a container runtime.
• HDFS round trip (260_hdfsreader_roundtrip): parquet and ORC through the reader on file:///, with the column types no other case writes, taken from the streamreader so it needs neither a database nor a container runtime.
• Elasticsearch reader (270_elasticsearchreader_scroll): 21 documents holding the values readers get wrong (both Integer bounds, a long below Integer.MIN_VALUE, an integer beyond the long range, an empty array, a missing object, a quote and a backslash, a document with none of the requested columns), one job scrolling with batchSize and no size, one filtering with a size in the body. The runner gains CASE_JVM_ARGS.
• Elasticsearch writer (280_elasticsearchwriter_types): one document of every field type the writer supports, read back with python rather than with the reader plugin, so the check cannot agree with a mistake the reader would make as well.
• Codec coverage (#1632): case 080 writes parquet with ZSTD, 090 ORC with LZ4, and 260 does the same for both formats it round trips, so a broken codec, a missing jar or a dropped native artifact surfaces as a case failure.
• txtfilewriter append (110_mysql2txt_append): flipped from EXPECTED_FAIL to a real assertion — the golden holds the rows of the first run followed by the rows of the append run (#1661).

📦 Dependencies

23 updates, among them zstd-jni 1.5.7-16 → 1.5.7-21, ngdbc 2.28.8 → 2.30.11, joda-time 2.14.3 → 2.15.0, snappy-java 1.1.10.8 → 1.1.10.11, netty 4.2.19.Final, kryo 5.6.2 → 5.7.1, ognl 3.4.13 → 3.4.16, commons-codec 1.23.0, commons-lang3 3.21.0, jackson 2.22.3, httpclient5-fluent 5.6.5 / httpcore 5.4.4, log4j-over-slf4j 2.0.20, duckdb_jdbc 1.5.5.1 → 1.5.6.0 (the driver jar is ~85MB: it bundles native libraries for every platform), and iceberg 1.12.0 with the writer ported to the FileWriterFactory API.

⬆️ Upgrading

• Distribution: addax-6.2.0.tar.gz and addax-6.2.0.tar.gz.sha256sum.asc — check the checksum before unpacking.
• Docker: wgzhao/addax:6.2.0 (Docker Hub) and quay.io/wgzhao/addax:6.2.0.
• Maven Central: com.wgzhao.addax:addax-core:6.2.0.
• Check first if you run the server (clients must handle 400 and CANCELLED), use the FTP writer's append mode or useKey=false, read Excel workbooks with blank cells or formulas, read HTTP with column: ["*"], write to InfluxDB 2 or Paimon, or use the Elasticsearch writer against a pre-7.0 cluster.
• Plugin documentation: https://github.com/wgzhao/addax-docs

Addax 6.2.0（中文版）

Addax 6.2.0 是一个稳定性版本:自 6.1.0 以来对引擎、公共库、插件与随发行版交付的脚本做了 38 项修复、新特性与重构,其中多项关闭了静默写坏、丢弃或截断数据的路径,另有 23 项依赖升级和扩充后的端到端测试套件(27 个用例,此前为 18 个)。本版还带来四项新能力:server 改为每个作业独立进程运行、新增 DuckDB 读写插件、HTTP reader 支持预热后续分页、Elasticsearch writer 支持多个 bulk 并发在途。另有九处不兼容变更,涉及 server API、两个 FTP 插件、Excel reader、HTTP reader、两个 InfluxDB 2 插件、Paimon writer 以及 Elasticsearch writer(现在要求 Elasticsearch 7.0 及以上)——升级前请先看迁移说明。

版本号定为 6.2.0 而不是 6.1.1 是有意的:本版本包含新特性与不兼容变更,按版本命名规范应走小版本号。

⚠️ 破坏性变更

1. server 改为每个作业独立进程运行

对 server 模块的复查发现:作业运行在 server 的 JVM 里,而引擎的作业状态是进程级的——作业之间会互相看到对方的设置(请求参数被写进了系统属性),无法取消,崩溃时会把 server 一起带走。失败的任务只报 java.lang.reflect.InvocationTargetException;格式错误的作业会被接受、随后异步失败;已完成任务表无上限增长。

现在每个作业都与命令行方式一致地运行——独立 JVM、参数以 -D 传入、独立日志文件、server 停止时进程被终止。失败的任务会返回它自己日志里的错误行加上退出码和日志文件名;格式错误的作业在创建任务前就被拒绝(400);任务可以经新增的 POST /api/cancel 进入 CANCELLED 状态。新增 --task-timeout、--max-tasks(限制已完成任务表)、--bind、ADDAX_SERVER_PARALLEL;接受 YAML 作业;超过 8 MiB 的请求体被拒绝;未知方法返回 405 并带 Allow 头。

• 迁移方式: 客户端需要处理提交时的 400 与 CANCELLED 状态。查询参数重新成为作业的 -D 属性(?jobName= 按文档喂给 report 载荷)。每个作业有一到两秒的 JVM 启动开销。

2. FTP writer 重写(#1651)

插件存在破坏 SFTP 写入、可能丢失数据的缺陷:mkDirRecursive 对每一级缺失目录执行两次 mkdir,第二次以 SSH_FX_FAILURE 失败,SFTP 只能写入已完整存在的目录;getOutputStream 追加而不是截断,重跑一遍行数翻倍,压缩输出把两个归档拼进一个文件;store 从不读取完成应答,被服务端拒绝的传输无人发觉;数据超时用 Duration.ofSeconds 传毫秒值;append 模式不保证文件名唯一;useKey=false 仍用私钥认证;connectPattern 有文档但被忽略。

• 迁移方式: append 模式重跑会在原文件旁写新文件,而不是追加进去——需要固定文件名请用 truncate 模式;useKey=false 却依赖私钥的作业必须把 useKey 改为 true;zip/gzip/bzip2 之外的压缩输出现在带对应编解码器的扩展名。
• SFTP 写入会创建缺失的目标目录;失败的传输会让任务失败,而不是报告成功;connectPattern(PORT 与 PASV)生效。

3. FTP reader 在建立控制连接之前完成设置

commons-net 在连接时用当时的控制编码构建读写器,login() 之后设置的编码永远不会生效:命令一直走 ISO-8859-1,所有非 ASCII 路径变成 ?。配置的超时同样在 connect() 之后才应用,控制连接从不受其约束。现在控制编码固定为 UTF-8,超时在连接前设置。

• 迁移方式: 非 ASCII 路径与文件名在 UTF-8 服务端上对全部命令(列目录、取文件)都能正常工作。服务端在线上使用其他编码的场景,需要一个目前尚不存在的控制编码选项。

4. Excel:空单元格现在读作 null

Excel reader 会丢列:一行中间被跳过的单元格会让该行之后所有值左移一位,静默写坏数据。它也无法读取大工作簿(WorkbookFactory 收到的是 POI 无法 seek 的流,50 万行的工作簿直接失败;把整张表当作 XMLBeans 对象持有,20MB 文件要占用 3.2GB 堆)。

• 迁移方式: 无值单元格现在读作 null 而不是空字符串;没有缓存结果的公式读作 null,而不是被求值。依赖空字符串的作业可以用 nullToDefault transformer 映射回来;公式从未计算过的工作簿只需用 Excel 打开并保存一次,结果就会被缓存下来、被 reader 读取。
• xlsx 改用事件模型读取:50 万行 × 8 列 3.6 秒、峰值内存 841MB。每条记录都会补齐到最宽行并以 null 填充,列不会再错位。新增 trim、sheetIndex、sheetName 选项;多 sheet 工作簿会记录读取的是哪一张;不是工作簿的文件以警告跳过,不再让整个作业失败。

5. HTTP reader 不再损坏或丢弃数据,并支持分页预热

通配符列只取第一条记录的键,之后每条记录新增的字段都会丢;请求参数被字符串化("NUM": 5 被发成 "5");URL 参数不做百分号编码,URL 里配置的查询串对 POST 会被丢弃;超时、空响应体和 3xx 都被读成"0 条记录"并报告成功;无视分页参数的端点会让 reader 无界循环(12 秒发出 75,000 个请求);客户端级 Authenticator 拿代理凭据去应答服务端的 401 质询。

• 迁移方式: column: ["*"] 下每页记录现在可能比之前多出若干列(所有页键的并集);固定 schema 的 writer 需要显式列清单,或保证所有记录的键一致。
• resultKey 写错、3xx 或空响应体现在会直接失败。新增选项:maxPages(配合重复页检查限制翻页)、sslVerify(开启证书与主机名校验)、prefetchPages——默认在途一页时,下一页的往返在当前页解析与写出期间完成(300ms 端点、写每页需要 100ms 的 writer,21 页的读取从 8.5 秒降到 6.5 秒),prefetchPages=4 时同一读取降到 2.3 秒。

6. InfluxDB 2 writer 遇到写入错误会失败

点数据交给异步 WriteApi 后从不检查结果:被拒绝的写入只进日志,作业却报 "Total N records, Error 0" 并以 0 退出——错误 token(401)、不存在的 bucket(404)、被保留策略丢弃的点(422)、以及全部被拒的服务端(503,20000 个点存进 0 个)看起来都像成功。时间戳转换只保留微秒/纳秒类型的亚秒部分,把同一秒内所有点坍缩到 1970-01-01T00:00:00Z;null 布尔值让作业 NPE;数字类型的 tag 值抛 ClassCastException。

• 迁移方式: 记录列数与 column + 1 不一致现在会让作业失败,而不是写入部分字段;第一个之外的 date/timestamp 列按 RFC3339 字符串写出,不再是框架的本地渲染;失败的写入不再由客户端重试,需重新运行作业(点写入幂等,重跑安全)。
• 改用 WriteApiBlocking(失败即让作业失败,并给出 bucket 与点数),按 batchSize 分批,开启 gzip(1000 点的请求体缩小 7.5 倍),并按 reader 切片拆任务,channel 大于 1 时不再失败。精度矩阵(s/ms/us/ns)让每条记录落在正确时间并精确保留微秒。新增 tagColumns,把指定列的值写为 tag,reader→writer 往返可保持来源的 tag。对照 InfluxDB 2.8 验证:29 个端到端作业与 4 项线路级检查通过。

7. InfluxDB 2 reader 从索引解析列并流式读取

读取多个 measurement 从来无法工作(生成的 flux 查询不能编译);字段探测只看第一张表(第二个 measurement 的字段被静默丢弃);空气的列探测会把原始列字符串留在 slice 配置里,任务以 ClassCastException 死掉;所有值都按探测时记录的类型强转(同一字段在一张表里是 long、另一张里是 double 就抛异常)。

• 迁移方式: 下游 writer 收到的 _time 现在是 timestamp 列,布尔字段是 boolean——此前都是字符串。需要旧表示形式的消费方请自行格式化,例如用 groovy transformer。
• 列在任务内从 Influx 索引解析,并对全部请求的 measurement 取并集校验;未知的 measurement 或列按名字报错;无数据的时间范围以 0 条记录正常结束,不再失败。读取改为流式:20 万行读取(InfluxDB 2.8,-Xmx256m)耗时从 7.76 秒降到 4.04 秒,峰值 RSS 从 944MB 降到约 628MB。

8. Paimon writer 把 bucket 交给 Paimon 决定(#1638)

writer 自己算 bucket(哈希整行)。Paimon 只在同一 bucket 内合并主键表的 key,因此同一个 key 的多个版本被分散到不同 bucket,表里留下重复主键。此外 DATE 被写成 Timestamp、TINYINT/SMALLINT 写成 Integer、FLOAT 写成 Double(Paimon 对应保留 int/byte/short/float,每次写入都因类型转换失败);每 batchSize 行单独提交,每千行产生一个 snapshot 和一个数据文件。

• 迁移方式: 写入动态 bucket 表(bucket = -1,Paimon 1.2 主键表默认值)的作业现在在 init 阶段直接失败,需要固定 bucket 数的表;batchSize 不再有任何作用(每任务单 writer、单次提交);列无法转换的记录不再写出(此前会写出半填充的行)。
• DATE 写为 epoch days,TINYINT/SMALLINT/FLOAT 按自身宽度,ARRAY/MAP 用 Paimon 自己的容器;新增可选列清单,把 reader 的列按名字映射到表的列;未知 writeMode 直接拒绝,不再按追加处理。

9. Elasticsearch writer 要求 Elasticsearch 7.0 及以上

writer 按 Elasticsearch 5 的约定构造所有请求:mapping 写到 /{index}/{type}/_mapping,bulk 发到 /{index}/{type}/_bulk,geo_shape 携带 6.0 已移除的 quadtree 参数。对 8.11 而言 mapping 请求返回 "no handler found",作业第一步就失败;插件也从未声明 jest 需要的 httpclient,连接根本建不起来。

• 迁移方式: 本 writer 面向 Elasticsearch 7.0 引入的无 type API,验证范围为 7.0+(测试套件对 8.11 验证);Elasticsearch 5.x 与 6.x 在本插件中不再受支持。作业里残留的 type、actionType 会被当作未知键忽略,不影响运行;geo_shape 上的 tree 与 precision 被忽略并给出警告。
• 索引连同 settings 与 mappings 一次创建;主键列为空的记录不再带 id 写出(此前每条这类记录都落在字面量 id "null" 上互相覆盖);byte 字段按数字写出;Elasticsearch 拒绝的 mapping 会让作业失败,而不是悄悄退化成它自己推断的 mapping。

🐛 修复

数据正确性

• elasticsearchreader:求值失败的过滤器——比如 keyword 列与 true 比较,Ognl 抛出的是普通 NumberFormatException 而不是 OgnlException——会让整个作业带着原始堆栈直接失败,而文档行为是保留记录并警告一次。现在会捕获所有异常,且唯一的警告会给出异常类名(#1660)。
• elasticsearchreader:低于 Integer.MIN_VALUE 的 long 被整数强转截断,超出 long 范围的整数让任务直接崩溃;连接被拒或凭据错误不再报成 "index not exist";search body 未配 size 的 scroll 会补上 batchSize:20 万文档从 42 秒/20000 页降到 6 秒/200 页。
• mongodbreader:_id 为 date、decimal128、uuid 或 binary 的集合读到 0 条却报成功——切片边界在 slice 配置的 JSON 往返中丢了类型,而 MongoDB 的比较是类型敏感的(#1620)。binary 现在读为文档与 writer 都使用的 BytesColumn 类型(此前一个 binary 单元格到达时是 BsonBinary{type=0, data=[1, 2, 3]});投影只取配置的叶子并排除未读取的 _id(20 万文档传输 7.48MB,此前 10.89MB);列清单同时含 nested 与 nested.a.b 时不再以 "Path collision" 失败。
• mongodbwriter:update 模式记录的更新键缺失或为 null 时构造出过滤器 {key: null}——MongoDB 会把它匹配到所有缺少该字段的文档——upsert 覆盖了无关文档,作业却报成功;这类记录现在会被收集为脏记录(#1621、#1622)。失败的批次改为无序写入,服务端保留它接受的全部文档,只收集被拒的位置:2000 条记录、4 条被拒的场景,失败记录从 1000 降到 4,耗时从 6 秒降到 3 秒。preSql 的 json/item 分支颠倒修复(item-only 配置构造出 null 过滤器,json-only 却按 item 清单删除),type: int 按文档存为 int32,无访问控制的集群不再强制要求用户名(#1600)。
• hivereader:binary 单元格每行都走异常路径(Hive 上 getBytes 永远抛异常),回退用 getString 解码再按 UTF-8 编码,unhex('00ff80') 从 3 字节变成 6 字节;NULL binary 单元格还会让整行变脏。BINARY/VARBINARY/BLOB/LONGVARBINARY 现在读 getObject,拿回精确字节(#1615)。插件改跟受管理的 hive 3.1.3 客户端,不再是被剥空依赖的 2.1.1:Hive 3.1.0 服务端恢复可读(test.time_test_orc 与 test.time_test_parquet 各 768 行,此前是 768 条脏记录),timestamp with local time zone 与 interval 列可解码,两个 Kerberos 入口也能走到 realm 查询。
• lib-rdbms:reader 对每个 ResultSetMetaData 调用 isSigned(),而它是 JDBC 可选方法、Hive 对每一列都拒绝——从 HiveServer2 读 768 行会以"成功、0 行"收场。BIGINT UNSIGNED 溢出改为按值本身判断,其余可选元数据方法走带兜底的访问器(#1614)。
• hdfsreader:Spark、Hive、Arrow 写出的 parquet 与 orc 恢复可读——时间戳按文件 schema 解码(INT96 与 INT64 的毫秒/微秒/纳秒),decimal 支持 int32/int64/定长字节数组,list 的元素与 map 的值按各自的字段解码,orc 的 date 按日历日解码(此前在 UTC 以西会差一天)。声明类型装不下的值不再让整个 parquet 文件剩下的部分被放弃;缺失的可选字段是 null;文件提供不了的列在启动时失败,而不是每行收集一条脏记录;格式改从文件魔数读取,不再为了判断格式打开列出的每个文本文件。
• s3reader:通配符模式转正则时没有引用其余部分——data/*.csv 里的点匹配任意字符,无关的 data/foo1csv 被读进作业;加号变成量词;方括号直接编译失败。现在字面部分会被引用,对象清单去重并排序(精确名与覆盖它的模式并列时同一对象会被列两次,记录出双份),列出后消失的对象让作业失败,而不是报一个不完整却成功的作业。csv/text 之外的 fileFormat 直接报错,不再按 CSV 解析。
• s3writer:对象名在第一个点处切分,report.2024.csv 在 bucket 里留下 report_<uuid>.2024——后缀现在取最后一个点。清单支持翻页越过前 1000 个键,truncate 能清掉整个前缀,nonConflict 能发现该页之外的冲突;删除按 1000 个一批发出。
• lib-storage:裸模式(path: ["*.csv"])相对工作目录解析,不再报 "Cannot find any file in path"。未配置 compress 时按流的前导字节匹配压缩魔数:不带 compress 读取 gzip 文件此前会被当作文本交给 CSV 解析器——产出一条二进制噪声记录,还报告成功。txt、ftp、hdfs、s3 reader 都受益。
• excelwriter:配置的表头会被第一条数据记录冲掉;含控制字符的字符串会产出任何合规 reader 都解析不了的包;字面量 _xHHHH_ 编码读回来变了值;DOUBLE 与 TIMESTAMP 列被写成文本;日期单元格指向的日期 style 从未在包里定义。全部修复,此外改为写临时文件再移入目标、换用 log4j-to-slf4j 让 POI 日志进入 addax 的日志。20 万 × 8 导出约 0.41 秒(此前约 0.67 秒),文件大约大 4%。
• jsonfilereader:每个列在每条记录上重新从作业配置解析,每个日期值新建一个 SimpleDateFormat,每条记录后 flush 通道——50 万 × 12 的文件里约四分之三的运行时间是这些开销(13.0 秒降到 4.0 秒)。json-path 选项真正生效(此前 addOptions() 的返回值被丢弃);嵌套对象与数组保留 json 文本,不再静默变 null;缺失路径产出 null 列而不是让作业失败;以 ] 结尾的路径项不再崩溃(#1601)。
• ftpreader:通配符模式手工转正则时只引用了 .、*、?,于是 /a+b*.csv 会匹配 aab.csv;根目录的 /*.csv 什么都列不出来;相对路径会崩溃、或把存在的路径静默报成缺失。改为用 JDK glob 匹配器编译模式,所有配置路径必须是绝对路径,匹配不到文件的路径以 WARN 级别报出。
• txtfilewriter:writeMode: append 在目标文件已存在时永远无法工作——prepare() 特意保留这些文件,随后 startWrite() 却要求 createNewFile() 成功,而该调用恰好对这个文件返回 false。append 现在以 CREATE|APPEND 打开目标(普通打开会静默截断已有行)。对已存在的压缩目标追加会以明确的配置错误拒绝,因为每次运行写出的是一个完整压缩流(#1661)。

稳定性与性能

• elasticsearchwriter:新增 parallelBulk 选项,允许若干 bulk 请求同时在途(默认 1,即原循环):20000 文档分 200 批,1 时 7.5 秒、2 时 3.7 秒、4 时 1.8 秒,且每份文档恰好送达一次。五个任务达到 71k records/s,单任务为 33k。
• icebergwriter:插件此前不可用——jackson-annotations 与 hadoop BOM 的版本错配让 catalog.loadTable 在一行数据都没读之前就抛 NoClassDefFoundError,parquet/orc 版本也与 iceberg 自身冲突。其后还有:BINARY 以 byte[] 交给需要 ByteBuffer 的写入器,TIME 以 LocalDateTime 交给需要 LocalTime 的写入器,list/map 的元素类型被忽略,转换失败仍写出半填充行,每 batchSize 行新建 writer 并单独提交——1000 行、batchSize 100 留下 10 个文件和 10 个 snapshot。现在每个任务一个 PartitionedFanoutWriter、结束时提交一次,文件按表的 write.target-file-size-bytes 滚动,失败的任务不写出任何东西。任务存活期间每个分区保持一个打开文件,分区基数极高的表需要相应控制单任务的数据量。
• server:(见破坏性变更 1)此外 escapeJson 不再原样输出控制字符,任务快照从三次不同步写入改为一个不可变记录原子替换,启动脚本传递 logback 与日志目录属性。
• mongodbreader:任务限定自己的连接池(此前每任务一个 client、按驱动默认允许 100 连接),游标免受慢 writer 可能触发的服务端空闲超时影响,密码解密到局部变量而不是写回框架在 debug 级别打印的配置。
• hivereader:列元数据每个 ResultSet 解析一次,不再对每个单元格调用 ResultSetMetaData#getColumnType;配置了 splitPk 会给出警告,因为在 Hive 上这意味着每通道一次全表扫描而非 seek。
• hdfsreader:列计划按文件解析一次,不再逐单元格解析;列目录不再为了猜测格式而打开每个文件。
• lib-rdbms:未知 SQL 方言改为通用解析,不再中断作业(#1595)——Druid 的 DbType 枚举里没有 DuckDB,而预检查对每个 reader 作业都会执行。新增 DuckDB 方言:双引号标识符、流式读取、INSERT OR REPLACE / ON CONFLICT 模板(#1596)。

脚本、打包与 CI

• install_plugins.sh 接受 -o 作为 --offline 的别名,与用法说明一致。
• 发布工作流等待 VALIDATED 而不是 PUBLISHED(6.1.0 发布时,21 分 45 秒的 deploy 步骤里有 17 分 51 秒花在这上面);绿色现在表示"已上传并校验",Central 会在几分钟后自行发布。
• commit-lint 工作流成为其文件里唯一的任务;verify 任务计算的内容与 E2E 构建步骤完全相同。E2E 任务对其不可能产生影响的 PR 跳过(源码、pom、assembly 描述符、插件元数据与测试套件本身),workflow_dispatch 始终运行。
• Dependabot 的 Hive 4 升级提案被搁置:Hive 4 把 hive-standalone-metastore 改成 pom 聚合器并挪走了类,导致 hivereader 无法解析它声明的构件——迁移到 Hive 4 客户端是一次需要对着目标服务端测试的迁移(#1619)。
• 文档:品牌统一为一套描述与一句 tagline(MESSAGING.md 为唯一事实来源),README 增加 DuckDB logo,CONTRIBUTING 改为要求本地验证及其结果,而不是本项目没有的单元测试套件。

🧪 测试

端到端套件从 18 个用例增长到 27 个:

• DuckDB(5 个用例,#1596):appender 正常路径与列顺序错位后必须回退的路径(两者都断言数据落在正确的列),每种原生类型一行,MySQL → DuckDB 往返,20 万行 querySql 读取。测试框架新增 addax_duckdb_sql,通过随发行版附带的驱动构建 DuckDB 文件——没有服务端可起,也没有 fixture 目录可灌。
• S3 reader(250_s3reader_objects):预置的 moto bucket——模式必须不匹配的对象、含正则元字符的模式、gzip 对象与普通对象在同一次作业中读取。setup.sh 现在可以以 77 退出表示跳过而不算失败;不需要数据库的用例选择也不再要求容器运行时。
• HDFS 往返(260_hdfsreader_roundtrip):在 file:/// 上用 reader 往返 parquet 与 ORC,使用了其他用例不会写到的列类型,数据取自 streamreader,既不依赖数据库也不依赖容器运行时。
• Elasticsearch reader(270_elasticsearchreader_scroll):21 个文档承载 reader 容易出错的取值(Integer 两端边界、低于 Integer.MIN_VALUE 的 long、超出 long 范围的整数、空数组、缺失对象、引号与反斜杠、没有任何所请求列的文档),一个作业用 batchSize 滚动、search body 不带 size,另一个带 size 并过滤。runner 新增 CASE_JVM_ARGS。
• Elasticsearch writer(280_elasticsearchwriter_types):writer 支持的每种字段类型各一个文档,用 python 而不是 reader 插件读回,校验不会与 reader 犯了同一个错误却一起通过。
• 编解码器覆盖(#1632):用例 080 用 ZSTD 写 parquet、090 用 LZ4 写 ORC、260 对两种格式同样处理——编解码器损坏、jar 缺失或原生库缺失都会以用例失败暴露。
• txtfilewriter append(110_mysql2txt_append):从 EXPECTED_FAIL 翻转成真实断言——golden 是第一次运行的行接着 append 运行的行(#1661)。

📦 依赖升级

共 23 项,其中 zstd-jni 1.5.7-16 → 1.5.7-21、ngdbc 2.28.8 → 2.30.11、joda-time 2.14.3 → 2.15.0、snappy-java 1.1.10.8 → 1.1.10.11、netty 4.2.19.Final、kryo 5.6.2 → 5.7.1、ognl 3.4.13 → 3.4.16、commons-codec 1.23.0、commons-lang3 3.21.0、jackson 2.22.3、httpclient5-fluent 5.6.5 / httpcore 5.4.4、log4j-over-slf4j 2.0.20、duckdb_jdbc 1.5.5.1 → 1.5.6.0(driver jar 约 85MB:打包了所有平台的原生库),以及 iceberg 1.12.0 并把 writer 迁移到 FileWriterFactory API。

⬆️ 升级方式

• 发行包: addax-6.2.0.tar.gz 与 addax-6.2.0.tar.gz.sha256sum.asc,解压前请校验校验和。
• Docker: wgzhao/addax:6.2.0(Docker Hub)与 quay.io/wgzhao/addax:6.2.0。
• Maven Central: com.wgzhao.addax:addax-core:6.2.0。
• 升级前请先确认是否:使用 server(客户端需处理 400 与 CANCELLED)、使用 FTP writer 的 append 模式或 useKey=false、读取含空单元格或公式的 Excel、以 column: ["*"] 读 HTTP、写入 InfluxDB 2 或 Paimon、或用 Elasticsearch writer 对接 7.0 之前的集群。
• 插件文档:https://github.com/wgzhao/addax-docs

📜 Full changelog (6.1.0 → 6.2.0)

• 1540116 [maven-release-plugin] prepare release 6.2.0
• 9f7c8d8 docs(release): add the 6.2.0 release notes
• 8355cd9 fix(plugin-txtfilewriter): append to the kept file instead of failing (#1661)
• 00ad4cc fix(plugin-elasticsearchreader): keep records whose filter cannot be evaluated (#1660)
• 6225593 chore(deps): bump netty.version from 4.2.18.Final to 4.2.19.Final (#1659)
• 361e13f chore(deps): bump com.esotericsoftware:kryo from 5.6.2 to 5.7.1 (#1658)
• 0ee1e5b chore(deps): bump org.apache.httpcomponents.client5:httpclient5-fluent (#1657)
• 16883cb chore(deps): bump ognl:ognl from 3.4.13 to 3.4.16 (#1656)
• 05553f3 chore(deps): bump commons-codec:commons-codec from 1.22.1 to 1.23.0 (#1655)
• 1535b0e feat(server)!: process-per-job execution with validation, cancellation and timeouts
• bd4295b chore(deps): bump org.xerial.snappy:snappy-java (#1652)
• 918799d fix(plugin-ftpwriter)!: rework the write path and the connection handling (#1651)
• 4b2c56e chore(deps): bump com.github.luben:zstd-jni from 1.5.7-20 to 1.5.7-21 (#1650)
• d48b76f chore(deps): bump org.xerial.snappy:snappy-java (#1649)
• 0867a9a refactor(plugin-ftpreader): model the protocol and the connection settings
• e711053 perf(plugin-ftpreader): stop probing directories a listing already identified
• 5bee877 fix(plugin-ftpreader): report the paths that yield no file
• 1650912 fix(plugin-ftpreader): resolve wildcard paths as globs
• 56b9ebb fix(plugin-ftpreader): require every configured path to be absolute
• 38f620f fix(plugin-ftpreader)!: set up the ftp control connection before it is opened
• 6cc96a8 chore(deps): bump joda-time:joda-time from 2.14.4 to 2.15.0 (#1647)
• 97dfb36 chore(deps): bump iceberg to 1.12.0 and port the writer to the FileWriterFactory api (#1642)
• ab592aa chore(deps): bump org.duckdb:duckdb_jdbc from 1.5.5.1 to 1.5.6.0 (#1643)
• c495067 fix(plugin-influxdb2writer)!: fail on a write error and keep the values the server can hold
• 459513c fix(plugin-influxdb2reader)!: resolve the columns from the index and stream the rows
• 8b8523a fix(plugin-icebergwriter): repair the dependency set, the value mapping and the commit cadence
• c5ffeab fix(plugin-paimonwriter)!: repair the bucket, the value layout and the batch commits (#1638)
• 28c15f0 chore(deps): bump org.apache.commons:commons-lang3 from 3.20.0 to 3.21.0 (#1635)
• 377bfd5 chore(deps): bump at.yawk.lz4:lz4-java from 1.11.3 to 1.12.0 (#1634)
• f02dc17 test(e2e): write parquet with ZSTD and ORC with LZ4 (#1632)
• 5092ae9 chore(deps): bump org.slf4j:log4j-over-slf4j from 2.0.19 to 2.0.20 (#1631)
• 68bcb9f chore(deps): bump com.github.luben:zstd-jni from 1.5.7-18 to 1.5.7-20 (#1630)
• df379b3 chore(deps): bump httpcore.version from 5.4.3 to 5.4.4 (#1629)
• 3482999 chore(deps): bump jackson.version from 2.22.2 to 2.22.3 (#1628)
• 7efc8c8 feat(plugin-httpreader): request the following pages while the current one is written
• 8107560 fix(plugin-httpreader)!: repair the request, the paging and the wildcard columns
• 334ead1 feat(plugin-elasticsearchwriter): keep several bulk requests in flight
• 502edbd fix(plugin-elasticsearchwriter): write to elasticsearch 7 and later
• 9f318b3 fix(plugin-elasticsearchreader): repair the read path and batch the scroll
• 2619e17 fix(plugin-mongodbwriter): refuse a wildcard record without an update key (#1622)
• d98ef8e fix(plugin-mongodbwriter): repair the update key and the batch write (#1621)
• ed17e01 docs(contributing): say what a pull request is expected to show
• e0a6f50 fix(plugin-mongodbreader): stop losing every row of a collection with a typed _id (#1620)
• 57e1094 ci(dependabot): hold off the Hive 4 bump (#1619)
• 86b3551 chore(deps): bump com.sap.cloud.db.jdbc:ngdbc from 2.30.7 to 2.30.11 (#1612)
• 3ca832d chore(deps): bump joda-time:joda-time from 2.14.3 to 2.14.4 (#1616)
• 565dca4 chore(deps): bump com.github.luben:zstd-jni from 1.5.7-17 to 1.5.7-18 (#1618)
• b276ae4 fix(plugin-hivereader): keep binary cells intact and read a Hive 3 server (#1615)
• 6816ad5 fix(lib-rdbms): stop requiring optional ResultSetMetaData methods (#1614)
• e0a855a fix(plugin-hdfsreader): read the parquet and orc files other engines write
• 78641d4 test(e2e): install the S3 test double into a virtualenv and report why an import fails
• a2dff17 test(e2e): read a seeded object store through the s3reader
• d655b8c fix(plugin-s3reader): validate the file format and page the object listing
• 381ac5e fix(plugin-s3writer): keep the object suffix and page through the bucket
• c5a0168 fix(lib-storage): resolve a file pattern without a directory against the working directory
• 9a7cbd2 fix(plugin-s3reader): match object patterns literally and read each object once
• 47d1c6d fix(lib-storage): detect the compression of a stream from its leading bytes
• 7f64303 fix(plugin-excelreader)!: read large workbooks and keep columns aligned
• 5717ef0 fix(plugin-excelwriter): write correct cell types and keep the header row
• 96531cb ci(github-actions): drop the disabled CodeQL workflow
• f4d5d94 chore(deps): bump com.sap.cloud.db.jdbc:ngdbc from 2.28.8 to 2.30.7 (#1602)
• e27144c chore(deps): bump com.github.luben:zstd-jni from 1.5.7-16 to 1.5.7-17 (#1603)
• ee78915 chore(deps): bump github/codeql-action from 4.38.0 to 4.38.1 (#1604)
• e8b2232 fix(plugin-jsonfilereader): stop re-reading the column configuration for every record (#1601)
• c2407e8 fix(plugin-mongodbreader,plugin-mongodbwriter): repair the filters and the stored types (#1600)
• bf076f0 docs(agents): fix the run instructions and the plugin development link
• 6494069 docs(agents): fix the addax.sh invocation and merge a duplicated doc link
• 6f68e12 fix(docs): center the tagline in the social preview image
• eb6531c ci(github-actions): keep only the commit lint job and rename the workflow (#1599)
• 9097e4b ci(github-actions): skip the E2E job for pull requests that cannot change it (#1598)
• 0e520c6 docs(brand): add the DuckDB logo to the supported data source grids (#1597)
• 5c82023 feat(lib-rdbms): add DuckDB support with a reader and a writer plugin (#1596)
• 6b0c0f4 fix(lib-rdbms): resolve the SQL dialect instead of throwing on unknown ones (#1595)
• 684249f fix(script): accept -o as an alias for --offline
• 5ac4892 ci(github-actions): stop waiting for the Central publish to finish
• 26bffae docs(brand): unify product messaging around one description and tagline
• 0048576 docs(release): point the 6.1.0 notes at absolute URLs
• ad905b5 [maven-release-plugin] prepare for next development iteration