Addax 6.3.0

Addax 6.3.0 is a minor release: it overhauls how transformers are loaded, validated and parameterised, removes the built-in date variables from addax.sh, fixes a permanent hang when the channel capacity is smaller than the exchange batch, restores the native transformers that a regression had broken, makes the reported write counters agree with the read ones, keeps DECIMAL fractions and DATETIME text through SQLite round trips and preserves the time-of-day of a DATETIME written to an ORC string column — plus 36 dependency updates. The end-to-end suite was trimmed to the core database and file cases (20 cases) and a reproducible performance baseline harness was added.

The version is 6.3.0 rather than 6.2.1 on purpose: this release contains a new feature and a backward-incompatible change, which is a minor bump under the versioning scheme.

⚠️ Breaking changes

1. addax.sh no longer computes date variables (#1715)

The launch script used to inject ten system properties — curr_date_short, curr_date_dash, curr_datetime_short, curr_datetime_dash, biz_date_short, biz_date_dash, biz_datetime_short, biz_datetime_dash, biz_datetime_0_short and biz_datetime_0_dash. The biz_* family was hard-coded to the previous calendar day, which is wrong on weekends, holidays, T+0 runs and historical fillback, and because the injected -D arguments came after the user's -p values they could not be overridden.

The computation block and the injection are gone. Date variables are now the caller's responsibility: addax-admin fills them per task business date, and a direct addax.sh run must pass them explicitly.

• Migration: pass every variable the job references with -p, for example -p "-Dbiz_date_short=20260101" (one -D per variable, in the same -p string). Unresolved placeholders are left verbatim, so a job that still references ${biz_date_short} will put the literal text into SQL or paths after the upgrade — check every job that uses ${curr_*} or ${biz_*} before upgrading.
• AGENTS.md and SKILL.md now state the new contract.

✨ Features

1. Transformer loading, validation and parameter handling (#1688)

The transformer module treated configuration mistakes as per-record failures: a transformer with a missing or malformed parameter turned every row into a dirty record instead of failing the job, the user-transformer loader resolved transformer.json against the working directory, dx_null_to_default never received its parameters, dx_pad could hang forever on an empty padding string, and every built-in re-parsed its positional parameters on each record.

• Named parameters. The five built-in transformers accept a named form: dx_substr (idx, pos, length), dx_pad (idx, flag, length, chr), dx_replace (idx, pos, length, str), dx_filter (idx, operator, expr) and dx_map (idx, operator, value). For example {"name": "dx_pad", "parameter": {"idx": 0, "flag": "l", "length": 8, "chr": "*"}}. The old columnIndex + paras form keeps working but logs a deprecation warning once per transformer name; mixing both forms in one transformer fails at startup.
• Startup validation. Parameters are parsed and validated once per execution instead of once per record. A missing, unknown or contradictory parameter fails the job before any task starts, with the transformer name and the expected form, instead of producing dirty records for the whole run.
• Compiled once. Groovy code and regular expressions are compiled at job start. dx_groovy can be invoked once only, which was documented before and is now enforced.
• User transformers. transformer.json is no longer resolved against the working directory: the registry loads the transformer directories the job references, validates the class (public, non-abstract, public no-arg constructor) once, and reports the transformer name and path instead of silently skipping it. The thread context classloader is restored correctly around the load.
• dx_null_to_default now receives its parameters and requires parameter.column; dx_pad no longer hangs on an empty padding string.
• Records filtered by a transformer are counted separately from dirty records, so the reported numbers no longer mix the two.
• New runnable examples: transformer_named_demo.json and transformer_legacy_demo.json; transformer_demo.json, transformer_groovy.json and transformer_null_to_default.json were refreshed.

🐛 Fixes

Data correctness

• sqlite (#1706): the JDBC driver types a column from the storage class of the row it is positioned on, so a DECIMAL/NUMERIC fraction was truncated when a PK split started on an integral value, and driver 3.54.1.0 also maps DATETIME to Types.TIMESTAMP, which would have written TEXT values as INTEGER epoch milliseconds. The reader now types DECIMAL/NUMERIC from the declared column type and the writer binds DECIMAL/NUMERIC through setBigDecimal and writes Types.TIMESTAMP as TEXT with milliseconds, so fractional values and DATETIME text survive round trips. The reader also defaults username/password, so a job copied from the plugin template starts instead of failing before the first row.
• hdfswriter, s3writer (#1707): ORC string targets rendered every non-TIME date column as yyyy-MM-dd, silently truncating a DATETIME (for example a BSON date from mongodbreader) to a date-only string. DATE/DATETIME now render through the common column formatters (common.column.dateFormat / datetimeFormat / timeZone), matching the text and parquet writers; DATE stays date-only and TIME keeps its nanos-aware path. Consumers of such ORC files now see the time-of-day.
• core (#1673): the write side counted every record it drained from the channel, terminate marker included, so a task reported one more written record than the reader produced — a 4-task job that moved 400 rows reported writeReceivedRecords=404. writeSucceedRecords now equals readSucceedRecords for a job that wrote everything it read, and the three channel waits restore the thread's interrupt status and report the same AddaxException type instead of dropping it or throwing a bare IllegalStateException.
• core (#1664): native transformers dx_substr, dx_replace, dx_map and dx_filter were broken by a regression — they demanded Groovy code and their columnIndex/paras were silently dropped, so even the bundled transformer_demo.json could not run. The pre-regression semantics are restored: only the Groovy functions consume code/codeFile, every other transformer requires columnIndex and takes optional paras, and a missing columnIndex fails with a clear message.
• plugin-paimonwriter (#1679): any warehouse reached through Hadoop failed with NoClassDefFoundError: com/google/protobuf/ServiceException raised from the addBlock path, because Paimon excludes protobuf from its Hadoop dependencies and the assembled plugin carried none. protobuf-java (3.25.5) is now bundled the way hdfswriter, hdfsreader, icebergwriter and s3writer already do; file:// warehouses were the only working case before.

Reliability

• core (#1668): with core.transport.channel.capacity smaller than the exchange batch (default 32), MemoryChannel waited for a whole batch to fit and never enqueued anything, so the reader blocked forever and the job sat at "Total 0 records" with no error. Batches are now enqueued record by record, waiting per record for room in the queue and the byte budget, and the consumer is signalled after every record instead of on the 200 ms poll timeout. A single record larger than the byte budget fails with OVER_LIMIT_ERROR instead of waiting forever, and clear() resets the byte accounting together with the queue.
• core (#1670): a fail-over retry could start while the previous attempt was still writing (duplicate rows), and the wait degraded to an unbounded join(0) whenever the retry budget was exactly exhausted. Each attempt now runs on its own two-worker executor, and its termination is the only "attempt is over" signal, so the previous attempt is dead before a retry starts; the next task is dispatched immediately instead of on a 100 ms tick.
• core (#1667): plugin entry classes are resolved and validated at job start — the class must exist, extend Reader.Job/Reader.Task or Writer.Job/Writer.Task, be public, non-abstract, static when nested and declare a public no-arg constructor. A broken plugin now fails before any task starts, with the plugin name, the plugin.json path and the attempted class name, instead of failing mid-run with a raw stack dump.
• core (#1665): job.content is normalized from the legacy array form to a dict at parse time; a multi-element array logs a warning and the first element wins, an empty array fails fast, and the post-split task list moved to an internal key.
• Internal cleanups with no intended behavior change: the task state machine now lives in an immutable TaskState (#1672), the dead single-record channel API and RecordExchanger are gone (#1669), framework keys are read through typed JobConfig/TaskConfig views (#1666), and the unused abstractions from the never-implemented distributed mode were removed (#1663).

Scripts, packaging and CI

• install_plugins.sh (#1695): validates plugin names and versions, installs into libs/ with staged atomic replacement and backups, restores required transitive dependencies, verifies metadata and the bootstrapped Maven archive, and adds force, locking, summaries, strict release checksums and interactive Maven output. Existing lib/ installs are reinstalled under libs/. The same commit renames the Elasticsearch writer's template resource so template-based job generation finds plugin_job_template.json.
• Distribution (#1719): dorisreader was missing from the root package.xml, so -Pdistribution produced a tree without plugin/reader/dorisreader even though the module is in the reactor. Release and local distributions now contain all 66 reactor plugins.
• Docker (#1662): the published image is now built from the same assembled addax-<version>/ tree as the GitHub release archive instead of being compiled again inside the Docker build, and a plain docker build . from a clone still builds from source. The release job no longer rebuilds the distribution inside the Docker build.
• YAML examples (#1674): the oracle2txt and stream2hdfs examples stored the field delimiter as a raw 0x01 byte, which SnakeYAML rejects with "special characters are not allowed". The four affected files use the escaped "\u0001" form and now load.
• Security (#1700): Guava 32.0.1-jre → 33.7.2-jre for CVE-2026-102554 — a small Java-serialized payload could force a multi-gigabyte allocation while deserializing CompactHashMap/CompactHashSet/MapMakerInternalMap, crashing the JVM with OutOfMemoryError.

🧪 Testing

• The end-to-end suite was trimmed to the core database and file cases (#1671): the DuckDB, S3 (moto) and Elasticsearch cases and their harness pieces were removed so the suite stays small enough to be read and trusted and needs neither a Python virtualenv nor an external cluster in CI. It now has 20 cases (010–190, 200_sqlite2sqlite, 260_hdfsreader_roundtrip), all green locally with no skips. The DuckDB, S3 and Elasticsearch plugins themselves are untouched.
• New case 200_sqlite2sqlite (#1706): ten PK splits, legacy TEXT and INTEGER-epoch DATETIME values, a scale-less NUMERIC target and a DECIMAL column whose fraction would be lost if the reader typed it from the current row.
• New reproducible performance harness under e2e/perf (#1689): a deterministic 2M-row MariaDB fixture, scenario jobs (MySQL → HDFS text/parquet/ORC, MySQL → local text/ORC, MySQL → MySQL, stream → HDFS/sink), a baseline runner with JFR/GC capture, a channel sweep and a data-phase report. The measured baseline report is linked from both READMEs; results are git-ignored.

📦 Dependencies

36 dependency and CI updates, among them:

• paimon 1.2.0 → 2.1.0, poi 5.5.1 → 6.0.0, duckdb_jdbc 1.5.6.0 → 2.0.0.1, clickhouse-jdbc 0.10.0 → 0.11.1, sqlite-jdbc 3.50.3.0 → 3.54.1.0
• jackson 2.22.3 → 2.23.0, ucanaccess 5.1.8 → 5.3.0, netty 4.2.19.Final → 4.2.20.Final, guava 32.0.1-jre → 33.7.2-jre (CVE-2026-102554)
• zstd-jni 1.5.7-21 → 1.5.7-26, lz4-java 1.12.0 → 1.14.0, snappy-java 1.1.10.11 → 1.1.10.12, jsch 2.28.7 → 2.28.8
• commons-lang3 3.21.0 → 3.22.0, json-smart 2.6.0 → 2.6.1, gson 2.14.0 → 2.15.0, ognl 3.4.16 → 3.4.17, javadbf 1.14.1 → 1.14.2, joda-time 2.15.0 → 2.15.1, fastjson2 2.0.65 → 2.0.66
• mysql-connector-j 8.2.0 (pinned, #1696): 8.2.0 is the last Connector/J release that works with MySQL 5.7 — 8.3.0 and later require MySQL Server 8.0 or newer — so dependabot no longer proposes routine upgrades while CVE-driven upgrades still open pull requests.
• Build and CI: maven-enforcer-plugin, central-publishing-maven-plugin, spotbugs-maven-plugin, actions/setup-java 6.2.0, docker/login-action 4.8.0.

⬆️ Upgrading

• Distribution: addax-6.3.0.tar.gz and addax-6.3.0.tar.gz.sha256sum.asc — check the checksum before unpacking.
• Docker: wgzhao/addax:6.3.0 (Docker Hub) and quay.io/wgzhao/addax:6.3.0.
• Maven Central: com.wgzhao.addax:addax-core:6.3.0.
• Check first if:
  • a job references ${curr_*} or ${biz_*} and runs directly through addax.sh (or any scheduler other than addax-admin) — pass the values with -p;
  • a downstream consumer parses a DATETIME column from an ORC string file as a date-only string;
  • a job still configures transformers with columnIndex + paras — it keeps working, but migrate to the named form, and do not mix the two styles in one transformer;
  • plugins were installed with install_plugins.sh — existing lib/ installs are reinstalled under libs/.
• Plugin documentation: https://github.com/wgzhao/addax-docs