dwh-migration-tools v1.11.1
Highlights
New: Databricks metadata connector (--connector databricks). Dumps Unity Catalog metadata over the Unity Catalog REST API (no SQL warehouse needed) and the hive_metastore catalog via SQL: catalogs, schemata, tables, columns, views, table constraints (PK/FK/UNIQUE/CHECK), column masks and row filters. Output archive: databricks.dump.zip. hive_metastore can be read on a SQL warehouse (--warehouse <id>) or, new in this release, on a running all-purpose cluster (--cluster <cluster-id>, Databricks Runtime 16.2+). On a warehouse, databricks.metadata.warehouse-transport selects jdbc (default on Java 11+, 1.7–1.9× faster), inline, or external-links (default on Java 8). Per-object calls now run in parallel (--thread-pool-size). --assessment is not supported; use the Databricks Discovery Notebook for usage data.
Security: dependency upgrades for 41 published advisories. jackson 2.21.7, netty 4.1.137 (incl. CRITICAL CVE-2026-75595), bouncycastle 1.85, guava 33.7.2, zookeeper 3.9.5, httpclient5 5.6.3 / httpcore5 5.4.3, snappy-java 1.1.10.9, postgresql 42.7.12, redshift-jdbc 2.2.2, Apache Kerby 2.1.2 (CVE-2026-57914). All Java 8 compatible. Advisories with no Java 8 fix (logback 1.3, spring-core 5.3, jetty 9.4, opentelemetry-api) are documented in .security/known_cves.yml.
Standalone jar. The build now produces dwh-migration-dumper.jar, a single runnable shadow jar (used by the bqms gcloud component). It no longer crashes on first DNS lookup on Java 18+ (dnsjava resolver provider excluded).
Snowflake. New Snowflake MDS TCO discovery scripts under migration-discovery-scripts/snowflake/ (workload structure, credit attribution, date ranges, IsAdaptiveWarehouse).
Telemetry. The dumper now records the runtime Java version in compilerworks-metadata.yaml and dumper-telemetry.jsonl.
⚠️ Breaking change
- The deprecated
snowflake-liteconnector has been removed (#1189). Usesnowflakeinstead.
What's Changed
- [b/567154873] build(dumper): package standalone dumper shadowJar for gcloud component by @haselbach in #1180
- feat(dumper): record runtime Java version in metadata and telemetry by @haselbach in #1181
- Introduce Snowflake migration discovery scripts by @sayuzbas-google in #1182
- [b/575552296] fix(dumper): exclude dnsjava resolver provider from shadowJar by @haselbach in #1183
- Add Databricks metadata connector by @jin-jj in #1184
- Bumping google api client version to 2.9.2 by @sayuzbas-google in #1185
- Update OWASP scan configuration to include OSS Index warning by @kaxuna in #1186
- Update labeler.yml (pull_request_target → pull_request) by @kaxuna in #1187
- Provide new ways to extract Snowflake MDS TCO data by @sayuzbas-google in #1188
- Remove deprecated snowflake-lite by @sayuzbas-google in #1189
- Refactor OWASP scan workflow and improve comments by @kaxuna in #1190
- Adding date ranges for MDS extraction by @sayuzbas-google in #1191
- Align Snowflake MDS workload categorization and credit attribution fallback by @sayuzbas-google in #1192
- Add IsAdaptiveWarehouse to Snowflake MDS workload structure extraction by @sayuzbas-google in #1193
- Bump vulnerable dumper dependencies (jackson, guava, netty, zookeeper, httpclient5, bouncycastle, snappy, JDBC drivers) by @prescottbreeden in #1195
- [b/579421681] Pin Apache Kerby to 2.1.2 (CVE-2026-57914) by @kaxuna in #1196
- Support hive_metastore using cluster resource and JDBC driver (Databricks) by @jin-jj in #1197
New Contributors
- @prescottbreeden made their first contribution in #1195
Full Changelog: v1.11.0...v1.11.1