github datahub-project/datahub v1.7.0.1
on GitHub

3 hours ago

DataHub v1.7.0.1

Patch release for the v1.7.0 line: authorization tightening, CVE dependency bumps, entity-graph-cache and lineage correctness fixes, optional optimistic locking for aspect writes, and the remainder of the custom-assertion stack.

Requirements

  • CLI / Python SDK: 1.7.0.9
  • Helm Chart: 1.1.0

Full upgrade guidance, including every breaking change and migration step: Updating DataHub — v1.7.0.1.

Upgrade path

  • From v1.7.0: drop-in image bump. No ZDU or Helm chart change is required. System-update runs a resumable backfill that copies embedded assertion notes to the new assertionNote aspect (tunable with SYSTEM_UPDATE_ASSERTION_NOTE_MIGRATION_ENABLED, _BATCH_SIZE, _DELAY_MS, _LIMIT).
  • From pre-1.7.0: the v1.7.0 path still applies — you must upgrade to v1.6.0 first (Helm chart 1.0.3, let system-update complete), then upgrade to v1.7.0.1 with Helm chart 1.1.0. Do not skip 1.6.0.

Model changes: additive only. The assertionNote aspect is now registered on the assertion entity so ingestion can fully upsert assertionInfo without clobbering user-authored notes. Assertion URNs and run history are unchanged.

Action required before upgrading

Area What changed Who is affected
System client secret The hardcoded systemClientSecret default (JohnSnowKnowsNothing) is removed from the server configs (#18886) Set DATAHUB_SYSTEM_CLIENT_SECRET on GMS, MAE/MCE/PE consumers, the frontend, and Actions before upgrading if you still rely on the built-in default
Timeseries reads Dataset profile / usage / operations timeseries (and dashboard usage stats) now require the matching View Dataset Profile / Usage / Operations privileges on Rest.li and OpenAPI as well as GraphQL (#19508) API clients that previously read these aspects with only entity GET — grant the privileges or rely on the default view-dataset-sensitive policy
Role & group membership writes roleMembership requires Manage Policies; groupMembership / nativeGroupMembership requires Edit Group Members; corpGroup ownership requires Edit Owners — checked at the aspect layer across GraphQL, OpenAPI, and Rest.li (#19360) Automation writing membership outside the UI. Only additions are checked; removals, unchanged re-ingestion, and system writes pass. Toggle with metadataChangeProposal.validation.aspectAuthorization.privilegeGrant.enabled (default true)
View authorization (VBAC) container and schemaField are view-restricted by default; schema-field View Entity Page inherits from the parent dataset (#19212) Deployments with VIEW_AUTHORIZATION_ENABLED=true — ensure View Entity Page policies cover containers and parent datasets

Security and authorization

  • Default self policy (#19297) — limited to VIEW_ENTITY_PAGE and GET_ENTITY on the actor's own entity.
  • GMS /auth helpers (#19316) — signUp, resetNativeUserCredentials, verifyNativeUserCredentials, and getSsoSettings now require system client credentials, matching generateSessionTokenForUser. Frontend login, signup, password reset, and SSO are unchanged.
  • System-write identification (#19384) — aspect authorization no longer treats client-supplied appSource as proof of a system write.
  • Schema-field incidents (#19405) — authorized against the parent entity.
  • Lineage scroll (#19463) — unauthorized OpenAPI lineage scroll endpoints are redacted as restricted.
  • UI sanitization (#19298, #19299, #19300) — query and incident descriptions, documentation PDF preview iframe src, and embed renderUrl schemes.
  • Ingestion secret masking (#19515, #19526, #19561, #19332) — masking is unified, idempotent, and fail-closed (mask before truncation; install once per process). Recorded HTTP cassettes are scrubbed.
  • AWS credentials (#19483) — OpenSearch IAM signing, Bedrock embeddings, STS, and object storage share a process-wide credential provider and client lifecycle.

Reliability and performance

  • Entity-graph cache (#19498, #19288, #19301, #19412) — relationships are no longer silently dropped; indexed search fields are fetched for the cache; null relationship endpoints are skipped; hierarchy graph fallback batches frontier scrolls.
  • Container contents (#19489) — Container.relationships(types: [IsPartOf], direction: INCOMING) again returns all contained entities from the live graph.
  • Lineage traversal (#19413) — scrollAcrossLineage is bounded to prevent GMS OOM on wide graphs.
  • Case-insensitive relatedEntityTypes (#19268).
  • Hot-path caching and batching (#19404, #19476, #18266) — corpGroup roleMembership cached via SystemEntityClient, per-request CorpUserFlags served from a cached aspect, and Dashboard usageStats buckets and metrics batch-loaded across a page.
  • Optional optimistic locking for aspect writes (#19249) — OPTIMISTIC_LOCKING_ENABLED replaces SELECT FOR UPDATE with compare-and-set on SystemMetadata.version for Ebean storage. SCOPED_RETRY_ENABLED retries only the conflicted URN's branch, and ENTITY_WRITE_LOCK_BACKEND=hazelcast serializes same-(urn, aspect) writers off the DB connection pool. PostgreSQL aspect writes also take row locks in consistent primary-key order. All of these default off.

Assertions

  • Custom-assertion stack completed on the 1.7 line (#18882) — API/SDK, UI, the assertionNote aspect, and the dbt / Great Expectations / YAML data-contract writers. Those writers emit AssertionType.CUSTOM instead of the deprecated DATASET shape on the next run; assertion URNs are unchanged and run history is preserved. Assertions that are never re-ingested keep their stored DATASET aspects and continue to render. See Custom Assertions.

Ingestion

  • nltk 3.10.1 excluded (#18889) — that version silently breaks document chunking.
  • GitPython floor raised to 3.1.58 (#19506).
  • Bundled default CLI / Python SDK is 1.7.0.9 (#19605).

Dependency CVE bumps

Quickstart

datahub docker quickstart (and --version v1.7.0) resolves to the v1.7.0.1 images once the version mapping update lands.

What's Changed

Full Changelog: v1.7.0...v1.7.0.1

Check out latest releases or
releases around datahub-project/datahub v1.7.0.1

Don't miss a new datahub release

NewReleases is sending notifications on new releases.

Get notifications