Notable Highlights
- Lineage Visualization
- Pipelines and Tasks, Flows and Jobs
- Airflow Lineage
- Editable Field Descriptions
- Nested Schema Viz
- Search Improvements
- datahub CLI
- Official PyPi packages
- Production-quality Helm scripts
- New Integrations
- Officially-supported Sources: Airflow, AWS Glue, dbt, Druid, Superset, MongoDB, Oracle
Changelog
- #2440 @dexter-mh-lee feat(k8s): Move helm charts out of contrib
- #2397 @gabe-lyons feat(lineage): implement support for datasets, charts and dashboards downstream lineage fetching in a generic way
- #2434 @adriaanslechten feat(ingest) LDAP groups ingestion
- #2438 @hsheth2 fix(ingest): use entrypoints lib instead of pkg_resources
- #2425 @gabe-lyons feat(ingest): adding superset ingestion source
- #2433 @topwebtek7 fix(react): fix lineage sidebar buttons
- #2436 @hsheth2 fix(ingest): support custom snowflake types
- #2419 @topwebtek7 feat(react): add dataJob, dataFlow entity pages, refactor with fragments
- #2418 @frsann Fix(search): fix datajob and dataflow search mappings
- #2429 @hsheth2 fix(ingest): fix chart type enum serialization and add tests for rest emitter
- #2431 @shirshanka docs: Update agenda for Apr 23 townhall
- #2427 @hsheth2 fix(ingest): ensure upstreams in airflow lineage emission are entities
- #2426 @hsheth2 fix(ingest): include database info for snowflake
- #2424 @hsheth2 feat: add s3 data platform and logo
- #2423 @topwebtek7 feat(react): schema visualization add support for nested structs
- #2422 @topwebtek7 fix(react): lineage sidebar buttons should refer to the selected entity
- #2421 @dexter-mh-lee fix(kafka-setup): Fix start script for kafka setup
- #2417 @topwebtek7 feat(react): update dataset entity default svg icon
- #2411 @thomasplarsson feature(ingestion): Adding the concept of transformers
- #2415 @dexter-mh-lee fix(k8s): Add credentials to kafka-setup job and clean up
- #2412 @hsheth2 feat(ingest): add Kafka-based emitter example
- #2413 @gabe-lyons fix(lineage): allow lineage viz to handle circular dependencies
- #2414 @dexter-mh-lee fix(kafka-setup): Add the correct context to the git workflow for pushing kafka-setup image
- #2403 @hsheth2 fix(ingest): bump avro-gen3
- #2406 @topwebtek7 feat(react): use default entity icon if lineageentity has no icon
- #2408 @hsheth2 fix(ingest): properly handle fieldDiscriminator with restli
- #2409 @hsheth2 fix(ingest): add sqlalchemy extra
- #2398 @G-nther feat(kafka-setup): add option for SSL and topic partition config via environment
- #2404 @dexter-mh-lee feat(k8s): add extraEnvs to setup jobs
- #2407 @topwebtek7 feat(react): add footer buttons in lineage sidebar
- #2405 @thomasplarsson feature(ingestion): Make origin/fabric_type configurable
- #2384 @topwebtek7 feat(react): add padding between tags and description on datasets profile page
- #2396 @gabe-lyons feat(sample): adding sample mces for dataflows and datajobs
- #2400 @hsheth2 fix(ingest): streamline codegen init methods
- #2382 @topwebtek7 feat(react): update schema table to have fixed description column, set line break with max description width
- #2402 @dexter-mh-lee fix: Fix env variable setup for kafka, mysql-setup docker containers
- #2401 @hsheth2 fix(ingest): add db name to postgres URNs
- #2393 @hsheth2 fix(ingest): enable mypy
disallow_incomplete_defs
anddisallow_untyped_decorators
- #2395 @gabe-lyons fix(react): fix access to pictureLink in charts and dashboards
- #2399 @gabe-lyons fix(tags): check description existence on tags
- #2383 @topwebtek7 feat(react): fix long descriptions overflow issue in lineage side panel
- #2392 @hsheth2 refactor(ingest): update test harness to use a compose file per test
- #2391 @topwebtek7 feat(react): fix browse link of last breadcrumb linked to unknown page
- #2385 @dexter-mh-lee feat(mysql-setup): Add the ability to specify database name for mysql-setup
- #2389 @hsheth2 feat(ingest): add generic sqlalchemy source
- #2390 @dexter-mh-lee feat(k8s): Add ability to add service accounts to setup jobs
- #2387 @dexter-mh-lee fix(kafka-topic-convention): Fix DAOs that do not refer to TopicConvention
- #2386 @dexter-mh-lee feat(index): Add index naming convention for elasticsearch
- #2388 @hsheth2 fix(ingest): report correct version status in dev mode
- #2368 @hsheth2 feat(ingest): add Airflow lineage backend
- #2380 @OddCN fix(docs): fix config example for file sink
- #2362 @dexter-mh-lee feat(k8s): Update pods with correct probes and remove unnecessary dependencies
- #2372 @thomasplarsson fix(ingestion): dont crash on non-RecordSchema topics
- #2360 @hsheth2 docs(ingestion): remove outdated data-source-onboarding.md docs
- #2376 @topwebtek7 feat(react): hide Owned By label in card if no owners
- #2373 @shubham49 fix(react): ownership rendering
- #2377 @topwebtek7 feat(react): add null state indicator in user profile when no entities
- #2379 @topwebtek7 feat(react): update avatar to use initial if no image, refactor all avatars with custom one
- #2369 @gabe-lyons feat(lineage): support arbitrary entity types in lineage viz
- #2364 @thomasplarsson fix(ingestion): Support mapping from avro "boolean" and "map" types t…
- #2343 @thomasplarsson fix(ingestion): properly detect optional fields in avro schemas
- #2370 @topwebtek7 feat(react): add empty state UI for browse when no entities
- #2242 @frsann feat(datajob): Datajob graphql query
- #2367 @topwebtek7 feat(react): add dropdown menu links, menu styling, removed warnings
- #2365 @frsann chore(dependabot): Update pyyaml version
- #2366 @topwebtek7 feat(react): add icons on entities, updated styling in LineageViz
- #2351 @hsheth2 fix(ingest): add test for avro serialization and deserialization
- #2361 @hsheth2 feat(cli): Add support for checking docker memory usage
- #2358 @topwebtek7 feat(react): original description shows in edit modal even when the description has been updated
- #2357 @gabe-lyons feat(react): improving error logging on dataset entity
- #2356 @dexter-mh-lee fix(elasticsearch): Fix inconsistencies between documents and elasticsearch mappings
- #2359 @hsheth2 fix(ingest): support
python3 -m datahub
- #2353 @hsheth2 chore(ingest): remove unused
plugin_requirements.txt
file - #2352 @hsheth2 fix(ingest): bump pybigquery version
- #2350 @hsheth2 fix(ingest): support
datahub --version
- #2349 @gabe-lyons feat(lineage): improve lineage re-focus experience
- #2341 @frsann feat(tags): Add tag graph builder
- #2348 @jjoyce0510 fix(Ember App): Allow ember build (disabled by default)
- #2345 @hsheth2 fix(cli): add --verbose flag for
datahub check plugins
- #2346 @gabe-lyons fix(lineage): add upstream arrows back in
- #2347 @hsheth2 feat(ingest): add Oracle db support
- #2336 @topwebtek7 feat(react): add description edit behavior along with modal
- #2340 @gabe-lyons feat(lineage): adding ghost edges indicating hidden dependencies
- #2331 @hsheth2 feat(ingest): start airflow integration + metadata builders
- #2339 @hsheth2 fix(ingest): add support for database and table patterns to glue source
- #2338 @hsheth2 fix(docker): remove
restart: always
from docker-compose for consistency - #2335 @gabe-lyons feat(lineage): adding directionality to lineage edges to make the visualization more clear
- #2337 @gabe-lyons fix(lineage): fixing lineage layout bugs
- #2319 @amonkhouse feat(ingest): adding support for AWS Glue
- #2312 @shakti-garg feat(es-setup): add logic in elasticsearch setup to compare-and-update index if already exists
- #2333 @gabe-lyons feat(lineage): expandable lineage visualization for dataset <> dataset lineage
- #2332 @hsheth2 docs: add wolt logo to frontpage
- #2315 @grantatspothero feat(ingest): adds experimental support for ingesting Looker metadata
- #2330 @luck02 fix(test): dbt-manifest files
- #2329 @topwebtek7 feat(react): moving filter panel from modal to drawer
- #2328 @hsheth2 build: remove deprecated ember app from build
- #2327 @hsheth2 feat(ingest): verify dynamic registry types at runtime
- #2316 @joemirizio feat(ingest): dynamically register plugins
- #2325 @hsheth2 fix(ingest): remove outdated metadata-ingestion scripts
- #2313 @shakti-garg fix(k8s): make es-setup job parameters more contextual
- #2322 @gabe-lyons docs(theme): making
yarn start
instructions more explicit - #2317 @hsheth2 doc: update slack links to https
- #2324 @frsann fix(datajob): Fix URI templates for datajob and dataflow
- #2320 @frsann fix(tags): Support creating tags with MCE
- #2323 @arunvasudevan fix(docs): Update metadata-serving.md
- #2318 @dexter-mh-lee fix(docker): Fix issue in gms start.sh
- #2321 @shirshanka docs: Update next townhall details, fixup links and misc docs
- #2251 @bernardino feat(Kubernetes): Add JMX exporter containers to all DataHub components
- #2308 @dexter-mh-lee fix(search): Fix styling for column match snippet
- #2302 @shakti-garg feat(k8s): Add k8s hook in datahub helm chart for setting up elasticsearch
- #2298 @dexter-mh-lee feat(docker): Add dockerfile for initializing an existing mysql server
- #2297 @shakti-garg feat(kafka-config): add variable KAFKA_CONSUMER_GROUP_ID to override group id value for kafka consumer
- #2303 @frsann feat(elasticsearch): Elasticsearch simple auth over https support
- #2306 @hsheth2 feat(ingest): support environment variables in recipes
- #2305 @hsheth2 fix(ingest): deal with breaking change from confluent_kafka
- #2304 @hsheth2 fix(cli): validate only required containers
- #2295 @hsheth2 feat: datahub check local-docker
- #2296 @gabe-lyons fix(docs): fix index builder link
- #2287 @gabe-lyons fix(react): improving layout of no data state
- #2290 @hsheth2 fix(ingest): resolve array serialization bug
- #2284 @pedro93 fix(ingest): Add druid-specific identification logic
- #2291 @dexter-mh-lee feat(healthcheck): Add Healthcheck servlet for GMS
- #2289 @hsheth2 feat(ingest): MongoDB ingestion source
- #2286 @dexter-mh-lee feat(search-by-field): Add the ability to search for field names
- #2288 @dexter-mh-lee feat(docker): Add healthchecks to datahub docker images
- #2283 @hsheth2 fix: remain as root until final datahub-frontend build stage
- #2267 @thomasplarsson feat(react): enable the react frontend to use SSL when talking with GMS
- #2277 @gabe-lyons feat(react): updating entity layouts to be friendly to smaller viewport sizes
- #2279 @hsheth2 fix(ingest): use custom pybigquery ref to get descriptions
- #2278 @hsheth2 docs(ingest): add python versions badge
- #2247 @luck02 feat(ingest): Add DBT source to metadata-ingestion
- #2276 @gabe-lyons feat(react): update search suggestion styling
- #2274 @gabe-lyons fix(react): fix randomizing suggestions on autocorrect
- #2250 @saxo-lalrishav feat : custom menu options in the main dropdown menu
- #2268 @hsheth2 fix(docker): use local datahub-frontend in dev.sh
- #2270 @gabe-lyons feat(react): show search suggestions
- #2269 @hsheth2 docs: update readme for v0.7.0 release
- #2264 @pedro93 feat(react): add druid logo
- #2266 @gabe-lyons fix: fixing tag formatting issues