Notable Highlights
- New React Application re-written from the ground up
- Support for GraphQL
- New Metadata Ingestion Framework (Python)
- Officially-supported Sources: Kafka, MySQL, SQL Server, Hive, Postgres, Snowflake, BigQuery, AWS Athena, Druid, LDAP
- New Homepage and Hosted Docs redesign at datahubproject.io
- Product Features: SSO (OIDC), Tags, Themes, Dashboards
- Metadata Backend Implementations: MLModel ecosystem, DataFlow ecosystem
- Move to Elasticsearch 7. Migration guide from 5.x here
Changelog
- #2263 @jplaisted feat(search) BREAKING Support ElasticSearch 7, drop ES5
- #2260 @gabe-lyons fix(tags): fixing margins on tags for long descriptions
- #2259 @hsheth2 docs: update roadmap progress
- #2258 @dexter-mh-lee refactor(demo): Add empty global tags to BigQuery demo data
- #2255 @jjoyce0510 feat(react): Adding shadow and deeper linear gradient
- #2254 @gabe-lyons feat(tags): improving elastic search templates for tags
- #2253 @gabe-lyons fix(tags): fix ownership on tag create
- #2256 @hsheth2 fix: update slack links
- #2248 @gabe-lyons feat(tags): editing tags from react client on datasets, schemas, charts & dashboards
- #2252 @jjoyce0510 refactor(react): React as the default UI
- #2246 @hsheth2 feat(ingest): various minor fixes
- #2245 @jjoyce0510 feat(react): Adding big query logo
- #2249 @gabe-lyons fix(react): enabling charts and dashboards to be supported by theme config
- #2235 @pedro93 feat(ingest): Add support for druid
- #2244 @gabe-lyons feat(react): moving schema tab to be default
- #2243 @shirshanka docs: adding mar-19 townhall agenda
- #2240 @dexter-mh-lee feat(tags): Enable search for datasets by tags
- #2236 @pedro93 feat(k8s): Add metadata-ingestion as a Helm component
- #2241 @shirshanka docs: Improving architecture docs
- #2239 @hsheth2 feat(docs): use gradle for building docs
- #2232 @hsheth2 fix(ingest): various avro codegen fixes
- #2237 @gabe-lyons fix(dataflow): fixing browse dao access
- #2166 @arunvasudevan feat: MLmodel Graphql Query
- #2197 @frsann feat(datajob): Backend implementation
- #2233 @jjoyce0510 refactor(react): All entity search UI + misc improvements
- #2234 @jjoyce0510 docs(react): Oidc React Doc Updates
- #2231 @dexter-mh-lee fix(docker): start issue when there are multiple kafka brokers in bootstrap config
- #2227 @jjoyce0510 refactor(React): Misc UI improvements
- #2230 @hsheth2 fix(ingest): pin version of avro-gen3
- #2226 @hsheth2 fix(ingest): use python extras in docker image
- #2224 @hsheth2 feat(ingest): use plugin system based on Python extras
- #2190 @jjoyce0510 feat(react): SSO support simple OIDC authentication
- #2223 @dexter-mh-lee Added images to es/kafka-setup
- #2222 @dexter-mh-lee fix(ci): rename file to match git workflow needs
- #2220 @dexter-mh-lee fix(ci): remove paths_ignore from workflow files
- #2219 @thomasplarsson refactor(ingest): improve athena source api and documentation
- #2221 @gabe-lyons fix(ci): setting CI to false for builds
- #2218 @gabe-lyons feat(react): hiding raw schema button when no raw schema exists
- #2216 @dexter-mh-lee fix(es-setup): Add git workflows to upload docker for elasticsearch and kafka setup
- #2213 @thomasplarsson feat(ingest): add aws athena ingestion source
- #2217 @gabe-lyons fix(ci): fail CI on react build errors
- #2215 @gabe-lyons fix(react): fix theming test in react and simplifying api
- #2209 @thomasplarsson feat(ingest): add option for optimized skipping of schemas
- #2212 @hsheth2 fix(ingestion): nullable types and timestamp precision
- #2207 @hsheth2 feat(ingest): standalone metadata emitters
- #2205 @dexter-mh-lee fix(ci): Fix github package path
- #2204 @dexter-mh-lee feat(ci): Add SHA based tagging before pushing to docker registries
- #2203 @gabe-lyons feat(tag): adding search for tags in gms layer
- #2193 @gabe-lyons feat(react): adding ability to support theming of datahub, with two themes included
- #2201 @hsheth2 feat: add date and time types to SQL model
- #2202 @thomasplarsson feat(mae-consumer): enable mae-consumer to use ssl when communicating with elasticsearch
- #2199 @thomasplarsson fix(mae-consumer): mae-consumer needs sslcontext bean
- #2181 @shirshanka chore: renaming business_glossary rfc directory with pull request number
- #2182 @shirshanka chore: renaming graphql_frontend rfc directory with pull request number
- #2183 @shirshanka chore: renaming react-app rfc directory with pull request number
- #2196 @shirshanka docs(roadmap): update project roadmap
- #2195 @jjoyce0510 fix(graphql): Add "fixed" SchemaFieldDataType mapping
- #2194 @gabe-lyons feat(tags): Enriching sample data for tags
- #2191 @hsheth2 feat(docs): automatically populate sidebar with RFCs
- #2192 @jplaisted (feat) Simple python script to carry over ES indices from 5 to 7.
- #2173 @brendansun93 feat(React): Ownership component of user profile
- #2189 @thomasplarsson feat(gms): add elasticsearch SSL support
- #2112 @frsann feat(tags): RFC for tags
- #2187 @gabe-lyons fix(react): fixing test issues that arose from ill-timed merges
- #2164 @gabe-lyons feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend.
- #2185 @jjoyce0510 feat(graphql): More forgiving for unknown data platforms during reads
- #2184 @jjoyce0510 test(React): Home page tests
- #2186 @hsheth2 fix(docs): fix broken links
- #2179 @gabe-lyons feat(react): adding raw schema view option for table schemas
- #2178 @hsheth2 feat(ingest): bigquery sample data
- #2176 @hsheth2 docs: point to hosted docs site
- #2177 @hsheth2 docs(ingest): clarify setuptools requirement
- #2175 @hsheth2 build(docs): only deploy docs on main repo
- #2174 @hsheth2 docs: hosted documentation website
- #2167 @jjoyce0510 feat(React): Impl browse UI for Dashboards and Charts
- #2168 @jjoyce0510 fix(React): Fix Browse Pagination Bug
- #2172 @hsheth2 fix(ingest): loosen Kafka broker validation
- #2165 @jjoyce0510 feat(DataPlatform Logos): Adding server driven logos
- #2171 @hsheth2 docs(ingest): clarify Kafka connection config
- #2169 @shirshanka doc(townhall): Add links for Feb 19, upcoming townhall on Mar 19
- #2161 @hsheth2 fix(ingest): bigquery source and dataset naming fixes
- #2163 @jjoyce0510 fix(graphql): Bubbling up exceptions logged in GraphQL resolvers
- #2159 @hsheth2 build(ingest): use multi-stage docker build for datahub-ingestion
- #2157 @hsheth2 feat(ingest): capture table descriptions
- #2158 @hsheth2 feat(ingest): switch quickstart to Python ingestion
- #2156 @pedro93 feat(ingest): support alternative authentication in sql ingestion
- #2152 @gabe-lyons fix(react): fixing format we propagate filters to graphql in
- #2154 @gabe-lyons feat(react): Redirecting /assets to index
- #2151 @hsheth2 build(docker): add large generated directories to dockerignore
- #2150 @hsheth2 ci(ingest): setup docker container for metadata ingestion
- #2145 @RickardCardell feat: neo4j Bolt TLS support (#2100)
- #2143 @dexter-mh-lee feat(dashboards): Add browse end point for charts and dashboards
- #2144 @RickardCardell feat: neo4j https support (#2101)
- #2147 @gabe-lyons docs(frontend): Update docs to clarify running local frontend w/ local react app
- #2148 @jjoyce0510 feat(gms): Add optional data platform display name
- #2149 @jplaisted Switch GMA dep from bintray to artifactory.
- #2146 @jjoyce0510 Fixing required audit stamps bug
- #2140 @jjoyce0510 feat(React): Search page UI improvements, 'all' entity search.
- #2133 @thomasplarsson feat(datahub-dao): enable services to access gms over https
- #2136 @hsheth2 feat(ingest): support Postgres PostGIS extensions
- #2139 @gabe-lyons docs(Ownership): making lack of support for ownergroups in frontend explicit in pdl
- #2137 @dexter-mh-lee refactor(docker-dev): set up elasticsearch using local mapping on docker-compose.dev
- #2135 @hsheth2 ci(ingest): run apt update
- #2134 @hsheth2 refactor(ingest): cleanup configuration models
- #2130 @jjoyce0510 feat(React UI): SearchPage and SearchResultsPage
- #2132 @jjoyce0510 Add URL to dashboard / chart page
- #2131 @gabe-lyons fix(React): Adding test coverage for search page & fixing filter select bug
- #2128 @jjoyce0510 fix(react): Fix authenticated user profile
- #2125 @hsheth2 fix(ingest): gracefully handle unknown types
- #2127 @jjoyce0510 feat: Introducing optional DataPlatform logo url
- #2124 @hsheth2 fix(ingest): update sample MCEs based on MLModel changes
- #2126 @jjoyce0510 fix(gms): fix getAllDataPlatforms bug
- #2123 @hsheth2 docs(ingest): add solutions for common install issues
- #2122 @hsheth2 feat(ingest): add support for LDAP ingestion
- #2120 @hsheth2 test(ingest): verify the output of mssql
- #2119 @jjoyce0510 feat(React): Adding basic chart + dashboard UI
- #2115 @brendansun93 feat(React): Avatar dropdown menu and logout function
- #2121 @hsheth2 feat(ingest): improve error reporting for pipelines
- #2117 @jjoyce0510 feat(GraphQL API): GQL implementation of Charts + Dashboards
- #2118 @jjoyce0510 bug(docker react): Fix react docker image build
- #2116 @hsheth2 docs(ingest): add example DAGs for Airflow
- #1896 @RyanHolstien feat: ML Model Backend Implementation
- #2109 @jjoyce0510 feat(graphql): Add Dashboards + Charts Models
- #2111 @jjoyce0510 bug(docker): Removing datahub-gms-graphql-service from default docker-compose.yml file
- #2108 @hsheth2 docs(ingest): clarify docs for new ingestion framework
- #2107 @shirshanka Update with Feb 19 Agenda
- #2104 @jjoyce0510 refactor(React): Renaming components/ dir to app/
- #2103 @jjoyce0510 docs(React): React README
- #2102 @jjoyce0510 fix(React): Fix Dataset Ownership Role Update Bug
- #2099 @gabe-lyons feat(React): Powering lineage page through graphql endpoint
- #2097 @gabe-lyons feat(React): deprecation status in header
- #2096 @gabe-lyons feat(React): Adding schema tab on dataset entity
- #2093 @jjoyce0510 feat(React): Adding Properties, Documentation tabs, user search, & supporting Dataset mutations
- #2095 @brendansun93 fix (React): Fix broken profile link on ownership + header
- #2094 @hsheth2 fix: Bump gma version
- #2091 @jjoyce0510 feat(GraphQL Mutation): Introducing updateDataset mutation
- #2061 @arpanm96 2056 | Change dataset lineage pendulum animation component
- #2090 @gabe-lyons feat(React Incubation): adding lineage page with sample data
- #2087 @allenhaozi fix(docs): update URN models and DatasetUrn links
- #2083 @gabe-lyons feat(React Incubation): User Profile
- #2084 @jjoyce0510 fix(GraphQL): Add schemaMetadata mapping & correctly form path string
- #2086 @gabe-lyons fix(react): including eslint in CI
- #2085 @jjoyce0510 fix(integ test): Bumping GMA to 0.2.40
- #2082 @hsheth2 fix(SQL ingest): Bump confluent-kafka version
- #2077 @jjoyce0510 refactor(React Incubation): Entity Interface & EntityRegistry
- #2081 @jjoyce0510 Enable mutation only in frontend
- #2076 @jjoyce0510 feat(GraphQL Query): Supporting complete Dataset, CorpUser with Search & Browse in GraphQL API
- #2071 @arunvasudevan feat (graphql): Datahub GMS Graphql Api Application for Querying Dataset
- #2075 @gabe-lyons refactor(React Incubation): Search routing refactor
- #2074 @shirshanka docs: Updating README with upcoming TownHall info
- #2073 @jjoyce0510 Fixing CorpUsers Search Bug
- #2072 @mars-lan Drop obsolete info on mysql-etl
- #2067 @jplaisted feat: update GMA to 0.2.35
- #2069 @gabe-lyons fix (react): fixing browse routing
- #2060 @jjoyce0510 feat(React Browse): Adding Browse Logic and misc refactorings
- #2066 @jjoyce0510 feat(GQL Queries): Productionalizing GraphQL Part 1: Dataset Query support + adding shared GraphQL module
- #2057 @nagarjunakanamarlapudi (refactor): Convert dataPlatforms to GMA aspect models and associated resource to GMA resource.
- #2063 @shirshanka Updating townhall history for Jan 15, and date for next townhall
- #2059 @jjoyce0510 feat(React Incubation): React Playground Proof of Concept
- #2055 @jjoyce0510 React Incubation RFC
- #2058 @jsotelo Support optional initContainers to setup extra initialization
- #2054 @nagarjunakanamarlapudi Update townhall schedule docs to third friday of every month
- #2053 @sattsb Fix for Kafka-UI Connectivity Error with Kafka-Rest Proxy
- #2050 @venkyvb Fix broken link in docs to the Restli resource base classes
- #2049 @shirshanka Townhall Agenda for Jan 15
- #2042 @jjoyce0510 feat(GraphQL support): RFC for GraphQL support in datahub-frontend (Part 1/2) - Queries
- #2045 @shirshanka docs: add links to GMA repo and RFC process from relevant places
- #2013 @mars-lan build(docker): add script to clean up docker environment
- #2041 @shirshanka Update README.md
- #2040 @nagarjunakanamarlapudi fix(search): Fix the rollback unintentional rollback(#2028) of dataset index to search by field paths.
- #2034 @immmus fix: The field connection with type URLConnection is not Serializable.
- #2032 @nagarjunakanamarlapudi Update town hall history for 12/4 and announce the next town hall date
- #2024 @nagarjunakanamarlapudi feat(Search): Inject restli client into index builders
- #2028 @jplaisted feat: add elasticsearch sanity integration tests