dgraph-io/dgraph v21.12.0

Dgraph v21.12.0 - Zion

on GitHub

Changed

• [BREAKING] feat(sroar): Bring sroar to Dgraph (#7840)
• [BREAKING] Return error for illegal math operations. (#7631)
• [BREAKING] feat: bring dgraph-lambda to dgraph, alpha launches lambda server (#7973)
• [BREAKING] fix json marshal unmarshal for namespace > 127 (#7810)
• [BREAKING] fix(usability): make force-namespace flag compulsory in live loader for galaxy user (#7731)
• [BREAKING] perf(Transactions): Run transactions concurrently (#7694)
• [BREAKING] feat(flags): expand badger to accept all valid options (#7677)
• [BREAKING] fix(commit): make txn context more robust (#7659)
• [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666)
• [BREAKING] fix(metrics): Rename Badger metrics. (#7507)
• [BREAKING] Make backup-restore an open source feature (#8067)

Added

• GraphQL

  • Feat(GRAPHQL): adds @default directive for setting default field values at create and update (#8017)
  • Feat(GRAPHQL): Support auth with custom DQL (#7775)
  • Feat(GRAPHQL): This PR allows updatable and nullable @id fields. (#7736)
  • Feat(GRAPHQL): Disallow DQL schema changes for predicates used in GraphQL schema (DGRAPH-3245) (#7742)
  • Feat(GRAPHQL): This PR allows @id field in interface to be unique across all the implementing types. (#7710)
  • Feat(GRAPHQL): Add language tag support in GraphQL (#7663)
  • Feat(GRAPHQL): Zero HTTP endpoints are now available at GraphQL admin (GRAPHQL-1118) (#6649)
  • Feat(GRAPHQL): Webhooks on add/update/delete mutations (GRAPHQL-1045) (#7494)
  • Feat(GRAPHQL): Allow Multipe JWKUrls for auth. (#7528)
  • Feat(GRAPHQL): Add support for passing OAuth Bearer token as authorization JWT (#7490)

• Core Dgraph

  • Feat(metrics): Add Badger metrics. (#8034)
  • Feat(magicNumber): Introduce magic number (#8032)
  • Feat(lambda): allow access to access jwt in lambda (#8023)
  • Feat(rdf-response): Support RDF response via http query request (#8004)
  • Feat(sroar): Use rank() API from sroar and some cleanup (#8002)
  • Feat(lambda): store lambda scripts within the dgraph (#7955)
  • Feat(query): handle extend keyword for Queries and Mutations (#7916)
  • Feat(Backup): Add native google cloud storage backup support (#7829)
  • Feat(Backup): Add native support for backup to Azure. (#7843)
  • Feat(cloud): add shared-instance flag in limit superflag in alpha (#7770)
  • Feat(Dgraph): Add task queue implementation (#7716)
  • Feat(DQL): @groupby on scalar fields and count duplicate (#7746)
  • Feat(Query): Add random keyword in DQL (#7693)
  • Feat(tool): Neo4j CSV to RDF Converter (#7545)
  • Feat(query): Add mechanism to have a limit on number of pending queries (#7603)
  • Feat(flag): remove unused badger.max-retries option from bulk command (#7591)
  • Feat(sentry): clusterID flag added for alpha sentry reports (gql-services) (#7580)
  • Feat(cmd/debuginfo) add new metrics to be collected (#7439)
  • Feat(flags): use Vault for ACL secrets (#7492)
  • Feat(Apollo): Add support for @provides and @requires directive. (#7503)
  • Feat(restore): Introduce incremental restore (#7942) (#7971)
  • Feat(schema): do schema versioning and make backup non-blocking for indexing (#7852)
  • Feat(zero bulk): adding bulk call for alpha to inform zero about the tablets (#8100)
  • Feat(cdc): Add superflag to enable TLS without CA or certs. (#8097)

• Enterprise Features

  • Feat(Multi-tenancy): Add namespaces field to state (#7808)
  • Feat(multi-tenancy): make drop data namespace aware (#7789) (#7795)
  • Feat(cdc): Add support for SCRAM SASL mechanism (#7765)
  • Feat(acl): allow access to all the predicates using wildcard (#7991)
  • Feat(cdc): Add superflag to enable TLS without CA or certs. (#7946)

Fixed

• GraphQL

  • Fix(GRAPHQL): add validation of null values with correct order of graphql rule validation (#8007) (#8008)
  • Fix(GRAPHQL): fix type assertion failure in graphql if resolver is not defined (#8003)
  • Fix(GRAPHQL): fixing graphql schema update when the data is restored (#7970)
  • Fix(GRAPHQL): Nested Auth Rules not working properly. (#7915)
  • Fix(GRAPHQL): optimize eq filter queries (#7895)
  • Fix(GRAPHQL): Fix duplicate XID error in case of interface XIDs (#7776)
  • Fix(GRAPHQL): Pass on HTTP request headers for subscriptions (#7806)
  • Fix(GRAPHQL): Make mutation rewriting tests more robust (#7768)
  • Fix(GRAPHQL): Fix error message of lambdaOnMutate directive (#7751)
  • Fix(GRAPHQL): fix auth query rewriting with ID filter (#7740)
  • Fix(GRAPHQL): Add filter in DQL query in case of reverse predicate (#7728)
  • Fix(GRAPHQL): Fix GraphQL encoding in case of empty list (#7726)
  • Fix(GRAPHQL): fix @cascade with Pagination for @auth queries. (#7695)
  • Fix(GRAPHQL): Fix Execution Trace for Add and Update Mutations (#7656)
  • Fix(GRAPHQL): Log query along with the panic (#7638)
  • Fix(GRAPHQL): Add error handling for unrecognized args to generate directive. (#7612)
  • Fix(GRAPHQL): Fix panic when no schema exists for a new namespace (#7630)
  • Fix(GRAPHQL): fixed output coercing for admin fields. (#7617)
  • Fix(GRAPHQL): fix lambda querying a lambda field in case of no data. (#7610)
  • Fix(GRAPHQL): Add extra checks for deleting UpdateTypeInput (#7595)
  • Fix(GRAPHQL): remove support of @id directive on Float (#7583)
  • Fix(GRAPHQL): Fix mutation with Int Xid variables. (#7565)
  • Fix(GRAPHQL): Fix custom(dql: ...) with __typename (GRAPHQL-1098) (#7569)
  • Fix(GRAPHQL): Change variable name generation for interface auth rules (#7559)
  • Fix(GRAPHQL): Apollo federation now works with lambda (GRAPHQL-1084) (#7558)
  • Fix(GRAPHQL): fix empty remove in update mutation patch, that remove all the data for nodes in filter. (#7563)
  • Fix(GRAPHQL): fix order of entities query result (#7542)
  • Fix(GRAPHQL): Change variable name generation from Type to Type_ (#7556)
  • Fix(GRAPHQL): fix duplicate xid error for multiple xid fields. (#7546)
  • Fix(GRAPHQL): Added support for exact index on field having @id directive. (#7534) (#7551)
  • Fix(GRAPHQL): fix query rewriting for multiple order on nested field. (#7523)
  • Fix(GRAPHQL) fix empty type Query with single extended type definition in the schema. (#7517)

• Core Dgraph

  • Fix(sort): Only filter out nodes with positive offsets. (#8077)
  • Fix(fragment): merge the nested fragments fields (#8075)
  • Fix(lambda): upgrade lambda dependencies to fix vulnerabilities (#8074)
  • Fix(magic): fix the magic version in bulk loader etc (#8070)
  • Fix(split): enable split of posting list with single plist (#8062)
  • Fix(restore): Do not retry restore proposal (#8058)
  • Fix(txn): Fix data races in transaction code (#8060)
  • Fix(shutdown): wait for pending queries to process on alpha shutdown (#8057)
  • Fix(restore-test): Make offline restore use separate map directory for each group (#8047)
  • Fix(lambda-logs): extract namespace from body.namespace (#8043)
  • Fix(lambda): make lambda active only after successful start (#8036)
  • Fix(probe): do not contend for lock in lazy load (#8037)
  • Fix(lambda): shutdown node processes when alpha gets killed (#8027)
  • Fix(snapshot): Fix snapshot calculation after restore (#8024)
  • Fix(badger): Upgrade badger to fix deadlock (#8025)
  • Fix(bulk): quote the schema correctly in bulk loader (#8019)
  • Fix(sbs): handle response error gracefully (#8018)
  • Fix(admin): make config changes to pass through gog middlewares (#8014)
  • Fix(lambda): fix race condition in lambda server spin up (#8013)
  • Fix(proposals): Incremental proposal key for zero proposals (#8005)
  • Fix(lambda): monitor lambda server, fix performance issue, remove lambda logs from extensions (#8006)
  • Fix(live): quote the xid when doing upsert (#7983)
  • Fix(sroar): Bring latest sroar to master (#7977)
  • Fix(query): Do not execute filters if there are no source uids (#7962) (#7969)
  • Fix(snapshot): update last snapshot time across members (#7968)
  • Fix(pool): use write lock when getting health info (#7963)
  • Fix(JoinCluster): Avoid retrying JoinCluster indefinitely (#7961)
  • Fix(rollups): Write rolled-up keys at ts+1 (#7957) (#7959)
  • Fix(conn): JoinCluster loop should use latest conn (#7950)
  • Fix(restore): Set kv version to restoreTs for rolled up keys and schema keys (#7930) (#7935)
  • Fix(backup): Fix full backup request (#7932)
  • Fix(cmd/debug): Print banned namespaces correctly. (#7929)
  • Reconnect via a redial in case of disconnection. (#7918)
  • Fix(metrics): Expose dgraph_num_backups_failed_total metric view. (#7900)
  • Fix(sroar): Fix TestAuthWithCustomDQL failure because of roaring bitmaps (#7902)
  • Fix(DQL): revert changes related to cascade pagination with sort (#7885)
  • Fix(restore): append galaxy namespace to type name (#7880)
  • Fix(Backup): use validReadTs from manifest for backward compatibility (#7601) (#7863)
  • fix the predicate move (#7862)
  • Fix(restore): consider the banned namespaces while bumping (#7839)
  • Fix(restore): update the schema and type from 2103 (#7838)
  • Fix(updatemanifest): update the version of manifest after update (#7828)
  • Fix(backup): handle manifest version logic, update manifest version to 2105 (#7825)
  • Fix(schema-update): Start opIndexing only when index creation is required. (#7845)
  • Fix(admin): remove exportedFiles field (#7835)
  • Fix(restore): reset the kv.StreamId before sending to stream writer (#7833)
  • Fix(auth): preserve the status code while returning error (#7832)
  • bug fix to permit audit streaming to stdout writer(#7803)
  • Fix(lease): don't do rate limiting when not limit is not specified (#7787) (#7801)
  • Fix(restore): Bump uid and namespace after restore (#7790)
  • Fix(txn): ensure that txn hash is set (#7782)
  • Fix(export-backup): Fix double free in export backup (#7780)
  • Fix(Dgraph): Forward task status requests to correct Alpha (#7774)
  • Fix(bulk): upsert guardian/groot for all existing namespaces (#7759)
  • Fix(export): Fix facet export of reference type postings to JSON format (#7744)
  • fix: Prevent proposal from being dropped accidentally (#7741)
  • Fix(live): make live loader progress on a cluster with very high maxUid (#7743)
  • Fix(Chunker): don't delete node with empty facet in mutation (#7737)
  • Fix(DQL): optimize query for has function with offset. (#7727)
  • fixing readme for dgraph after latest release (#7732)
  • Fix(lease): prevent ID lease overflow (#7724)
  • Fix(lsbackup): Fix profiler in lsBackup (#7729)
  • Fix(bulk): throw the error instead of crashing (#7722)
  • Fix(ee): GetKeys should return an error (#7713)
  • Fix(raftwal): take snapshot after restore (#7719)
  • Fix(pagination): Fix after for regexp, match functions (#7700)
  • Fix(query): Prevent multiple entries for same predicate/type in schema mutations. (#7715)
  • Fix(vault): Hide ACL flags when not required (#7701)
  • Fix(flag): fix bulk loader flag and remove flag parsing from critical path (#7679)
  • Fix(upgrade): make upgrade tool to work with non-acl cluster (#7674)
  • Fix(query): Fix pagination with match functions (#7668)
  • Fix(postingList): Acquire lock before reading the cached posting list (#7632)
  • Fix(zero): add a ratelimiter to limit the uid lease per namespace (#7568)
  • Fix(export): use UriHandler for exports (#7690)
  • Fix s3 backup copy (#7669)
  • return if no uids exist in queries for Geo (#7651)
  • Fix(DGRAPH): fix @normalize response when multiple fields at different levels with same alias are selected. (#7639)
  • Fix(/commit): protect the commit endpoint via acl (#7608)
  • Use GetString for vault path (#7605)
  • Fix query logging for mutations (#7646)
  • Fix(login): fix login based on refresh token logic (#7637)
  • Fix(Query): Fix cascade pagination with 0 offset. (#7636)
  • feat(flags): Add query timeout as a limit config (#7599)
  • Fix(flags): Add empty defaults to Vault superflag (#7598)
  • Fix(persistent): make persistent query namespace aware (#7570)
  • Fix(rollups): Fix splits in roll-up (#7609)
  • fix for xgo version to use (#7620)
  • Fix(flags): Expose global flags to dgraph subcommands. (#7530)
  • Fix(telemetry): fix zero crash due to telemetry (#7575)
  • Fix(telemetry): Track enterprise feature usage (#7495)
  • Fix(release): update support for xgo tool (#7576)
  • Fix(super-flags): Use GetPath for path arguments in superflags (#7541)
  • Fix(dql): Fix error message in case of wrong argument to val() (#7543)
  • Fix(Roaring): Remove pack from posting list (#7535)
  • Fix(Flags): immediately panic on SuperFlag user errors (#7529)
  • Fix(Rollups): Don't try splitting a posting list with cardinality less than 2. (#7525)
  • Fix(export): fix namespace parameter in export (#7524)
  • Fix(live): fix usage of force-namespace parameter in export (#7526)
  • Fix largeSchemaUpdate test (#7522)
  • Fix(Configs): Allow hierarchical notation in JSON/YAML configs (#7498)
  • Fix(Bulk): Remove stale allocator in reduce (#7510)
  • Fix upsert mutations (#7515)
  • Fix(standalone): Set whitelist flag using superflag. (#7512)
  • Fix(admin-endpoints): Error out if the request is rejected by the server (#7511)
  • Fix(Dgraph): Throttle number of files to open while schema update (#7480)
  • Fix(metrics): Expose Badger LSM and vlog size bytes. (#7488)
  • Fix(schema): log error instead of panic if schema not found for predicate (#7502)
  • Fix(tool): Don't ban namespace in export_backup (#8099)
  • Fix(state): fix hex to uint64 response of list of namespaces (#8101)
  • Fix(restore): return nil if there is error (#8098)

• Enterprise Features

  • Fix(audit): fixing audit logs for websocket connections (#8048)
  • Fix(acl): subscribe for the correct predicates (#7992)
  • Fix(acl): filter out the results based on type (#7978) (#7980)
  • Fix(groot): do not upsert groot for all namespaces on restart (#7917)
  • Fix(cdc): Show namespace info in event meta (#7721)
  • Fix(learner): Don't start a learner node with no peers (#7582)
  • Fix(audit): logs not getting deleted after N days (#7567)
  • Fix(release/v21.03) - Use worker.GetEEFeatureList instead of ee.GetEEFeatureList (#7564)
  • Fix(multi-tenancy): Format namespace to human readable form (#7552)
  • Fix(learner nodes): Reconnect to learner nodes after restart (#7554)
  • Fix(multi-tenancy): fix live loader for case when namespace does not exist for data (#7505)

Performance:

• Opt(schema): Optimize populateSchema() by avoiding repeated lock acquisition (#8068)
• Perf: Speed up parsing of a huge query with a lot of conditional mutations (#7871)
• Opt(Restore): Make restore map phase faster (#8038)
• Opt(codec): return nil instead of a new bitmap (#7997)
• Opt(cache): Use Ristretto to store posting lists (#7995)
• Opt(rdf-output): Make RDF output generation concurrent (#7988)
• Opt(recurse): Optimise recurse and bring range iterators from sroar (#7989)
• Opt(restore): Sort the buffer before spinning the writeToDisk goroutine (#7984)
• Perf(sroar): Use latest sroar and add histogram in the sbs tool (#7982)
• Opt(Alpha): Load schema and types using Stream framework (#7938) (#7940)
• Opt(query): Use sroar in pb.List (#7864)
• Opt(snapshot): use full table copy when streaming the entire data (#7870)
• Opt(snapshot): Optimize snapshot by using sinceTs (#7826)
• Opt(predMove): iterate Phase I till there is major data to move (#7792)
• Opt(dropPrefix): allow logical drop for deleting predicates and indexing (#7764)
• Opt(txn commits): Optimize txns by passing Skiplists to Badger (#7777)
• Opt(GraphQL): filter existence queries on GraphQL side instead of using @filter(type) (#7757)
• Opt(predMove): hot tablet move (#7703)
• Opt(Backup): Make backups faster (#7680)
• Perf(restore): Implement map-reduce based restore (#7664)
• Opt(reindex): do not try building indices when inserting a new predicate (#7109)
• Perf(txn): de-duplicate the context keys and predicates (#7478)
• perf(rollup): use NSplit API from sroar to improve rollup performance (#8092)