Relational database schemas (#1020 via #1159)

Subgraph data is now stored in Postgres using relational schemas. Instead of a storing all entities of a subgraph in a single table using JSONB, each entity type gets a dedicated table, with each field being stored in its own column.

This has several advantages:

1. Reading and writing are faster, more scalable.
2. Entity data can be validated and enforced better, preventing bugs in subgraphs.
3. Introspecting the data is much easier when debugging.

The stricter validation of entities written to the store may cause subgraphs that work today to fail when being deployed. A note from @lutter, who implemented this feature:

As part of the switch to the relational storage scheme, subgraph entities are validated more strictly than they were in the past. In particular, the store will reject attempts to store entities that have null in fields that are marked as non-null in the GraphQL schema. If your subgraph reports such errors after you deploy a new version, you can address that either by marking the field as nullable in the GraphQL schema by leaving the ! off its type, or by changing your mappings to ensure that the field is always set to a non-null value.

Note: This only affects new subgraph deployments. Existing deployments will keep using the JSONB schema. On the hosted service we will slowly clean up unused deployments over time until we can drop the old JSONB-based schema entirely.

Start block for data sources (#1262, #1294, #1320)

Until now, subgraphs were always indexing from the genesis block. This involved scanning the entire chain for relevant blocks, logs and traces, even if the subgraph contracts were only deployed recently.

This was fine initially. However, with more expensive features—such as block and call handlers—being added and more advanced subgraphs being developed, it has often become desirable to skip irrelevant old blocks entirely to speed up indexing.

This is now possible. As of this release, data sources can specify an optional startBlock number in the manifest:

dataSources:
  - name: Gravity
    source:
      address: '0x2E645469f354BB4F5c8a05B3b30A929361cf77eC'
      abi: Gravity
      startBlock: 6000000

The subgraph will then start indexing from this block. If there are multiple data sources with or without start blocks, the earliest of these blocks is used as the starting point. The absence of a startBlock is equivalent to 0 (aka the genesis block).

Other changes

GraphQL

• Fix querying interfaces by id (#1204)
• Fix following interface references (#1303 via #1307)
• Support where filter on relationship fields (#1212)
• Validate @derivedFrom directives on subgraph schemas (#1247)

Ethereum

• Add an Ethereum contract call cache to speed up indexing (#1230 via #1270)
• Detect failed Solidity assertions in try_ calls (#1162)
• Detect Bad instruction fd errors in try_ calls (#1327)
• Allow recursive data source template creation within the same block (#1185)
• Handle Alchemy timeouts in eth_getLogs gracefully (#1242)
• Make eth_getLogs scanning more efficient (#1260)
• Only fetch transaction receipts when necessary (#1308, #1326)

Mappings

• Reduce memory usage and number of threads by using one thread per mapping instead of one per data source (#1281)

Database

• Allow subgraphs to make progress while Postgres is autovacuuming the subgraphs table (#1175)
• Use a single connection pool per node (#1232)
• Preload pg_stat_statements in Docker Compose setup (#1163)
• Fix default Postgres port in the README (#1181, thanks @akeem!)

Monitoring

• Add Prometheus instrumentation support (#1259 via #1264)

Configuration

• Add new GRAPH_IPFS_SUBGRAPH_LOADING_TIMEOUT environment variable for loading subgraph manifests from IPFS (#1091, #1238, #1244)
• Add optional limit for GraphQL/WS operations using the GRAPH_GRAPHQL_MAX_OPERATIONS_PER_CONNECTION environment variable (#1160)

Misc

• Don't skip the genesis block when indexing (#1295 via #1297)
• Allow Docker image to wait for other services using full URLs (#1275)
• Fix Docker builds by passing --locked to Cargo
• Support multiple Ethereum networks in Docker image by sperating networks in the ethereum environment variable with spaces (#1332)
• Fix yarn commands in README (#1282, thanks @ana0!)
• Update dependencies (bs58, chrono, clap, crossbeam-channel, graphql-parser, hex, hex-literal, hyper, indexmap, jsonrpc-core, jsonrpc-http-server, lru_time_cache, num-bigint, serde, serde_derive, serde_yaml)
• Remove dependencies on itertools and sentry