[!WARNING] This version contains a thread leak, leading to eventual higher memory usage, bump to v2.11.13
Substreams improvements v1.15.8
Rework the execout File read/write to improve memory efficiency:
-
This reduces the RAM usage necessary to read and stream data to the user on tier1,
as well as to read the existing execouts on tier2 jobs (in multi-stage scenario) -
The cached execouts need to be rewritten to take advantage of this, since their data is currently not ordered:
the system will automatically load and rewrite existing execout when they are used. -
Code changes include:
- new FileReader / FileWriter that "read as you go" or "write as you go"
- No more 'KV' map attached to the File
- Split the IndexWriter away from its dependencies on execoutMappers.
- Clock distributor now also reads "as you go", using a small "one-block-cache"
-
Removed
SUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENTenv var (since this is not a RAM issue anymore) -
Add
uncompressed_egress_bytesfield tosubstreams request statslog message
Poller
- add
--headersflag tofireeth tools pollerto allow auhenticated calls to ETH_RPC providers - add
--allow-empty-receipts-on-block-0bool flag to work with tron-evm-mainnet - add
--parallel-workersint flag to allow increasing from the default (which is now 20 instead of 10)
Various
- (dstore) Add storageClass query parameter for s3:// urls on stores (@fschoell)
- Update the firehose-beacon proto to include the new Electra spec in the 'well-known' protobuf definitions (@fschoell)
- Use The Graph's Network Registry to recognize chains by genesis blocks and fill the 'advertise' server on substreams/firehose