This is a big release that contains support for a new HAPI service and whole new runtime component to dramatically improve performance. Due to the magnitude of the changes, it did take us a little longer to mark it as generally available as we wanted to ensure it was tested as much as possible beforehand.
First up is support for the Hedera Token Service (HTS) that was recently announced and rolled out to previewnet. A lot of work was put into supporting the new transaction types on the parser side including enhancing the schema with new tables and ingesting them via the record stream. HTS also required a new balance file version that adds token information to the CSV. Token information is now returned for our existing REST APIs while the next release will contain token specific REST APIs for further granularity. Check it out in previewnet and let us know if you have any feedback!
We made a lot of strides in improving the ingestion performance in previous releases, but since we also wanted to ensure low end to end HCS latency via our gRPC API we had to sacrifice some of that speed. As a result, we could only ingest at about 3,000 transactions per second (TPS) before latency spiked above 10 seconds. This was entirely due to our use of PostgreSQL notify/listen to notify the gRPC API of new data.
In this release, we add a new notification mechanism without sacrificing speed or latency with our support for Redis pub/sub. With Redis, the mirror node can now ingest at least 10,000 TPS while still remaining under 10 seconds from submitting the topic message and receiving it back via the mirror node's streaming API. Redis is enabled by default, but it can be turned off if HCS latency is not a concern and you want to avoid another runtime dependency.
We also added support for the HAPI protobuf changes that are coming in v0.9.0. The protobuf is removing two deprecated fields while adding a new signedTransactionBytes field. Since the mirror node still needs to support historical transactions we retain support for parsing transactions that contain the old payload format.
Enhancements
- Bump versions for 0.20.0 #1200
- Bump version to rc1 from alpha1 w HTS Acceptance Tests #1137
- Bump versions for v0.20.0-alpha1 #1109
- Enable code scanning for vulnerabilities #1092
- Add Redis support to Helm #1091
- HTS: Support new balance file version #1090
- HTS: Add support for token transactions to Importer #1089
- HTS: Add token balances to accounts REST API #1088
- HTS: Add new fields to accounts, balances and transactions REST API #1087
- HTS: Minor updates to design and schema #1086
- Add topic messages and stateproof to monitor #1076
- Add a Redis HCS topic listener #1075
- stateproof: download record file from stored node #1074
- HTS: DB schema changes #1073
- Replace account_realm_num and account_num with account_id in account_balance table #1071
- Migrate account_balance table to use entity_id for account id #1070
- State proof alpha: download record file from the node used to verify and download the file in importer #1069
- HTS: Design document #1061
- allow individual maxLimit config in rest service monitor #1060
- Bump hedera-protobuf to 0.9.0-alpha2 and support new fields #1056
- HTS: Import token transactions #1051
- HTS: Support new balance file version #1049
- HTS: Database Schema #1048
- HTS: Add new fields to balances and transactions REST API #1047
- HTS: Design #1043
- Allow for configuration of specific checks on the monitor #1033
- Protobuf changes for 0.9.0 #1027
- Pre-OA Mirror Node #913
- Reduce subscribe topic latency #893
- Monitoring API lacks topics and stateproof endpoint coverage #667
- Aggregate monitor API should return 4xx if individual server fails #462
Bug Fixes
- Fix NPE parsing token transactions #1157
- NullPointerException parsing token dissociate #1156
- Fix memory leak in performance tests #1072
- Re add deleted metrics #1058
- gRPC notify thread blocked improvement #1057
- Fix polling cloud storage more often than the configured poll period #1054
- Fix race condition in check-state-proof #1053
- Check-state-proof error #1052
Documentation
- Fix wrong command in check-state-proof readme #1055
Dependency Upgrades
- Bump docker-maven-plugin from 0.33.0 to 0.34.1 #1100
- Bump jacoco-maven-plugin from 0.8.5 to 0.8.6 #1099
- Bump testcontainers from 4.0.9 to 4.3.0 in /hedera-mirror-rest #1097
- Bump eslint-plugin-import from 2.22.0 to 2.22.1 in /hedera-mirror-rest #1096
- Bump sinon from 9.0.3 to 9.1.0 in /hedera-mirror-rest #1093
- Bump spring-cloud-dependencies from Hoxton.SR7 to Hoxton.SR8 #1084
- Bump eslint-config-prettier from 6.11.0 to 6.12.0 in /hedera-mirror-rest #1083
- Bump aws-sdk from 2.756.0 to 2.761.0 in /hedera-mirror-rest #1082
- Bump supertest from 4.0.2 to 5.0.0 in /hedera-mirror-rest #1081
- Bump mathjs from 7.2.0 to 7.3.0 in /hedera-mirror-rest #1080
- Bump prettier from 2.1.1 to 2.1.2 in /hedera-mirror-rest #1079
- Bump mathjs from 7.2.0 to 7.3.0 in /hedera-mirror-rest/monitoring/monitor_apis #1078
- Bump supertest from 4.0.2 to 5.0.0 in /hedera-mirror-rest/monitoring/monitor_apis #1077
- Bump cucumber.version from 6.6.0 to 6.7.0 #1066
- Bump sdk from 1.2.0 to 1.2.2 #1065
- Bump aws-sdk from 2.747.0 to 2.756.0 in /hedera-mirror-rest #1064
- Bump testcontainers from 3.4.4 to 4.0.9 in /hedera-mirror-rest #1063
- Bump eslint-plugin-jest from 24.0.0 to 24.0.2 in /hedera-mirror-rest #1062
- Bump embedded.testcontainers.version from 1.81 to 1.82 #1041
- Bump testcontainers from 3.4.3 to 3.4.4 in /hedera-mirror-rest #1039
- Bump pretty-quick from 3.0.0 to 3.0.2 in /hedera-mirror-rest #1036
- Bump husky from 4.2.5 to 4.3.0 in /hedera-mirror-rest #1035
- Bump protobuf-java from 3.11.1 to 3.13.0 #1024
- Bump reporting-plugin from 4.0.77 to 4.0.84 #1023
Contributors
We'd like to thank all the contributors who worked on this release!