RPC optimization

Next Milestone Date

February 14, 2022

Assign

Jenya Piskunov

Jack Chan

Status

Live on Devnet

Github

Collaborator

Grant-1

Grant-2

Grant-3

Release Date

2022 Q2

Goals

The goal of this project is to build a separate endpoint to serve the heavy RPC calls like eth_getLogs calls.

Context

eth_getLogs call is a very heavy call as it may go through all the blockchain blocks from genesis to the latest blocks to find the transaction data. The current implementation of getLogs RPC handler in Harmony wasn’t optimized to serve thousands of requests as it was only designed to handle a single user’s request.

Based on the shared experience of Infura, the getLogs RPC request could serve users better using a separate indexed DB, such like postgres.

Architecture

From the public end point, the payload needs to be parsed to identify the eth_getLogs call and redirect the request to the specific load balancer to serve the requests. The logLB is using the reader/writer architecture to serve scalable requests.

Additional indexer has to be built to index the receipt and put into the indexed postgres DB. This is the writer of the DB.

Scalable readers can read from the DB and serve the RPC requests. This is the reader of the DB.

Action Items

~~look into existing indexed DB, prototype RPC handler to serve getLogs call (@Jenya Piskunov)~~

~~benchmarking the performance of getLogs on indexed DB vs existing code (@Jenya Piskunov)~~

investigate cloudflare and aws on jsonrpc parsing (@Jack Chan)

architecture review

Reference

https://blog.infura.io/ethereum-rpcs-methods/

Infura has a cap on requests of 10,000 events per query.
Best practice is to request a single block, as we’ve done in this example,
and do that for each mined block.

https://blog.infura.io/building-better-ethereum-infrastructure-48e76c94724b/?&utm_source=infurablog&utm_medium=referral&utm_campaign=tutorials&utm_content=eth_call_tutorial

RPC ethLogs PR

currently 2 instances behind lb (rpcapi1.explorer.t.hmny.io and rpcapi2.explorer.t.hmny.io)

curl --location --request POST 'https://indexed-db-api.hmny.io/v0/rpc' \ --header 'Content-Type: application/json' \ --data-raw '{ "jsonrpc": "2.0", "method": "eth_getLogs", "id": 1, "params": [{"fromBlock": "0x1340423", "toBlock": "0x1340423"}] }'

benchmarks one by one queries (no load) show the same response speed

200 generated requests (same set used for node and postgres) NODE internal.s0.t.hmny.io Total time: 231577 ms POSTGRES EXPLORER RPC Total time: 218401 ms

Explore benchmarking using Infura’s Versus tool:

https://blog.infura.io/compare-ethereum-api-performance-with-versus/

Todo

RO replicate to read by the RPC nodes handled by RDS / postgres

any configuration (@Jack Chan)

API gateway (@Nita Neou (Soph) )

search for getLogs in the Payload, faster for the lambda
monitor the performance of the parser
dedicated getlog endpoint behind ELB apilog.s0.t.hmny.io (internal only)

monitoring and scaling

auto scaling support

HA indexer for the explorer Db (@Jenya Piskunov)

instability of the indexer
dedicated indexer for getLogs
new DB to save the logs

a new separate endpoint for the new getLogs service by @February 18, 2022 (@Nita Neou (Soph))