🕸️

Decentralized RPC

Next Milestone Date
March 31, 2022
Assign
Nita Neou (Soph)Leo ChenJack Chan
Status
Execution Started
Github
Collaborator

Grant-1

Grant-2

Grant-3

Release Date

2022 Q3

milestones and todo checklist along the way of fully decentralized ops: deploys, monitors, supports of testnets / rpc / gateways:

Weekly Milestones

1/3/2021

  • Regional build out matured: DNS-based latency routing, healthchecks across the board, monitoring dashboards, global logging and forensics reporting, etc.
  • Multi-cloud POC is now available, next is to test the new setup while we look at rolling out with Pocket Networks.

12/10/2021

  • Multiple events occurred in the new EU region nodes, adding rate limiters next week
  • Proof-of-Concept using other cloud providers to reduce cost and improve disaster recovery (DR) scenarios, and decentralize further

12/03/2021

  • Both Chainstack and Covalent team delayed their deliverables to Jan 2022

11/26/2021

  • Websockets are also unlocked to global infra on 11/22, i.e. connected wallets update balances quicker and more reliably now across the globe
  • Covalent contract signed, launch ETA mid-Dec
  • Chainstack grant approved 11/19, launch ETA late-Dec

11/19/2021

  • Unlocked global infra presence on 11/15, at the same time doEVM calls reached 1B in total last 30 days
  • Chainstack agreement reached, new service offering for developers targeted by end of year,
  • Covalent still in progress

11/12/2021

  • Found issue with latency and LB stickiness for Pocket Networks, no ETA (traffic back with us)
  • Began rolling out multi-geo nodes for shard 0 RPC endpoints, targeting 11/17
  • Covalent setup in progress, targeting 12/15 launch
  • Finalized draft with Chainstack, will receive a proposal by next week

11/5/2021

  • Pocket Network exhaustion for a day on 11/3, now back to a steady state exceeding our 50M txn/day goals (60M/day and growing!)
  • Covalent contract signed, discussion next Tuesday
  • Researching rclone replacement backed by decentralized storage.

11/1/2021

  • Service degraded on 10/24 — multiple solutions in-progress, re-organized load balancers (root caused and solved)
  • Pocket Networks — worked, issues root caused and solved.
  • Covalent — slow to respond due to EthLisbon
  • Chainstacks — discussion in progress

10/25/2021

  • Pocket Network POC completed up to 25% traffic from bridge + developer RPC routed over

Oct 18, 2021

  • Pocket Network - Unblocked all known issues, dialed up to 10% over the weekend, 25% by end of the month
  • Covalent Network - still pushing through, some members in Lisbon. Need to sync on the indexing solution with Sam
  • ChainStack - will explore with them next week

Oct 11, 2021

  • Pocket Network - Traffic shifting testing began, SSL certificates being put in place on Pocket gateways
  • Covalent Network - agreement drawn up

Oct 4, 2021

  • Pocket Network - Finalized closing plan, to discuss internally on traffic shifting next week
  • Covalent Network - Kickoff discussion on RPC and indexing mechanisms, to follow up on drafting an agreement between teams
  • QuickNode - spoke to Solana (no outages), Polygon (team members were new).
    • Community Feedback: QuickNode made similar promises to both Polygon and Fantom, and failed in part because of the use of LevelDB by the archival nodes. LevelDB causes the database to compact quite frequently, and without notice when not triggering it manually. To ensure availability, they have to run 3-4 high end servers, and rotate compacting manually in order to have a solid uptime. This will get worse as the chain grow and that’s to support a single geographic location.

9/27/2021

  • Pocket Network - met with them in NYC, plan to go live 1st week of October. Marketing plan in place

9/20/2021

  • Pocket Network - drafted closing plan between Harmony and Pocket
  • QuickNode - asked for references

9/13/2021

  • Pocket Network
    • One time setup cost token swap worth $50K POKT tokens, rest is incentivized by POKT's staking-and-reward mechanism
    • Two planned phases:
      • Phase 1 — route 25% RPC traffic to Pocket Network via token swap agreement, fully online by end of October
      • Phase 2 — add Harmony Validators into the Pocket Network of nodes, lead by a Validator node runner, to onramp 8 to 16 Validators, with POKT tokens sponsored by Harmony worth ~$4,500 per node
    • Latency & routing concern answered — client requests are routed to the nearest API Gateway, from there they measure the lowest latency nodes from the 5 nodes that are elected in a 15-min session window. There are likely more than 1 session running at any point in time for our level of traffic
    • Goal - Pocket node runners take 10% of RPC network traffic by the first week of October, and 25% by end of October
    • Scope - Pocket Network takes RPC and GraphQL services
  • QuickNode
    • begun discussion with QuickNode
    • they provide full-service with pay-as-you-go model
    • $250K setup fee plus a 5c/10K req cost, equiv. to $7.5K/month for 50% of our RPC traffic ~ 50M relays/day)
    • when signed by end of this month, they'll be ready to deliver by Dec 30

Weekly On-Call Summary

Sep/9: https://talk.harmony.one/t/weekly-on-call-summary-august-31-september-6-2021/2406

RPC node stabilization

RPC node are reaching critical level of disk space usage - upgrade completion on 30th September 2021

Call revert exception - https://github.com/harmony-one/harmony/issues/3860 - 30th September 2021

need to follow up with dapp projects if issue is still ongoing since 4.2.1 upgrade

New self-hosted API gateway - 15th October 2021