Streaming Sync & Fast Sync

Next Milestone Date

June 20, 2022

Assign

Jacky WangRongjian Lan

Status

Live on Testnet

Github

Collaborator

Grant-1

Grant-2

Grant-3

Release Date

2022 Q3

Fast Sync & Gossips: streaming state sync, information dispersal algorithm

We are replacing gRPC in state sync with p2p streaming with a rate limiter and downloader batch. Later we will implement RapidChain’s gossiping protocol based on an information dispersal algorithm (IDA) for sending large blocks. Both approaches, along with further performance analysis and parameter tuning, will improve our network resiliency and latency.

Current problem at hand:

1. The CPU and memory usage will grow to 100% at the light sail nodes after running for 2~3 days.
2. Some of the in-sync RPCs are redirected to gRPC sync after some iterations of code change.
3. Sometimes, the sync loop is stuck in short-range sync. Need to investigate.

After these three problems being solved, the stream sync protocol can be launched.

Harmony's Networking Story

At Harmony, we don't just look at what is needed to get the job done today - we want to build something that will get the job done even years from now. This goes for all layers of our technology...

medium.com

2021/10/26

1. Continue debugging for stream sync. 1. The previous found suspect do impact CPU (very occasionally), but it is still not the root cause. The nodes dies after 3 days' running... 2. The next suspect is... still the discovery protocol... Keep researching

2021/10/19

1. Continue working on stream sync. Found a suspect for CPU explosion
1. The problem lies in two peers sending out the INIT package in two end's streaming protocol because of the immediate discovery after stream number is lower than lower threshold. And this will fail the stream setup and result in a large number of discovery and stream setup operations (which is heavy)
2. Have reproduced the issue in two nodes in different regions resulting in high CPU cost (which is a hipe, not a slow growth).
3. Starting with the fix - Adding a cooldown mechanism in discovery and stream manager.

2021/10/08

1. Working on stream sync. First looking at the potential memory / CPU leak. Running two machines running customized code on mainnet with stream sync turned on. The customized code include:
1. Frequent discovery call (10s/discovery).
2. Unstable stream sync protocol (injected error in stream sync, occasionally fail)

The result remains to be observed.

2. Finished code revisit and RPC fix concerning stream sync.
3. Next step is to
1. Investigate the CPU / memory leak
2. Spin up stressnet for reproducing the stucking short range sync.