War Mode
In 2021, Harmony grew significantly in ecosystem and market recognition. We have 159 unique validators on mainnet, 4.5 billion ONE staked, various dapps launched on mainnet with average 4m+ transactions processed daily and half a million unique wallets registered on mainnet. We've earned it through years of hard work and #buidling ever since 2018. And we should be proud of ourselves for these achievements.
It wasn't always so rosy and smooth. We've had our fair share of hard times when we were not sure about the strategy and how to win the market, when we only had a few months of runway and the market was ruthless, and when we had to launch mainnet in one month with only limited engineering resources. But we went all through it, overcame every formidable obstacle and achieved things which are almost impossible.
We went through the hardship with the spirit that we will give it all we've got and fight like it's a war - you either win or die. We Fight in the War Mode. I still remember the three major war mode times 1) when we had to launch a demo Dapp for Binance in 1 week (April 2019) in order to get launched on Binance IEO, which was a critical event to keep the project alive; 2) when we were trying to launch mainnet in one month (June 2019) to be the first sharded mainnet (earlier than other IEO projects such as Matic and Elrond); 3) when we pushed to launch open staking (May 2020) and capture the last wave of staking. The work was tough and endless but we all made it with just a few of us engineers working tirelessly doing 996 (monday to saturday, 9am-9pm) style schedule. In fact, it's more than 996, before launches, we were working until midnight as late as 3am-4am and many Sundays had to be skipped to get the last piece in place for launch. That's what we did to keep Harmony alive, to achieve missions impossible, to give it all we've got.
As we got into the era of covid and everybody worked from home, we also stopped doing the 996 schedule since we knew it's not sustainable for the long term. Working normal schedule and away from the teammates could be wearing on our motivation and the spirit. We may not need to work until 4am now, but the war mode spirit shouldn't be gone. The war mode may not be about working long hours anymore, BUT it's still about:
- Being serious on our product launches with careful planning and resource allocation early on;
- Launching on time with quality;
- Being customer-focused and making our users happy through our quality products and reliable services. Not settling on inferior experience for our users;
- Having full ownership on the product you own and deliver end-to-end.
We should bring back the war mode spirit that helped Harmony go through all the obstacles and accomplish the impossibles. It's part of the gene of Harmony team and culture and we should always have it, to prepare for the unpredictable future and to make Harmony rise to the next level of success.
Engineering Productivity
Besides the war mode spirit, we should also consider doing the following to increase the engineering productivity:
- Having daily standup to sync and make steady progress every day.
- Work together in office for more than 1 days a week (could be 3), to get more conversation and planning done effectively.
- Having (bi-)weekly project demo or knowledge sharing session to learn/share progress together.
Hiring
More importantly, we should resume the hiring effort to get more engineering resources. Solders don't fight alone in war. As Harmony grows in product scope and ecosystem scale, our team is still in the similar size as in 2019 (it's even smaller for engineering compared to 2019) and it significantly affects the development speed and the the product quality due to lack of resources. We should admit that some of the core product line and services can not be easily outsourced. Core protocol, bridges, rpc services and many more simply cannot be handled fast and with quality by external contributors or through bounties, either due to lack of knowledge/context or the skin in the game to deliver with quality.
Take Polygon for example, they had around 40 people 1 year ago and with limited product lines. They grew from 40 to 200 people so far and at the same time their valuation grew from 4b to 17b with multiple product lines (commit chain, hermes, avail etc.). They have 17 product managers working on many different areas such as protocol design, layer2 solution design, wallet, explorer, bridges etcs, lead by a head of product who thinks more on the bigger picture and strategy of growing Polygon as a everything-layer2 platform. Not to mention their big team of engineers behind the chain development etc. As we grow bigger in scope and initiatives, having enough human resources in the team to support that scope is needed.
We may not need 200 people, but we do need urgently:
- 2-3 Golang engineers to work on core protocol development and optimization (e.g. resharding, cross-shard txns, leader incentive and rotation etc.). One of them could focus on node binary rpc service and transaction pool logic optimization to make sure our rpc and user txn sending experience is good.
- 1 more devops engineer to share burden of Soph and remove the signle-point-of-failure on Soph.
Particularly, lacking engineering and technical resources have shown bad impact on our rpc service quality.
Lack of devops resource who can respond to issues quickly has caused delay in incident response time for rpc outage. Soph has been a single point of failure since he’s been covering most of the devops tasks and launches, and a week without soph has led to issues on our production service that impact real users.
Lack of blockchain engineering resource has caused the void of expertise knowledge on rpc and transaction pool component. There are certainly rooms of optimization on the node binary level to make rpc code more efficient and to solve node out-of-sync issue.
And for sure, we need the war mode spirit on our rpc issues even before we get the new hires since it’s the most critical component for our partners and users, it is literally the single point of failure for the whole harmony ecosystem and reputation.