Nita Neou (also known as Soph in the community) has been working in the infrastructure dealing with Network and network Security for more than 15 years in the bank and oil and gas industry. He likes technics and design so to understand how things works. Architecture design is one of his passion and will always have drawing to represent his idea and thought. Prior to joining Harmony, he has been a fierce and active member of P-OPS helping with major milestone in the Harmony journey from staking, 8 to 2s finality, EVM compatibility, bridges, testnets, and more. He loves teaching the community and his co-worker in the hope of everyone to be equal and in harmony. Nita has been graduated from a French Engineering school with a major in System Network and Security. During his free time, he love spending time with his kids and teaching them about computer and programming.

Documents

Operationalize Devnet for the developer

Monthly Goals

April/2022

Launch Snap Database/Shard LevelDB on 5 internal nodes/S0 Epoch Sync
Launch proof-of-concept of Global LoadBalancer,
Onboard new hire Konstantin/Max/Socheat
other goal

March/2022

RPC: Beta Launch new endpoint using the new architecture and optimization based on offchain indexed data solution and new RPC design setup using cloudflare
System: Multisig service improvement of 25%
Onboard - Socheat Khauv

Feb/2022

RPC: Implement for launch new endpoint using the new architecture and optimization based on offchain indexed data solution and new RPC design setup using cloudflare
System: CI/CD for Contract Verifier

Jan/2022 (Deliverable)

System: Devnet launch and support network
Network Release: v4.3.2 (Network outage) / v4.3.3 with Launch of deep beacon prune chain

Leo: Soph managed the release process very well and communicated effectively with all validators and partners. Appreciate!

Upcoming Plans

Quarterly Milestones: 2021 Q4

centralized log server solution and monitoring
harmony binary node CI and testing
decentralized ops (https://harmonyone.notion.site/Decentralized-Ops-82087ce8eef647b98a430d4449f43a5d)

Yearly Planning: 2022 Q1 - Q3

hire one part-time DevOps
take over the testing framework of CI and release
Q1 support external leader rotation and slashing;
Q1-Q2 testing framework, CI/CD of go-sdk, pyhmy, bridge btc
Q2-Q3 testing framework, CI/CD of explorer dashboard
Q3 support resharding implementation, fast state sync in order to have harmony 100% decentralized
Q1-Q4 bridge support cosmos, polygon, solana, celo, layer-2

Past 3-Month Achievements (since July)

Network and Application Service delivery (devops) implementation/improvement/testing:

explorer dashboard (https://explorer.harmony.one)
Sushi integration, bridges (ETH, BTC) leading to a TvL $172M
contract verification / multisignature (gnosis)
Harmony network upgrade (v4.2.1 / v4.3.0) fixing the harmony block time from 3~4s back to 2s and helping with further decentralization (external validator nodes from 640 to 900 and external voting power from 32% to 51%)

Deliverables & 6-Week Focus

Launch readiness review and ORR for BTC bridge (https://github.com/harmony-one/harmony/issues/3911)
Decentralized RPC endpoints in 2 more regions (https://github.com/harmony-one/harmony/issues/3910)
Build new DNS sync nodes (https://github.com/harmony-one/harmony/issues/3909)

Weekly Update

2021-06-13 - 2021-06-19

Cost reduction activities (instance removal consolidation, and downgrade, new s3 storage)
Testnet rollback
Harmony private investigation work
Bridge hack investigation/remediation
Partner support : binance, sushi team,

2021-06-13 - 2021-06-19

Cost reduction activities (instance removed)
Testnet outage and rollback preparation
testing of PR 4070 (epoch sync)
partner support - Binance, quicksync, okex

2021-06-06 - 2021-06-12

Cost reduction activities (instance removal consolidation, and downgrade, new s3 storage)
multisig API monitoring
Elastic RPC rollout to s0 and DFK issue troubleshooting
Network Shard 0 sync issue troubleshooting and fixing

2021-05-29 - 2021-06-05

support TJ in oncall activities
v4.3.10 archival node update
partner support (binance, safepal,

2021-05-23 - 2021-05-29

2 last training session with new hires with recorded video
Test PR 4181 for EIP1898 support and make it available for the graph team
Cost analysis and removed 5 nodes
support TJ in oncall activities

2021-05-23 - 2021-05-29

2 training session with new hires with recorded video
Graph partner support and troubleshoot subgraph indexing issue ⇒ EIP1898 not supported
Follow up and preparation for v4.3.9 hardfork
working/guiding XP/Socheat on HIP18
working with partners : binance, kucoin, okex, …

2021-05-16 - 2021-05-22

on leave

2021-05-09 - 2021-05-15

CICD for staking statistic and monitoring
Planning for v4.3.9 and mainnet hard fork for epoch 999
Planning for Training session
Planning for testnet hardfork HIP18/Chain-id
Partner support : graph, sushi, opolis.

2021-05-02 - 2021-05-08

Helping Jacky/Artem/Socheat/TJ/Max with Ops/network/general questions
Helping Gheis with Grafana RPC count issue and fix duplicate peer id on the network
Implement staking statistic backend https://statistic.stake.hmny.io/
Streamsync discussion troubleshooting and next steps with Jacky
Grafana dashboard and devops updates with the Explorer-v2 RPC related metrics
finalizing bounty RPC test and perform payment
Partner support : ANKR, Opolis DAO, thegraph, kucoin,

2021-04-18 - 2021-04-24

Helping Jacky/Artem/Socheat with Ops questions / operation
btc-bridge troubleshooting
v4.3.8 preparation with backward compatibility with staking RPCs and deployment
PR4100 testing and build of a Localnet with external validator stable config
Binance tx loss issue troubleshooting with Konstantin

2021-04-11 - 2021-04-17

Helping Socheat/Boris/Max with Ops questions / operation
Partner support : bitrue, binance, AAG
Testing PR 4070/4100/4097
Update install-node playbook to help out with the node reinstall with smaller disk
RDS work with External contractor to reduce IO and AWS RDS cost
investigating Security reports on explorer.ps.hmny.io and wiki.harmony.one

2021-04-04 - 2021-04-10

Helping Socheat/Boris/Max with Ops questions / operation
Working with socheat S1/2/3 Validator node update with snapDB
working with Harjas on updating localnet test for 30 gwei gas price min
Partner support : binance/bithumb/guarda/thegraph/honest mining/smartstake/AAG
Network Consensus issue and Post mortem x2
Testing PR 4070/4100

2021-21-02 - 2021-27-02

Cloudflare getLogs OffChain solution tested internally and confirmed working
Support launch of the snapshot DB with daily snapshot of the new DB implemented
DFK issue troubleshooting

Other

Partner support : bithumb

2021-21-02 - 2021-27-02

Daily test/troubleshoot/code analysis of DFK issues
DO SFO 10 new nodes build to increase capacity and work towards multi region setup with cloudflare
getLogs Offchain solution

AWS - API gateway and lamdba function - testcode improvement and troubleshoot gzip issue
Cloudflare - research on cloudflare worker

2021-14-02 - 2021-20-02

Partner support x6 (including Binance, and other exchanges)
streamsync test on testnet activated on S3
API gateway initial code has been pushed and working, left to work on the redundancy and scalability
renewed the SSL cert LHR/JFK/SIN DO LB regions
Multiple issue fixed related to btc vault deployment, Sushi subgraph sync,
working on multiple issue still on going : snapshot sync, multisig slow sync/stuck with node rate limits, DFK RPC issue

2021-07-02 - 2021-13-02

2 x BTC bridge security node built
2 x new explorer API node to support offchain get_logs rpc (total of 4) with multi region setup and redundancy (pending DNS health check condition)
Support binance/huobi and withdrawal blocked issue
10 x FRA DO node removed from the network to save 10k a month
Streamsync test on testnet
2 new archival node build for internal testing and to support internal ops
research on lambda function and api proxy to support the offchain get_logs rpc

2021-31-01 - 2021-06-02

Contract verifier update and deployment
2 x new explorer API node to support offchain get_logs rpc
BTC bridge deployment update and fix after btc bridge relayer code change
btc.api.s0.t.hmny.io SSL issue troubleshooting

2021-24-01 - 2021-30-01

Testing Ledger Nano S issue with
Complete RPC node out of disk upgrade to 2TB (JFK/FRA/LHR)
Release v4.3.4 release and network upgrade Mainnet
btc.api.s0.t.hmny.io created for btc vault node runner

2021-24-01 - 2021-30-01

Metamask Staking test
Test RPC node out of disk upgrade to 2TB on DO
Release v4.3.3 with beacon chain prune feature and support with external validator
Synthetic and monitoring test and alarms updates
Graph node / subgraph sync issue with Viper Dev

Other

working/helping partner (Honest Mining, chainstack)
Immunefy entry review

2021-17-01 - 2021-23-01

Beacon Chain prune feature test
RPC Network troubleshooting and upgrade with v4.3.3 release
New bridge infrastructure with multi region node to ensure availability of the bridge api endpoint
Fix snapshot process
Sushi graph node fix and sync

Other

working/helping partner (Binance, Kucoin, OKEX, AAG, Honest Mining, chainstack)
Immunefy entry review

2021-12-05

RPC issues troubleshooting and fixing
Explorer v2 sync issue
Release 4.3.1 is finally out

Other

working with partner (Polynetwork, Okex)

2021-11-28

Auth/private port RPC implementation troubleshooting and fixing multiple PR #3948 3934 submitted
troubleshoot and fix testnet RPC instability and explorer stuck
DNS rate limiting PR test
Testing and Preparation for next release 4.3.1

Others :

lots of time taken for hotel/visa/flight/covid injection/research on US/Cambodia restrictions for COVID
working with PolyNetwork on header verification for the integration they are working with us
review and detail analysis of pocket network new solution node pilot looking for grant

2021-11-21

RPC decentralization :

Regional node built with 9 new instances split between SIN, FRA, IAD
Geo routing implementation so DNS request are resolved based on user location
Worked with chainstack for a revised proposal and now approved

Working on next release of the harmony code

Coordinate a few more PR to ensure proper node operation with the private api port
testing and ensuring the latest code with canary node

Working with Jenya to have integrate with our monitoring VFAT RPC test
Node out of sync

Upgrade of DNS node to C5.2xl
Adding another region with 3 DNS nodes for S1/S2/S3 so dedicated DNS node all have 4 DNS node and avoid shard 2 DNS node all out of sync causing a 75% total voting power which is near to losing consensus

S1/S2/S3 mainnet validator node upgrade to r5.xlarge so to prevent issue issue
Testnet fork issue troubleshooting and fixing

Other

Discussion with VDAO, multiple HIP, attended VDAO AMA of the week

2021-11-14

RPC decentralization :

Regional node built with 9 new instances split between SIN, FRA, IAD
Completed for api.s0.t.hmny.io and rpc.s0.t.hmny.io now monitoring and pending user feedback

Working with Jenya to have first hand script to emulate VFAT behavior, next step to integrate with our monitoring
RPC issue / troubleshooting again, working with partner reverse protocol to identify root cause
Multiple testnet incident node stuck / shard (S1) down / s0 explorer dashboard indexing down

2021-11-07

RPC decentralization :

Working closely with pocket node for more traffic swapping traffic

BTC-Bridge

write runbook + architecture diagram
Implement monitoring and build backup btc mainnet node

mainnet node validator node replacement
Testing of PR 3918/3919 / troubleshooting on 3915 and fix high memory usage
datadog logging / alerting / monitoring test

Hiring, interview 2 candidates

2021-10-31

RPC decentralization :

pocket network work decentralization continues as RPC issues calmed down
new potential partner initial discussion which would most likely not to through

BTC-Bridge

Study design/architecture
Implement deployment support for relayer/vault testnet and mainnet, bcoin mainnet node

Degraded RPC service :

dedicated S0 DNS sync node built so to avoid having overhead on the RPC node
troubleshooting RPC slowness and out of sync issue
Working with Openswap and defi kingdom devs directly to fix/troubleshoot issues

Harmony code update deployment in attempt to improve DNS service / DNS node cpu usage
Testnet multiple fixes and issues worked with POPS due to more update being pushed now
multiple disk space issue and fixes

2021-10-24

RPC decentralization :

pocket network continued work for more uptake 25% and 50% (if no issue)
new potential partner initial discussion

multi-sig troubleshooting
S1/S2/S3 validator node disk upgrade to 500GB
2 new panels added to https://monitor.harmony.one/d/joo9Q1m7z/s0-explorer-rpc-metrics?orgId=1 so we can now see the each RPC duration
sushi subgraph resource improvement and monitoring integrated with pagerduty
troubleshooting UnHealthyHostCount GreaterThanOrEqualToThreshold 1.
RPC slowness and out of sync issue

2021-10-17

RPC decentralization :

pocket network
new potential partner initial discussion

Finalize GCP bigquery bounty for indexing of harmony mainnet data
Operational work

multi-signature node troubleshooting and fixing
sushi node error 504
explorer-v2 api1 node down fix
Prune RPC node replacement from c5.4xlarge to c5.2xlarge
Experiment to replace i3en.12xlarge with c5.2xlarge with 2 EBS with raid 0
api.harmony.one REVERT CALL error troubleshooting

Nita "Soph" Neou

Documents

Monthly Goals

April/2022

March/2022

Feb/2022

Jan/2022 (Deliverable)

Upcoming Plans

Past 3-Month Achievements (since July)

Deliverables & 6-Week Focus

Weekly Update

2021-06-13 - 2021-06-19

2021-06-13 - 2021-06-19

2021-06-06 - 2021-06-12

2021-05-29 - 2021-06-05

2021-05-23 - 2021-05-29

2021-05-23 - 2021-05-29

2021-05-16 - 2021-05-22

2021-05-09 - 2021-05-15

2021-05-02 - 2021-05-08

2021-04-18 - 2021-04-24

2021-04-11 - 2021-04-17

2021-04-04 - 2021-04-10

2021-21-02 - 2021-27-02

2021-21-02 - 2021-27-02

2021-14-02 - 2021-20-02

2021-07-02 - 2021-13-02

2021-31-01 - 2021-06-02

2021-24-01 - 2021-30-01

2021-24-01 - 2021-30-01

2021-17-01 - 2021-23-01

2021-12-05

2021-11-28

2021-11-21

2021-11-14

2021-11-07

2021-10-31

2021-10-24

2021-10-17