⚔️

Nita "Soph" Neou

Nita Neou (also known as Soph in the community) has been working in the infrastructure dealing with Network and network Security for more than 15 years in the bank and oil and gas industry. He likes technics and design so to understand how things works. Architecture design is one of his passion and will always have drawing to represent his idea and thought. Prior to joining Harmony, he has been a fierce and active member of P-OPS helping with major milestone in the Harmony journey from staking, 8 to 2s finality, EVM compatibility, bridges, testnets, and more. He loves teaching the community and his co-worker in the hope of everyone to be equal and in harmony. Nita has been graduated from a French Engineering school with a major in System Network and Security. During his free time, he love spending time with his kids and teaching them about computer and programming.

Documents

Operationalize Devnet for the developer

Monthly Goals

April/2022

March/2022

  • RPC: Beta Launch new endpoint using the new architecture and optimization based on offchain indexed data solution and new RPC design setup using cloudflare
  • System: Multisig service improvement of 25%
  • Onboard - Socheat Khauv

Feb/2022

  • RPC: Implement for launch new endpoint using the new architecture and optimization based on offchain indexed data solution and new RPC design setup using cloudflare
  • System: CI/CD for Contract Verifier

Jan/2022 (Deliverable)

  • System: Devnet launch and support network
  • Network Release: v4.3.2 (Network outage) / v4.3.3 with Launch of deep beacon prune chain

Leo: Soph managed the release process very well and communicated effectively with all validators and partners. Appreciate!

Upcoming Plans

  • Quarterly Milestones: 2021 Q4
  • Yearly Planning: 2022 Q1 - Q3
    • hire one part-time DevOps
    • take over the testing framework of CI and release
    • Q1 support external leader rotation and slashing;
    • Q1-Q2 testing framework, CI/CD of go-sdk, pyhmy, bridge btc
    • Q2-Q3 testing framework, CI/CD of explorer dashboard
    • Q3 support resharding implementation, fast state sync in order to have harmony 100% decentralized
    • Q1-Q4 bridge support cosmos, polygon, solana, celo, layer-2

Past 3-Month Achievements (since July)

  • Network and Application Service delivery (devops) implementation/improvement/testing:
    • explorer dashboard (https://explorer.harmony.one)
    • Sushi integration, bridges (ETH, BTC) leading to a TvL $172M
    • contract verification / multisignature (gnosis)
    • Harmony network upgrade (v4.2.1 / v4.3.0) fixing the harmony block time from 3~4s back to 2s and helping with further decentralization (external validator nodes from 640 to 900 and external voting power from 32% to 51%)

Deliverables & 6-Week Focus

Weekly Update

2021-06-13 - 2021-06-19

  • Cost reduction activities (instance removal consolidation, and downgrade, new s3 storage)
  • Testnet rollback
  • Harmony private investigation work
  • Bridge hack investigation/remediation
  • Partner support : binance, sushi team,

2021-06-13 - 2021-06-19

  • Cost reduction activities (instance removed)
  • Testnet outage and rollback preparation
  • testing of PR 4070 (epoch sync)
  • partner support - Binance, quicksync, okex

2021-06-06 - 2021-06-12

  • Cost reduction activities (instance removal consolidation, and downgrade, new s3 storage)
  • multisig API monitoring
  • Elastic RPC rollout to s0 and DFK issue troubleshooting
  • Network Shard 0 sync issue troubleshooting and fixing

2021-05-29 - 2021-06-05

  • support TJ in oncall activities
  • v4.3.10 archival node update
  • partner support (binance, safepal,

2021-05-23 - 2021-05-29

  • 2 last training session with new hires with recorded video
  • Test PR 4181 for EIP1898 support and make it available for the graph team
  • Cost analysis and removed 5 nodes
  • support TJ in oncall activities

2021-05-23 - 2021-05-29

  • 2 training session with new hires with recorded video
  • Graph partner support and troubleshoot subgraph indexing issue ⇒ EIP1898 not supported
  • Follow up and preparation for v4.3.9 hardfork
  • working/guiding XP/Socheat on HIP18
  • working with partners : binance, kucoin, okex, …

2021-05-16 - 2021-05-22

  • on leave

2021-05-09 - 2021-05-15

  • CICD for staking statistic and monitoring
  • Planning for v4.3.9 and mainnet hard fork for epoch 999
  • Planning for Training session
  • Planning for testnet hardfork HIP18/Chain-id
  • Partner support : graph, sushi, opolis.

2021-05-02 - 2021-05-08

  • Helping Jacky/Artem/Socheat/TJ/Max with Ops/network/general questions
  • Helping Gheis with Grafana RPC count issue and fix duplicate peer id on the network
  • Implement staking statistic backend https://statistic.stake.hmny.io/
  • Streamsync discussion troubleshooting and next steps with Jacky
  • Grafana dashboard and devops updates with the Explorer-v2 RPC related metrics
  • finalizing bounty RPC test and perform payment
  • Partner support : ANKR, Opolis DAO, thegraph, kucoin,

2021-04-18 - 2021-04-24

  • Helping Jacky/Artem/Socheat with Ops questions / operation
  • btc-bridge troubleshooting
  • v4.3.8 preparation with backward compatibility with staking RPCs and deployment
  • PR4100 testing and build of a Localnet with external validator stable config
  • Binance tx loss issue troubleshooting with Konstantin

2021-04-11 - 2021-04-17

  • Helping Socheat/Boris/Max with Ops questions / operation
  • Partner support : bitrue, binance, AAG
  • Testing PR 4070/4100/4097
  • Update install-node playbook to help out with the node reinstall with smaller disk
  • RDS work with External contractor to reduce IO and AWS RDS cost
  • investigating Security reports on explorer.ps.hmny.io and wiki.harmony.one

2021-04-04 - 2021-04-10

  • Helping Socheat/Boris/Max with Ops questions / operation
  • Working with socheat S1/2/3 Validator node update with snapDB
  • working with Harjas on updating localnet test for 30 gwei gas price min
  • Partner support : binance/bithumb/guarda/thegraph/honest mining/smartstake/AAG
  • Network Consensus issue and Post mortem x2
  • Testing PR 4070/4100

2021-21-02 - 2021-27-02

  • Ops
    • Cloudflare getLogs OffChain solution tested internally and confirmed working
    • Support launch of the snapshot DB with daily snapshot of the new DB implemented
    • DFK issue troubleshooting
  • Other
    • Partner support : bithumb

2021-21-02 - 2021-27-02

  • Ops
    • Daily test/troubleshoot/code analysis of DFK issues
    • DO SFO 10 new nodes build to increase capacity and work towards multi region setup with cloudflare
    • getLogs Offchain solution
      • AWS - API gateway and lamdba function - testcode improvement and troubleshoot gzip issue
      • Cloudflare - research on cloudflare worker

2021-14-02 - 2021-20-02

  • Ops
    • Partner support x6 (including Binance, and other exchanges)
    • streamsync test on testnet activated on S3
    • API gateway initial code has been pushed and working, left to work on the redundancy and scalability
    • renewed the SSL cert LHR/JFK/SIN DO LB regions
    • Multiple issue fixed related to btc vault deployment, Sushi subgraph sync,
    • working on multiple issue still on going : snapshot sync, multisig slow sync/stuck with node rate limits, DFK RPC issue

2021-07-02 - 2021-13-02

  • Ops
    • 2 x BTC bridge security node built
    • 2 x new explorer API node to support offchain get_logs rpc (total of 4) with multi region setup and redundancy (pending DNS health check condition)
    • Support binance/huobi and withdrawal blocked issue
    • 10 x FRA DO node removed from the network to save 10k a month
    • Streamsync test on testnet
    • 2 new archival node build for internal testing and to support internal ops
    • research on lambda function and api proxy to support the offchain get_logs rpc

2021-31-01 - 2021-06-02

  • Ops
    • Contract verifier update and deployment
    • 2 x new explorer API node to support offchain get_logs rpc
    • BTC bridge deployment update and fix after btc bridge relayer code change
    • btc.api.s0.t.hmny.io SSL issue troubleshooting

2021-24-01 - 2021-30-01

  • Ops
    • Testing Ledger Nano S issue with
    • Complete RPC node out of disk upgrade to 2TB (JFK/FRA/LHR)
    • Release v4.3.4 release and network upgrade Mainnet
    • btc.api.s0.t.hmny.io created for btc vault node runner

2021-24-01 - 2021-30-01

  • Ops
    • Metamask Staking test
    • Test RPC node out of disk upgrade to 2TB on DO
    • Release v4.3.3 with beacon chain prune feature and support with external validator
    • Synthetic and monitoring test and alarms updates
    • Graph node / subgraph sync issue with Viper Dev
  • Other
    • working/helping partner (Honest Mining, chainstack)
    • Immunefy entry review

2021-17-01 - 2021-23-01

  • Ops
    • Beacon Chain prune feature test
    • RPC Network troubleshooting and upgrade with v4.3.3 release
    • New bridge infrastructure with multi region node to ensure availability of the bridge api endpoint
    • Fix snapshot process
    • Sushi graph node fix and sync
  • Other
    • working/helping partner (Binance, Kucoin, OKEX, AAG, Honest Mining, chainstack)
    • Immunefy entry review

2021-12-05

  • Ops
    • RPC issues troubleshooting and fixing
    • Explorer v2 sync issue
    • Release 4.3.1 is finally out
  • Other
    • working with partner (Polynetwork, Okex)

2021-11-28

  • Ops
    • Auth/private port RPC implementation troubleshooting and fixing multiple PR #3948 3934 submitted
    • troubleshoot and fix testnet RPC instability and explorer stuck
    • DNS rate limiting PR test
    • Testing and Preparation for next release 4.3.1
  • Others :
    • lots of time taken for hotel/visa/flight/covid injection/research on US/Cambodia restrictions for COVID
    • working with PolyNetwork on header verification for the integration they are working with us
    • review and detail analysis of pocket network new solution node pilot looking for grant

2021-11-21

  • RPC decentralization :
    • Regional node built with 9 new instances split between SIN, FRA, IAD
    • Geo routing implementation so DNS request are resolved based on user location
    • Worked with chainstack for a revised proposal and now approved
  • Ops
    • Working on next release of the harmony code
      • Coordinate a few more PR to ensure proper node operation with the private api port
      • testing and ensuring the latest code with canary node
    • Working with Jenya to have integrate with our monitoring VFAT RPC test
    • Node out of sync
      • Upgrade of DNS node to C5.2xl
      • Adding another region with 3 DNS nodes for S1/S2/S3 so dedicated DNS node all have 4 DNS node and avoid shard 2 DNS node all out of sync causing a 75% total voting power which is near to losing consensus
    • S1/S2/S3 mainnet validator node upgrade to r5.xlarge so to prevent issue issue
    • Testnet fork issue troubleshooting and fixing
  • Other
    • Discussion with VDAO, multiple HIP, attended VDAO AMA of the week

2021-11-14

  • RPC decentralization :
    • Regional node built with 9 new instances split between SIN, FRA, IAD
    • Completed for api.s0.t.hmny.io and rpc.s0.t.hmny.io now monitoring and pending user feedback
  • Ops
    • Working with Jenya to have first hand script to emulate VFAT behavior, next step to integrate with our monitoring
    • RPC issue / troubleshooting again, working with partner reverse protocol to identify root cause
    • Multiple testnet incident node stuck / shard (S1) down / s0 explorer dashboard indexing down

2021-11-07

  • RPC decentralization :
    • Working closely with pocket node for more traffic swapping traffic
  • BTC-Bridge
    • write runbook + architecture diagram
    • Implement monitoring and build backup btc mainnet node
  • Ops
    • mainnet node validator node replacement
    • Testing of PR 3918/3919 / troubleshooting on 3915 and fix high memory usage
    • datadog logging / alerting / monitoring test
  • Hiring, interview 2 candidates

2021-10-31

  • RPC decentralization :
    • pocket network work decentralization continues as RPC issues calmed down
    • new potential partner initial discussion which would most likely not to through
  • BTC-Bridge
    • Study design/architecture
    • Implement deployment support for relayer/vault testnet and mainnet, bcoin mainnet node
  • Ops
    • Degraded RPC service :
      • dedicated S0 DNS sync node built so to avoid having overhead on the RPC node
      • troubleshooting RPC slowness and out of sync issue
      • Working with Openswap and defi kingdom devs directly to fix/troubleshoot issues
    • Harmony code update deployment in attempt to improve DNS service / DNS node cpu usage
    • Testnet multiple fixes and issues worked with POPS due to more update being pushed now
    • multiple disk space issue and fixes

2021-10-24

  • RPC decentralization :
    • pocket network continued work for more uptake 25% and 50% (if no issue)
    • new potential partner initial discussion
  • Ops
    • multi-sig troubleshooting
    • S1/S2/S3 validator node disk upgrade to 500GB
    • 2 new panels added to https://monitor.harmony.one/d/joo9Q1m7z/s0-explorer-rpc-metrics?orgId=1 so we can now see the each RPC duration
    • sushi subgraph resource improvement and monitoring integrated with pagerduty
    • troubleshooting UnHealthyHostCount GreaterThanOrEqualToThreshold 1.
    • RPC slowness and out of sync issue

2021-10-17

  • RPC decentralization :
    • pocket network
    • new potential partner initial discussion
  • Finalize GCP bigquery bounty for indexing of harmony mainnet data
  • Operational work
    • multi-signature node troubleshooting and fixing
    • sushi node error 504
    • explorer-v2 api1 node down fix
    • Prune RPC node replacement from c5.4xlarge to c5.2xlarge
    • Experiment to replace i3en.12xlarge with c5.2xlarge with 2 EBS with raid 0
    • api.harmony.one REVERT CALL error troubleshooting