A repo for tracking resources related to resolving incidents on the Juno Network.
For security disclosures, follow these instructions.
- Create a folder with the date of the incident
- Collect resources related to root cause, and anything validators/devs will need to re-bootstrap the network
- If you need to recover from a hard fork, both
5-April-22and28-July-22have anupgrade.mdwith the steps involved - To restore IBC clients, follow the
ibc_client_recovery.mdguide
If there is a hardfork caused by an attack, your two options are to fork the chain to a new network, e.g. juno-2 or to continue using juno-1 but skip a block to avoid double-signs.
TL;DR as of 2022-08-08 we have chosen the latter on both occasions. To take this course:
- Export state from the problem block -1
- Bear in mind that this state export will contain things like registrations that have taken place in upgrade handlers. A more minimal state would require re-initialization. This is mainly a concern for testing.
- Change the export to both a new genesis time in the future and skip a block. i.e. in the first attack the problem block was
2578097. The chain was restarted at block2578099. Finally, change the state to remove any problem state, e.g. smart contract state that could be problematic - Patch any problems with the Juno binary to fix the security issue (or other consensus issue)
- Get validators to turn off nodes
- Get validators to switch binary
- Get validators to add the export that you've sanitized as a new genesis (instructions for steps 5-7 are in the
upgrade.mdfiles) - Get validators to turn on nodes and wait for new genesis. Invariant checks likely mean the network won't come back up for between 13-23hrs.
- When a decent number of validators (say, 5 on different OS/CPU/host strategy) have completed invariant checks, the remaining nodes can either skip invariant checks, or sync from a snapshot if the network is already live.