Recommendation to sync to 2.5.13-stable. Announcement for next release.
claberus opened this issue · comments
After trying to catch heisenbugs for several months affecting disk and memory usage of OpenEthereum, Parity agreed with us the best way to move forward is to backport to 2.5.13-stable and include all of the bug fixes and support for hard forks since 2.7 and 3.0.
This decision was critical to ensure the long term viability of a project we took over to ensure client diversity.
We expect it to be available the second week of September approximately and we are evaluating options to automate converting 3.0x db to the 2.5 db format without having to resync.
Edit: Hardcoded bootnodes in 2.5.13-stable are not yet mantained, but you can use the Gnosis maintained bootnodes by using
--bootnodes enode://68f46370191198b71a1595dd453c489bbfe28036a9951fc0397fabd1b77462930b3c5a5359b20e99677855939be47b39fc8edcf1e9ff2522a922b86d233bf2df@144.217.153.76:30303,enode://ffed6382e05ee42854d862f08e4e39b8452c50a5a5d399072c40f9a0b2d4ad34b0eb5312455ad8bcf0dcb4ce969dc89a9a9fd00183eaf8abf46bbcc59dc6e9d5@51.195.3.238:30303
edit2: see Vision for OpenEthereum (ex-Parity client) post on Medium
Wait, just for my understanding. For our usecase we operate multiple archiving, tracing nodes. Syncing that from scratch again is not an option, it took several months already last time around a year ago. Did I read correctly that you intend to demand that node operator resync instead of applying a database migration?
Wait, just for my understanding. For our usecase we operate multiple archiving, tracing nodes. Syncing that from scratch again is not an option, it took several months already last time around a year ago. Did I read correctly that you intend to demand that node operator resync instead of applying a database migration?
Which version are you currently running @quietnan ?
One machine on 3.0.1 and one machine on rakita/11758_use_std_rwlock.
Both machines fail to keep up these days, despite sufficient hardware resources.
One machine on 3.0.1 and one machine on rakita/11758_use_std_rwlock.
Both machines fail to keep up these days, despite sufficient hardware resources.
I'm referring to your multiple archiving and tracing nodes.
Me too. Both were using 3.0.1 a while ago. Now we switched one of them to rakita/11758_use_std_rwlock to see if that solves our problems. When I say 'multiple archiving and tracing nodes', then this does not mean we do so on different versions. We do so for georedundancy. Those two are running far apart but are (or rather used to before switching to rakita/11758_use_std_rwlock) set up the same.
Hi,
I have tried to run v2.5.13 and it cannot find any peers:
2020-08-24 12:43:11 UTC Keys path /data/parity/base/keys/ethereum
2020-08-24 12:43:11 UTC DB path /data/parity/db/ethereum/db/906a34e69aec8c0d
2020-08-24 12:43:11 UTC State DB configuration: fast +Trace
2020-08-24 12:43:11 UTC Operating mode: active
2020-08-24 12:43:11 UTC Configured for Ethereum using Ethash engine
2020-08-24 12:43:11 UTC Removed existing file '/data/parity/base/jsonrpc.ipc'.
2020-08-24 12:43:12 UTC Updated conversion rate to Ξ1 = US$406.73 (11707778 wei/gas)
2020-08-24 12:43:16 UTC Public node URL: enode://xxxxxxxxxxxxxx@<ip>:30303
2020-08-24 12:43:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 166 µs
2020-08-24 12:44:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 166 µs
2020-08-24 12:44:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs
2020-08-24 12:45:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs
2020-08-24 12:45:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs
2020-08-24 12:46:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs
2020-08-24 12:46:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs
Is there any special requirement to run this version?
Hi,
I have tried to run v2.5.13 and it cannot find any peers:2020-08-24 12:43:11 UTC Keys path /data/parity/base/keys/ethereum 2020-08-24 12:43:11 UTC DB path /data/parity/db/ethereum/db/906a34e69aec8c0d 2020-08-24 12:43:11 UTC State DB configuration: fast +Trace 2020-08-24 12:43:11 UTC Operating mode: active 2020-08-24 12:43:11 UTC Configured for Ethereum using Ethash engine 2020-08-24 12:43:11 UTC Removed existing file '/data/parity/base/jsonrpc.ipc'. 2020-08-24 12:43:12 UTC Updated conversion rate to Ξ1 = US$406.73 (11707778 wei/gas) 2020-08-24 12:43:16 UTC Public node URL: enode://xxxxxxxxxxxxxx@<ip>:30303 2020-08-24 12:43:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 166 µs 2020-08-24 12:44:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 166 µs 2020-08-24 12:44:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs 2020-08-24 12:45:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs 2020-08-24 12:45:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs 2020-08-24 12:46:16 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µs 2020-08-24 12:46:46 UTC 0/25 peers 832 bytes chain 512 KiB db 0 bytes queue 448 bytes sync RPC: 0 conn, 0 req/s, 131 µsIs there any special requirement to run this version?
Thank you for commenting this. Added a comment about how to change the bootnodes used in parity 2.5.13.
Me too. Both were using 3.0.1 a while ago. Now we switched one of them to rakita/11758_use_std_rwlock to see if that solves our problems. When I say 'multiple archiving and tracing nodes', then this does not mean we do so on different versions. We do so for georedundancy. Those two are running far apart but are (or rather used to before switching to rakita/11758_use_std_rwlock) set up the same.
We are thinking on creating a tool to migrate the database, this could be nice since clients will not have to resync again. The team is still very small for this project and all old paritytech developers ( the only that knows how the database internally works ) are not in the project now. So maybe it will take time if we do not have more contributors to the project.
We have our Devs sync meeting open to the community tomorrow. If you want to participate, message me or write your questions on our Discord channel. openethereum/pm#4
Me too. Both were using 3.0.1 a while ago. Now we switched one of them to rakita/11758_use_std_rwlock to see if that solves our problems. When I say 'multiple archiving and tracing nodes', then this does not mean we do so on different versions. We do so for georedundancy. Those two are running far apart but are (or rather used to before switching to rakita/11758_use_std_rwlock) set up the same.
We are thinking on creating a tool to migrate the database, this could be nice since clients will not have to resync again. The team is still very small for this project and all old paritytech developers ( the only that knows how the database internally works ) are not in the project now. So maybe it will take time if we do not have more contributors to the project.
@adria0 We are using OpenEthereum in a productive environment. Is this still advisable at this point (with apparently a lot of experience no longer on the project) or should we move away from OpenEthereum as soon as possible? We'd rather know sooner than later.
Me too. Both were using 3.0.1 a while ago. Now we switched one of them to rakita/11758_use_std_rwlock to see if that solves our problems. When I say 'multiple archiving and tracing nodes', then this does not mean we do so on different versions. We do so for georedundancy. Those two are running far apart but are (or rather used to before switching to rakita/11758_use_std_rwlock) set up the same.
We are thinking on creating a tool to migrate the database, this could be nice since clients will not have to resync again. The team is still very small for this project and all old paritytech developers ( the only that knows how the database internally works ) are not in the project now. So maybe it will take time if we do not have more contributors to the project.
@adria0 We are using OpenEthereum in a productive environment. Is this still advisable at this point (with apparently a lot of experience no longer on the project) or should we move away from OpenEthereum as soon as possible? We'd rather know sooner than later.
Well, the idea of the current OE team is make it fully operable, updated to the last spec (we have already implemented all berlin EIPs), and as much stable as possible. We think that client diversity is a primary goal, so we will encourage users to use OpenEthereum. To be honest, I would love an OE that have more stuff there: GraphQL, shared transaction pool, more performance, database split to archive, fully native rust async with tokio 0.2, support for retrieving revert reasons (omg, why we are not still supporting this!), modular system, plug-ins, etc... so I expect to release those things in a closer future, but not we are just focused in one: make it usable and stable for Berlin, and I think that we are on the road for it, so I expect this to be the final zigzag.
We're also running archive nodes as well as multiple nodes that need to be guaranteed to have at least two years of logs for filters available which we can do with standard "fast"/warp nodes as well right now but not if we ever need to throw away the DB and re-sync. We are diligently updating our software, so we are running OE 3.0 on all nodes now. I really really hope we'll have a way to upgrade the DB from the 3.0 format to whatever the new releases are, and do that with a little downtime as possible as we have production DApps running on those nodes.
Just to add some of our experience. We are running 3.0.0 (fatdb) since it's been out and it has been working extremely well, fast and responsive. The only issue we get is the parking_lot bug that is being tracked in #11758. And even that now seems very likely to be solved by using std_rwlock.
Regarding downgrading a database, https://github.com/ordian/open-ethereum-downgrade-db-to-2-5 does not work for 3.0+. See feature request there, issue 1.
Any chance of getting some vetted downgrade script?
Happy to participate in testing with archiving tracing node.
Hello @claveros @adria0 , is there any news on this front? I echo the concerns raised by others in this thread; if the decision is made to NOT merge in the bugfix PRs and instead continue from the 2.5 codebase, it would be great to know this in advance, considering how long it takes to sync Ethereum from scratch. Many people are running this code in production and need some indication of future plans. Thanks!
@edobry https://medium.com/openethereum/vision-for-openethereum-ex-parity-client-eb7b11f6eef8
Please be more explicit on the "We expect to deliver a migration utility" part. Will you or will you not deliver such a migration utility. You are well aware that syncing an archiving tracing node from scratch takes several months, there is no chance to have that finished in time for Berlin fork.
And what does "We will seed the database file to facilitate the migration." mean? Are you talking about a torrent? For which configuration? Archiving yes/no, tracing yes/no, fat yes/no? Will you be seeding all 8 combinations to facilitate all the uses that are currently being deployed in live, production systems?
@claveros Thank you for the update! This does indeed help clarify future plans. Like @quietnan mentions, this holds significant implications for many OpenEthereum operators, and we would all greatly benefit from either a. a real downward migration utility or b. seeding all possible state versions as mentioned in the comment above.
Also, very important: you need to bump the major version and make this into a 4.0.0 release; as this is a breaking, non-backwards-compatible change, SemVer requires you to signal this explicitly in the version number.
Also, the post does not tell people running v2.6.7-beta-b463487-20191231 what they need to do. It says 2.5.13 is on path, and 3.0.x is not, but ignores in-between.
Hello guys, thank you for the great work and for helping to maintain client diversity.
And thank you for sharing your vision in this post.
Any expected delivery date for this release ?
Also, for 3.0.1 users like me who have synced for years, it is very important that you provide a way to start with a full database without having to resync from scratch. Either a migration tool or a way to seed the full database, as mentioned in your post.
Again, a warm thank you for your work !
When I look into the git history of the migration.rs file, I see that the last database migration (from V14 to V15) seems only to have deleted data (namely the COL_ACCOUNT_BLOOM column), as you can see here in the implementation of VacuumAccountsBloom.
Are these "Account Bloom entries" needed? If so, how long will it take to restore this data for the downgrade to database version V14?
@johnzweng, the accounts bloom was removed in 3.0.1, this is the reason why a database downgrade from 3.0.1 to 2.5.13 is not possible. OE 3.1 will remove also this column, making it possible to upgrade both 2.5.13, 2.7.2 and 3.0.1 to 3.1. The rocksdb engine inside changes the version, we hope that this rocksdb downgrade is not going to create problems.
@adria0 thanks for clarification. Looking forward to OE 3.1 🙂
@adria0 What do you think about adding disclaimers to the GitHub release notes of the affected versions, mark them as "pre-release" or even unpublishing them?
@adria0 What do you think about adding disclaimers to the GitHub release notes of the affected versions, mark them as "pre-release" or even unpublishing them?
are you talking about 2.7.2 and 3.0.1 ?