Crash on servers very far from each other.
HyCore opened this issue · comments
Hi, i was making some test and got that wierd behavior, i have a server in london with it's master, and other paper in los angeles, and an other one in Japan. For some reason, i get those error and the server enter in a state of perpertual crash / freeze:
[01:34:06 WARN]: java.lang.Exception: Tried to request a chunk world,region,-57,118 from serverLA, but we are not connected to them!
[01:34:06 WARN]: at puregero.multipaper.MultiPaper.lambda$readRegionFileAsync$13(MultiPaper.java:339)
[01:34:06 WARN]: at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
[01:34:06 WARN]: at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[01:34:06 WARN]: at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
[01:34:06 WARN]: at puregero.multipaper.MultiPaperConnection.lambda$sendAndAwaitReply$7(MultiPaperConnection.java:168)
[01:34:06 WARN]: at puregero.multipaper.MultiPaperConnection.lambda$send$6(MultiPaperConnection.java:159)
[01:34:06 WARN]: at puregero.multipaper.mastermessagingprotocol.messages.MessageHandler.channelRead0(MessageHandler.java:46)
[01:34:06 WARN]: at puregero.multipaper.mastermessagingprotocol.messages.MessageHandler.channelRead0(MessageHandler.java:12)
[01:34:06 WARN]: at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[01:34:06 WARN]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
[01:34:06 WARN]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[01:34:06 WARN]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
[01:34:06 WARN]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[01:34:06 WARN]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[01:34:06 WARN]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[01:34:06 WARN]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[01:34:06 WARN]: at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
[01:34:06 WARN]: at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
[01:34:06 WARN]: at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
[01:34:06 WARN]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
[01:34:06 WARN]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[01:34:06 WARN]: at java.base/java.lang.Thread.run(Thread.java:833)
Servers have to establish peer to peer network which is done automatically by master unless you are running container eg like petrodactle panel, docker etc
// https://discord.com/channels/937309618743427113/938648997445390356/939851715589570602
The external server port has been removed from MultiPaper. All peer to peer connections now use the same port as the Minecraft server does. This allows for a faster and easier setup of servers since you only need to worry about a single port.
MultiPaper is able to automatically discover its peers' ip addresses, however some environments such as docker don't allow this. You can manually enter the MultiPaper's IP address using the `-Dserver.address=192.168.0.1` argument. Eg:
`java -Xmx1G -Dserver.address=192.168.194.65 -jar server.jar`
Also, official Pterodactyl eggs are now available for download:
https://multipaper.io/egg-multi-paper.json
https://multipaper.io/egg-multi-paper--master.json
This guide can help connecting the servers to eachother on Pterodactyl:
https://pterodactyl.io/community/games/minecraft.html
also there is issue with your design as multipaper does not function well with high latency connections, i would advise running servers in same datacenter
i don't use perodactyl, (never did as of rn) but thanks for those information.
I'm already running most of my instance from one big server, but i want to spread accross the server to allow a better ping for all player. (and it works fine)
it's not completed ??
error you posted indicates issue with networking so you have to fix that part
but how, as those issue are only happening over server far away and not internal ones
so you think "-Dserver.address=" will fix it?
so you think "-Dserver.address=" will fix it?
i think yes for internal servers, as i think master is broadcasting 127.0.01 which what confuses the servers
i'm running 8 dedicated server on ubuntu 20.04, and in the multipaper.yml i clearly put the main server ip and port, why adding this extra info would go anything?
i will try your fix, but i'm really not sure it will be fixed like that, and please, don't put it as closed like that, without further more investigation
thanks, i will also post the log on how it happen and what's my exact config later on for easier debugging
i will try your fix, but i'm really not sure it will be fixed like that, and please, don't put it as closed like that, without further more investigation
sorry about this, might be mistake due my phone broken screen
@HyCore This is normal behavior, the developer explained before that this is not how the software is supposed to be used and it is only supposed to work well with a ping of <10ms. I have also tried doing what you did about 6 months ago, and my servers also generated errors for no clear reason.
so you think "-Dserver.address=" will fix it?
Did this fix it?
after a few test, no still not. also is it now updated to 1.20.2 (and do i also see a loose in interest in multipaper when folia came along? + in that case, how hard would it be to have a multifolia)
Multifolia has been mentioned and discussed in the Discord. There's no real gain.or benefit from the effort required to put into developing it.
Here's the fix for the issue:
- DO NOT RUN THEM THAT FAR APART!!!
The developer says that each "shard" has to be within MAX of 2 ms to the Master. You should have ALL servers in the same datacenter. Even if you and your next door neighbor would have a machine each and run servers, the ms would be too high unless you connect qires directly between the systems.
Here's the fix for the issue:
- DO NOT RUN THEM THAT FAR APART!!!
The developer says that each "shard" has to be within MAX of 2 ms to the Master. You should have ALL servers in the same datacenter. Even if you and your next door neighbor would have a machine each and run servers, the ms would be too high unless you connect qires directly between the systems.
sure, however imagine the benefits of running a node on each continent of the world... it would really help people far away. If it was possible to implement a fix or countermeasure to support that inherent latency, it would make multipaper so much more interesting.
As of right now, a australian player trying to connect and play on a US or canadian server will get insanely slow connection and very bad ping...
Finding a way to manage chunk and mods in high latency scenario with be def an upgrade.