Make Parallel in FVCOM

Question

Make Parallel in FVCOM

dharmaraharja96 opened this issue 3 years ago · comments

Hi!

I have an issue to make a parallel setup for FVCOM. When I'm running it with 8 processors on one pc, it is running quickly.
then when I was combined in with 2 pc, with 16 total of processors. the running time is getting lower and also when using 24 processors when using 3 pc. For the set up I used 1 server and 3 clients. have anyone have solutions on this case? thanks

Ricardo Torres · Answer 1 · Tue Mar 09 2021 16:30:55 GMT+0800 (China Standard Time)

Hi I would assume the pc to pc communications are a bottle neck in your setup. Without something like infiniband as interconectors fvcom will not scale up. For efficient parallel setup you need an HPC rather than a pc grid setup (if that is what your using).

…

________________________________ From: dharmaraharja96 <notifications@github.com> Sent: 09 March 2021 03:16 To: pwcazenave/fvcom-toolbox <fvcom-toolbox@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [pwcazenave/fvcom-toolbox] Make Parallel in FVCOM (#12) Hi! I have an issue to make a parallel setup for FVCOM. When I'm running it with 8 processors on one pc, it is running quickly. then when I was combined in with 2 pc, with 16 total of processors. the running time is getting lower and also when using 24 processors when using 3 pc. For the set up I used 1 server and 3 clients. have anyone have solutions on this case? thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#12>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGHWP24VHR3P55MOVQPJODTCWHHXANCNFSM4Y2UIGQQ>. Visit our website www.pml.ac.uk and follow us on Twitter @PlymouthMarine. This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses. Our Privacy Notice is available at http://www.pml.ac.uk/System-pages/Privacy_Notice . Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. PML Applications Ltd, www.pml-applications.co.uk , is the wholly owned subsidiary of PML, registered as a company in England & Wales, company number 4232750.

dharmaraharja96 · Answer 2 · Wed Mar 10 2021 10:23:28 GMT+0800 (China Standard Time)

thanks for your response, for my setup parallel I'm using this link:
https://www.youtube.com/watch?v=gvR1eQyxS9I (for the communication parallel I'm using LAN)
and for the setup in fvcom I'm using the example tutor, how about that? Do you have an assumption for my setup? thanks

Simon Waldman · Answer 3 · Wed Mar 10 2021 15:42:07 GMT+0800 (China Standard Time)

Hi dharmaraharja,

I haven't watched all of the video, but i'm guessing that the two computers are connected to each other using standard gigabit ethernet (or maybe even 100mbps, depending on the network cards). While some software will work just fine on this sort of home-built cluster, FVCOM usually won't. FVCOM requires a lot of communication between subdomains, and hence the network connections can easily be a bottleneck.
HPCssystems that are designed for things like FVCOM use higher performance network links, eg things like Infiniband, Omnilink, etc

Sorry for the bad news!

dharmaraharja96 · Answer 4 · Wed Mar 10 2021 19:57:09 GMT+0800 (China Standard Time)

Hi Simon

thanks for your information, I'm also trying to setup fvcom parallel on my campus. in my campus using gigabit switch with LAN cable (cat 5e) and the transfer rate data is 1000mbps, but I have the same problem, communication parallel setup not optimum.
Do we have to use the above 1000 mbps also? and my specific problem is the network?

Ricardo Torres · Answer 5 · Wed Mar 10 2021 20:19:58 GMT+0800 (China Standard Time)

Infiniband transfer rate is 200Gbps. I don't know at what value the transfer rate becomes a bottle neck with FVCOM but from your tests it is clear that they are a problem at 1Gbps. There is very little you can do to overcome that limitation other than use multicore computers that don't rely on cpu to cpu transfer.

…

________________________________ From: dharmaraharja96 <notifications@github.com> Sent: 10 March 2021 11:57 To: pwcazenave/fvcom-toolbox <fvcom-toolbox@noreply.github.com> Cc: Ricardo Torres <rito@pml.ac.uk>; Comment <comment@noreply.github.com> Subject: Re: [pwcazenave/fvcom-toolbox] Make Parallel in FVCOM (#12) Hi Simon thanks for your information, I'm also trying to setup fvcom parallel on my campus. in my campus using gigabit switch with LAN cable (cat 5e) and the transfer rate data is 1000mbps, but I have the same problem, communication parallel setup not optimum. Do we have to use the above 1000 mbps also? and my specific problem is the network? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#12 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGHWP4KYMN7T2K7W26ECA3TC5NCNANCNFSM4Y2UIGQQ>. Visit our website www.pml.ac.uk and follow us on Twitter @PlymouthMarine. This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses. Our Privacy Notice is available at http://www.pml.ac.uk/System-pages/Privacy_Notice . Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. PML Applications Ltd, www.pml-applications.co.uk , is the wholly owned subsidiary of PML, registered as a company in England & Wales, company number 4232750.

Simon Waldman · Answer 6 · Wed Mar 10 2021 20:21:57 GMT+0800 (China Standard Time)

I can't promise that the network is the problem, but it is very likely. Gigabit ethernet (the 1000mbps that you mention) is generally not good enough for FVCOM. I'm not sure whether the bandwidth or the latency is the problem.

As Ricardo mentioned, infiniband (often used for HPC networks) goes up to 200 times faster, and it also has much lower latency. Unfortunately this is probably not a problem you can solve cheaply. If you have some money to spend, then best to talk to an HPC expert.

dharmaraharja96 · Answer 7 · Wed Mar 10 2021 20:47:04 GMT+0800 (China Standard Time)

thank you so much Ricardo and Simon for the information
your information really gives me a lot of information

Simon Waldman · Answer 8 · Thu Mar 11 2021 01:28:02 GMT+0800 (China Standard Time)

@riquitorres do you have the ability to close this "issue"? Don't think I can.