how to set all cpu cores usage

Question

how to set all cpu cores usage

jhcuarta opened this issue 2 years ago · comments

Hi
I was wondering if you could help me out to set the main script for using all the cpu cores avaible. I count with 8 threads and it takes days for opening a single clustal file, I've tried linux and windows and no success, its worth noting it is a big file, nevertheless it is taking more than ten days for just opening the file, and I'm sure I'm no using all cores although I modified the variables -Xmx and -Xss despite no expertise in the subject.
Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz 1.99 GHz
Installed RAM 32,0 GB (31,9 GB usable)

Best regards

Wayne Maddison · Answer 1 · Sun Oct 09 2022 10:55:58 GMT+0800 (China Standard Time)

The file reading is single threaded. It would take a lot of reprogramming to change that. I think something is likely going wrong. Is there any chance you could send me the file? How big is it? Wayne

…

On 8 Oct 2022, at 18:49, jhcuarta ***@***.***> wrote: [CAUTION: Non-UBC Email] Hi I was wondering if you could help me out to set the main script for using all the cpu cores avaible. I count with 8 threads and it takes days for opening a single clustal file, I've tried linux and windows and no success, its worth noting it is a big file, nevertheless it is taking more than ten days for just opening the file, and I'm sure I'm no using all cores although I modified the variables -Xmx and -Xss despite no expertise in the subject. Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz 1.99 GHz Installed RAM 32,0 GB (31,9 GB usable) Best regards — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

-------------- Wayne Maddison Professor and Canada Research Chair, Depts. of Zoology and Botany 6270 University Blvd. University of British Columbia Vancouver, BC V6T 1Z4 Canada President of the International Society of Arachnology: https://arachnology.org Personal web page: http://waynemaddison.wordpress.com Lab web page: http://waynemaddisonlab.wordpress.com Mesquite: http://www.mesquiteproject.org; MacClade: http://macclade.org Beaty Biodiversity Museum: http://beatymuseum.ubc.ca I am grateful to live, learn and work on the unceded, traditional, and ancestral lands of the xʷməθkʷəy̓əm (Musqueam), Səl̓ílwətaʔ/Selilwitulh (Tsleil-Watuth), Stó:lō, and Skwxwú7mesh (Squamish) Nations of the Coast Salish peoples.

jhcuarta · Answer 2 · Sun Oct 09 2022 11:09:24 GMT+0800 (China Standard Time)

Hi
I linked the file

https://drive.google.com/file/d/1oUbvI-_urYoEIgUYORoL9I59QjmAF0uP/view?usp=sharing

Wayne Maddison · Answer 3 · Tue Oct 11 2022 16:09:12 GMT+0800 (China Standard Time)

Mesquite read the file in about 15 minutes on my iMac, whether I gave it 8GB or 16GB of memory. I’m not sure what’s happening for you. Can you tell if Mesquite continues to be using CPU cycles? Wayne

…

On 8 Oct 2022, at 20:09, jhcuarta ***@***.***> wrote: [CAUTION: Non-UBC Email] Hi I linked the file https://drive.google.com/file/d/1oUbvI-_urYoEIgUYORoL9I59QjmAF0uP/view?usp=sharing — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

-------------- Wayne Maddison Professor and Canada Research Chair, Depts. of Zoology and Botany 6270 University Blvd. University of British Columbia Vancouver, BC V6T 1Z4 Canada President of the International Society of Arachnology: https://arachnology.org Personal web page: http://waynemaddison.wordpress.com Lab web page: http://waynemaddisonlab.wordpress.com Mesquite: http://www.mesquiteproject.org; MacClade: http://macclade.org Beaty Biodiversity Museum: http://beatymuseum.ubc.ca I am grateful to live, learn and work on the unceded, traditional, and ancestral lands of the xʷməθkʷəy̓əm (Musqueam), Səl̓ílwətaʔ/Selilwitulh (Tsleil-Watuth), Stó:lō, and Skwxwú7mesh (Squamish) Nations of the Coast Salish peoples.

jhcuarta · Answer 4 · Wed Oct 12 2022 07:24:05 GMT+0800 (China Standard Time)

Hi
On Windows I've been using the K8 starter, I even modified the SK batch file setting -Xmx28G -Xss16G and no success for the your time frame, while on linux using java 8 I run the following command line "java -Xmx28G -Xss16G -d64 -Djava.library.path=lib -Djri.ignore.ule="yes" -cp /home/jason/Documents/Mesquite_Folder -jar Mesquite.jar", and same result.
Honestly dont know whats going on

Wayne Maddison · Answer 5 · Wed Oct 12 2022 22:42:05 GMT+0800 (China Standard Time)

I presume you are able to read small files? The confusing thing is that even if there’s a memory problem, it should simply give you an Out of Memory error rather than taking forever. Perhaps try -Xmx8000M rather than 8G??? I can try to add code that gives updates about file reading, so you know where it’s stuck. Perhaps ask me next week if I haven’t contacted you? Wayne

…

On 11 Oct 2022, at 16:24, jhcuarta ***@***.***> wrote: [CAUTION: Non-UBC Email] Hi On Windows I've been using the K8 starter, I even modified the SK batch file setting -Xmx28G -Xss16G and no success for the your time frame, while on linux using java 8 I run the following command line "java -Xmx28G -Xss16G -d64 -Djava.library.path=lib -Djri.ignore.ule="yes" -cp /home/jason/Documents/Mesquite_Folder -jar Mesquite.jar", and same result. Honestly dont know whats going on — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

-------------- Wayne Maddison Professor and Canada Research Chair, Depts. of Zoology and Botany 6270 University Blvd. University of British Columbia Vancouver, BC V6T 1Z4 Canada President of the International Society of Arachnology: https://arachnology.org Personal web page: http://waynemaddison.wordpress.com Lab web page: http://waynemaddisonlab.wordpress.com Mesquite: http://www.mesquiteproject.org; MacClade: http://macclade.org Beaty Biodiversity Museum: http://beatymuseum.ubc.ca I am grateful to live, learn and work on the unceded, traditional, and ancestral lands of the xʷməθkʷəy̓əm (Musqueam), Səl̓ílwətaʔ/Selilwitulh (Tsleil-Watuth), Stó:lō, and Skwxwú7mesh (Squamish) Nations of the Coast Salish peoples.

jhcuarta · Answer 6 · Thu Oct 13 2022 03:09:16 GMT+0800 (China Standard Time)

Hi
On linux I'm modified the -Xmx flag as follows: "java -Xmx27000M -Xss6000M -d64 -Djava.library.path=lib -Djri.ignore.ule="yes" -cp /home/jason/Documents/Mesquite_Folder -jar Mesquite.jar", the issue persists, although in the terminal window the following line is displayed
(Translate File) #
(Mesquite) # Dialog closed
Chosen: Clustal (DNA/RNA)
Oct 12, 2022 1:56:58 PM org.apache.commons.httpclient.HttpMethodDirector executeMethod
"SEVERE: Narrowly avoided an infinite loop in execute"

Wayne Maddison · Answer 7 · Thu Oct 13 2022 06:53:10 GMT+0800 (China Standard Time)

Oh wow. The file is a FASTA file. Try that importer instead? Wayne

…

On 12 Oct 2022, at 12:09, jhcuarta ***@***.***> wrote: [CAUTION: Non-UBC Email] Hi On linux I'm modified the -Xmx flag as follows: "java -Xmx27000M -Xss6000M -d64 -Djava.library.path=lib -Djri.ignore.ule="yes" -cp /home/jason/Documents/Mesquite_Folder -jar Mesquite.jar", the issue persists, although in the terminal window the following line is displayed (Translate File) # (Mesquite) # Dialog closed Chosen: Clustal (DNA/RNA) Oct 12, 2022 1:56:58 PM org.apache.commons.httpclient.HttpMethodDirector executeMethod "SEVERE: Narrowly avoided an infinite loop in execute" — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

-------------- Wayne Maddison Professor and Canada Research Chair, Depts. of Zoology and Botany 6270 University Blvd. University of British Columbia Vancouver, BC V6T 1Z4 Canada President of the International Society of Arachnology: https://arachnology.org Personal web page: http://waynemaddison.wordpress.com Lab web page: http://waynemaddisonlab.wordpress.com Mesquite: http://www.mesquiteproject.org; MacClade: http://macclade.org Beaty Biodiversity Museum: http://beatymuseum.ubc.ca I am grateful to live, learn and work on the unceded, traditional, and ancestral lands of the xʷməθkʷəy̓əm (Musqueam), Səl̓ílwətaʔ/Selilwitulh (Tsleil-Watuth), Stó:lō, and Skwxwú7mesh (Squamish) Nations of the Coast Salish peoples.

jhcuarta · Answer 8 · Thu Oct 13 2022 07:52:09 GMT+0800 (China Standard Time)

Hi
I imported as fasta and worked rapidly. Thanks a lot

Wayne Maddison · Answer 9 · Thu Oct 13 2022 08:42:29 GMT+0800 (China Standard Time)

Good! W

…

On 12 Oct 2022, at 16:52, jhcuarta ***@***.***> wrote: [CAUTION: Non-UBC Email] Hi I imported as fasta and worked rapidly. Thanks a lot — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

-------------- Wayne Maddison Professor and Canada Research Chair, Depts. of Zoology and Botany 6270 University Blvd. University of British Columbia Vancouver, BC V6T 1Z4 Canada President of the International Society of Arachnology: https://arachnology.org Personal web page: http://waynemaddison.wordpress.com Lab web page: http://waynemaddisonlab.wordpress.com Mesquite: http://www.mesquiteproject.org; MacClade: http://macclade.org Beaty Biodiversity Museum: http://beatymuseum.ubc.ca I am grateful to live, learn and work on the unceded, traditional, and ancestral lands of the xʷməθkʷəy̓əm (Musqueam), Səl̓ílwətaʔ/Selilwitulh (Tsleil-Watuth), Stó:lō, and Skwxwú7mesh (Squamish) Nations of the Coast Salish peoples.