CPU overuse
kfuku52 opened this issue · comments
I just realized that estimate_shift_configuration() tries to use almost all CPU resources no matter what nCores is. I guess this is not expected behavior. For example, when this code (nCores=1) is executed...
library(l1ou)
tree = rtree(100)
tree = ape::chronoMPL(tree)
trait = matrix(runif(1000, 0, 3), 100, 10)
rownames(trait) = tree$tip.label
colnames(trait) = paste0('trait', 1:ncol(trait))
adj_data = l1ou::adjust_data(tree=tree, Y=trait, normalize=FALSE)
fit_ind = l1ou::estimate_shift_configuration(tree=adj_data$tree, Y=adj_data$Y, nCores=1, rescale=FALSE)
top says the R session is using almost all CPU resources (6236/6400%).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3710 kfuku 20 0 2836076 481560 10460 R 6236 0.1 71:33.71 R
4899 root 20 0 5221332 3.0g 66304 S 3.6 0.6 213:35.75 cmd
10244 root 20 0 51560 7836 2520 S 1.3 0.0 0:00.04 sample_ilo
10227 kfuku 20 0 173380 3532 1704 R 1.0 0.0 0:00.06 top
Here is the result from nCores=8. It initiated with the single process with ~6400/6400 %CPU...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3710 kfuku 20 0 2838736 490788 10460 R 6332 0.1 115:28.19 R
17267 root 20 0 89352 7308 1556 S 1.7 0.0 196:08.84 zabbix_agentd
11635 kfuku 20 0 173380 3524 1700 R 0.7 0.0 0:00.12 top
Each of 8 processes started using more than one core without limit.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11666 kfuku 20 0 2838772 481720 2392 R 1100 0.1 1:44.52 R
11664 kfuku 20 0 2838772 481784 2456 R 843.0 0.1 0:53.74 R
11662 kfuku 20 0 2838772 481704 2376 R 802.0 0.1 0:49.33 R
11663 kfuku 20 0 2838772 481720 2392 R 782.7 0.1 0:52.42 R
11921 kfuku 20 0 2838772 481708 2380 R 750.5 0.1 0:45.36 R
11730 kfuku 20 0 2838772 481724 2396 R 740.1 0.1 0:48.67 R
11665 kfuku 20 0 2838772 481716 2388 R 700.0 0.1 0:47.18 R
11736 kfuku 20 0 2838772 481720 2392 R 645.9 0.1 0:41.93 R
12180 kfuku 20 0 173380 3528 1700 R 1.0 0.0 0:00.06 top
My session info:
> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /lustre7/home/lustre4/kfuku/.pyenv/versions/miniconda3-4.3.30/lib/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] l1ou_1.42 genlasso_1.3 Matrix_1.2-17 MASS_7.3-51.4 magic_1.5-9
[6] abind_1.4-5 grplasso_0.4-6 lars_1.2 phylolm_2.6 ape_5.3
[11] igraph_1.2.4.1
loaded via a namespace (and not attached):
[1] Rcpp_1.0.1 lattice_0.20-38 codetools_0.2-16 listenv_0.7.0
[5] future_1.12.0 digest_0.6.18 grid_3.5.1 nlme_3.1-139
[9] magrittr_1.5 future.apply_1.2.0 tools_3.5.1 compiler_3.5.1
[13] pkgconfig_2.0.2 globals_0.12.4
I would appreciate it if you could fix this problem.
probably way too later ... potentially some other algorithm uses multi-threading that I cannot control. Eg. a simple matrix multiply is multi-threading.