n slighly > p

[Mariam Rizkallah]

Dear OEM,

I have been using cv.oem. I was trying to keep n > p. The problem is that once n is slightly closer to p, R session either crashes or does not finish, both on Windows and Linux (both in RStudio or the console). The issue gets worse with grouping and cross validation. Please find my example.

Thank you.

nobs  <- 150
nvars <- 60

X <- matrix(rnorm(nobs * nvars), ncol = nvars)
group.indicators <- rep(1:(60/10), each = 6)
y <- rbinom(nobs, 1, 
  prob = 1 / (1 + exp(-drop(X %*% c(0.15, 0.15, -0.15, -0.15, 0.25, rep(0, nvars - 5)))))
)

input <- X

train_rows <- sample(1:nobs, 0.66 * nobs)
x.train <- as.matrix(input[train_rows, ])
x.test <- as.matrix(input[- (train_rows), ])

y.train <- y[train_rows]
y.test <- y[-train_rows]

cvfit <- oem::cv.oem(
  x = x.train, y = y.train,
  penalty = "grp.lasso",
  groups = group.indicators,
  type.measure = "auc",
  nlambda = 100,
  grouped = TRUE,
  family = "binomial"
)

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.4 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.3 tools_3.4.3    yaml_2.1.19 

[Jared Huling]

The OEM algorithm has poor convergence properties for this scenario and I would recommend you use another algorithm instead. OEM is explicitly designed for the n >> p case.