stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)

Home Page:https://mc-stan.org/loo

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

loo_moment_match_i computes wrong p_loo

avehtari opened this issue · comments

loo_moment_match_i recomputes lpd using the transformed draws

lpd <- matrixStats::logSumExp(log_liki) - log(length(log_liki))
, which causes p_loo computation to be wrong. lpd needs to be computed with the original draws. PR coming.

Below within numerical accuracy, p_loo should be 22.0 without and with moment_match, but now the latter is wrong
``
fit1 <- brm(count ~ zAge + zBase * Trt, data = epilepsy, family = poisson());

fit1 <- add_criterion(fit1, criterion='loo', save_psis=TRUE)
loo(fit1)
Computed from 4000 by 236 log-likelihood matrix.

     Estimate    SE

elpd_loo -874.2 89.9
p_loo 22.0 5.4
...

fit1b <- add_criterion(fit1, criterion='loo', save_psis=TRUE, moment_match=TRUE, overwrite=TRUE)
loo(fit1b)
Computed from 4000 by 236 log-likelihood matrix.

     Estimate    SE

elpd_loo -874.1 89.9
p_loo 21.3 4.8
...