stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)

Home Page:https://mc-stan.org/loo

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Update E_loo khat diagnostic

avehtari opened this issue · comments

  1. @sethaxen pointed out that E_loo h-specific khat diagnostic is using the old approach sqrt(1 + h_theta^2) * r_theta, while the Appendix E of 2022 version of PSIS paper recommends using max khat from left and right tail. posterior package has now handy function for computing max khat from left and right tail. We could copy that or add posterior dependency and use also other Pareto smoothing and diagnostic functions from there.

  2. @sethaxen also pointed out that h-specific khat is computed always assuming type="mean".
    For type="var" and type="sd" we could use khat <- E_loo_khat.default(x^2, psis_object, log_ratios) as computation of those require reliable estimation of E[x^2]. For type="quantile" we could use r-specific khat (ie khat for ratios).

  3. It would probably be good to report maximum of h-specific khat and r-specific khat, as some h can hide the problems in r which would be fine without self-normalization, but in case of self-normalization the reliability of normalization always matters.

Currently, loo suggests posterior. if I would use posterior to fix this issue, would it stay as suggested?

It’s possible to keep it in Suggests, check for posterior when the user runs E_loo, and error if it’s not found. But if we think E_loo is an important function and there’s no option to run E_loo without posterior, we should probably move it to Imports instead of Suggests, which will automatically install posterior when loo is installed. The downside is that it forces the user to have all the additional dependencies that posterior requires in order to use anything in the loo package, even the functions that don’t need posterior.

Importing posterior would add dependency on methods, abind, rlang, tibble, vctrs, tensorA, pillar, distributional
rstanarm, cmdstanr, and brms already import posterior
rstan imports methods, but no posterior, but I think it would be good if rstan would start using posterior, too
other packages depending on loo, depend or import rstanarm, cmdstanr, brms, or rstan, so some packages would get the extra dependencies

I will first make a PR with dependency on posterior, and if decided that no dependency, then I'll copy the functions