validating speculator
changhoonhahn opened this issue · comments
Comparison between inferred properties using speculator versus FSPS with the same setup (NMF SFH and ZH bases) reveals some significant discrepancies, especially for log SFR.
The speculator model call was validated at some point so this is likely not the cause (see notebook).
A comparison of the posteriors of the two methods seems to reveal the issue: the FSPS chain has not converged... This is a bit strange because the chains have the same number of iterations and walkers. And the likelihood should be identical, except that FSPS takes >10x the amount of time.
317e2d0 : compared model=speculator with model=fsps for the mini mock challenge ([notebook])(https://github.com/changhoonhahn/gqp_mc/blob/317e2d0ffb6c00c2227e7f9c06a6da5c8adeddeb/nb/baseline_vs_fsps.ipynb)
The comparison revealed a noticeable bias in the inferred stellar mass for model=speculator:
@changhoonhahn Okay. I'll queue emulator chains for the first 10 galaxies.
@changhoonhahn Hmm, the dust chains haven't finished yet. But for the rest, (emulator, fsps) I got these plots. Black is old and red is new. They look surprisingly similar, but it seems I still need to run append on some chains. I don't suspect my code is outdated as my fsps version is 0.3.0 and I git pulled and also ran setup.py.
Hm, the chains for emulator and fsps seem significantly different even with fsps.
Lets take a step back. Could you try printing out the log posterior for model=emulator and model=fsps for a set of parameter values within the prior? We expect the log posterior to be very similar.
I've compared the SED outputs frommodel=emulator and model=fsps over the SFH basis coefficient parameter space in speculator_accuracy.ipynb.
It seems the emulator is particularly inaccurate in the region of parameter space occupied by the Dirichlet prior:
.
Over the larger parameter space the emulator does a better job:
Closing this issue since this falls under #52