retrain speculator
changhoonhahn opened this issue · comments
The current version of the speculator FSPS emulator is inaccurate in the SFH basis coefficient parameter space that we're interested in (see speculator_accuracy.ipynb). We need to update the speculator FSPS emulator and validate its accuracy.
The speculator package now has a demo for training speculator: speculator_training_demo.ipynb.
- Retrain emulator following its instructions.
- Validate retrained emulator
Scripts for generating speculator training set for simple (calzetti) dust model has been implemented and deployed on NERSC (d7bd33e):
@kgb0255 lets try to keep track of the speculator training using the task list I've added here: https://github.com/changhoonhahn/gqp_mc/blob/master/run/todo.speculator_training.md
As you train speculator models with different training set sizes, pca components, etc, lets fill out the table to keep track.
simpledust, 300 training batches, 20 PCAs, 200 pca training batches: ~4% accuracy for 99 percentile
see https://github.com/changhoonhahn/gqp_mc/blob/ae611fa0c5471e82fda90aaa6f31b843b2477b43/nb/training_desi_simpledust_speculator.ipynb for details
simpledust, 300 training batches, 20 PCAs, 200 pca training batches: ~4% accuracy for 99 percentile
simpledust, 300 training batches, 40 PCAs, 200 pca training batches: ~5% accuracy for 99 percentile
I've uploaded the collection of dPCA plots here.
PCAs in intervals
Since we've been struggling to train speculator for the entire wavelength range (2305 - 11025), let's try to divide it into three wavelength ranges and train a NN separately for each one.
Here are the wavelength ranges I've settled on:
- 2305 < wave < 4500
- 4500 < wave < 6500
- 6500 < wave < 11025
These were chosen based on the accuracy level at different wavelengths (e.g. we struggle to get good accuracy at < 4500) and so that each bin has similar dimensions.
I've added two notebooks to train and validate speculator in separate wavelength bins:
- training_desi_complexdust_speculator_wavebins.ipynb: for training the speculator model for a specific wavelength bin
- validate_trained_desi_complexdust_speculator_wavebins.ipynb: for validating the separate wavelength bin speculator model
Not sure why I only included wave > 3000 in the validation plot in the past, but we definitely speculator to reach ~1% accuracy over the entire 2305 - 11025 wavelength. The lower and higher wavelengths, make things worse...
Turns out the situation is not so dire. Here's an updated validation plots with a larger test set (1e5 test samples):
The dip we see above is due to numerical precision issues.
Our mean fractional error isn't at <1% yet, but at 1.5% so we're almost there!
*these plots were generated using the new validation script because the RAM available on colab couldn't handle the new test set.
<1% achieved!
Finally, we've achieved <1% accuracy for wavebin0. Using Npca=50, Ntrain=5e6, and a [256, 256, 256, 256] architecture did the trick.
<1% for 99 percentile.
For consistency, I'm running wavebin1 and wavebin2 with the same architecture.
Ntrain=5,000,000, Npca=[50, 30, 30], architecture=4x256
Ha! I mixed up log base 10 with natural log, so we actually do way better...
