Use of append_iter_outfile

[Jonathan Ozik]

Is there an issue with using append_iter_outfile to mean both append to data.table e.g.:
all_iter_parm_draws <- rbind(all_iter_parm_draws, iter_parm_draws[!is.na(iter_parm_draws$draw), ])
and to append to existing output file, e.g.:
write.table in save_results

I'm thinking that for the case of outputting the all_iter_parm_draws, all_iter_sim_target, and all_iter_target_dist (lines 424-441 in imabc.R here), we should use the "append_iter_outfile" flag (where I think we may want to change the name of this variable to indicate that it's about appending data rather than appending to an out file) to just indicate if we're accumulating the data into these data.tables or not, and then when calling save_results, we'd just say append = FALSE? Does this make sense?

Similarly for lines 1133-1138 when we're outputting the good_parm_draws, good_sim_target, and good_target_dist, it doesn't make sense to me to have an append option, since we'd never want to append to the good_XXX files?

[Chris Maerzluft]

The reason I use append to save the files is because it is much faster and more memory efficient to save all the data to one file using append. The only reason I even create all_iter_parm_draws is for the case when a person does not want imabc to save output to a file (i.e. they only want the results in R). Of course that assumes the data is to be written to a single file. If you wanted, we could save iteration results to separate files (e.g. output_dir/SimulatedParameters_iter1.csv, etc.) and do more along the lines of what you are suggesting.

I do think some people would want to see the different iterations that go into making the good_XXX files. That can be a helpful tool in diagnostics of the model calibration