Make settings done in the testing script persist.

[Timo Könnecke]

Beroset noted on matrix it would be cool if the checkmarks in the script settings would be "sticky".
I figure this would need some config file read and written by the script and ignored by git.