Back-test TOML versions?
brandonchinn178 opened this issue · comments
I'm investigating some TOML libraries in Haskell, and I'm not completely sure what version of TOML they support, but I'm pretty sure they support v0.4.0 or v0.5.0. I'd be open to upgrading them to v1.0.0, but in the meantime, it would be great if I could run toml-test -version v0.4.0 on them to quickly see how they conform to spec.
Alternatively, if you want to avoid maintaining old versions' tests in perpetuity, you could add releases for old versions of toml-test (or add docs somewhere containing github commit hashes) that clearly indicate what version of the spec they adhere to.
There isn't a great way to do this right now. I wouldn't be against such a flag, but I (or someone) would have to identify which tests belong to which version. It's quite a bit of "boring gruntwork" I don't really have the time for right now. In hindsight I should have done this when I added the tests for 0.5 and 1.0 last year, but I didn't think of it.
If someone else comes up with a list then I can do the code for it.
The quickest way is to use an older version. I believe that if you use commit 06d0a36 you've got TOML 0.5, and commit b201e81 is TOML 0.4, but it could be that you need to use a slightly earlier commit – I'm not entirely sure since I've only been maintaining this since last year so it may be some TOML 0.5 tests were in here before that. If you identify specific commits then let me know so I can add them to the readme, or maybe even "backdate" a 0.4.0 and 0.5.0 tag/release.
I went ahead and added a -toml flag to select the version (-version was already used to show the toml-test version) as it'll be useful to avoid similar problems with future TOML versions, but still need to expand the list of tests here: https://github.com/BurntSushi/toml-test/blob/master/version.go#L17
Might have to split out some tests, as they test both 0.5.0 and 1.0.0 features.
That was fast! Thanks, looks like a good start!
Yeah, figured it would be useful for the future in any case. For the older versions just need to expand the list of tests; if you get it running against one of those 0.5 or 0.4 Haskell libraries then let me know (or send a PR) to add the appropriate tests in there. Some may have the be split up a bit as they test both newer and older features in the same file.
I'll close this because the feature is implemented and I'll use it for future versions (the upcoming TOML 1.1).
Not all the older tests are tagged but I'm not sure how much interest there is to tag all the pre-1.0 tests considering TOML 1.0 is two years old at this point, but if someone wants to work on it then I'll merge it (or help out, if needed).