Improve snapshot testing support: output exact same result for each run

[Kim T]

Background
Testing frameworks can record a snapshot of command line output:
https://jestjs.io/docs/snapshot-testing

Testing frameworks create a diff of all changes in the output vs the previous run e.g.

+ This is a new line which was added
- This was the old version which was removed

Problem
When pluginval is run through snapshot testing using a command such as:
pluginval ./plugins/Surge XT.vst3

You see a diff every time:

- Random seed: 0x4e45580
+ Random seed: 0x57d0fc6
- Validation started: 10 Jul 2022 2:54:09pm
+ Validation started: 10 Jul 2022 2:55:25pm
- Time taken to open plugin (cold): 243 ms
+ Time taken to open plugin (cold): 254 ms
- Time taken to open plugin (warm): 95 ms
+ Time taken to open plugin (warm): 126 ms
- Time taken to open editor (cold): 501 ms
+ Time taken to open editor (cold): 504 ms
- Time taken to open editor (warm): 495 ms
+ Time taken to open editor (warm): 510 ms
- Time taken to run test: 664 ms
+ Time taken to run test: 678 ms
- Time taken to run test: 45 ms
+ Time taken to run test: 25 ms
- Time taken to run test: 49 ms
+ Time taken to run test: 44 ms

Solution
I have a workaround for now using RegEx to remove the diff lines. But it's very hacky!

function cleanOutput(output: string): string {
  // Replace app path with variable
  const regex: RegExp = new RegExp(APP_DIR, 'g');
  const output2: string = output.replace(regex, '${APP_DIR}');
  // Replace script path with variable
  const regex2: RegExp = new RegExp(SCRIPT_DIR, 'g');
  const output3: string = output2.replace(regex2, '${SCRIPT_DIR}');
  // Replace random seed with variable
  const regex3: RegExp = new RegExp('Random seed: (.+)', 'g');
  const output4: string = output3.replace(regex3, '${RANDOM_SEED}');
  // Replace validation started with variable
  const regex4: RegExp = new RegExp('Validation started: (.+)', 'g');
  const output5: string = output4.replace(regex4, '${VALIDATION_STARTED}');
  // Replace time taken with variable
  const regex5: RegExp = new RegExp('Time taken to (.+)', 'g');
  return output5.replace(regex5, '${TIME_TAKEN_TO}');
}

Could you add a parameter to disable outputting any lines containing date/times in the output?
pluginval will then output an identical output every time, which will improve snapshot testing.
Thanks!

EDIT: If the output could be consistent between Linux, Mac and Windows, that that would an added bonus!

[drowaudio]

How about if those were verbose log messages? Would that be enough?
I would have thought verbose logging would change a bit between runs anyway?

[drowaudio]

Or could you use the junit output format instead?
I'm not sure I can make any guarantees about the output being consistent. I think it's more useful to have info logging like the seed used etc, especially if there is a standard output format that can be used like junit.

[Kim T]

How would JUnit solve the issue? Whichever testing framework I use, wouldn't the output from pluginval be the same?

Or are you suggesting not doing snapshot testing and only checking the pass/fail status of the overall run?

I would say that timing output could be considered as verbose. Most users don't need to know how long each test took do they? That would be my preferred solution 🙂

[drowaudio]

Yes, I mean just using the pass/fail output of junit.

I think the log output of pluginval could always change, especially for things like the random seed. You'd want that to be different every time in order to fuzz your plugin and you'd need it to be displayed or you'd lose the info if you have a failed validation.

Because I can't guarantee that the log output will never contain any random data I think it's better to use the more parsable and standard junit output format.

[Kim T]

Instead of omitting verbose logs, and not being able to guarantee the output, could I pass a new parameter?
pluginval --results-only which only outputs the test results?

[drowaudio]

That's probably a better approach. I think it's fairly common to have a --quiet option which would only output test failures.