Feature request: Test for issues relating to multiple plugin instances
ianacaburian opened this issue · comments
- Would there be a way to do some basic stability tests when opening multiple instances ? E.g. Tests to catch fatal errors relating to mismanagement of static resources used across multiple instances.
- Further since it does some basic profiling - could there also be a feature to show how many instances it takes to reach X% CPU levels with some stats on the current machine.
Great work btw.
There's no current support for these things but it probably wouldn't be too difficult to add opening a second instance.
The real question is what do you do when you open the second instance?
It might actually be easier to run several copies of pluginval at once (you can do this from bash using & I think), just make sure they are all running the testing in-process or their pipes will clash.
But if you wanted to add some multi-instance testing yourself I'm open to PRs!
For the profiling, that's a bit more difficult as I think it just renders as fast as possible at the moment? To do proper profiling you'd need to do a real-time render or it will just eat as many cycles as it can to complete in as little time as possible?
Then there's also the question of what are you testing? Presumably your input or at the very least plugin settings will drastically change the CPU usage? There's a few too many variables here to make this an easy question I'm afraid.
I didn't think of running multiple instances of pluginval - thanks for that.
I was just playing around with how I can get pluginval to fail and it seems to be quite robust apart from catching anything that will cause a DAW to crash when multiple plugin instances are involved. A simple example would be if the plugin uses some static resource, opening a second instance then closing it again would usually cause a crash - if the static resource is being manually destructed by the second instance somewhere.
With the profiling part, it's true it doesn't sound as easy as I originally thought, but it would be nice if there could be a stage in the pluginval routine where it goes into some "free edit" mode, where the user can manually adjust the plugin settings to induce some state, and then tell pluginval to continue, and then it would run some further allocation tests, minor profiling.
This could catch things happening in a more realistic user scenario than analyzing how a plugin behaves in its default state.
I've already poked around inside to see how I could do this so I may get around to it given the time.
Thanks for your help and ideas!