What do you think of allowing an ordered run of `Experiment.conduct`?
nickdirienzo opened this issue · comments
I know with version 1.0, #13 was shipped which always randomizes the order of execution, like GitHub Scientist. This is great for most use cases of laboratory, but I have a use case where I'd like to always run the control before the candidate. Is this something you'd support?
For some additional context, I'm using laboratory for an online data migration and when dual-writing, I'd like to have the control (i.e. primary database) always run before the candidate (i.e. secondary database). We have some constraints that make this functionality a bit of a requirement for our migration, unfortunately.
I know I can subclass Experiment and rewrite conduct to not shuffle, but I'd prefer to call conduct with a kwarg to disable the functionality so I can use a plain old Experiment instead of hand-rolling my own. Does this sound reasonable to you?
If so, I can open up a PR based on my fork: nickdirienzo@ba4ee41.
Thanks for reading @joealcorn! And nice work with this framework 😄
This sounds reasonable - subclassing conduct is a bit of a pain. I'll gladly accept that patch!
It suddenly occurs to me that we should be recording the execution order to help debug issues too, have opened issues #19 for that.
It's interesting to hear that you're using this library for a data migration too, that's not a use case I'd considered before but I'm glad it's coming in handy. Would be interesting in hearing you got on with it once you're done.
Yeah, ha. I was going to remove one line in the override, but as changes happen to this library, it'll be hard to keep up. Thanks for accepting a PR for this issue! I'll send one over after this.
That would be helpful too! I commented on #19 and given that I'm already here, I can add a PR for that too. Plus, it'd help me out with the project I'm on with more verbosity.
Stripe wrote a blog post on online migrations and they use GitHub's Scientist for this. When I looked for the Python variant, I found this library. I'm really happy with how it fits into this problem space so far. No promises, but I'm hoping to write a blog post once we see a success using this pattern and I'll definitely include a thanks to laboratory 😄