Improve Namespace performance
TimBeyer opened this issue · comments
I've noticed that when having many namespaces, and many experiments to set up and tear down again, which is the recommended way of handling changing experiments that are added and removed without reshuffling all assignments, performance starts to become an issue.
We've recently had to rewrite all of our experiments code to use caching via cookies because calculating those on every request significantly slowed down our application. The whole stateless aspect of planout is one of the major selling points though, so I do not find it a particularly elegant solution.
I've put up a benchmark on http://requirebin.com/?gist=a584dd4d020dabd830fa
Try playing around with the numSegments and experiments that get added and removed and you'll see that currently namespaces don't scale really well.
Assuming you are using several namespaces in parallel with 100 segments each and several running experiments, you'll quickly end up with hundreds of milliseconds of processing time.
I wonder if there's room for optimization so Planout.js scales better, or if this is because of the hashing and cannot be improved on.
In any case I'd be happy to help out with this if you have any ideas for low hanging fruits.
@TimBeyer that definitely doesn't sound good, but we have not experienced the same perf issues. From my perf testing a while ago, random assignment should be incredibly quick (usually no more than 1ms overhead, usually less but I never recorded my results so this may be my memory playing tricks on me :) ). It's definitely possible that some recent changes may have caused some perf issues, though. I can look into if there is any low-hanging fruit around the initialization of namespaces that is causing some slowdown.
Couple of questions
- iirc you are using this on the server-side with node.js and not client-side. Is that still the case?
- How many namespaces / how many experiments are you running?
- Are you using singletons for your namespace classes? This is probably only relevant if you're defining your namespaces on the client-side.
In any case, I'm going to take a deep look into this this week, I'll let you know if I find anything. Thanks for letting me know about this.
@rawls238 When I tested just a normal experiment assignment without namespaces, things were indeed around the 1ms mark. However, when using namespaces things started adding up.
Currently we're doing 4 experiments (there were 5 in total, but one was concluded and so to get consistent assignments it too needs to be added and removed during namespace initialization anyway until all the remaining tests have been concluded) in a 1000 segment namespace, and the time it took to instantiate the namespace was around 150ms - 200ms. We're planning to expand the amount of namespaces in the future to about 5. For some quick performance gains we'll reduce the amount of segments per namespace though.
We used to do this both in the client and on the server in a universal JS app because it was easier to set up that way, but now we're going server-side only to avoid the overhead and only calculate the assignments once, storing all of them in a cookie with a timestamp.
The namespaces used to be singletons scoped to every request on the server so that we need to instantiate them only once. https://github.com/othiym23/node-continuation-local-storage comes in handy for those kind of things
FWIW at Facebook namespaces / experiments are stored in databases, so there is no sequential logic, and all of the param values are computed / filled in server side. Tim, if you are mostly working server side, that might be a more efficient and scalable approach.
On Oct 27, 2015, at 6:33 AM, Tim Beyer notifications@github.com wrote:
@rawls238 When I tested just a normal experiment assignment without namespaces, things were indeed around the 1ms mark. However, when using namespaces things started adding up.
Currently we're doing 4 experiments (there were 5 in total, but one was concluded and so to get consistent assignments it too needs to be added and removed during namespace initialization anyway until all the remaining tests have been concluded) in a 1000 segment namespace, and the time it took to instantiate the namespace was around 150ms - 200ms. We're planning to expand the amount of namespaces in the future to about 5. For some quick performance gains we'll reduce the amount of segments per namespace though.
We used to do this both in the client and on the server in a universal JS app because it was easier to set up that way, but now we're going server-side only to avoid the overhead and only calculate the assignments once, storing all of them in a cookie with a timestamp.
The namespaces used to be singletons scoped to every request on the server so that we need to instantiate them only once. https://github.com/othiym23/node-continuation-local-storage comes in handy for those kind of things
—
Reply to this email directly or view it on GitHub.
agreed with eytan about that approach if you are doing things server-side, but I have added a few bells and whistles to the namespace class in the recent past and have not verified that they don't have performance implications. Will let you know what you find
@eytan does that mean you have a large database containing all params for all visitors, and when a new experiment is released you update the whole database? I don't feel like I fully understand yet how the server-side approach would speed things up unless you already precompute all assignments before the requests come in. Either that, or you accept the performance hit on the first request of the user after the experiments have changed.
@TimBeyer, check out Section 5 of the PlanOut paper. In a production system, you typically wouldn't have these .addExperiment() and .removeExperiment() calls, and the code that manages the experiments would be completely separate from the code that retrieves the experiment definition and performs the assignment. All of the assignment is done online when params are requested for the user.
In some cases we cache the assignment server-side, but that can often end up occupying a ton of memory. If you do go that route, there are some subtleties to making it work in a way in a way that lets you change the definition or turn the experiment off if things go sour.
@rawls238 @eytan From looking at the code, it seems like the segmentation code is pretty much independent from the experiment assignment code in namespaces.
I'm wondering if it would make sense to split those things up, which would also have the benefit of making namespaces a bit less stateful, removing the need for the requireExperiment side-effect.
I'm thinking something like this: (apologies for the non ES6 syntax, it was quicker to copy paste from existing code that way)
var Namespace = extend.call(planout.Namespace.SimpleNamespace, {
setupDefaults: function () {
this.numSegments = 100;
},
setup: function () {
this.setName('Test');
this.setPrimaryUnit('userId');
},
setupExperiments: function () {
this.addExperiment('foo', Foo, 10);
this.addExperiment('bar', Bar, 10);
this.removeExperiment('foo');
}
});
var namespace = new Namespace();
var experiment1 = namespace.getExperiment({
userId: '1337'
});
var experiment2 = namespace.getExperiment({
userId: '7331'
});This way the namespace can be created only once and then reused forever.
Logging is delegated to the experiments anyway.
I also think it might clean up the overrides code.
It would also clear up the confusion where a Namespace basically is an Experiment. What I mean by this is that the Namespace shares mostly the same interface as the Experiment and almost every call on a namespace method causes the requireExperiment() side effect which turns the once generic Namespace instance into something that just delegates to its assigned Experiment.
@eytan after reading through rawls238/react-experiments#11 (comment) I realized we might be abusing namespaces as well, but pretty much the same reasons @gusvargas mentioned apply to us too.
Often you don't want to deal with the overhead in analytics and with making all variations work properly with all other variations until you have proven that they are worth the effort. Plus sometimes the business wants to try out new things with a smaller percentage of users first and then scale it up when it can be seen that things seem to be stable for that smaller audience.
I think this is a valid use case for namespaces, so the implementation of segmentation should probably be decoupled from the idea of using segmentation to run truly mutually exclusive tests touching the same kind of parameters.
It definitely can add a bit of fragility that sometimes you get undefined as the result of the parameter you request, but I believe with proper architecture this can be addressed.
@TimBeyer requireExperiment should only ever do experiment assignment once, even if it is called many times. I'm going to be spending some time today digging into the issues you described and see if I can find where we can make the biggest perf gains.
I think that your use of namespaces is fine - we talked about this and those are legitimate cases for using namespaces, though the primary use for it is for iterating on the same experiment with the same parameters it works equally well for the cases you describe.
I think that your suggestion re: breaking up the experiment segment assignments and the individual user assignment is a good idea. One way to do this is to use @eytan's suggestion of just persisting this information, but the other thing we could do to make this easier to integrate into an app without having to deal with any persistence mechanism is make it so that you can kick off experiment segment assignments for all namespaces when you start up your app and then all client requests that require experiment parameters just have to do individual experiment assignments (i.e. assignments with units that are not experiment names) on the fly as needed, but this should be <2 ms per assignment fetch. This is roughly speaking how we do it on the client-side, but there are some complications to the server-side approach. I'll see if I can sketch up a solution to make this work on the server-side.
In any case, I'll let you guys know what I find
yikes! If you put segment size down to 100 then you have reasonable load times which is mainly what we do. That explains why I haven't run into this yet...
@TimBeyer I think I know how to make this better - PR incoming soon
@TimBeyer good news is that I found some things that could lead to huge performance increases, not sure if I'll finish them today but definitely by next week.
@TimBeyer I'm planning on publishing a new version tomorrow or Tuesday with the perf changes. Should be significantly faster than it was before. As stated previously, they won't be compatible with previous experiment assignments but you should be able to migrate things using the toggleCompatible flag.
@rawls238 sounds nice! Is it already in master?
@DeTeam @TimBeyer just published v2.0 which should have substantial perf improvements. As noted before, it is incompatible with old experiment and namespace assignments so be careful when upgrading (you can always toggle it to be compatible with the old version by doing ExperimentSetup.toggleCompatibleHash(true)
Closing this but file another issue if you encounter any other issues or email me at garidor at hubspot.com
@TimBeyer did the updates fix the performance issues you were seeing?