Pipelined fetching of multiple objects from ModelSet

[Jakub Roztocil]

I've implemented simple pipelined fetching of multiple objects from ModelSet. Not sure if it's useful for others in this form, but you can see the code bellow. For efficiency reason, you can also pass raw=True to get plain dicts instead of model instances.

class ModelSet(Set):

    def get_in_pipeline(self, raw=False):        
        key = self.model_class._key
        ids = list(self._set)
        pipe = self.db.pipeline()

        for id in ids:
            pipe.hgetall('%s:%s' % (key, id))

        for i, hash in enumerate(pipe.execute()):
            id = ids[i]
            if raw:
                hash['_id'] = id
                yield hash
            else:
                model = self.model_class(**hash)
                model._id = id
                yield model

[sebastien requiem]

I find that globally useful but I'd rather have to configure the modelset to have all operations fetched in a (or in fact two) pipeline(s).

But Redisco should offer such a thing.

[Dmitriy Kruglyak]

So is this going to be merged into the code base?

[sebastien requiem]

Not yet.

@jkbr and I discussed the need for such a function but I would like something a bit more transparent than another function call on the modelset.

[John Loehrer]

Any progress made on this front? This is almost the only thing missing from redisco. I'd like to be able to do:

foo_list = Foo.objects.get_by_id(ids)

If it pipelined the loading of the objects instead of fetching one by one, that'd be a huge improvement.